Zimbra Strong SSL/TLS Cipher Suites

Apparently, even in 2016, Java (and by extension Zimbra) ships without support for strong crypto. Geopolitics == retarded.

Anyway, in order to get strong crypto up and running in Zimbra Webmail, you’ll need to download the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files for JDK/JRE 8.

Unzip the file and overwrite the JARs in /opt/zimbra/java/jre/lib/security.

But wait, that’s not enough. A bunch of default cipher suites are old and weak, or are vulnerable to the Logjam Attack. You can disable all that cruft as follows:

zmprov mcf +zimbraSSLExcludeCipherSuites SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_DHE_DSS_WITH_DES_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_DHE_RSA_WITH_DES_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_RSA_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_RSA_EXPORT_WITH_RC4_40_MD5 \
+zimbraSSLExcludeCipherSuites SSL_RSA_WITH_DES_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_RC4_128_MD5 \
+zimbraSSLExcludeCipherSuites SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_RSA_WITH_3DES_EDE_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_RSA_WITH_RC4_128_MD5 \
+zimbraSSLExcludeCipherSuites SSL_RSA_WITH_RC4_128_SHA \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_ECDHE_RSA_WITH_RC4_128_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_3DES_EDE_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_AES_128_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_AES_128_CBC_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_AES_128_GCM_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_DES_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_128_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_256_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

Finally, restart zmmailboxdctl. You’re good to go :-).

JAX-RS Client Filters

Life goes on with JAX-RS/Jersey. I wasted a couple of moments figuring out how to add custom headers to a Jersey generated JAX-RS client. Might as well write it down in the hope of saving someone a couple of minutes.

For starters, you’ll need a Client Filter that does the actual heavy(ish) lifting.

import javax.ws.rs.client.ClientRequestContext;
import javax.ws.rs.client.ClientRequestFilter;
 
/**
 * Add the X-GARBAGE header to all requests.
 */
public class GarbageFilter implements ClientRequestFilter {
	@Override
	public void filter(final ClientRequestContext requestContext) throws IOException {
		requestContext.getHeaders().add("X-GARBAGE", "This is added to all requests");
	}
}

And then you’ll have to register the filter with the Client(Config).

// import org.glassfish.jersey.client.ClientConfig;
 
final ClientConfig clientConfig = new ClientConfig();
clientConfig.register(new GarbageFilter()); // Yes, you could use JDK8 magic :-)
 
final Client client = ClientBuilder.newClient(clientConfig);

And that’s all. Every request you launch using the generated client will now contain your X-GARBAGE header.

JAX-RS Client File Upload

Another hiccup in using the wadl2java client generated from a (Jersey) JAX-RS app. This time, it concerns multipart/form-data.

The method:

	@POST
	@Produces(MediaType.APPLICATION_JSON)
	@Consumes(MediaType.MULTIPART_FORM_DATA)
	public JsonFoo create(@FormDataParam("file") InputStream data, 
		@FormDataParam("file") FormDataContentDisposition fileDetail, 
		@FormDataParam("file") FormDataBodyPart bodyPart) {
			// Implementation foo bar baz
	}

The WADL exposes something that looks like this.

<method id="create" name="POST">
	<request>
		<representation mediaType="multipart/form-data"/>
	</request>
	<!-- Response omitted for the sake of brevity -->
</method>

And the generated client has a method to go along with it. Unfortunately, it gives you no hints whatsoever as to how to actually provide a file/data.

// Long class names shortened
public static Create create(Client client, URI baseURI) {
	return new Create(client, baseURI);
}
 
// The Create object contains this little gem
public<T >T postMultipartFormDataAsJson(Object input, GenericType<T> returnType);

That’s wonderful. Unfortunately, if you pass in a java.io.File, nothing happens. The client barfs.

Many DuckDuckGo-searches, StackOverflow hunts and headscratchings later, I came up with a working solution:

// import import org.glassfish.jersey.media.multipart.FormDataMultiPart;
// import org.glassfish.jersey.media.multipart.file.FileDataBodyPart;
 
File file = new File(); // Your file!
FormDataMultiPart form = new FormDataMultiPart();
form.field("filename", file.getName());
form.bodyPart(new FileDataBodyPart("file", upload, new MediaType("image", "jpeg")));
 
Api.create(client, uri).postMultipartFormDataAsJson(form, new GenericType<CreateResponse>() {});

But wait! That won’t cut it. You also need to tell your Client that you want to use the Multipart Feature. Makes sense. If you don’t, you’ll end up with this exception.

org.glassfish.jersey.message.internal.MessageBodyProviderNotFoundException: MessageBodyWriter not found for media type=multipart/form-data, type=class org.glassfish.jersey.media.multipart.FormDataMultiPart, genericType=class org.glassfish.jersey.media.multipart.FormDataMultiPart.
// import org.glassfish.jersey.client.ClientConfig;
// import org.glassfish.jersey.media.multipart.MultiPartFeature;
 
final ClientConfig clientConfig = new ClientConfig();
clientConfig.register(MultiPartFeature.class);

And there you have it. File upload with JAX-RS and a wadl2java generated client.

JAX-RS vs Collections vs JSON Arrays

The Problem

Looks like I’ve managed to get myself into a pickle. I have a (Jersey) JAX-RS app, which automagically publishes a WADL. If I then generate a Client (using wadl2java), I end up with generated code that doesn’t work.

	@GET
	@Produces(MediaType.APPLICATION_JSON)
	public List<Foo> get() {
		return myList;
	}

Foo is a simple POJO with an @XmlRootElement annotation. The resulting JSON is perfectly fine: I get a JSON Array with a bunch of Foo objects. Perfect!

The generated WADL is a different matter:

    <method id="get" name="GET">
        <response>
            <!-- foo is a reference to an XSD Complex Type ... but where's my list? -->
            <ns2:representation element="foo" mediaType="application/json"/>
        </response>
    </method>

For some reason, the WADL generator is smart enough to realise that we’re dealing with Foo instances. But it’s too stupid to realise that we’re dealing with more than one.

If you then generate a Client using wadl2java, you’ll end up with something like this:

    public Foo getAsFoo() {
        return response.readEntity(Foo.class);
    }

Well that’s not going to work, is it? Trying to read a single Foo when you’ve got an array of them. And indeed … you get a wonderful exception.

Internal Exception: java.lang.ClassCastException: Foo cannot be cast to java.util.Collection

This seems to be a fundamental XML vs JSON problem. After all there is no such thing as an “XML Array”. Either you have one element, or you have multiple elements nested under a common root.

I could solve this by not relying on a plain JSON Array and encapsulating the result.

// Not this:
[
   {
      "foo":"bar"
   },
   {
      "bar":"baz"
   }
]
 
// But this:
{
   "listOfFoos":[
      {
         "foo":"bar"
      },
      {
         "bar":"baz"
      }
   ]
}

But then that’s ugly. And instead of using a native Java collection I’ll have to create a useless intermediary object. And it would break compatibility with current API clients.

The Solution

Whelp … I haven’t found one yet. To be continued, I hope. But I did manage to find a workaround, thanks to Adam Bien’s blog.

The incorrectly generated getAsFoo() method doesn’t work. But we can use getAsJson() instead — which doesn’t necessarily have to return a JSON string.

    List<Foo> foos = client.getFoo().getAsJson(
        new GenericType<List<Foo>>() {
        }
    );

GenericType is a dirty JAX-RS hack in my opinion, but it works. It’s a shame that I have to rely on getAsJson(), though. It would’ve been much cleaner to use the getAsFoo() method directly.

Endless Locale Bugs

This isn’t the first time I’ve ranted about Locale related issues. It probaly won’t be the last, either.

Currently my biggest gripe is with convenience methods in Java which rely on platform defaults, and are thus platform-specific. For instance, the venerable System.out.println() will terminate your line with whatever your platform thinks a line ending should be. Printing dates or numbers will try to use your platform Locale. Writing a string to a file will default to platform encoding.

Some of these defaults can be controlled at run time, others require JVM startup options. This is all horribly wrong. This results in all kinds of unexpected behaviour. It’s error-prone. None of this should be allowed to happen. Any method that assumes platform-specific magic should be deprecated. Which is exactly what I’ll do, as soon as I can figure out how to write a Sonar plugin to detect all this nonsense.

Getting started with JAX-RS & Gradle

Managed to waste an hour of my life today, trying to set up a JAX-RS + Gradle project. Gradle wasn’t really the problem, nor was JAX-RS, but putting it all together in a way that works took longer than expected.

So without further ado, I present to you a simple quickstart project you can use to kickstart JAX-RS + Gradle projects.

The Gradle build file is pretty minimalistic. There’s no need for web.xml. And I’ve included an example resource.

build.gradle:

description = 'JAX-RS Quickstart'
group = 'org.lick.me.jaxrs'
version = '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'eclipse'
sourceCompatibility = 1.8
 
repositories {
        mavenCentral()
}
 
 
buildscript {
        repositories {
                jcenter()
        }
        dependencies {
                // Embedded Jetty 9 Gradle Plugin
                classpath (group: 'com.sahlbach.gradle', name: 'gradle-jetty-eclipse-plugin', version: '1.9.+')
        }
}
apply plugin: 'jettyEclipse'
apply plugin: 'war'
 
dependencies {
        // JAX-RS 2.0(.1)
        compile 'javax.ws.rs:javax.ws.rs-api:2.0.1'
 
        // Jersey 2.15
        compile 'org.glassfish.jersey.containers:jersey-container-servlet:2.15'
 
        // Servlet 3
        compile 'javax.servlet:javax.servlet-api:3.1.0'
}

Extra quick quickstart:

git clone git@github.com:Nimlhug/jax-rs-quickstart.git
cd jax-rs-quickstart
gradle jettyEclipseRun

Java Date Performance Subtleties

A recent profling session pointed out that some of our processing threads were blocking on java.util.Date construction. This is troubling, because it’s something we do many thousands of times per second, and blocked threads are pretty bad!

A bit of digging led me to TimeZone.getDefault(). This, for some insanely fucked up reason, makes a synchronized call to TimeZone.getDefaultInAppContext(). The call is synchronized because it attempts to load the default time zone from the sun.awt.AppContext. What. The. Fuck. I don’t know what people were smoking when they wrote this, but I hope they enjoyed it …

Unfortunately, Date doesn’t have a constructor which takes a TimeZone argument, so it always calls getDefault() instead.

I decided to run some microbenchmarks. I benchmarked four different ways of creating Dates:

// date-short:
    new Date();
//date-long: 
    new Date(year, month, date, hrs, min, sec);
// calendar:
    Calendar cal = Calendar.getInstance(TimeZone);
    cal.set(year, month, date, hourOfDay, minute, second)
    cal.getTime();
// cached-cleared-calendar:
//    Same as calendar, but with Calendar.getInstance() outside of the loop, 
//    and a cal.clear() call in the loop.

I tested single threaded performance, where 1M Dates were created using each method in a single thread. Then multi-threaded with 4 threads, each thread creating 250k Dates. In other words: both methods ended up creating the same number of Dates.

Lower is beter.
Click to enlarge. Lower is beter.

With exception of date-long, all methods speed up by a factor of 2 when multi-threaded. (The machine only has 2 physical cores). The date-long method actually slows down when multi-threaded. This is because of lock contention in the synchronized TimeZone acquisition.

The JavaDoc for Date suggests replacing the date-long call by a calendar call. Performance-wise, this is not a very good suggestion: its single-threaded performance is twice as bad as that of Date unless you reuse the same Calendar instance. Even multi-threaded it’s outperformed by date-long. This is simply not acceptable.

Fortunately, the cached-cleared-calendar option performs very well. You could easily store a ThreadLocal reference to an instance of a Calendar and clear it whenever you need to use it.

More important than the raw duration of the Date creation, is the synchronization overhead. Every time a thread has to wait to enter a synchronized block, it could end up being rescheduled or swapped out. This reduces the predictability of performance. Keeping synchronization down to a minimum (or zero, in this case) increases predictability and liveness of the application in general.

Before anyone mentions it: yes, I’m aware that the long Date constructors are deprecated. Unfortunately, they are what Joda uses when converting to Java Dates. I’ve proposed a patch, but while doing a bit more research for this blog post, I’ve come to the conclusion that my patch needs a bit of refining as it is still too slow (though it no longer blocks). In the mean while, I hope that the -kind?- folks at Oracle will reconsider their shoddy implementation.

I’ve also heard rumours that Joda will somehow, magically, replace java.util.Date in JDK 8. Not sure how that’s going to work with backwards compatibility. I’d be much happier if java.util.Date would stop sucking quite as much. And if SimpleDateFormat were made thread-safe. And … the list goes on.

Java 7 Performance

I decided to compare Java 6 & 7 performance for $employer’s $application. Java 7 performs better — as expected. What I did not expect, was that the difference would be so big. Around 10% on average. That’s not bad for something as simple as a version bump.

Jave 6 vs Java 7

Ideally I’d like to investigate where this difference comes from. I suspect improved ergonomics have a lot to do with it.

$application uses Apache Solr rather extensively. In fact, most of the time querying is spent in Solr. With indexing it’s probably about 50% of the time. With querying it’s probably closer to 90%. All tests are run in a controlled environment, so I have a fair amount of confidence in these results.

The indexing test inserts 3 million documents in Solr. Creating these documents takes up the bulk of the time. It involves a lot of filesystem access — something which Java versions have very little influence over and heavily multi-threaded CPU-intensive processing.

If you’re not using Java 7, you really should consider upgrading. If you’re stuck with people who live in the past, maybe you can convince them with a bunch of pretty performance graphs of your own.

OutOfMemoryError while running Maven Surefire tests

Imagine you have a project which works perfectly fine and well. All tests pass, each and every time. Then one day you commit a couple of new classes with related tests. Of course you ran all tests before committing, and everything worked just fine. Then, a minute or so later, you get a mail from Hudson (or whatever you’re using for CI) saying that there are test failures. “Maybe I forgot a file”, I thought. Checked the test results on Hudson. About a dozen tests were failing, unrelated to anything I touched. Odd. OutOfMemoryErrors all over the place. Most odd. Hudson’s tomcat has 1G, which should be plenty. Same with each build’s MAVEN_OPTS.

Apparently, someone who wrote the Maven Surefire Plugin thought that it would be a GREAT idea to ignore things like MAVEN_OPTS and other memory settings. The plugin seems to start a new JVM instance to run the tests. Without any of the arguments you so carefully selected. No. Apparently you have to explicitly tell the Surefire plugin that maybe, just maybe, it would be a good idea to use the memory settings you already provided elsewhere.

Anyhoo, this fixed it:

      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.6</version>
        <configuration>
          <argLine>-Xmx512m</argLine>
        </configuration>
      </plugin>

DRY, you say? Not so much, eh.

Maven 3 resource filtering weirdness

Maven 3 is all nice and fast(er) and shiny, so I decided to upgrade a Maven 2 project to Maven 3. It (cl)aims to be backwards-compatible, so my consternation was pretty great when my build failed straight away. That’s to say, my tests failed. For some reason, my resources were no longer being filtered. Yup, ${property.keys} weren’t being replaced by values.

This struck me as being somewhat odd, because it worked fine with 2.2.1. A bit of debugging led me to the cause of the problem:

<!-- @Transactional can now be used as well -->

… apparently, the @ symbol is an escape character of sorts.

Considering that blurb on their website doesn’t even qualify as English, I’m not sure if this is a feature or a bug. But whatever. Removing that comment fixed the problem. Whoever came up with that bright idea (especially in an age where @annotations are as rampant as the black plague in the 14th century) probably deserves a spanking.