PDF image conversion

Someone silly sent me a metrick fuckton of images embedded in PDFs. They wanted this stuff on the intertubez and they wanted it now. I’m lazy, so I automated the process. Figured I would write it down so I can remember it.

Imagemagick’s convert tool was my first weapon of choice, but this resulted in really poor output. So I decided to combine it with the pdfimages tool. The following snippet assumes the PDF contains only one image of interest.

pdfimages input.pdf tmp
convert tmp-000.ppm output.jpg

Wrap a loop around it and you’re done. It’s almost too simple to be true.

“Failed to start domain – Host CPU does not provide required features: spec-ctrl”

[root@foo ~]# virsh start bar
error: Failed to start domain bar
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: spec-ctrl

After a recent CentOS update and reboot, certain VMs refused to start, bailing out with the error message above. The interwebz didn’t really offer much in terms of advice. After talking to people with more clue, a working theory was formed: the version combination of kernel/libvirt/kvm/qemu is messed up. Rolling back to an older version was not an option.

The root cause is Intel’s Spectre vulnerability and its mitigation. The fix proved to be surprisingly simple. Simply edit the VM definition (“virsh edit foo”) and remove “-IBRS” from the CPU definition. Bear in mind that this does disable the Indirect Branch Restricted Speculation mitigation, so consider this a security disclaimer.

To recap:

  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Haswell-noTSX-IBRS</model>
  </cpu>

becomes

  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>Haswell-noTSX</model>
  </cpu>

I’m not sure how this CPU model definition came to be. Did updating libvirt update the definition? Was the definition automatically detected when the VM was created, and did it stop working after an upgrade?

On pointless Java 8 embellishments, or an exercise in simplicity

While performing a code review, I stumbled upon this little gem:

Long num = someFunctionThatNeverReturnsNull();
if(Optional.ofNullable(num).orElse(0l) > 0) {
  // ...
}

The salient part, of course, is the if-statement. It’s rare to come across a single line with so many layers of wrongness.

  1. First, let’s talk about 0l. Depending on your font, that might look like zero one, zero el, o one or o el. By convention in Java, this should be zero EL: 0L. This is easier to read, and the suffix L makes it clear that we’re dealing with a long instead of an integer. That would look something like this:
    if(Optional.ofNullable(num).orElse(0L) > 0) {
      // ...
    }
  2. Second, what’s going on with these data types? 0L is now obviously a primitive long which will be auto-boxed to a Long object. num is a Long object. And then there’s the dangling 0. Which is a primitive integer (int). For one reason or another. Now, I admit that casting to long or int is pretty cheap. This is never going to be a performance issue. But consistency is a good thing. You probably want to compare like with like.
    There’s also a bit of weird auto-(un)boxing going on here. 0L will be auto-boxed to Long. But the result of the orElse() bit will be unboxed to a primitive long. The comparison will then be comparing a long to an int, which causes the int to be widened to a long.
    There isn’t much we can do about the auto-(un)boxing in this case, considering the comparison we want to perform. But we can at least ensure we’re using consistent data type width. So this:

    if(Optional.ofNullable(num).orElse(0L) > 0L) {
      // ...
    }
  3. And lastly, why the fuck are they using Optional in the first place? Never mind the fact that num can’t even be null here. The “plain”, non-embellished form of this statement would have been shorter and easier to read. That would look something like this:
  4. if(null != num && num > 0L) {
      // ...
    }

Here is the “corrected” version:

long num = someFunctionThatNeverReturnsNull();
if(num > 0L) {
  // ...
}

I can’t fathom why anyone would write garbage like this. It’s pointless. It’s hard to read. It’s ugly. And it’s bloody inefficient.

On brute force stupidity

Everyone who’s ever managed any internet-facing server is aware of the ridiculous amount of brute force SSH login attempts by all kinds of botnets. Some folks decide to move their SSHD to a non-standard port, some rely on complicated shenanigans like port knocking, and some use tools like fail2ban. I’m unfortunate enough to manage a little over a dozen servers, so I decided to have some fun with fail2ban.

[sshd]
enabled = true
banaction = %(banaction_allports)s
 
[recidive]
enabled = true

My configuration is pretty straightforward. You fuck up, you get banned. You fuck up repeatedly, you get banned for a longer time. Nothing special there. Given that I’m running a similar config on many boxes, I decided to compile some data relating to the origins of login attempts. This data was collected over a period of ~2 months on ~12 servers.

Here’s a quick plot of the number of times a certain IP address was banned. Only the top 100 abusers are included, because the chart has a very long tail indeed. I removed the IP addresses from the X-axis because there’s no way to include them without turning into a black blob.

It should be immediately obvious that a relatively small number of IP addresses is responsible for a metric fuckton of unwelcome activity. Remember that this represents the number of times an IP was banned. Left unchecked, the number of attempts increases by orders of magnitude.

The top offender (and the only one whose full IP address I’ll publish) is 116.31.116.38. It’s part of a Chinese subnet. It managed to get banned a staggering 4466 number of times. More than the next 5 abusers combined.

As the following chart illustrates, a whopping 76% of these IP addresses belong to Chinese subnets.

I daresay the internet would be a slightly better place if those 100 machines were permanently disconnected. It’s likely they’re just unsuspecting folks with compromised machines. But I for one am permanently firewalling all of them on any box I have access to.

Zimbra Strong SSL/TLS Cipher Suites

Apparently, even in 2016, Java (and by extension Zimbra) ships without support for strong crypto. Geopolitics == retarded.

Anyway, in order to get strong crypto up and running in Zimbra Webmail, you’ll need to download the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files for JDK/JRE 8.

Unzip the file and overwrite the JARs in /opt/zimbra/java/jre/lib/security.

But wait, that’s not enough. A bunch of default cipher suites are old and weak, or are vulnerable to the Logjam Attack. You can disable all that cruft as follows:

zmprov mcf +zimbraSSLExcludeCipherSuites SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_DHE_DSS_WITH_DES_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_DHE_RSA_WITH_DES_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_RSA_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_RSA_EXPORT_WITH_RC4_40_MD5 \
+zimbraSSLExcludeCipherSuites SSL_RSA_WITH_DES_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_RC4_128_MD5 \
+zimbraSSLExcludeCipherSuites SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_RSA_WITH_3DES_EDE_CBC_SHA \
+zimbraSSLExcludeCipherSuites SSL_RSA_WITH_RC4_128_MD5 \
+zimbraSSLExcludeCipherSuites SSL_RSA_WITH_RC4_128_SHA \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_ECDHE_RSA_WITH_RC4_128_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_EXPORT_WITH_DES40_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_3DES_EDE_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_AES_128_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_AES_128_CBC_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_AES_128_GCM_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_RSA_WITH_DES_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_128_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_256_CBC_SHA \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 \
+zimbraSSLExcludeCipherSuites TLS_DHE_RSA_WITH_AES_256_GCM_SHA384

Finally, restart zmmailboxdctl. You’re good to go :-).

JAX-RS Client Filters

Life goes on with JAX-RS/Jersey. I wasted a couple of moments figuring out how to add custom headers to a Jersey generated JAX-RS client. Might as well write it down in the hope of saving someone a couple of minutes.

For starters, you’ll need a Client Filter that does the actual heavy(ish) lifting.

import javax.ws.rs.client.ClientRequestContext;
import javax.ws.rs.client.ClientRequestFilter;
 
/**
 * Add the X-GARBAGE header to all requests.
 */
public class GarbageFilter implements ClientRequestFilter {
	@Override
	public void filter(final ClientRequestContext requestContext) throws IOException {
		requestContext.getHeaders().add("X-GARBAGE", "This is added to all requests");
	}
}

And then you’ll have to register the filter with the Client(Config).

// import org.glassfish.jersey.client.ClientConfig;
 
final ClientConfig clientConfig = new ClientConfig();
clientConfig.register(new GarbageFilter()); // Yes, you could use JDK8 magic :-)
 
final Client client = ClientBuilder.newClient(clientConfig);

And that’s all. Every request you launch using the generated client will now contain your X-GARBAGE header.

JAX-RS Client File Upload

Another hiccup in using the wadl2java client generated from a (Jersey) JAX-RS app. This time, it concerns multipart/form-data.

The method:

	@POST
	@Produces(MediaType.APPLICATION_JSON)
	@Consumes(MediaType.MULTIPART_FORM_DATA)
	public JsonFoo create(@FormDataParam("file") InputStream data, 
		@FormDataParam("file") FormDataContentDisposition fileDetail, 
		@FormDataParam("file") FormDataBodyPart bodyPart) {
			// Implementation foo bar baz
	}

The WADL exposes something that looks like this.

<method id="create" name="POST">
	<request>
		<representation mediaType="multipart/form-data"/>
	</request>
	<!-- Response omitted for the sake of brevity -->
</method>

And the generated client has a method to go along with it. Unfortunately, it gives you no hints whatsoever as to how to actually provide a file/data.

// Long class names shortened
public static Create create(Client client, URI baseURI) {
	return new Create(client, baseURI);
}
 
// The Create object contains this little gem
public<T >T postMultipartFormDataAsJson(Object input, GenericType<T> returnType);

That’s wonderful. Unfortunately, if you pass in a java.io.File, nothing happens. The client barfs.

Many DuckDuckGo-searches, StackOverflow hunts and headscratchings later, I came up with a working solution:

// import import org.glassfish.jersey.media.multipart.FormDataMultiPart;
// import org.glassfish.jersey.media.multipart.file.FileDataBodyPart;
 
File file = new File(); // Your file!
FormDataMultiPart form = new FormDataMultiPart();
form.field("filename", file.getName());
form.bodyPart(new FileDataBodyPart("file", upload, new MediaType("image", "jpeg")));
 
Api.create(client, uri).postMultipartFormDataAsJson(form, new GenericType<CreateResponse>() {});

But wait! That won’t cut it. You also need to tell your Client that you want to use the Multipart Feature. Makes sense. If you don’t, you’ll end up with this exception.

org.glassfish.jersey.message.internal.MessageBodyProviderNotFoundException: MessageBodyWriter not found for media type=multipart/form-data, type=class org.glassfish.jersey.media.multipart.FormDataMultiPart, genericType=class org.glassfish.jersey.media.multipart.FormDataMultiPart.
// import org.glassfish.jersey.client.ClientConfig;
// import org.glassfish.jersey.media.multipart.MultiPartFeature;
 
final ClientConfig clientConfig = new ClientConfig();
clientConfig.register(MultiPartFeature.class);

And there you have it. File upload with JAX-RS and a wadl2java generated client.

JAX-RS vs Collections vs JSON Arrays

The Problem

Looks like I’ve managed to get myself into a pickle. I have a (Jersey) JAX-RS app, which automagically publishes a WADL. If I then generate a Client (using wadl2java), I end up with generated code that doesn’t work.

	@GET
	@Produces(MediaType.APPLICATION_JSON)
	public List<Foo> get() {
		return myList;
	}

Foo is a simple POJO with an @XmlRootElement annotation. The resulting JSON is perfectly fine: I get a JSON Array with a bunch of Foo objects. Perfect!

The generated WADL is a different matter:

    <method id="get" name="GET">
        <response>
            <!-- foo is a reference to an XSD Complex Type ... but where's my list? -->
            <ns2:representation element="foo" mediaType="application/json"/>
        </response>
    </method>

For some reason, the WADL generator is smart enough to realise that we’re dealing with Foo instances. But it’s too stupid to realise that we’re dealing with more than one.

If you then generate a Client using wadl2java, you’ll end up with something like this:

    public Foo getAsFoo() {
        return response.readEntity(Foo.class);
    }

Well that’s not going to work, is it? Trying to read a single Foo when you’ve got an array of them. And indeed … you get a wonderful exception.

Internal Exception: java.lang.ClassCastException: Foo cannot be cast to java.util.Collection

This seems to be a fundamental XML vs JSON problem. After all there is no such thing as an “XML Array”. Either you have one element, or you have multiple elements nested under a common root.

I could solve this by not relying on a plain JSON Array and encapsulating the result.

// Not this:
[
   {
      "foo":"bar"
   },
   {
      "bar":"baz"
   }
]
 
// But this:
{
   "listOfFoos":[
      {
         "foo":"bar"
      },
      {
         "bar":"baz"
      }
   ]
}

But then that’s ugly. And instead of using a native Java collection I’ll have to create a useless intermediary object. And it would break compatibility with current API clients.

The Solution

Whelp … I haven’t found one yet. To be continued, I hope. But I did manage to find a workaround, thanks to Adam Bien’s blog.

The incorrectly generated getAsFoo() method doesn’t work. But we can use getAsJson() instead — which doesn’t necessarily have to return a JSON string.

    List<Foo> foos = client.getFoo().getAsJson(
        new GenericType<List<Foo>>() {
        }
    );

GenericType is a dirty JAX-RS hack in my opinion, but it works. It’s a shame that I have to rely on getAsJson(), though. It would’ve been much cleaner to use the getAsFoo() method directly.

Endless Locale Bugs

This isn’t the first time I’ve ranted about Locale related issues. It probaly won’t be the last, either.

Currently my biggest gripe is with convenience methods in Java which rely on platform defaults, and are thus platform-specific. For instance, the venerable System.out.println() will terminate your line with whatever your platform thinks a line ending should be. Printing dates or numbers will try to use your platform Locale. Writing a string to a file will default to platform encoding.

Some of these defaults can be controlled at run time, others require JVM startup options. This is all horribly wrong. This results in all kinds of unexpected behaviour. It’s error-prone. None of this should be allowed to happen. Any method that assumes platform-specific magic should be deprecated. Which is exactly what I’ll do, as soon as I can figure out how to write a Sonar plugin to detect all this nonsense.

Getting started with JAX-RS & Gradle

Managed to waste an hour of my life today, trying to set up a JAX-RS + Gradle project. Gradle wasn’t really the problem, nor was JAX-RS, but putting it all together in a way that works took longer than expected.

So without further ado, I present to you a simple quickstart project you can use to kickstart JAX-RS + Gradle projects.

The Gradle build file is pretty minimalistic. There’s no need for web.xml. And I’ve included an example resource.

build.gradle:

description = 'JAX-RS Quickstart'
group = 'org.lick.me.jaxrs'
version = '1.0-SNAPSHOT'
apply plugin: 'java'
apply plugin: 'eclipse'
sourceCompatibility = 1.8
 
repositories {
        mavenCentral()
}
 
 
buildscript {
        repositories {
                jcenter()
        }
        dependencies {
                // Embedded Jetty 9 Gradle Plugin
                classpath (group: 'com.sahlbach.gradle', name: 'gradle-jetty-eclipse-plugin', version: '1.9.+')
        }
}
apply plugin: 'jettyEclipse'
apply plugin: 'war'
 
dependencies {
        // JAX-RS 2.0(.1)
        compile 'javax.ws.rs:javax.ws.rs-api:2.0.1'
 
        // Jersey 2.15
        compile 'org.glassfish.jersey.containers:jersey-container-servlet:2.15'
 
        // Servlet 3
        compile 'javax.servlet:javax.servlet-api:3.1.0'
}

Extra quick quickstart:

git clone git@github.com:Nimlhug/jax-rs-quickstart.git
cd jax-rs-quickstart
gradle jettyEclipseRun