SSL Gotchas

Here are some of the things that I've found out the hard way.

• OpenSSL

  Since writing the applications, I got hold of a copy of the book "Network Security with OpenSSL" by John Viega, Matt Messier and Pravir Chandra. I think if I'd had this to start with it would have saved me a lot of time - it has example code and documentation on the "openssl" utility that is much better than the current online stuff. The book is available from Amazon.

  A lot of things I've done wrong resulted in the error no shared cipher, which seems to be a bit of a catch-all.

  1. You must seed the (pseudo-random number generator) before using SSL. if you don't do this, then you sometimes get an error PRNG not seeded, but not always. Sometimes you get no shared cipher.
  2. If you want a server to use DH ciphers, you have to to initialize DH parameters (although the client side doesn't seem to have this requirement). If you don't do this, then although DH ciphers show up as enabled for the SSL context, any attempt by a client to negotiate one of these will result in a no shared cipher error. The fact that you have to do this is is mentioned on SSL_CTX_set_cipher_list(3), but I must not have read it.
  3. When you call SSL_CTX_set_cipher_list(3), you provide a string representing ciphers, the format of which is described in ciphers(1). The documentation for ciphers(1) talks about modifying a list of ciphers, which might lead you to believe that you could call SSL_CTX_set_cipher_list multiple times, modifying the context's list of ciphers. But when ciphers(1) refers to "list of ciphers", it's talking about the string that you build up, not any existing SSL context. I.e. when you call SSL_CTX_set_cipher_list, any previous cipher list attached to the context is overwritten.
  4. The type of certificate you have determines what kind of cipher suites you can negotiate. The certificates I used (as described elsewhere on these web pages) had RSA encrypted keys. This meant that when the server was using them, it would never negotiate any "DSS" cipher suites. It is possible to create certificates with DSA keys, in which case the "DSS" cipher suites do get used (but then the RSA ones don't). For example, to create a certificate with RSA encrypted key, I used:

     openssl> req -x509 -new -out nickcert.pem -config myconfig.cnf -keyout nickkey.pem
     

     Note that the documentation for "req(1)" says that if the "-key" option is not used, then an RSA key will be generated. The commands I've used to get a certificate with DSA key (thus allowing DSS ciphers) is:

     openssl> dsaparam -genkey -out nickdsaparam.pem 128
     openssl> gendsa -out nickdsakey.pem -des3 nickdsaparam.pem  (asks for pw)
     openssl> req -x509 -new -out nickdsacert.pem -config myconfig.cnf -key nickdsakey.pem
     

     It seems like java's "keytool" utility is the other way round - i.e. by default you get DSA key encryption, and you use "-keyalg" to specify RSA if you want it.

• JSSE

  1. When you create a keystore, you need (in the default implementation) to use the same password for the keystore as for any self-signed certificates you generate with keytool -genkey. In the standard implementation, JSSE appears to use javax.net.ssl.keyStorePassword for both passwords, and doesn't let you specify the password for an individual key. If you have different passwords, you're likely to get Cannot recover key exceptions.
  2. When you create a SSLSocket or SSLServerSocket, the JSSE doesn't do the SSL handshake until it has to. Typically this happens when you do the first I/O to the socket. This is so that after you've created the socket, you have the chance to set various options before the handshake. If the handshake fails in this case, you get an SSLException

     However, another thing that causes the handshake to occur is Socket.getSession(), which you might use to get info on, say, what ciphers are enabled. If the handshake fails in this case, you don't get an SSLException, and if you subsequently try and do I/O to the socket you'll get a IOException.

     You can look at SSLSession.getCipherSuite() which will be SSL_NULL_WITH_NULL_NULL in this case.

• Differences between JSSE and OpenSSL

  1. The way that the default JSSE implementation decides to trust a certificate is that it "starts at the server certificate and proceeds up the chain until a trusted certificate is found" (this quote is taken from a response from Sun in the java security archives). In other words, any certificate in the truststore counts as one that is trusted.

     The way that OpenSSL decides to trust a certificate is that it looks in its verify path to find a chain of certificates which must include one for the issuer (a certificate for the subject isn't good enough). I.e. it follows the chain and requires that the chain ends with a self-signed certificate.

  2. For an OpenSSL client, the default behaviour is not to verify server certificates. You have to call, e.g. SSL_CTX_set_verify to enable this. For a JSSE client, this verification is turned on by default, and I couldn't see an easy way to turn it off. You can achieve this if you want by implementing your own TrustManager and overriding the 'checkServerTrusted()' method.
  3. If you run an OpenSSL server and enable non-anonymous cipher suites but don't provide a server certificate, then you don't get an error until someone tries to connect with you. Then you get no shared cipher. If you do this with JSSE, you get an exception No available certificate
  4. If you run an OpenSSL server and provide a server certificate file but don't provide a private key, or if you provide the wrong private key,then you don't get an error until someone tries to connect with you. Then you get no shared cipher. On JSSE, the server certificate in a keystore contains its private key, so you don't see this situation.
  5. If you run an OpenSSL server and provide a server certificate and suitable private key, but give the wrong key-password, you don't get an error until someone tries to connect with you. Then you get no shared cipher. On JSSE you get an exception with the text Keystore was tampered with, or password was incorrect