How we protect against Poodlebleed (CVE-2014-3566)

Operations is driven by chaos. Yet we have another brilliant example: Poodlebleed. This is how we at the Flying Circus face it.

Yesterday, an exploit to a severe security issue in SSLv3 has been released. This exploit is known as Poodlebleed. SSLv3 is involved in encrypting communication over networks, e.g. when securing websites with HTTPS or encrypting email traffic using IMAPs/POP3s and STMPs. More information about the exploit can be read here and downloaded here (more details).

The bug

The exploit allows an attacker to expose data over time that is encrypted between SSLv3 compatible servers and clients. SSLv3 is quite old but still widely supported by nearly any software. Although using the exploit is not trivial and stealing information takes some time, we take Poodlebleed very serious.

We have verified that our managed components do not support SSLv3 by default. Further, we have scanned our whole platform for customer installations that have overridden these defaults and informed customers accordingly. Right now, our systems are reasonably safe again.

In the following, we will explain in more detail which measures have been taken and what our customers and others can do to come up against Poodlebleed in their own installations.

The fix in brief

Because SSLv3 is a general protocol, the issue can't be fixed by updating a specific software to a non-vulnerable version. The approach is to avoid the usage of SSLv3 and vulnerable ciphers in any software that supports them.

HTTPS via nginx and Apache2

Our managed nginx configuration was not vulnerable since SSLv3 is forbidden as a valid protocol by default. Customers that have overridden this default have been informed.

Some of our managed Apache2 installations were vulnerable and have been fixed by disabling SSLv3 and limiting the available cipher suites (Thanks Hynek!) in every Virtual Host configuration block that has SSL enabled:

<VirtualHost _default_:443> 
... 
SSLProtocol ALL -SSLv2 -SSLv3 
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
... 
</VirtualHost>

Vulnerable Apache2 customer configurations have not been found.

In the future, we will provide configuration snippets for both, nginx and Apache2 that can easily be included in own configurations. It will contain a safe set of SSL configuration options which we will maintain. This way, customers do have to worry less about safe defaults.

SMTPs via Postfix

We have decided not to disable SSLv3-Support on our central mail server. In our opinion, mail servers (not clients!) have to be treated differently because emails need to reach their recipients, even if the next mail server we pass them on to does not apply the same safety measures as we do.

Customers who want to disable SSL support in their installations can find further information here.

POP3s/IMAPs via Cyrus and dovecot

We have disabled SSLv3 support in our central IMAP and POP3 services (Cyrus) by limiting available ciphers (Thanks again Hynek!) in imapd.conf to:

tls_cipher_list: ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

However, tools judge differently when determining whether we are still vulnerable. We are in contact with Hynek to further track this down.

We do not provide managed Cyrus installations and therefore no customer action is necessary (unless you have a self-compiled Cyrus).

For our managed dovecot installations we have also limited available ciphers in /etc/dovecot/conf.d/10-ssl.conf like we did for Cyrus:

ssl_protocols = !SSLv2 !SSLv3
ssl_cipher_list = ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS

LDAPs via OpenLDAP

We have limited the cipher suites that our managed OpenLDAP installations allow in /etc/openldap/slapd.10tls.conf:

TLSCiphersuite SECURE256:-VERS-SSL3.0:+VERS-TLS1.2:+VERS-TLS1.0:+VERS-TLS1.1

Customer configurations have already been adjusted by hand and will be permanently fixed upon the next Flying Circus release which we expect in the next days.

NRPE

In NRPE, which we heavily use to monitor our infrastructure, SSL configuration options unfortunately can't be changed and even worse, supporting SSL is the fixed default. Luckily, SSLv3 has been fixed-disabled, too. Our NRPE installations are therefore not vulnerable.

MySQL

Our managed MySQL-Server installation has never been vulnerable because it is not configured to support OpenSSL by default. This can easily be checked by:

litdev81 ~ # mysql
mysql> SHOW VARIABLES LIKE 'have_openssl';
| Variable_name | Value |
| have_openssl | DISABLED |
1 row in set (0.02 sec)
mysql>

PostgreSQL

Our PostrgreSQL installations are basically not vulnerable since they are not listening on public interfaces. However, customers should check whether they have changed the default listening ports and addresses.

Nevertheless, we will try to disable SSLv3 support in our managed PostgreSQL by default. Tomorrow is another day...

Aftermath and future improvements

In the next days we will roll out a Flying Circus release which contains permanent fixes for adjustments that have for now been made by hand.

Further, we have dependency issues on our platform which prohibit us from upgrading openssl to version 1.0.1 on our current Gentoo snapshot. This will change when rolling out our yearly OS update in the next weeks. openssl-1.0.1 brings support for TLSv1.2. At the moment we run on openssl-1.0.0m which only supports earlier versions of TLS that are no longer recommended. Having TLSv1.2 on board will allow us to disable SSL support completely (in favor of TLS) in a lot of our managed components.

Providing a safe set of options for a piece of software in a snippet file that can easily be included in own configurations is a good idea in general and we will provide this w.r.t. SSL-related options for nginx and Apache2. We will also evaluate how to bring this to other components.