Deutsch

Current Texts Comic Imprint Calendar Search PHP-Classes Container-Wizard main.s21

Categories

The Debian SSL Issue

Yesterday an entry appeared at the heise newsticker which might become one of the most severe software bugs ever: a flaw within Debian's version of OpenSSL that has existed for about two years and results in very weak keys, compromising not only the keys, but any transmission protected by them.

The Course of Events

Testing OpenSSL with Valgrind, a Debian developer stumbles across warnings issued by that compiler about the use of uninitialized memory. Looking into the source, he finds that there is already a command left in the code for another compiler to ignore the offending line. He then decides to remove it altogether, which would not have impacted the randomness of the number generator. However, he finds another line which looks similar and decides to remove it as well, breaking the RNG in the process: from now on, the RNG would produce predictable numbers, limiting the effective length of all keys generated by that version of OpenSSL to about 16 effective bit, a bitlength that is easily broken.
He however goes to the openssl-dev mailing list, asking about the impact of his changes and gets no stiff opposition. He therefore issues his patch to the Debian repository. No further testing is done and no person in charge objects to it. It is therefore inserted into the mainstream distribution and released five days later. From thereon, every key generated with OpenSSL in Debian is ineffective, and every encryption done by them can be broken easily.
Discovering the Bug nearly two years later, the Debian team immediately issues a patch and offers a tool to check keys. Up to now, (14.05.2008 21:29:56 CET), Debian however has failed to bring up conclusive hints on how to handle the issue.

The failures

Looking at the case, this is a typical chain of events that lead to a catastrophic outcome. The mechanisms which should have prevented such an accident where either ignored, not effective or not even there:

1. The OpenSSL team uses an unusual pattern to add more entropy to the key. I lack knowledge in C, but I trust that other people mentioned it to be an uncommon pattern. They fail to document the code.
   It is in general a good idea to document code, especially when you're doing something unusual. While you of course expect that no one will meddle with what he does not understand, it is clear that people will do anyway - and that it is certainly better to give them a clear warning and explaination why they should not.
2. A contributor to Debian works with Valgrind on OpenSSL and becomes annoyed by the warnings issued. He does not really know what the code does and he does not issue it to OpenSSL.
   It is not a very good idea to meddle with code you don't understand. But you also have to take into consideration what are the stakes. We're talking about an aesthetic nuisance in a very important library. Additionally, crypto code is hard to understand and failures might be subtle. Even a botched kernel patch would have been less serious - once it's been fixed, it's fixed, but here we talk not only about the present, but the past two years.
3. The people he asks on the mailing list fail to see that he is breaking the code. They also fail to see the possible consequence, as the contributor is not just some noob messing around with code, but a Debian contributor instead.
   I read in discussions how they complained about 'noobs' annoying them on their list. Surely, guys, the arrogant attitude of some people is a serious problem when relying upon the community developing software superior to some Closed Source products. Open Source is about sharing knowledge, which frankly also means to answer "stupid" questions from time to time.
4. No testing is done on the code even though it is clear that a very important component has been changed; the maintainer does not object.
   It is wrong to just blaime the contributor. The whole process seems to be badly broken when some contributor can change crypto code without a very, very good reason. Also, no tests where done, which would have needed to be extra strict here-
5. After it was deployed with the new release of Debian, apparently no one noticed the error for two years. It could have been noticed in the code and by checking the strength of the keys.
   Open Source proselytizers always recite the mantra that so many people look into the source, that bugs are fewer than with that black box Closed Source. Frankly, I'm pretty sure that a similar bug within Windows Vista would have been uncovered much more quickly, as people would have raced to find a severe bug within crypto modules supplied by Microsoft.
6. Debian finally notices the error and issues a patch, but fails to immediately deliver tools necessary to check all possibly compromised keys and they do not give recommendations on what to do.
   Just seeing a homepage with 'more on that later' forever really ticked me out. What to do? I would not care if I'm just talking about my private development server, but I have an infrastructure to maintain and people relying on it. I can really imagine that people with real infrastructures found it even less funny.
7. People in the community start arguing about who is to blame. Discussion about what happened often enough degenerate into personal insults or attacks on closed source software.
   Im just not interested in whether Microsoft would have done worse or would have waited until the next 'Patch Day'. I'm interested in how to deal with the problem and what consequences will be taken from it by those in charge. Additionally, I even think Microsoft would have handled better, given their detailed Knowledge Base-Articles. Also, it does not help if Debian-Fanbois and OpenSSL-people duke it out on who's got to blame.

The Aftermath / What to learn

The flaw is surely one of the most serious bugs ever, I even go as far to think of it as the worst bug ever, due to its massive impact not only the present, but the past too. We have to see that in worst case, two years worth of encrypted communication is broken. People relying on that encryption will suddenly find themselves exposed. And even only one compromised machine can very well mean a larger damage. If you generated a good key over a bad connection, that good key has to be seen as compromised also, which could be very well the start of a long chain. In the end, would my own infrastructure had been compromised, I would have needed to recreate the whole infrastructure from scratch, as I could not have been sure anymore what I can consider secure and what not. One might say that this is over the top, but in the end the only safe way to ensure that you're secure again.

Given that, this bug will haunt the Linux world for a time to come. I found out that a major german hosting provider is vulnerable as well - we're talking a few million websites here. Now knowing about reality, I find it hard to imagine that this problem is fixed quick; those with experience know well enough that often the Powers That Be care less about seemingly elevated security issues when they have other pressing business to be done.

Also, the bug will surely shake the confidence of some in their "mine is longer than yours"-keys. Personally, I greet that; I'm fed up with "know it alls" boasting about their secure operating system and how stupid everyone else is. Even though they think they are in "full control" of their system, the fact that such a bug can fester undiscovered for about two years surely demonstrates that just knowing how to operate a bash does not mean to be in full control. Frankly, not even compiling everything from scratch guarantees that unless you've read and understood the sources. Everything of it.

Said that, the Open Source community would do best really adressing to that problem instead of going to patterns like "It would have been even worse with Closed Source" and "It was someone else's fault". Face it - Open Source has been beaten on it's very own hometurf: one of the major arguments of OS apologetics is that bugs in Open Source are more likely to be found and that it can be especially trusted for high security applications such as cryptography. Debian and the community surely has failed to cash in on that one. If they fail to accept it and learn from it - Microsoft and other closed source vendors surely will.

Now as we surely all agree that it was a Bad Idea to mess with the source for no important reason in the beginning, one should learn from it and first begin to see that vendor patches as such should be thought over. And even if distributors want to continue with their practice of enhancing third party products downstream, they should at least think about which packets to modify and out of what reasons. Crypto is so critical that it should almost never be touched on the vendor side, not even by the most intelligent code gurus. As of now, I can think of no viable reason why it should be touched altogether, but certainly not because some debugging tool throws warnings. Even if it fails to build on a certain platform, I deem it to be better to have no crypto at all than a broken one.

Comments

Please note: comments posted won't be visible immediately, as they will be checked for harmful content.