June and July were bad months for free software. First the
chunked encoding vulnerability, and just when we’d finished
patching that, we get the
of these are pretty scary - the first making every single webserver
potentially exploitable, and the second makes every remotely managed
machine vulnerable.
But we survived that, only to be hit just days later with the
problems. Would it ever end? Well, there was a brief respite, but
then, at the end of July, we had the OpenSSL buffer overflows.
All of these were pretty agonising, but it seems we got through it
mostly unscathed, by releasing patches widely as soon as possible. Of
course, this is painful for users and vendors alike, having to
scramble to patch systems before exploits become available. I know
that pain only too well: at The
Bunker we had to use every available sysadmin for days on end to
fix the problems, which seemed to be arriving before we’d had time to
catch our breath from the previous one.
But I also know the pain suffered by the discoverer of such problems,
so I thought I’d tell you a bit about that. First, I was involved in the
Apache chunked encoding problem. That was pretty straightforward,
because the vulnerability was released without any consultation with
the Apache Software Foundation, a move I consider most ill-advised,
but it did at least simplify our options: we had to get a patch out as
fast as possible. Even so, we thought we could take a little bit of
time to produce a fix, since all we were looking at was a denial of
service attack, and, let’s fact it, Apache doesn’t need bugs to suffer
denial of service - all this did was make it a little cheaper for the
attacker to consume your resources.
That is, until Gobbles came out with the
for the problem. Now, this really is the worst possible position to be
in. Not only is there an exploitable problem, but the first you know
of it is when you see the exploit code. Then we really had to
scramble. First we had to figure out how the exploit worked. I figured
that out by attacking myself and running Apache under gdb. I have to
say that the attack was rather marvellously cunning, and for a while I
forgot the urgency of the problem while I unravelled its
workings. Having worked that out, we were in a position to finally
fix the problem, and also, perhaps more importantly, more generically
prevent the problem from occurring again through a different
route. Once we had done that, it was just a matter of writing the
releasing the patches, and posting the advisory to the usual places.
The OpenSSL
problems were a rather different story. I found these whilst
working on a security review of
the USAF. OpenSSL is a rather large and messy piece of code, that I
had, until DARPA funded it, hesitated to do a security review of,
partly because it was a big job, but also partly because I was
sure I was going to find stuff. And sure enough, I found
problems (yes, I know this flies in the face of conventional wisdom -
many eyes may be a good thing, but most of those eyes are not trained
observers, and the ones that are do not necessarily have the time or
energy to check the code in the detail that is required). Not as many
as I expected, but then, I haven’t finished yet (and perhaps I never
will, it does seem to be a never-ending process). Having found some
problems, which were definitely exploitable, I was then faced with an
agonising decision: release them, and run the risk that I would find
more, and force the world to go through the process of upgrading
again, or sit on them until I’d finished, and run the risk that
someone else would discovered them and exploit them?
In fact, I dithered on this question for at least a month - then
one of the problems I’d found was fixed in the development version
without even being noted as a security fix, and another was reported
as a bug. I decided life was getting too dangerous and decided to
release the advisory, complete or not. Now, you might think that not
being under huge time pressure is a good thing, but in some ways it is
not. The first problem came because various other members of the team
thought I should involve various other security alerting
mechanisms. For example, CERT or a
mailing list operated by most of the free OS vendors. But there’s a
problem with this: CERT’s process is slow and cumbersome and I was
already nervous about delay. Vendor security lists are also dangerous
because you can’t really be sure who is reading them and what their
real interests are. And, more deeply, I have to wonder why vendors
should have the benefit of early notification, when it is my view that
they should arrange things so that their users could use my patches as
easily as I can? I build almost everything from original source, so
patches tend to be very easy to use. RPMs and ports make this harder,
and vendors who release no source at all clearly completely screw up
their customers. Why should I help people who are getting in the way
of the people who matter (i.e. the users of the software)?
Then, to make matters worse, one of the more serious problems was
reported independently to the OpenSSL team by CERT, who had been
alerted by Defcon. I was going, and there
was no way I was delaying release of the patches until after
Defcon. So, the day before I got on a plane, I finally released the
advisory. And the rest is history.
So, what’s the point of all this? Well, the point is this: it
was a complete waste of time. I needn’t have agonised over CERT or
delay or any of the rest of it. Because half the world didn’t do a
damn thing about the fact they were vulnerable, and because of that,
as of yesterday, a
is spreading through the ‘net like wildfire.
Why do I bother?
Your comments are welcome.
How do we fix this problem?
Please keep bothering
As a member of the other half of the world
that did jump on your advisories and patch
my systems, let me just say -- thank you!
Yeah, I bitched about one patch after
another. But I'd much rather go that route
than get cracked, and my many thousands of
users with uninterrupted service would
also appreciate it if they only knew.
That worm will not spread to my half the
world.
Bob Goldstein
For the other half ...
Yeah, for the half that DID do something! And eventually half of the other half will learn too. And then maybe some more.
Kudos
Hey man, keep fighting the good fight. *insert better cliche here*
No really. I managed to patch up everybody so far.
And if I can do it, so can the lusers you're speaking of.
Now, about that version.75765.number.876876.Delta scheme of yours ..
Thanks,
adric@adric.net
(verdammt plaintext wesite acct password forgeting, mailing me, changing fogetting, mailing me ..)
I agree: Please keep bothering
I also installed each patch as soon as I found out about them. Thus, I didn't participate in the recent excercise.
Please keep bothering. There are those of us who do care.
Automatic updates
The solution is for more vendors to have services that allow users to automatically update their machines with security fixes - like the Red Hat Network, apt-get, and so on. Then the open source groups and people like you can continue to work to reduce vulnerabilities even in the absense of known threats and that will reduce the window of risk for anyone who bothers to keep their systems current.