Related link: http://kerneltrap.org/node/view/3513

Recently the news came out (mostly from this kerneltrap article) that the 2.6 kernel tree was to continue to receive active development. Torvalds and Morton (the current 2.6 maintainer) have noted good results in incorporating patches into the 2.6 tree that in the past would have gone into a development tree. Because of this, they have decided to keep with the 2.6 tree for the foreseeable future, and only announce a “development” 2.7 tree when enough patches are submitted that could seriously disrupt the kernel’s stability.

Linus, who normally can do no wrong, received quite a bit of flack back in the middle of the 2.4 series when he allowed a completely different virtual memory manager to be dropped in. Linus argued that the code was good and the improvements were needed, but sysadmin who had finally trusted 2.4 enough to migrate servers to it from 2.2 complained that any major development should have gone in 2.5. That combined with some fairly serious stability problems in certain 2.4 releases (2.4.11 comes to mind) have made quite a few administrators, myself included, a bit gun shy about the “stable” kernel series.

As a sysadmin, one of my main goals is to ensure that the servers I run stay up. Unless I’m forced to upgrade due to an exploit, some of my servers might be a few kernel revisions behind, as it simply isn’t worth downtime just to have the latest kernels. Currently, all of my machines are on the 2.4 series, along with some of my desktops. I have started to play with 2.6 on a desktop or two since around 2.6.3 or so, but I still am a bit reluctant to use it on many machines, and definitely not any production servers. It’s a shame, because the 2.6 series is definitely faster, works better with ACPI, and has a lot of other interesting features.

In many ways, I have always looked at the early releases of the “stable” kernel series a bit like I look at Debian testing/unstable. Most of the time, Debian unstable packages are pretty solid (I’d argue often more stable than the packages other distributions put on their *.0 releases), and even when they aren’t, if you have some Debian experience, you can generally fix the problems that do crop up.

I love Debian unstable, and I use it on all of my desktops, however I don’t use it on any of my production servers. I’m starting to get a similar feeling about the 2.6 kernel series. Before this shift in the way Linus and Morton handle patches to the stable series, I was planning to start the process of migrating servers to 2.6 “after it stabilized” ie. some time after 2.6.10 or so. Now, I’m really not so sure.

The way it appears now, the 2.6 series will be treated like Debian unstable–use it, but you should know what you are doing, and if you use it on a server, you deserve what you get–or possibly a blend between that and testing. The 2.7 kernel, or the -mm series for 2.6 will be more like Debian experimental.

So, when will you deploy 2.6 on your servers? On your desktops?

Related link: http://www.garloff.de/kurt/linux/ddrescue/

One thing I like about Linux is that there are always tools out there to discover that save you a lot of time or in this case data. I always love to walk a new user through transferring files from one drive to another and tell them “now run watch df to check the output.”

“Watch? Oh WOW why didn’t I know about this before?”

Well, I had another one of those moments the other day when I ran across dd_rescue. This utility addresses the issue of using dd to image a drive that has bad blocks or otherwise has errors. When dd hits an error, it fails, so you are left with using skip and seek arguments to try to route around the bad blocks. Dd_rescue addresses this by simply skipping bad blocks it finds (or trying X number of times before skipping if you wish), so you can image even failing drives full of bad blocks.

Of course, dd_rescue has its problems as well. While it does skip bad blocks, it takes it some time to do so, and since bad blocks are often grouped together, it can take a very long time to dig through a group of bad blocks. If this group is at the front of your data, you are wasting the valuable last breaths of your hard drive looking at bad blocks, instead of recovering good ones.

The dd_rhelp script addresses this by acting as a front-end for dd_rescue. When dd_rescue finds a bad block, dd_rhelp has it jump ahead a bit, and then read blocks in reverse until it finds the tail end of the bad block group. It then continues forward and grabs all of the good blocks first. Then, after it finishes grabbing your good data, it goes back to the bad data and attempts to recover what it can from there.

dd_rhelp has an added advantage in that it can also resume at any point in the recovery by keeping track of where it is in dd_rescue’s logs. So if you have to Ctrl-C from the process, you can just re-run it and dd_rhelp will pick up where it left off. Also, it has a nice little ASCII progress meter, which is useful for keeping track of where you are.

Of course, it makes sense to not run this rescue while you are booted into the bad drive, so either move the drive to another system, or use some sort of rescue disk like Knoppix.

So, to use this, download dd_rescue version 1.03 or later (1.03 is also packaged in Debian unstable at the moment, but testing seems to have 1.02) from its site, and then get the latest version of dd_rhelp. While older versions of dd_rescue will work by themselves, dd_rhelp requires 1.03. Building dd_rhelp is a simple matter of untarring and running ./configure && make, and it even compiled just fine under Knoppix.

Once the tools are ready, you can recover a drive with

# dd_rhelp /dev/hda1 /mnt/hdb1/hda1_rescue.img

Then run fsck on the image you have created, mount it loopback (mount -o loop /mnt/hdb1/hda1_rescue.img /mnt/hda1 in my case) and then browse through it and recover the files you need.

Now you can see if you still have a warranty on the drive and return it, demolish it with a sledgehammer, or take it apart and use it as a desk accessory.

What are some of your “click of death” horror stories? IBM Deathstar or otherwise!

Related link: http://www.securityfocus.com/news/9100

“Zero day exploits are upon us” and “zero day vulnerabilities are imminent” claims this paper. Funny they’d mention :-) Some “industry insiders” have long claimed that zero-days have been with us for a long time, we just didn’t know (which is the whole point of a novel attack - a “zero day” - people don’t know when its used…)

I should note that the whole concept of exposure window (”window shrinks between the discovery of vulnerabilities and the exploits that follow them”) doesn’t really apply to zero days. Here is why: the public “discovery” of a vulnerability happens after somebody is actually hacked through one of those holes using an exploit.

Another funny part is that they imply that timely patching will somehow help with zero-days. While system hardeding might help (if you are lucky), patching probably won’t - zero day implies there is no patch…

Related link: http://www.post-gazette.com/pg/04187/341822.stm

You always hear people talking about “ubiquitous computing”–the idea that at some point computers will become so pervasive that you will just hop from one computer to the next throughout your daily life.

According to this article on post-gazette.com, researchers in Pittsburgh are developing an application to manage virtual machines capable of suspending an application on one computer, moving to another computer, and having all of the applications and files resume locally on the new computer.

In many ways it’s somewhat like the screen command, or something like the work done by the AT&T Labs in Cambridge except in those cases the application stays running on a server somewhere and you just connect and input and output moves across the network. In this case it seems files are actually transferred locally to whatever machine you are connected to.

It’s an interesting concept, and I look forward to seeing where their research leads. In the meantime, though, you can do a lot of the same things just with a Knoppix CD and a usb key drive.

Here’s how:

One of the big things that has made Knoppix so popular is just how well its automatic hardware configuration scripts work. You can take a Knoppix CD to basically any computer, boot it, and most if not all of the hardware is working by the time the desktop environment comes up.

Knoppix has packed in all sorts of useful tools from a complete office environment, web browsers, instant messengers, email clients, games, the list goes on, so that many people have started using Knoppix CDs so that they could use their favorite Linux tools when they weren’t in front of their main Linux machine. Knoppix is really good about not doing anything potentially dangerous to the underlying system until it is explicitly told, so you could boot up Knoppix on your friend’s computer, or even a computer at Best Buy, and shut it down without leaving any tracks or damage.

Knoppix has the ability to use a writable device like a usb key drive, as a persistent home directory. This means all of your files and settings stay on the usb drive, so you can carry it around with you, boot on any machine you are in front of, and pick up where you left off.

A persistent home directory is pretty simple to set up. Basically you need some kind of writable media you can carry around with you. Usb drives, floppy drives, or even your iPod qualifies. Boot into Knoppix with the device plugged in, and make sure you can access it, then click Kmenu->KNOPPIX->Create a persistent KNOPPIX home directory (or run /usr/sbin/mkpersistenthome from a console).

When the wizard launches, pick the drive to use, and tell Knoppix to create a loopback file on the device (basically a complete filesystem written to a single file, like CD Images). You can choose how much space you will allow for this home directory, so you don’t have to worry about filling up your USB drive or iPod if you don’t want to. The wizard will create the loopback file and copy your complete home directory to the loopback file, and then you are done.

Next time you boot, if you add the cheat code home=scan to the boot prompt, Knoppix will scan all drives on the system (including your usb key drive, iPod, etc) for the loopback file. All of your settings and files on your desktop are where you left them, and you can get right back to work. Any changes you make in your home directory are written to this loopback file, so changes stay persistent across reboots.

There are a lot of different persistent options in Knoppix, so many I can’t really fit them here, but if you want to get a head start on this whole “ubiquitous computing” thing, I recommend grabbing a usb key and giving Knoppix a look.

Related link: http://www.immunitysec.com/education-unethicalhacking.shtml

That caught my attention since I suspect lots of people are tired of various hacking classes, teaching you how to use nmap, then go search PacketStorm for an exploit and then “penetrate” (and then pretend they now “think like a blackhat” :-)). But what if no exploit is posted - does that mean the target is secure? Of course not! Here is this course from a bunch of really smart people at Immunity Inc.

It says “We do not teach you how to run Nessus, Nmap, or that you should have a policy against SNMP on your network. We do teach you how to write exploits…” The outline covers exploit writing and the real penetrating methodology of an expert attacker, not that of a “script kiddy”…

Related link: http://www.securityfocus.com/columnists/252

Finally somebody said it soooo well: “Windows is easily the most complicated of the three operating system families that I’m most familiar with, those being BSD, Linux, and Windows.”

I just smile when people say that UNIX is hard and Windows is simple. Recenly I was playing with various security settings in Windows 2000, trying to harden an exposed machine, running file sharing and IIS. Wooow, UNIX hardening is so easy and straightforward by comparison… Just thinking of “local policy objects” makes one’s mind dizzy :-)

The article summarizes it nicely: “Windows is horribly complicated.”

For the longest time, virus writers (I mean viruses and worms both when I say “virus” here, just for brevity’s sake) have mostly been motivated by curiosity. In the “old days,” with distribution methods limited primarily to floppy disks, part of the “fun” of writing a virus was to see how far it could spread, and I suppose in some cases, how much damage it could cause.

Today most viruses spread via the Internet, whether in an email, p2p network, or just by exploiting a known security hole. The primary motive for a long time has still seemed to be curiosity, with the main payoff seeing how far a virus would spread.

The payload in viruses has typically either been very destructive (infection leading to deleting some or all data on a drive or otherwise destroying a system) or fairly harmless (asking the user for a cookie in the case of Cookie Monster, or just focusing on spreading to other hosts). Most of the major viruses in the past 5 years really only did harm because of the how zealously they searched for new hosts, resulting in lots of network traffic.

Recently, I’ve noticed quite a few viruses with a slightly different payload. Instead of just primarily being focused on spreading, a lot of recent viruses have also starting using infected hosts as zombie machines for other purposes–either DDOSing an enemy, or (and this is becoming more and more common) using the infected host as a spam relay, or even both.

This new payload has introduced a profit motive into the virus-writing business. With spam relays being cracked down upon and blacklisted more and more, an enterprising programmer could probably make quite a bit of money through writing a successful virus that was able to send X million spam messages.

The money involved in such an enterprise is fairly predictable too. You can fairly accurately estimate how many hosts would be vulnerable to a particular exploit, and there is a baseline number of machines that are never patched, and that will almost always be infected. These machines would be able to be used for quite some time by the virus writer, and yield the most profit. Above that are the machines that will eventually get patched, but probably not until and unless the virus does something rather disruptive, like make the computer or Internet connection incredibly slow. Finally are the computers that are patched frequently, which can still possibly be used for a short time by the virus writer.

All in all, some simple math and a little research can tell an enterprising virus writer how much money could be made with a virus exploiting a random vulnerability. Take a typical machine:

seconds of infection * spam sent per second * dollars per spam = profit per machine

I say “profit” because other than time, the main “cost” of distributing the virus, bandwidth, is managed and shared by every infected machine on the Internet–the costs are negligible to the virus writer.

Now, introducing a profit motive to writing viruses really changes the landscape a bit. For one, money will attract some people that may not have been willing to accept the risk of being arrested purely for curiosity’s sake. Secondly, money will attract unscrupulous people who may not even know how to write a virus, but might be willing to subcontract out that part to someone who does. Third, you might just see some spammers cut out the middleman and hire people to handle this part of the distribution themselves.

The result ends up being more people writing viruses (in some cases more talented people, or at least more dedicated people) which results in more viruses. This also means extra motivation for people to exploit security holes (especially unpatched ones) more quickly, as that means more hosts spreading the virus, which means more money.

I wouldn’t be surprised if in the future, that pimply-faced teenager in a t-shirt and jeans you see the FBI hauling off to jail for writing the latest virus, is replaced by a young adult in a suit and tie.

Do you think a profit motive has changed / will change virus writing at all?