The previous poll (vote here, live results here, analysis here) proved to be a success so the next one is here.

This time the question is: “Assuming that you centrally COLLECT system, network or security logs from their originating sources, what is THE MAIN reason for doing it?”

Vote on!

UPDATE 11/11/2007: results and analysis are posted here

See all my polls here.

Mark Wade wrote a firsthand account on tracking a purchase that was originally solicited from a SPAM email.

There are a couple of points where the article is vague on technical details, but I’m probably not the intended audience. I found the article to be informative, but it also left me with more questions.

First, I wanted more details on the domain registrations that were encountered along the way. Are the domains still active, or are they on administrative hold? I’m wondering if spammers pay for domains with stolen credit cards, knowing they may only have a couple of days to use it; or if they keep a cache of domains that they cycle through, and just move the site contents from domain to domain.

Second, the article mentions that somewhere along the chain of purchasing his item that a SSL certificate was encountered. A trusted SSL certificate is going to be a little harder for a spammer to get compared to a domain name. Remember, the whole point of SSL certificates are to make Internet commerce safe, and to give consumers confidence with a system of verifiable trust. Unless, the cert is free, payment for a certificate is up front; and the certificate itself should have accurate owner information. The author never mentions who the signing authority was, nor did he mention whether or not there were any signing errors.

Third, he mentions a tracking log of the package, but does not reveal which shipping company it was. Since the package never arrived, I’m left wondering whether the entire log was a fake. Usually, a shipping company cannot provide tracking information for a package that goes through other shippers. For example, a tracking number for something shipped via Canada Post isn’t too helpful once the parcel is handed off to the US Postal System for its final delivery. The log shown in the article mentions multiple hops in China, and multiple hops in Virginia. Was the comment about the package being lost in the USPS a real loss by the post office, or sarcastic wit?

Compromising hosts to send spam is pretty easy; but setting up an actual commerce website to handle transactions risks more exposure. If a spammer really wants to make money this way, there has to be a path leading from the victim to the criminal.

At the same time, there are other companies involved in the purchasing process that create more leads. Domain registrars, web hosting providers, certificate providers, shippers, and credit card processors. As a consumer, it would be more interesting to me to know which of these vendors have more extensive track records of dealing with spammers. Are these companies being duped, or are they more swayed by easy money compared to building consumer confidence?

The original title of Mr. Wade’s article was “Following the SPAM”, and in that regard he seems to have done a good job. As a follow-up, I would recommend a more detailed investigation, and to repeat the advice once given to Woodward and Bernstein: “Follow The Money.”

The legal maneuvering between two of my more favored tech companies just kicked it up a notch. Johnathan Schwartz looks like he’s not going to take NetApp’s lawsuit lying down.

I’m not even going to pretend I’m a lawyer, but a counter-suit is probably a suitable response to the original NetApp suit. In the end, the best hope for everyone (customer wise), is a mutual understanding between these two behemoth companies.

There are too many companies out there that have major investments in NFS, ZFS, and Netapp filers. For enterprise users of any of these technologies (or all of them), a one sided victory probably sounds pretty scary.

Read part one, before continuing.

Ten minutes later, Dave the DNS administrator was in the garage. He joined Kip, Tom, Sally, Douglas, Velma, Suse, and myself. “Before we begin, let’s review exactly what this script does,” I said.

Kip looked around, and decided to retell his overview, “The script reads database records from a table, and then for each record, it reads a HTML file on the disk, updates the database record, and then loops to the next record.”

“You forgot something,” I said.

“Oh,” he said. “After the database update, it waits before going on to the next record.”

“You didn’t say wait the first time,” I said. “You said rest.”

“Yeah,” Kip said. “Actually, it’s the sleep call.”

“How long does the script sleep?” I asked.

“Just one second,” Kip replied. He looked over at Sally and Tom, “We were concerned that the script could hammer the database unless we put in the sleep statement.”

I turned to look at Sally and Tom, “So you were concerned about performance?”

“Yes,” Tom said. “We’re dealing with a lot of data and file activity. The script runs 4 times a day.”

“How much data?” I asked.

“The table has sixty thousand records,” Sally said.

I grabbed a piece of chalk from Velma, and wrote 60,000 on the garage floor. “We are starting with 60,000 records”. Underneath it, I wrote 1+. “And we know that each loop iteration will take at least one second, because we explicitly sleep for one second.”

Everyone nodded their heads in agreement.

I turned to Tom, “How many seconds are in a minute?”

“Sixty,” he said.

“How many seconds are in an hour?”

“Three thousand, six hundred.”

“And how many seconds are in a day?”

Everybody looked around, quickly trying to do the math in their head. I turned to Dave, the DNS administrator. “How many seconds are in a day, Dave?”

“Eighty-six thousand, four hundred, ” he said. I wrote 86,400 in chalk on the floor.

“How did you know that?” Douglas asked.

“Most DNS administrators are familiar with that number, because it’s commonly used the set the Time To Live value of DNS records.”

“And the script only has a window of twenty-one thousand, six hundred seconds to run in, since it runs four times a day.”

Everyone looked at the numbers on the floor. Sally was the first one to speak. “So, in an effort to reduce the impact of the script, we made it worse with the delay.”

“Exactly,” I said. I picked up the source code. “Without thoroughly looking at this code, I would estimate that ninety-eight percent of the time, this script is doing nothing but waiting. On decent enterprise hardware, the file operations, and the database updates should only take milliseconds.”

“That’s right,” Velma said.

“So, how do we fix this?” Kip asked.

“Well, if there is a concern regarding impacting other systems, just change your sleep iteration. The simplest thing to do is to not sleep for every iteration, but for a percentage of iterations. If the script slept for 1 second every 10 iterations, it should finish in under two hours.”

“That’s an easy fix,” Kip said.

Velma took the chalk and wrote down some additional figures. “For the best case, the script should know how many records it has, and how many seconds it has in the given window to process all of those records.”

“Right, ” I said. “Turn it into a mathematical function, but decrease the window size by an hour, just to be on the safe side.”

Dave looked around and raised his hand. “Uh, is there anything else you needed me for?”

“No, ” I said. “That was it.”

Dave walked back to his bike, mumbling about the 10 mile bike ride back to his house.

“How did you know I didn’t use the perl Time::HiRes module?” Kip asked. “I could have slept for fractions of a second.”

“If you did, the use statement would have been visible in the first page of your code printout,” I replied.

“Well, how did you know I didn’t use threading?”

“Because nobody casually implements threading in perl scripts.”

There was a somber moment until Tom spoke up. “Wow. I never expected the sleep statement to be the problem with this.”

I picked up all of the graphs and charts from the table. “This is the most obvious solution, given the information you all have told me. You guys had the answer all along, you just never expected it to be that simple.”

“Remember, ” I said. “A good systems administration team always works together, and is not afraid to look at the code written by a developer.” I turned to Kip. “At the same time the developer should not be afraid to discuss performance metrics, and profiling code behavior with a systems administrator.”

Douglas was about to say something when a wild haired CTO walked into the garage. She was carrying a sleek 2 rack-unit server with a hatchet embedded into the casing. “Somebody hacked my Linux server!” she exclaimed.

I looked back at the others. “Sorry guys, I have another case.”

The small team of geeks said their thanks and walked out of the garage. I turned to the CTO holding up a jar full of quarters, “Payment up front, and I can’t guarantee anything if you didn’t preserve the ARP cache.”

Parody. Based on a true story.

It was a lazy, summer Saturday morning, just like any other. From my vantage point, I could see kids riding their bikes and playing street hockey. I sat in the middle of an empty garage with my feet propped up on a table, reading a book on how to lift fingerprints from a CVS repository. Saturdays were good days for the detective business, so I knew it wouldn’t be long before customers came in with their problems.

Less than an hour after opening up, Kip, Sally, and Tom walked in. It was Kip that spoke first. “We want to hire you,” he said. He backed up his statement by dropping twenty-five cents into the jar sitting on the table. I put down my book and assessed my clients.

Kip was a good perl programmer; always keeping his parents happy with his ability to quickly write scripts to handle any problem. Sally was the DBA, who held the record for winning the Oracle performance competition at the county fair. Tom was the new systems administrator, who was known for carrying his pet lizard (Suse) with him everywhere he went. All three of them were good at their job.

“I have a perl script that’s taking way too long,” Kip said while dropping a few pages of source code on the table. “It’s a looping function that impacts the database and the filesystem.”

“What exactly does the script do?” I asked.

“The script grabs a lot of records through a SELECT call; each record contains information on a HTML file on the disk. For every record, I run a loop that stats the file, reads the first few lines from the file, makes an UPDATE call to the database with new information, and rests before moving on to the next record.”

Tom jumped in, “The job is scheduled through cron to run every six hours: 6am, noon, 6pm, and midnight. But when I look at the process table, there’s multiple instances running. The jobs are taking too long and not finishing within six hours.”

Sally added her two cents, “The updates don’t seem to have a serious impact against the Database. The table itself is only sixty thousand rows, and it’s fully indexed.” Sally brought me graphs generated from an Oracle management program.

“And the filesystem is fine, too.” Tom said. “The average number of disk requests remains constant. There’s no iowait or contention.” To make his point, he placed a stack of SAR reports on my desk.

I looked at the data sitting on my desk and thought about it for a second.

“Gentlemen,… and Sally,” I said. “This isn’t a problem with code, and it’s not a problem system performance. You could say that the problem lies with not understanding the very nature of the planet.”

All three of them looked at me, perplexed. “But you didn’t even look at my source code!” Kip exclaimed.

“I don’t need to,” I replied. “You already gave me everything I need to know.” I reached into my baseball mitt and grabbed my iPhone. “In five minutes I can have a DNS administrator over here, and he can give you an important fact that you all missed.”

WHAT DID I KNOW THAT THEY DID NOT? THE SOLUTION WILL APPEAR MONDAY, BUT FEEL FREE TO THROW IN YOUR COMMENTS.

Update: The solution has been posted.

I figured I’d do a poll a week since people really like it. So, my first poll-a-week: Which Logs Do You Collect?

Vote away! I will post and comment on results here after a few weeks.

UPDATE: poll results and analysis are posted here. Enjoy!

We rolled out mod_evasive across a pool of servers the other day. Since we already had Apache running, you can rightfully assume that installing this module was done in response to user bahavior.

No, we weren’t selling Hannah Montana tickets, or seeing if Ron Paul would make a nice president; but we did attract a regional based script kiddie. If you give teenagers an online poll asking who has the better football team, and the winner of that poll will be announced on television; it’s a good bet that a few people are going to stuff the ballot box in their favor.

Ironically, nobody even cares about the results; but we have to deal with the people running libwww-perl, or specially crafted JavaScript pages that resubmit form values hundreds of times. Since this isn’t online banking, we decided that using captcha wasn’t worth the effort, so the goal was to block excessive attempts.

The case for mod_evasive is pretty clear. In most cases, it’ll stop successive hits repeatedly sent to the same URL multiple times. Fifty hits enter, one hit leaves. The hit per second parameters are fully configurable. It also logs to syslog; so its behavior can be monitored.

The case against mod_evasive is scalability. Mod_evasive does not use shared memory between child processes. It also won’t work in a load balanced server pool unless the client IP is persistently tied to the same web server in the pool. For larger web server environments, a better solution should be implemented into the load balancing front-end. Finally, in some cases, mod_evasive may not be enough; because even though it still returns 403s, you’re still dealing with a hit and an open TCP socket connection on your server. If your infrastructure is under attack, mod_evasive will never replace firewall blocking or upstream filtering.

But, if your environment is relatively small, or if application abuse does not have a high impact, mod_evasive is a pretty good tool to have around.