When I first started using LVM I got bit by a few bugs. It’s all part of being an early adopter. As a result I never really used it on production hardware. It wasn’t until about 2 years ago that I gave it another look. In a similar manner I never really thought much of software raid beyond a novelty. Much of that has changed now and I use them both on a regular basis for a number of reasons.

Last week I attended a virtualization seminar. I did not expect a lot from the event at first, but I was surprised by the qualities of the guest speakers. Both had strong backgrounds with VM environments, and they did a good job of explaining what it takes to migrate to VM.

One of the speakers made an interesting statement, saying that the hypervisor is now commoditized. The market for virtual solutions has gotten so big, it’s unavoidable. VMWare has ESX, Xen has their system, and Microsoft is coming out with Hyper-V. If everybody offers what is essentially the same thing, then how do these products stand out from one another?

Now your incentives for buying virtualization have changed. You don’t buy VMWare just because it offers virtualization; you buy VMWare because it has the best service, and the best hot migration features. You might buy Hyper-V because your familiar with Microsoft internal APIs and management tools. On top of virtualization, I’m not sure what else Xen has to offer, but there could be new features coming out from Citrix.

When I left the seminar, I started to re-evaluate hardware decisions that were made in the past. The nature of the beast has changed. Eight years ago, hardware decisions were taken for granted, because it too, was commoditized.

Everything runs on an x86, and everyone makes an x86, so the low price usually won out. Anything that the vendor offers on top of the low price might have clinched the deal. Better support, better service, free shipping? Whatever it took to sell a server and get it out the door. Hundreds of IT departments packed data centers full of tight 1ru servers. Virtualization has now made those servers worthless.

When a single server failed, it was no big deal. You probably had another one just like it running the same application. If that same server is now running multiple virtual hosts, then the service impact is higher. Two machines may now be fighting for access to the same mirrored local disks. What are the chances that they’re impacting each other?

If your server can only handle 2 running virtual hosts, then you cut hardware costs by 50%; but in order to win, your hardware savings still need to be higher than the support and licensing costs of your enterprise VM solution. A 2 to 1 hardware savings ratio isn’t good. it’s expected. In order to maximize your investment, you should aim for a 4 to 1 hardware savings ratio, maybe higher

Migrating to a VM environment does not mean building a VM solution into your servers; it means building your servers around a VM solution. If the hypervisor really is treated like a commodity, then the same can no longer be said about the hardware.

Following the tradition of posting a tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #14: More access_log Fun: What Are You Not GETting?

In this tip, we will look at some bizarre artifacts that show up in web server access logs today. Here we have a production log from an Apache web server that is full of interesting (and sometimes ominous!) little mysteries that we will investigate in order to determine their impact on security and operational health of the site.

Logs do contain more mysteries than we have time, so we will focus on a few of them: specifically, unusual web request methods. Let’s see who is trying to POST or use some other method (OPTIONS, HEAD, PUT or something - see a list here) on our site, instead of just GET’ting the content (GET command is used by web browsers to retrieve the pages, while POST is used to upload content, press buttons, etc - at least in “web 1.0″ land - see earlier tip #12 where POST request was found in proxy logs)

Here is one little artifact that attracted my attention due to a POST request vs a web forum as well as a battery of slashes (which actually increases in subsequent request - of which there were many)

10.10.102.250 - - [12/Feb/2008:16:10:50 -0500] “POST /phpBB3////ucp.php?mode=register&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0″ 200 14397 http://www.example.com/phpBB3///ucp.php?mode=confirm&id=7640df05c7e24b7acf7a68800fe6dc59&type=1&sid=e5efaa77a777066c61f71808e9e57b19 “Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126″

… more…

10.10.102.250 - - [12/Feb/2008:16:12:29 -0500] “POST /phpBB3///////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0″ 200 9355 “http://www.example.com/phpBB3//////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19″ “Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126″

This one really is a mystery; what do we know about it? The server responded to the request OK (code 200), so the POST actually happened. The first request was a request to register with a web discussion board and the second was a request to login. Multiple slashes are actually ignored by the web server, so why put them in the request (no answer)? Also, I think that the User-Agent is spoofed … do you know why? Finally, if I see something like that in my logs, I will definitely investigate it, primarily due to the fact that Apache responded with 200 OK code.

The next one is so classic it it dumb (and so dumb, it’s a classic :-))

10.10.123.226 - - [12/Feb/2008:03:46:54 -0800] “POST /_vti_bin/shtml.exe/_vti_rpc HTTP/1.1″ 404 - “-” “MSFrontPage/6.0″

10.10.123.226 - - [12/Feb/2008:03:46:55 -0800] “OPTIONS / HTTP/1.1″ 200 20210 “-” “Microsoft Data Access Internet Publishing Provider Protocol Discovery”

It is probably one of the ancient IIS attacks (check out this fun BlackHat preso on that, circa 2003) - why would someone probe for it now is beyond me. In any case, Apache on Linux and “*.exe” don’t mix :-)

The final log record is also fun:

10.10.101.222 - - [12/Feb/2008:15:33:22 -0800] “PUT /zk.txt HTTP/1.0″ 405 223 “-” “Microsoft Data Access Internet Publishing Provider DAV 1.1″

The above uses a PUT request which is pretty much deprecated now; the purpose of the above is clearly malicious. In fact, modern Apache shouldn’t even allow it, thus it responds with code 405 “Method Not Allowed.” Nothing to worry about (even though some poor critter got owned with that! BTW, if you follow that link, check out HTTP response code 201 - if you see it in your logs, run! :-))

Overall, if you see too many POSTs or too many “GET then POST” sequences from the same IP in rapid succession, investigate it since no legitimate access should produce such a pattern…

As further reading, I heartily recommend this paper: “Detecting Attacks on Web Applications from Log Files“

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

My next fun logging poll is here - please vote! It is about tools for centralized collection of Windows Event Log from servers and other systems. One of the somewhat surprising discoveries from my previous poll was that few people look at Windows logs; this poll drills down into it.

UPDATE: the results and analysis posted here. Enjoy!

Past logging polls and their analysis:

This is my 6th logging poll (vote here now!)- links to the previous five polls below.

This one is deceptively similar to the #1 below, but it is not. This poll is What logs do you actually LOOK at? and not Which Logs Do You Collect? In other words, are you a log packrat? Are you collecting and never using the log data? You are making a mistake, if you don’t.

Past polls:

UPDATE: analysis of this poll posted here. Enjoy!

A podcasting friend of mine ran into the problem of always having to send the new episodes to his co-hosts for review. Once everyone agreed that it was okay, the show was put live on the RSS feed. Their method of distributing a raw mp3 file? Email.

If only there was a way to distribute the file electronically, without the overhead of email, and yet still get the file automatically once it’s ready. How about RSS?

ITunes and other feed aggregators have the feature of handling RSS feeds that are protected by HTTP authentication. When you download the feed, your client will prompt you for a username and password before downloading the RSS XML.

My friend’s podcast now has two RSS feeds:

http://www.example.com/feeds/public/podcast.xml
http://www.example.com/feeds/private/podcast.xml

The first URL is what’s submitted to all of the podcast directories. The second one is strictly for preview purposes. All of the responsible parties for the show subscribe to the private feed. This allows them to test new episodes, and verify that the RSS <item> content for the episode is correct.

Once everyone has agreed that the episode is ready, the RSS tags for the episode are copied over to the public feed XML file. Now outside users can see the episode and download it.

This is pretty good for most situations, but there is still one risk: HTTP URLs can contain authentication encoding in them, like so…

http://username:password@www.example.com/feeds/private/podcast.xml

Avoid using this convention in your bookmarks, or feed entires. If that URL were to be copied to an outside data source, there’s a chance that it could get into the wild. When that happens, outsiders may end up listening to your private, not production ready, podcast.

To reduce the chances of that happening, ITunes won’t list a podcast in their directory if the URL contains an embedded username and password. They won’t even list a podcast if the server makes a request for HTTP authentication.

Years ago I visited Danga back before the Six Apart acquisition, when the company had its headquarters a couple of miles away and when Brad lived a block and a half down the street. Brad showed me some of their management tools — almost all home-grown.

I mention that because today I stumbled on Dormando’s [crappy] Operations Mantras. Dormando works for Six Apart, and he has the same philosophy I see in Brad. Relentless automation and merciless monitoring are the two secrets of efficient and effective system administration and operations management.

I only wish that someone had handed me this list of mantras when I started as a system administrator in the ’90s. (Puppet and Xen would have been nice too.)

This poll is especially fun: What are your top challenges with logs and logging? Vote here.

Past polls were:

UPDATE: the analysis for this poll is posted. Enjoy!

Here is my next poll about logs: Who looks at logs at your organization?

Vote here!

Also, my past polls and analysis are here.

Let’s say you have evidence of network errors. Here’s the symptoms that you see:

1. A lot of TCP retransmits (layer4)
2. No Ethernet frame errors, dropped packets, or CRC errors (layer 2)
3. No ICMP errors, or IP level errors. Pings report no lag or dropped packets. (layer 3)
4. Failures are only reported on two nodes in your network, but no errors on the switch between the two nodes.

Given the above evidence, would you look at the wiring between the two nodes, including the patch panel ports? If so, why?

No wrong answers, just trying to bring about an open discussion of opinions.

Brendan Gregg wrote up a performance comparison between the utilities Top and Prstat. And just for good measure, he throws in a lot of Dtrace example code to show how he came to his conclusions.

Here’s some advice if you notice your system may be having a performance problem. Run the ls of your /proc filesystem. This will tell you how many active processes are running on your system.

If you’re dealing with a large process count, consider running either command with a short number of iterations, otherwise your monitoring attempts will only contribute to the overall load. I would argue that in a lot of cases, people running Top or Prstat in a background window are actually contributing to any performance problems that they eventually discover through the use of those tools.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #13: Into the Darkness … or The Ominous World of Unix Binary Audit Logs

In this tip, we will take a peek at one of the most esoteric areas of logging: Unix binary audit logs. Solaris BSM and Trusted Solaris auditing is the least unknown :-) example of it, even though other Unix vendors have similar auditing capabilities - see this for HP-UX Audit and this for IBM AIX audit. Linux kernel audit is also pretty much the same thing. If you look for information on ‘Solaris BSM audit logs’ , you’d find plenty of tips on how to enable such logging, a little on how to manage/rotate the log files, a bit on how to survive the resulting data deluge and ALMOST NOTHING on what to do with the log data, which is kinda sad :-) After looking at BSM logs for a while, I developed an opinion that nobody has ever looked at them on a regular basis :-)

So, let’s assume you enabled Solaris BSM kernel audit for user “root” and few other “interesting” users (there is no per-object logging in Solaris; other Unix’es do have it) via the following commonly recommended per-user configuration in /etc/security/audit_user:

root:lo,ad,fw:no

anton:all,-all:no

jsmith:all,-all:no

This config pretty much records all the actions by the users listed. Now, you have audit files growing like shrooms in you /var/audit. What good does it give us? First, we need to convert the binary audit files into text - something along the lines of

# auditreduce -A /var/audit/20071127193515.not_terminated.SunUltra10 | praudit -l > /tmp/sol_box_11272007

will do. Now what? In this tip we will learn how use the audit logs to see who is trying to copy sensitive files off the system.

First, who is connecting out - lets’s search the logs for ‘connect’ calls (if you are using LogLogic for it, use Index Search for this task; if not, grep will have to do, but be prepared to wait). A few recommended searches:

• “connect AND 172.16.10.*” or “connect AND NOT 172.16.10.*” (to look for connection to specific IPs or to the outside networks) or simply ‘connect AND username’

Here is an example found (with connect, IP and user in bold):

header,103,2,connect(2),,Tue Nov 27 11:36:46 PST 2007, + 193 msec,argument,1,0x4,so,socket,0x0002,0x0002,0x80d6,SunUltra10,0x0016,10.1.1.41,subject,root,anton,other,anton,other,29902,29720,0 1611 172.16.0.173,return,success,0

At this point we already know the user name of the user who run that connecting process since it will be in the results (you can also the user to search as I showed above).

Next, what are those connections - let’s try to uncover which programs actually connected (BSM logs don’t make that easy). Let’s search for process starts in the same time frame:

• “execve AND NOT ls AND NOT <whatever other commands you don’t care to see>” will give you a list of started programs.

Example:

header,124,2,execve(2),,Tue Nov 27 11:36:46 PST 2007, + 115 msec,path,/usr/bin/scp,attribute,100555,root,bin,136,1573,0,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,0

Notice that both records have the same timestamps. Sadly, time and parent process ID ( which is in our case 29720) is all that correlates them together.

Finally, what file was affected (i.e. copied off the system via scp in this case) - more digging is in order; we again use the process ID and time. The easiest is to search for a file name or browse all records around the same time frame (might be A LOT!):

• “*secret.zip* AND anton” will work; we can add the above process ID and look for “anton AND 29720″ (but expect a lot of data since this is a shell process ID)

For example:

header,135,2,open(2) - read,,Tue Nov 27 11:36:47 PST 2007, + 900 msec,path,/tmp/not-so-secret.zip.gz,attribute,100600,anton,other,0,32743959,18446744073709551615,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,4

What do we know now? This user connected to this system and MAYBE copied this file via, MAYBE via scp. How cool is that? (A: not cool at all, since we are not sure!)

To conclude, if you can avoid dealing with Solaris BSM logs, please do so :-) On a more serious note, you now know why these logs were called “the ugliest logs ever.”

Even more seriously (but still pretty humorously), these logs are a classic example of trees that make every effort to obscure the forest, because they record syscalls and not processes or user actions (and connect, execve and read are all logged separately). There are also many, many more idiosyncrasies (and, in fact, idiocies) where these come from :-)

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Time for another fun logging poll: What Do You Do With Collected Logs?

Vote here!

This is my Logging Poll #3, links to past polls:

• Poll#2 “Why Collect Logs?” (vote here, results so far, my analysis)
• Poll #1 “Which Logs Do You Collect?” (vote here, results so far here, analysis here)

UPDATE: analysis and results posted at here

The idea came from Jeremiah Grossman (here) when he described “The Best Web Application Vulnerability Scanner in the World” thus: “Within a few moments of pressing the scan button it’ll find every vulnerability, with zero false positives, generate a pretty looking report, and voila you’re compliant with GLBA, HIPAA, and PCI-DSS. Of course, we all know such a web application scanner is simply not possible to create for a variety of reasons.”

So, let’s imagine the idea log management application.

1. Logging configuration: the ideal log app will go and find all possible log sources (systems, devices, applications, etc) and then enable the right kind of logging on them according to a high level policy given to it (required: God-like powers)
2. Log collection: it will collect all the above logs securely (and without using any risky super-user access ) and with little to no impact to networks and systems (required: God-like powers)
3. Log storage: it can security store the above logs in the original format for as long as needed and in a manner allowing quick access to them - in both raw and summarized/enriched form (required: plenty of hardware)
4. Log analysis: this ideal application will be able to look at all kinds of logs, known to it and previously unseen, from standard and custom log sources, and tell the user what they need to know about their environment and based on their needs: what is broken? what is hacked? where? what is in violation of regulations/policies? what will break soon? who is doing this stuff? The analysis will power all of the following: automated actions, real-time notifications, long-term historical analysis as well as compliance relevance analysis (required: AI)
5. Information presentation: this tool will distill the above data, information and conclusions generated by the analytic components and present then in a manner consistent with the user’s role: from operator to analyst to engineer to executive. Interactive visual and drillable text-based data presentation across all log sources. The users can also customize the data presentation based on their wishes and job needs, as well as information perception styles (required: nothing more than a bunch of daring UI designers)
6. Automation: the ideal log management tool will be able to take limited automated actions to resolve discovered and confirmed issues as well as generate guidance to users so that they know what actions to take, when full-auto mode is not appropriate. The responses will range from full-auto actions to assisted actions (’click here to fix it’) to issuing detailed remediation guidance. The output will include a TODO-list of discovered items complete with actions suggested, ordered by priority (required: AI + some luck + some user stupidity :-))
7. Compliance: this tool can also be used directly by auditors to validate or prove compliance with relevant regulations by using regulation-specific content and all the collected data. The tool will also point at gaps in data collection as it applies to specific regulations that the user is interested in complying (required: God-like powers)

In other words, this magic black box will have crap shoveled from one side and will have answers to questions about the meaning of Life :-) coming out the other side…

What? :-) Am I nuts? Well, can I dream for a second? :-)

Following my now-famous Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs, here is the promised “Top 11 Reasons to Secure and Protect Your Logs”

1. Let’s review why you are reviewing logs. Will logs that might have been changed by somebody, somewhere, somehow still be useful for items 1-11 from here? No? Secure them!
2. Oooh, logs in court? Challenges abound! To respond to them, one needs to protect the logs so you can claim that they are both authentic and reliable.
3. A human error still beats an evil hacker as the main cause of IT problems. Are your logs safe from it? Available when needed? Protect them from crashes and other faults!
4. PCI DSS just says so: “Secure audit trails so they cannot be altered.” Wonna do it- or pay the fines?
5. Do you protect financial records? Identity info? Passwords? Some of it ends up in logs - thus making them more sensitive. Secure the C-I-A of logs!
6. Do you look at logs during incident investigation? Do you want them to be “true” or full of random (if creative…) cr*p, inserted by the guilty party? Secure the logs!
7. Think that “attacks vs logging” are theoretical? Think again. Are your logs safe or vulnerable? Is your logging tool 0wned?
8. Syslog + UDP = log injection. Are you protected (reliable TCP, confirmed delivery, encryption - SSH, SSL, VPN)?
9. Why change logs? No, really, why change logs? If you never change logs - and you never should - hash them right away after collection to make them immutable.
10. Logs are backed up on tape - who will see them? Well, whoever restores the tape, that’s who! Encrypt them to protect them from accidental and malicious disclosure if tape is lost.
11. Why log access to logs? Same reason why you had the logs in the first place - to review who did what. Who broke through and stole the logs? Who browsed them without permission? Only logs will tell - if you have them!

Overall, one need to strive for having no holes in log safeguards from log birth to analyst conclusion based on log information…

Possibly related posts:

• Top 11 Reasons to Look at Your Logs

• Top 11 Reasons to Collect and Preserve Computer Logs

The previous poll (vote here, live results here, analysis here) proved to be a success so the next one is here.

This time the question is: “Assuming that you centrally COLLECT system, network or security logs from their originating sources, what is THE MAIN reason for doing it?”

Vote on!

UPDATE 11/11/2007: results and analysis are posted here

See all my polls here.

Read part one, before continuing.

Ten minutes later, Dave the DNS administrator was in the garage. He joined Kip, Tom, Sally, Douglas, Velma, Suse, and myself. “Before we begin, let’s review exactly what this script does,” I said.

Kip looked around, and decided to retell his overview, “The script reads database records from a table, and then for each record, it reads a HTML file on the disk, updates the database record, and then loops to the next record.”

“You forgot something,” I said.

“Oh,” he said. “After the database update, it waits before going on to the next record.”

“You didn’t say wait the first time,” I said. “You said rest.”

“Yeah,” Kip said. “Actually, it’s the sleep call.”

“How long does the script sleep?” I asked.

“Just one second,” Kip replied. He looked over at Sally and Tom, “We were concerned that the script could hammer the database unless we put in the sleep statement.”

I turned to look at Sally and Tom, “So you were concerned about performance?”

“Yes,” Tom said. “We’re dealing with a lot of data and file activity. The script runs 4 times a day.”

“How much data?” I asked.

“The table has sixty thousand records,” Sally said.

I grabbed a piece of chalk from Velma, and wrote 60,000 on the garage floor. “We are starting with 60,000 records”. Underneath it, I wrote 1+. “And we know that each loop iteration will take at least one second, because we explicitly sleep for one second.”

Everyone nodded their heads in agreement.

I turned to Tom, “How many seconds are in a minute?”

“Sixty,” he said.

“How many seconds are in an hour?”

“Three thousand, six hundred.”

“And how many seconds are in a day?”

Everybody looked around, quickly trying to do the math in their head. I turned to Dave, the DNS administrator. “How many seconds are in a day, Dave?”

“Eighty-six thousand, four hundred, ” he said. I wrote 86,400 in chalk on the floor.

“How did you know that?” Douglas asked.

“Most DNS administrators are familiar with that number, because it’s commonly used the set the Time To Live value of DNS records.”

“And the script only has a window of twenty-one thousand, six hundred seconds to run in, since it runs four times a day.”

Everyone looked at the numbers on the floor. Sally was the first one to speak. “So, in an effort to reduce the impact of the script, we made it worse with the delay.”

“Exactly,” I said. I picked up the source code. “Without thoroughly looking at this code, I would estimate that ninety-eight percent of the time, this script is doing nothing but waiting. On decent enterprise hardware, the file operations, and the database updates should only take milliseconds.”

“That’s right,” Velma said.

“So, how do we fix this?” Kip asked.

“Well, if there is a concern regarding impacting other systems, just change your sleep iteration. The simplest thing to do is to not sleep for every iteration, but for a percentage of iterations. If the script slept for 1 second every 10 iterations, it should finish in under two hours.”

“That’s an easy fix,” Kip said.

Velma took the chalk and wrote down some additional figures. “For the best case, the script should know how many records it has, and how many seconds it has in the given window to process all of those records.”

“Right, ” I said. “Turn it into a mathematical function, but decrease the window size by an hour, just to be on the safe side.”

Dave looked around and raised his hand. “Uh, is there anything else you needed me for?”

“No, ” I said. “That was it.”

Dave walked back to his bike, mumbling about the 10 mile bike ride back to his house.

“How did you know I didn’t use the perl Time::HiRes module?” Kip asked. “I could have slept for fractions of a second.”

“If you did, the use statement would have been visible in the first page of your code printout,” I replied.

“Well, how did you know I didn’t use threading?”

“Because nobody casually implements threading in perl scripts.”

There was a somber moment until Tom spoke up. “Wow. I never expected the sleep statement to be the problem with this.”

I picked up all of the graphs and charts from the table. “This is the most obvious solution, given the information you all have told me. You guys had the answer all along, you just never expected it to be that simple.”

“Remember, ” I said. “A good systems administration team always works together, and is not afraid to look at the code written by a developer.” I turned to Kip. “At the same time the developer should not be afraid to discuss performance metrics, and profiling code behavior with a systems administrator.”

Douglas was about to say something when a wild haired CTO walked into the garage. She was carrying a sleek 2 rack-unit server with a hatchet embedded into the casing. “Somebody hacked my Linux server!” she exclaimed.

I looked back at the others. “Sorry guys, I have another case.”

The small team of geeks said their thanks and walked out of the garage. I turned to the CTO holding up a jar full of quarters, “Payment up front, and I can’t guarantee anything if you didn’t preserve the ARP cache.”

Parody. Based on a true story.

It was a lazy, summer Saturday morning, just like any other. From my vantage point, I could see kids riding their bikes and playing street hockey. I sat in the middle of an empty garage with my feet propped up on a table, reading a book on how to lift fingerprints from a CVS repository. Saturdays were good days for the detective business, so I knew it wouldn’t be long before customers came in with their problems.

Less than an hour after opening up, Kip, Sally, and Tom walked in. It was Kip that spoke first. “We want to hire you,” he said. He backed up his statement by dropping twenty-five cents into the jar sitting on the table. I put down my book and assessed my clients.

Kip was a good perl programmer; always keeping his parents happy with his ability to quickly write scripts to handle any problem. Sally was the DBA, who held the record for winning the Oracle performance competition at the county fair. Tom was the new systems administrator, who was known for carrying his pet lizard (Suse) with him everywhere he went. All three of them were good at their job.

“I have a perl script that’s taking way too long,” Kip said while dropping a few pages of source code on the table. “It’s a looping function that impacts the database and the filesystem.”

“What exactly does the script do?” I asked.

“The script grabs a lot of records through a SELECT call; each record contains information on a HTML file on the disk. For every record, I run a loop that stats the file, reads the first few lines from the file, makes an UPDATE call to the database with new information, and rests before moving on to the next record.”

Tom jumped in, “The job is scheduled through cron to run every six hours: 6am, noon, 6pm, and midnight. But when I look at the process table, there’s multiple instances running. The jobs are taking too long and not finishing within six hours.”

Sally added her two cents, “The updates don’t seem to have a serious impact against the Database. The table itself is only sixty thousand rows, and it’s fully indexed.” Sally brought me graphs generated from an Oracle management program.

“And the filesystem is fine, too.” Tom said. “The average number of disk requests remains constant. There’s no iowait or contention.” To make his point, he placed a stack of SAR reports on my desk.

I looked at the data sitting on my desk and thought about it for a second.

“Gentlemen,… and Sally,” I said. “This isn’t a problem with code, and it’s not a problem system performance. You could say that the problem lies with not understanding the very nature of the planet.”

All three of them looked at me, perplexed. “But you didn’t even look at my source code!” Kip exclaimed.

“I don’t need to,” I replied. “You already gave me everything I need to know.” I reached into my baseball mitt and grabbed my iPhone. “In five minutes I can have a DNS administrator over here, and he can give you an important fact that you all missed.”

WHAT DID I KNOW THAT THEY DID NOT? THE SOLUTION WILL APPEAR MONDAY, BUT FEEL FREE TO THROW IN YOUR COMMENTS.

Update: The solution has been posted.

I figured I’d do a poll a week since people really like it. So, my first poll-a-week: Which Logs Do You Collect?

Vote away! I will post and comment on results here after a few weeks.

UPDATE: poll results and analysis are posted here. Enjoy!

We rolled out mod_evasive across a pool of servers the other day. Since we already had Apache running, you can rightfully assume that installing this module was done in response to user bahavior.

No, we weren’t selling Hannah Montana tickets, or seeing if Ron Paul would make a nice president; but we did attract a regional based script kiddie. If you give teenagers an online poll asking who has the better football team, and the winner of that poll will be announced on television; it’s a good bet that a few people are going to stuff the ballot box in their favor.

Ironically, nobody even cares about the results; but we have to deal with the people running libwww-perl, or specially crafted JavaScript pages that resubmit form values hundreds of times. Since this isn’t online banking, we decided that using captcha wasn’t worth the effort, so the goal was to block excessive attempts.

The case for mod_evasive is pretty clear. In most cases, it’ll stop successive hits repeatedly sent to the same URL multiple times. Fifty hits enter, one hit leaves. The hit per second parameters are fully configurable. It also logs to syslog; so its behavior can be monitored.

The case against mod_evasive is scalability. Mod_evasive does not use shared memory between child processes. It also won’t work in a load balanced server pool unless the client IP is persistently tied to the same web server in the pool. For larger web server environments, a better solution should be implemented into the load balancing front-end. Finally, in some cases, mod_evasive may not be enough; because even though it still returns 403s, you’re still dealing with a hit and an open TCP socket connection on your server. If your infrastructure is under attack, mod_evasive will never replace firewall blocking or upstream filtering.

But, if your environment is relatively small, or if application abuse does not have a high impact, mod_evasive is a pretty good tool to have around.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #12: Proxy Log Fun - Proxy Logs vs Information Leakage

You probably know that web proxies (such as Squid, BlueCoat SG, BlueCoat Netcache and others) produce a lot of fun logs. Indeed, they are fun since they can be used for a whole range of things, from routine monitoring for AUP compliance to malware detection as well as possibly looking for the security scourge of 2007 - web client attacks.

Specifically, in this tip we will learn how proxy logs can be used for detection of file uploads and other outbound information transfers vie the web. First, think what is the legitimate use of file upload functionality for your web users. If web mail is allowed, then sending an attachment will include an upload. What else? The rest will be considered at least suspicious…

In addition to file uploads, some spyware application will also use similar methods to steal data. Looking for methods and content-type in combination with either known suspicious URL or user-agent (i.e. web client type) can often reveal spyware infections, actively collecting data. Admittedly, a well-written spyware can certainly fake the user-agent field so it is clearly not reliable, but still useful to add to our query above. Here are some of the criteria we will use to look for uploads in Squid and BlueCoat SG proxy logs:

• HTTP method (logged as “cs-method” by BlueCoat) = POST (as opposed to the usual GET, used to retrieve web content).
• For information uploads: content type (logged as “RS(content-type)” by BlueCoat) = anything but “html/text” (which is the type used for uploading web form contents) - especially try content types “application/octet-stream“, “application/msword“, “application/powerpoint“, “application/vnd.ms-excel“, “application/pdf” and a few others to look for common file uploads
• For spyware and application data transfers: user-agent set to anything but the common ones (i.e. not Mozilla, iTunes, LiveUpdate, etc) or even to “unknown.” One can also try user-agent containing your favorite messaging app (e.g. “MSN Messenger”, etc)

(if you feel adventurous, other interesting content-types to try are “application/x-javascript” and “text/javascript”)

Here are the examples of the above, including some “classics” (while spyware specimen are a bit dated, this method of detecting them via logs is relevant):

1. 1124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
2. 1124392388.975 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc.cometsystems.com/dss/cc.2_0_0.report_u
3. 2007-05-19 03:55:12 160 10.1.1.3 - - - OBSERVED “Spyware/Malware Sources;Spyware Effects;Web Advertisements” - 200 TCP_NC_MISS POST text/html;%20charset=utf-8 http bis.180solutions.com 80 /versionconfig.aspx ?did=5342&ver=1.0 aspx - 10.1.1.2 273 175 - - none - -
4. 2007-05-21 03:10:40 4 10.1.1.3 Joanna- authentication_redirect_to_virtual_host PROXIED “Search Engines/Portals” - 307 TCP_AUTH_REDIRECT POST - http storage.msn.com 80 /storageservice/schematizedstore.asmx - asmx “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSN Messenger 7.5.0324)” 10.1.1.2 791 2566 - - none - -

Here are some other signs that will make the above log entry extra-suspicious is:

• A dead giveaway: upload happens to a “known bad” URL (e.g containing “gator” and others above)
• Upload happens to an unresolved IP address (do a “whois” on it!)
• Uploads happens to a port not equal to 80 (i.e. the URL contains a port such as http://10.1.10.10:31337)
• Upload has confidential file name in the log entry (e.g. somebody dumb emailing a sensitive file to himself - as discussed here)

Overall, this log analysis method is good for casting a broad net to catch not just spyware-infected systems, but also unauthorized applications (e.g. method=POST and user-agent=iTunes), instant messaging (e.g. method=POST and then by user-agent, content or URL), simple forms of data theft and document handling policy violations (emailing files to self via web mail: method=POST and sensitive file name present in the entry; also content type set to popular file types) as well as other abuses of web access. As a result, proxy logs provide an extremely rich AND readily available source of data about threats that users face!

To top it off, one promising direction of future research is using web proxy logs to detect client-side exploits by malicious web servers (more on this in the near future!)

Possibly related posts:

• Anton Security Tip of the Day #6: The Other Web Log
• All my past security and logging tips

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

I recently had reason to spend a while working with Nessus on Windows XP (Service Pack 2). Usually, I use a Nessus Server running on Linux, either running locally if I am onsite, or one installed on our company infrastructure for scanning from the Internet. In fact, you read the documentation don’t you?, Tenable specifically recommends in the Nessus Installation Guide that you _not_ run Nessus on XP, and instead use a Windows Server product, such as Windows Server 2003.

The reason for this is that in Windows XP Service Pack 2, Microsoft introduced a number of Network Protection Technologies for mitigating the spread of malware. One of these limits the number of simultaneous incomplete outbound TCP connection attempts to 10, with additional attempts being queued and potentially dropped. This impacts the reliability of at least port scanning, and possibly other security checks.

Unfortunately the scenario I was working with required me to be running Nessus through a VPN client (never ideal), in reality requiring me to be on XP. Tenable does, however, have some recommendations for running Nessus as reliably as possible on XP:

• Max number of hosts: 10
• Max number of security checks: 4
• Max number of packets per second for port scan: 50

The maximum hosts/security checks setting is standard in all of the Nessus clients I’ve used, however the packets per second setting seems to only be available within the client shipped with the Windows Nessus server. If you, like me, are using the new NessusClient 3.0 beta for Windows, you need to make the following change to the Nessus server’s configuration to ensure that 50 is the default value:

• Go to the “config” directory in your Nessus server installation. By default this is C:\Program Files\Tenable\Nessus\config
• Open config.default.xml for editing - just use Notepad if you don’t have an XML editor
• Find the SYN Scan:Max number of packets per second for port scan node, and edit the value (the CDATA bit) from 500 to 50

This value should now be the default for all new scans.

This worked well for me, however needless to say that running a Nessus scan in VMWare (slowdown factor one), over a VPN link (slowdown factor two), over a transatlantic Internet connection (slowdown factor three), the scan took quite a while to complete…

Update: Ron Gula corrected me on this - this is available on the free registered feed.

A little while back I spotted this article on the Tenable Blog in reading my morning RSS feeds - Tenable have added a plugin with the ability to interrogate Windows machines for the wireless SSID that they are currently associated to. Why would this be handy? How about to identify clients on your network that are bypassing network controls through using the local Starbucks’ wireless network, and therefore providing a possible entry point back into your network.

This does of course have a few prerequisites:

• You need the Direct Feed (commercial) of Nessus plugins, or Security Center, to get this functionality. If you’re a security professional using Nessus as a core tool you of course have this, don’t you? Because then you get all sorts of useful stuff like SCADA plugins, and configuration/compliance auditing.
• You need to be doing a credentialed scan for the plugin to be able to use WMI to extract this information.

This should be able to give you a point in time view of whether hosts that you are scanning are connected to a wireless network when they are scanned. You can then match this against the list of known/authorised SSID’s to identify where clients are associated to unauthorised access points (i.e. the local Starbucks).

Does this solve the problem of identifying clients bridging to a wireless network? Well, no - it has a couple of weaknesses:

• It is at a point in time, so you only have the view of what wireless networks your clients connect to when you’re scanning them.
• This just identifies the SSID, not the access point itself (i.e. the access point’s MAC address), so it’s still possible it’s a rogue access point.

However, it is certainly handy to have this kind of functionality for those who don’t necessarily have a full blown wireless security solution in place.

As promised, I am following my Top 11 Reasons to Collect and Preserve Computer Logs with just as humorous and hopefully no less insightful ”Top 11 Reasons to Look at Your Logs.” 

1. The first reason is again disarmingly simple (is it, really? :-)). Read <