Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #16: Virtually Screwed - Journey Into VMWare ESX Log Analysis

CISecurty guide for VMWare (here) and DISA STIG for virtual machines (here) both mandate collection and analysis of VM platform logs; none goes into enough details on what to look for in logs. Let’s try to shed some light on security-focused log analysis of VMWare ESX v. 3.x logs.

First, at least until ESXi becomes the default choice, one needs to keep in mind that ESX as “Linux-inside” and thus diving into /var/log will not reveal any “alien technology” (well, not much :-)). However, one of the most useful logs is /var/log/hostd.N which is not a descendant of Linux standard logs. Extensive VM event records are written into this file.

Let’s focus on various types of logins to the ESX platform and identify logs that indicate a successful and failed attempts to log in. Here are a few useful examples to analyze:

Successful logins:

• May 30 09:20:42 esx2 su(pam_unix)[9405]: session opened for user root by jhonny(uid=1626)

This is a classic Linux root login message; you can watch for these by searching VMWare ESX logs for “session AND opened AND user AND root.” Notice the user name of the user who switched to root.

• May 30 09:20:34 esx2 sshd(pam_unix)[9364]: session opened for user jhonny by (uid=0)

This is also a classic Linux message for a normal (non-root) user login.

• [2008-05-25 06:57:48.774 ‘ha-eventmgr’ 111639472 info] Event 40645 : User jhonny@1.1.1.1 logged in

This is a VMWare -specific application login to ESX. You can track such events by username, by event ID or by keywords “event AND logged AND user” (if you are using search)

Failed logins:

• May 30 09:20:31 esx2 sshd[9356]: Failed password for jhonny from 1.1.1.1 port 54773 ssh2

Another classic Linux message from the ESX system; a failure to login due to incorrect password.

• May 27 12:06:59 esx2 sshd[4756]: Failed password for illegal user jonny from 1.1.1.1 port 30594 ssh2

A message indicating a failure to login due to incorrect username (note a typo).

• May 25 07:03:48 esx1 sudo: jhonny : 3 incorrect password attempts ; TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/bin/bash

This ESX Linux platform message should also be familiar to Linux/Unix admins: it indicates multiple sudo password failures; look for such messages in the logs.

BTW, do you need to be reminded to track NOT only failed, but also successful login events?!

Overall, you must prepare for the future by learning to analyze VMWare logs, just like you handled “legacy OS”, such as Linux/Unix and Windows.

As I said before, I am tagging all the tips on my del.icio.us feed; here is the link: All Security Tips of the Day.

My next logging poll is out - with it I set out to figure out the old mystery of mine, why people don’t protect their log data (e.g. see this lamentation “Top 11 Reasons to Secure and Protect Your Logs“)

Vote away! As always, results will be posted.

Past polls and analysis are all here. Enjoy!

I’ve been seeing this SQL Server code running wild for the past few days:

DECLARE @T varchar(255), @C varchar(255);
DECLARE Table_Cursor CURSOR FOR
SELECT a.name, b.name
FROM sysobjects a, syscolumns b
WHERE a.id = b.id AND a.xtype = 'u' AND
(b.xtype = 99 OR
b.xtype = 35 OR
b.xtype = 231 OR
b.xtype = 167);
OPEN Table_Cursor;
FETCH NEXT FROM Table_Cursor INTO @T, @C;
WHILE (@@FETCH_STATUS = 0) BEGIN
EXEC(
'update [' + @T + '] set [' + @C + '] =
rtrim(convert(varchar,[' + @C + ']))+
''Explot JavaScript goes here'''
);
FETCH NEXT FROM Table_Cursor INTO @T, @C;
END;
CLOSE Table_Cursor;
DEALLOCATE Table_Cursor;

Actually, the insertion of this code into web servers happens from a DECLARE statement that encodes the entire payload in hexadecimal characters, which is then helpfully translated into exploit code by your own database server. In a way, your SQL Server database hacks itself.

It’s been around since January, but the payloads have been different. Either multiple people are using the exploit, or the exploits are modified on a per-hire basis and delivered through the same bot network. One hacker with a client hack pays some other hacker with a server hack, and they go to town. The process attacks hundreds of insecure websites, which in turn attacks thousands of client hosts.

The interesting thing is that this code doesn’t really have a catchy name like all of the other exploits. Server exploits never get much attention in the media compared to viruses that attack millions of workstations at once, like Nimda, Melissa, or others.

DBA1: “Hey, did you hear that one website got compromised by ‘Column Smasher’?”

DBA2: “No, I thought it was called ‘Lemon Pledge’.”

DBA1: “Why would a database exploit be called ‘Lemon Pledge’?”

DBA2: “Because it cleans everything from your tables.”

There have been a few reports of these attacks hitting Cold Fusion servers. Thanks to Google and the .cfm file extension, it isn’t too hard to find a Cold Fusion server out there. And if someone is using Cold Fusion, they’re probably just coding in CFML, which isn’t a very robust language.

Remember FormMail? Formmail was that horrible CGI script that everyone abused to send out spam. Well, it seems like people haven’t taken the hint. All that information passed from a web client to the server through a GET or POST method should be considered dangerous. Web page constraints, JavaScript/AJAX validators, and hidden form fields can’t protect your database. Depending on how your web forms and server applications are written, you’re allowing outside input from unknown sources to be inserted into the middle of your humble SQL statement. The most important firewall to protect your database is your server side application.

Here’s a few things you can do to protect your database from SQL injection attacks. Suggestions 1 through 3 range from low level sanitation to high level extreme SQL programming. Suggestion 4 is geared more towards administrative efforts for a Database Administrator to protect their system from a web developers badly programmed application.

1. Sanitize the input. Run regular expresison filters that will ideally work on a pattern of allowed characters, Accept only alphabetical characters and numerals, but strip everything else out.

2. Use SQL bind variables to contain web application input, after it’s been filtered.

3. Using stored procedures can give you the benefit of limiting what statements your web application can execute on the database server. Keep in mind that stored procedures are still pretty complex, and unless they’re coded properly, they may not add additional security from the application.

4. Block select privileges to the sysobjects and other system tables. And just because you’re not running SQL Server, don’t assume you’re in the clear. Check with your DB vendor to see specific instructions on how your server handles the Information Schema portion of the SQL-92 standard.

When I first started using LVM I got bit by a few bugs. It’s all part of being an early adopter. As a result I never really used it on production hardware. It wasn’t until about 2 years ago that I gave it another look. In a similar manner I never really thought much of software raid beyond a novelty. Much of that has changed now and I use them both on a regular basis for a number of reasons.

Last week I attended a virtualization seminar. I did not expect a lot from the event at first, but I was surprised by the qualities of the guest speakers. Both had strong backgrounds with VM environments, and they did a good job of explaining what it takes to migrate to VM.

One of the speakers made an interesting statement, saying that the hypervisor is now commoditized. The market for virtual solutions has gotten so big, it’s unavoidable. VMWare has ESX, Xen has their system, and Microsoft is coming out with Hyper-V. If everybody offers what is essentially the same thing, then how do these products stand out from one another?

Now your incentives for buying virtualization have changed. You don’t buy VMWare just because it offers virtualization; you buy VMWare because it has the best service, and the best hot migration features. You might buy Hyper-V because your familiar with Microsoft internal APIs and management tools. On top of virtualization, I’m not sure what else Xen has to offer, but there could be new features coming out from Citrix.

When I left the seminar, I started to re-evaluate hardware decisions that were made in the past. The nature of the beast has changed. Eight years ago, hardware decisions were taken for granted, because it too, was commoditized.

Everything runs on an x86, and everyone makes an x86, so the low price usually won out. Anything that the vendor offers on top of the low price might have clinched the deal. Better support, better service, free shipping? Whatever it took to sell a server and get it out the door. Hundreds of IT departments packed data centers full of tight 1ru servers. Virtualization has now made those servers worthless.

When a single server failed, it was no big deal. You probably had another one just like it running the same application. If that same server is now running multiple virtual hosts, then the service impact is higher. Two machines may now be fighting for access to the same mirrored local disks. What are the chances that they’re impacting each other?

If your server can only handle 2 running virtual hosts, then you cut hardware costs by 50%; but in order to win, your hardware savings still need to be higher than the support and licensing costs of your enterprise VM solution. A 2 to 1 hardware savings ratio isn’t good. it’s expected. In order to maximize your investment, you should aim for a 4 to 1 hardware savings ratio, maybe higher

Migrating to a VM environment does not mean building a VM solution into your servers; it means building your servers around a VM solution. If the hypervisor really is treated like a commodity, then the same can no longer be said about the hardware.

Following the tradition of posting a tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #14: More access_log Fun: What Are You Not GETting?

In this tip, we will look at some bizarre artifacts that show up in web server access logs today. Here we have a production log from an Apache web server that is full of interesting (and sometimes ominous!) little mysteries that we will investigate in order to determine their impact on security and operational health of the site.

Logs do contain more mysteries than we have time, so we will focus on a few of them: specifically, unusual web request methods. Let’s see who is trying to POST or use some other method (OPTIONS, HEAD, PUT or something - see a list here) on our site, instead of just GET’ting the content (GET command is used by web browsers to retrieve the pages, while POST is used to upload content, press buttons, etc - at least in “web 1.0″ land - see earlier tip #12 where POST request was found in proxy logs)

Here is one little artifact that attracted my attention due to a POST request vs a web forum as well as a battery of slashes (which actually increases in subsequent request - of which there were many)

10.10.102.250 - - [12/Feb/2008:16:10:50 -0500] “POST /phpBB3////ucp.php?mode=register&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0″ 200 14397 http://www.example.com/phpBB3///ucp.php?mode=confirm&id=7640df05c7e24b7acf7a68800fe6dc59&type=1&sid=e5efaa77a777066c61f71808e9e57b19 “Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126″

… more…

10.10.102.250 - - [12/Feb/2008:16:12:29 -0500] “POST /phpBB3///////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0″ 200 9355 “http://www.example.com/phpBB3//////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19″ “Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126″

This one really is a mystery; what do we know about it? The server responded to the request OK (code 200), so the POST actually happened. The first request was a request to register with a web discussion board and the second was a request to login. Multiple slashes are actually ignored by the web server, so why put them in the request (no answer)? Also, I think that the User-Agent is spoofed … do you know why? Finally, if I see something like that in my logs, I will definitely investigate it, primarily due to the fact that Apache responded with 200 OK code.

The next one is so classic it it dumb (and so dumb, it’s a classic :-))

10.10.123.226 - - [12/Feb/2008:03:46:54 -0800] “POST /_vti_bin/shtml.exe/_vti_rpc HTTP/1.1″ 404 - “-” “MSFrontPage/6.0″

10.10.123.226 - - [12/Feb/2008:03:46:55 -0800] “OPTIONS / HTTP/1.1″ 200 20210 “-” “Microsoft Data Access Internet Publishing Provider Protocol Discovery”

It is probably one of the ancient IIS attacks (check out this fun BlackHat preso on that, circa 2003) - why would someone probe for it now is beyond me. In any case, Apache on Linux and “*.exe” don’t mix :-)

The final log record is also fun:

10.10.101.222 - - [12/Feb/2008:15:33:22 -0800] “PUT /zk.txt HTTP/1.0″ 405 223 “-” “Microsoft Data Access Internet Publishing Provider DAV 1.1″

The above uses a PUT request which is pretty much deprecated now; the purpose of the above is clearly malicious. In fact, modern Apache shouldn’t even allow it, thus it responds with code 405 “Method Not Allowed.” Nothing to worry about (even though some poor critter got owned with that! BTW, if you follow that link, check out HTTP response code 201 - if you see it in your logs, run! :-))

Overall, if you see too many POSTs or too many “GET then POST” sequences from the same IP in rapid succession, investigate it since no legitimate access should produce such a pattern…

As further reading, I heartily recommend this paper: “Detecting Attacks on Web Applications from Log Files“

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

My next fun logging poll is here - please vote! It is about tools for centralized collection of Windows Event Log from servers and other systems. One of the somewhat surprising discoveries from my previous poll was that few people look at Windows logs; this poll drills down into it.

UPDATE: the results and analysis posted here. Enjoy!

Past logging polls and their analysis:

This is my 6th logging poll (vote here now!)- links to the previous five polls below.

This one is deceptively similar to the #1 below, but it is not. This poll is What logs do you actually LOOK at? and not Which Logs Do You Collect? In other words, are you a log packrat? Are you collecting and never using the log data? You are making a mistake, if you don’t.

Past polls:

UPDATE: analysis of this poll posted here. Enjoy!

A podcasting friend of mine ran into the problem of always having to send the new episodes to his co-hosts for review. Once everyone agreed that it was okay, the show was put live on the RSS feed. Their method of distributing a raw mp3 file? Email.

If only there was a way to distribute the file electronically, without the overhead of email, and yet still get the file automatically once it’s ready. How about RSS?

ITunes and other feed aggregators have the feature of handling RSS feeds that are protected by HTTP authentication. When you download the feed, your client will prompt you for a username and password before downloading the RSS XML.

My friend’s podcast now has two RSS feeds:

http://www.example.com/feeds/public/podcast.xml
http://www.example.com/feeds/private/podcast.xml

The first URL is what’s submitted to all of the podcast directories. The second one is strictly for preview purposes. All of the responsible parties for the show subscribe to the private feed. This allows them to test new episodes, and verify that the RSS <item> content for the episode is correct.

Once everyone has agreed that the episode is ready, the RSS tags for the episode are copied over to the public feed XML file. Now outside users can see the episode and download it.

This is pretty good for most situations, but there is still one risk: HTTP URLs can contain authentication encoding in them, like so…

http://username:password@www.example.com/feeds/private/podcast.xml

Avoid using this convention in your bookmarks, or feed entires. If that URL were to be copied to an outside data source, there’s a chance that it could get into the wild. When that happens, outsiders may end up listening to your private, not production ready, podcast.

To reduce the chances of that happening, ITunes won’t list a podcast in their directory if the URL contains an embedded username and password. They won’t even list a podcast if the server makes a request for HTTP authentication.

Years ago I visited Danga back before the Six Apart acquisition, when the company had its headquarters a couple of miles away and when Brad lived a block and a half down the street. Brad showed me some of their management tools — almost all home-grown.

I mention that because today I stumbled on Dormando’s [crappy] Operations Mantras. Dormando works for Six Apart, and he has the same philosophy I see in Brad. Relentless automation and merciless monitoring are the two secrets of efficient and effective system administration and operations management.

I only wish that someone had handed me this list of mantras when I started as a system administrator in the ’90s. (Puppet and Xen would have been nice too.)

This poll is especially fun: What are your top challenges with logs and logging? Vote here.

Past polls were:

UPDATE: the analysis for this poll is posted. Enjoy!

Here is my next poll about logs: Who looks at logs at your organization?

Vote here!

Also, my past polls and analysis are here.

Let’s say you have evidence of network errors. Here’s the symptoms that you see:

1. A lot of TCP retransmits (layer4)
2. No Ethernet frame errors, dropped packets, or CRC errors (layer 2)
3. No ICMP errors, or IP level errors. Pings report no lag or dropped packets. (layer 3)
4. Failures are only reported on two nodes in your network, but no errors on the switch between the two nodes.

Given the above evidence, would you look at the wiring between the two nodes, including the patch panel ports? If so, why?

No wrong answers, just trying to bring about an open discussion of opinions.

Brendan Gregg wrote up a performance comparison between the utilities Top and Prstat. And just for good measure, he throws in a lot of Dtrace example code to show how he came to his conclusions.

Here’s some advice if you notice your system may be having a performance problem. Run the ls of your /proc filesystem. This will tell you how many active processes are running on your system.

If you’re dealing with a large process count, consider running either command with a short number of iterations, otherwise your monitoring attempts will only contribute to the overall load. I would argue that in a lot of cases, people running Top or Prstat in a background window are actually contributing to any performance problems that they eventually discover through the use of those tools.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #13: Into the Darkness … or The Ominous World of Unix Binary Audit Logs

In this tip, we will take a peek at one of the most esoteric areas of logging: Unix binary audit logs. Solaris BSM and Trusted Solaris auditing is the least unknown :-) example of it, even though other Unix vendors have similar auditing capabilities - see this for HP-UX Audit and this for IBM AIX audit. Linux kernel audit is also pretty much the same thing. If you look for information on ‘Solaris BSM audit logs’ , you’d find plenty of tips on how to enable such logging, a little on how to manage/rotate the log files, a bit on how to survive the resulting data deluge and ALMOST NOTHING on what to do with the log data, which is kinda sad :-) After looking at BSM logs for a while, I developed an opinion that nobody has ever looked at them on a regular basis :-)

So, let’s assume you enabled Solaris BSM kernel audit for user “root” and few other “interesting” users (there is no per-object logging in Solaris; other Unix’es do have it) via the following commonly recommended per-user configuration in /etc/security/audit_user:

root:lo,ad,fw:no

anton:all,-all:no

jsmith:all,-all:no

This config pretty much records all the actions by the users listed. Now, you have audit files growing like shrooms in you /var/audit. What good does it give us? First, we need to convert the binary audit files into text - something along the lines of

# auditreduce -A /var/audit/20071127193515.not_terminated.SunUltra10 | praudit -l > /tmp/sol_box_11272007

will do. Now what? In this tip we will learn how use the audit logs to see who is trying to copy sensitive files off the system.

First, who is connecting out - lets’s search the logs for ‘connect’ calls (if you are using LogLogic for it, use Index Search for this task; if not, grep will have to do, but be prepared to wait). A few recommended searches:

• “connect AND 172.16.10.*” or “connect AND NOT 172.16.10.*” (to look for connection to specific IPs or to the outside networks) or simply ‘connect AND username’

Here is an example found (with connect, IP and user in bold):

header,103,2,connect(2),,Tue Nov 27 11:36:46 PST 2007, + 193 msec,argument,1,0x4,so,socket,0x0002,0x0002,0x80d6,SunUltra10,0x0016,10.1.1.41,subject,root,anton,other,anton,other,29902,29720,0 1611 172.16.0.173,return,success,0

At this point we already know the user name of the user who run that connecting process since it will be in the results (you can also the user to search as I showed above).

Next, what are those connections - let’s try to uncover which programs actually connected (BSM logs don’t make that easy). Let’s search for process starts in the same time frame:

• “execve AND NOT ls AND NOT <whatever other commands you don’t care to see>” will give you a list of started programs.

Example:

header,124,2,execve(2),,Tue Nov 27 11:36:46 PST 2007, + 115 msec,path,/usr/bin/scp,attribute,100555,root,bin,136,1573,0,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,0

Notice that both records have the same timestamps. Sadly, time and parent process ID ( which is in our case 29720) is all that correlates them together.

Finally, what file was affected (i.e. copied off the system via scp in this case) - more digging is in order; we again use the process ID and time. The easiest is to search for a file name or browse all records around the same time frame (might be A LOT!):

• “*secret.zip* AND anton” will work; we can add the above process ID and look for “anton AND 29720″ (but expect a lot of data since this is a shell process ID)

For example:

header,135,2,open(2) - read,,Tue Nov 27 11:36:47 PST 2007, + 900 msec,path,/tmp/not-so-secret.zip.gz,attribute,100600,anton,other,0,32743959,18446744073709551615,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,4

What do we know now? This user connected to this system and MAYBE copied this file via, MAYBE via scp. How cool is that? (A: not cool at all, since we are not sure!)

To conclude, if you can avoid dealing with Solaris BSM logs, please do so :-) On a more serious note, you now know why these logs were called “the ugliest logs ever.”

Even more seriously (but still pretty humorously), these logs are a classic example of trees that make every effort to obscure the forest, because they record syscalls and not processes or user actions (and connect, execve and read are all logged separately). There are also many, many more idiosyncrasies (and, in fact, idiocies) where these come from :-)

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Time for another fun logging poll: What Do You Do With Collected Logs?

Vote here!

This is my Logging Poll #3, links to past polls:

• Poll#2 “Why Collect Logs?” (vote here, results so far, my analysis)
• Poll #1 “Which Logs Do You Collect?” (vote here, results so far here, analysis here)

UPDATE: analysis and results posted at here

The idea came from Jeremiah Grossman (here) when he described “The Best Web Application Vulnerability Scanner in the World” thus: “Within a few moments of pressing the scan button it’ll find every vulnerability, with zero false positives, generate a pretty looking report, and voila you’re compliant with GLBA, HIPAA, and PCI-DSS. Of course, we all know such a web application scanner is simply not possible to create for a variety of reasons.”

So, let’s imagine the idea log management application.

1. Logging configuration: the ideal log app will go and find all possible log sources (systems, devices, applications, etc) and then enable the right kind of logging on them according to a high level policy given to it (required: God-like powers)
2. Log collection: it will collect all the above logs securely (and without using any risky super-user access ) and with little to no impact to networks and systems (required: God-like powers)
3. Log storage: it can security store the above logs in the original format for as long as needed and in a manner allowing quick access to them - in both raw and summarized/enriched form (required: plenty of hardware)
4. Log analysis: this ideal application will be able to look at all kinds of logs, known to it and previously unseen, from standard and custom log sources, and tell the user what they need to know about their environment and based on their needs: what is broken? what is hacked? where? what is in violation of regulations/policies? what will break soon? who is doing this stuff? The analysis will power all of the following: automated actions, real-time notifications, long-term historical analysis as well as compliance relevance analysis (required: AI)
5. Information presentation: this tool will distill the above data, information and conclusions generated by the analytic components and present then in a manner consistent with the user’s role: from operator to analyst to engineer to executive. Interactive visual and drillable text-based data presentation across all log sources. The users can also customize the data presentation based on their wishes and job needs, as well as information perception styles (required: nothing more than a bunch of daring UI designers)
6. Automation: the ideal log management tool will be able to take limited automated actions to resolve discovered and confirmed issues as well as generate guidance to users so that they know what actions to take, when full-auto mode is not appropriate. The responses will range from full-auto actions to assisted actions (’click here to fix it’) to issuing detailed remediation guidance. The output will include a TODO-list of discovered items complete with actions suggested, ordered by priority (required: AI + some luck + some user stupidity :-))
7. Compliance: this tool can also be used directly by auditors to validate or prove compliance with relevant regulations by using regulation-specific content and all the collected data. The tool will also point at gaps in data collection as it applies to specific regulations that the user is interested in complying (required: God-like powers)

In other words, this magic black box will have crap shoveled from one side and will have answers to questions about the meaning of Life :-) coming out the other side…

What? :-) Am I nuts? Well, can I dream for a second? :-)

Following my now-famous Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs, here is the promised “Top 11 Reasons to Secure and Protect Your Logs”

1. Let’s review why you are reviewing logs. Will logs that might have been changed by somebody, somewhere, somehow still be useful for items 1-11 from here? No? Secure them!
2. Oooh, logs in court? Challenges abound! To respond to them, one needs to protect the logs so you can claim that they are both authentic and reliable.
3. A human error still beats an evil hacker as the main cause of IT problems. Are your logs safe from it? Available when needed? Protect them from crashes and other faults!
4. PCI DSS just says so: “Secure audit trails so they cannot be altered.” Wonna do it- or pay the fines?
5. Do you protect financial records? Identity info? Passwords? Some of it ends up in logs - thus making them more sensitive. Secure the C-I-A of logs!
6. Do you look at logs during incident investigation? Do you want them to be “true” or full of random (if creative…) cr*p, inserted by the guilty party? Secure the logs!
7. Think that “attacks vs logging” are theoretical? Think again. Are your logs safe or vulnerable? Is your logging tool 0wned?
8. Syslog + UDP = log injection. Are you protected (reliable TCP, confirmed delivery, encryption - SSH, SSL, VPN)?
9. Why change logs? No, really, why change logs? If you never change logs - and you never should - hash them right away after collection to make them immutable.
10. Logs are backed up on tape - who will see them? Well, whoever restores the tape, that’s who! Encrypt them to protect them from accidental and malicious disclosure if tape is lost.
11. Why log access to logs? Same reason why you had the logs in the first place - to review who did what. Who broke through and stole the logs? Who browsed them without permission? Only logs will tell - if you have them!

Overall, one need to strive for having no holes in log safeguards from log birth to analyst conclusion based on log information…

Possibly related posts:

• Top 11 Reasons to Look at Your Logs

• Top 11 Reasons to Collect and Preserve Computer Logs

The previous poll (vote here, live results here, analysis here) proved to be a success so the next one is here.

This time the question is: “Assuming that you centrally COLLECT system, network or security logs from their originating sources, what is THE MAIN reason for doing it?”

Vote on!

UPDATE 11/11/2007: results and analysis are posted here

See all my polls here.

Read part one, before continuing.

Ten minutes later, Dave the DNS administrator was in the garage. He joined Kip, Tom, Sally, Douglas, Velma, Suse, and myself. “Before we begin, let’s review exactly what this script does,” I said.

Kip looked around, and decided to retell his overview, “The script reads database records from a table, and then for each record, it reads a HTML file on the disk, updates the database record, and then loops to the next record.”

“You forgot something,” I said.

“Oh,” he said. “After the database update, it waits before going on to the next record.”

“You didn’t say wait the first time,” I said. “You said rest.”

“Yeah,” Kip said. “Actually, it’s the sleep call.”

“How long does the script sleep?” I asked.

“Just one second,” Kip replied. He looked over at Sally and Tom, “We were concerned that the script could hammer the database unless we put in the sleep statement.”

I turned to look at Sally and Tom, “So you were concerned about performance?”

“Yes,” Tom said. “We’re dealing with a lot of data and file activity. The script runs 4 times a day.”

“How much data?” I asked.

“The table has sixty thousand records,” Sally said.

I grabbed a piece of chalk from Velma, and wrote 60,000 on the garage floor. “We are starting with 60,000 records”. Underneath it, I wrote 1+. “And we know that each loop iteration will take at least one second, because we explicitly sleep for one second.”

Everyone nodded their heads in agreement.

I turned to Tom, “How many seconds are in a minute?”

“Sixty,” he said.

“How many seconds are in an hour?”

“Three thousand, six hundred.”

“And how many seconds are in a day?”

Everybody looked around, quickly trying to do the math in their head. I turned to Dave, the DNS administrator. “How many seconds are in a day, Dave?”

“Eighty-six thousand, four hundred, ” he said. I wrote 86,400 in chalk on the floor.

“How did you know that?” Douglas asked.

“Most DNS administrators are familiar with that number, because it’s commonly used the set the Time To Live value of DNS records.”

“And the script only has a window of twenty-one thousand, six hundred seconds to run in, since it runs four times a day.”

Everyone looked at the numbers on the floor. Sally was the first one to speak. “So, in an effort to reduce the impact of the script, we made it worse with the delay.”

“Exactly,” I said. I picked up the source code. “Without thoroughly looking at this code, I would estimate that ninety-eight percent of the time, this script is doing nothing but waiting. On decent enterprise hardware, the file operations, and the database updates should only take milliseconds.”

“That’s right,” Velma said.

“So, how do we fix this?” Kip asked.

“Well, if there is a concern regarding impacting other systems, just change your sleep iteration. The simplest thing to do is to not sleep for every iteration, but for a percentage of iterations. If the script slept for 1 second every 10 iterations, it should finish in under two hours.”

“That’s an easy fix,” Kip said.

Velma took the chalk and wrote down some additional figures. “For the best case, the script should know how many records it has, and how many seconds it has in the given window to process all of those records.”

“Right, ” I said. “Turn it into a mathematical function, but decrease the window size by an hour, just to be on the safe side.”

Dave looked around and raised his hand. “Uh, is there anything else you needed me for?”

“No, ” I said. “That was it.”

Dave walked back to his bike, mumbling about the 10 mile bike ride back to his house.

“How did you know I didn’t use the perl Time::HiRes module?” Kip asked. “I could have slept for fractions of a second.”

“If you did, the use statement would have been visible in the first page of your code printout,” I replied.

“Well, how did you know I didn’t use threading?”

“Because nobody casually implements threading in perl scripts.”

There was a somber moment until Tom spoke up. “Wow. I never expected the sleep statement to be the problem with this.”

I picked up all of the graphs and charts from the table. “This is the most obvious solution, given the information you all have told me. You guys had the answer all along, you just never expected it to be that simple.”

“Remember, ” I said. “A good systems administration team always works together, and is not afraid to look at the code written by a developer.” I turned to Kip. “At the same time the developer should not be afraid to discuss performance metrics, and profiling code behavior with a systems administrator.”

Douglas was about to say something when a wild haired CTO walked into the garage. She was carrying a sleek 2 rack-unit server with a hatchet embedded into the casing. “Somebody hacked my Linux server!” she exclaimed.

I looked back at the others. “Sorry guys, I have another case.”

The small team of geeks said their thanks and walked out of the garage. I turned to the CTO holding up a jar full of quarters, “Payment up front, and I can’t guarantee anything if you didn’t preserve the ARP cache.”

Parody. Based on a true story.

It was a lazy, summer Saturday morning, just like any other. From my vantage point, I could see kids riding their bikes and playing street hockey. I sat in the middle of an empty garage with my feet propped up on a table, reading a book on how to lift fingerprints from a CVS repository. Saturdays were good days for the detective business, so I knew it wouldn’t be long before customers came in with their problems.

Less than an hour after opening up, Kip, Sally, and Tom walked in. It was Kip that spoke first. “We want to hire you,” he said. He backed up his statement by dropping twenty-five cents into the jar sitting on the table. I put down my book and assessed my clients.

Kip was a good perl programmer; always keeping his parents happy with his ability to quickly write scripts to handle any problem. Sally was the DBA, who held the record for winning the Oracle performance competition at the county fair. Tom was the new systems administrator, who was known for carrying his pet lizard (Suse) with him everywhere he went. All three of them were good at their job.

“I have a perl script that’s taking way too long,” Kip said while dropping a few pages of source code on the table. “It’s a looping function that impacts the database and the filesystem.”

“What exactly does the script do?” I asked.

“The script grabs a lot of records through a SELECT call; each record contains information on a HTML file on the disk. For every record, I run a loop that stats the file, reads the first few lines from the file, makes an UPDATE call to the database with new information, and rests before moving on to the next record.”

Tom jumped in, “The job is scheduled through cron to run every six hours: 6am, noon, 6pm, and midnight. But when I look at the process table, there’s multiple instances running. The jobs are taking too long and not finishing within six hours.”

Sally added her two cents, “The updates don’t seem to have a serious impact against the Database. The table itself is only sixty thousand rows, and it’s fully indexed.” Sally brought me graphs generated from an Oracle management program.

“And the filesystem is fine, too.” Tom said. “The average number of disk requests remains constant. There’s no iowait or contention.” To make his point, he placed a stack of SAR reports on my desk.

I looked at the data sitting on my desk and thought about it for a second.

“Gentlemen,… and Sally,” I said. “This isn’t a problem with code, and it’s not a problem system performance. You could say that the problem lies with not understanding the very nature of the planet.”

All three of them looked at me, perplexed. “But you didn’t even look at my source code!” Kip exclaimed.

“I don’t need to,” I replied. “You already gave me everything I need to know.” I reached into my baseball mitt and grabbed my iPhone. “In five minutes I can have a DNS administrator over here, and he can give you an important fact that you all missed.”

WHAT DID I KNOW THAT THEY DID NOT? THE SOLUTION WILL APPEAR MONDAY, BUT FEEL FREE TO THROW IN YOUR COMMENTS.

Update: The solution has been posted.

I figured I’d do a poll a week since people really like it. So, my first poll-a-week: Which Logs Do You Collect?

Vote away! I will post and comment on results here after a few weeks.

UPDATE: poll results and analysis are posted here. Enjoy!

We rolled out mod_evasive across a pool of servers the other day. Since we already had Apache running, you can rightfully assume that installing this module was done in response to user bahavior.

No, we weren’t selling Hannah Montana tickets, or seeing if Ron Paul would make a nice president; but we did attract a regional based script kiddie. If you give teenagers an online poll asking who has the better football team, and the winner of that poll will be announced on television; it’s a good bet that a few people are going to stuff the ballot box in their favor.

Ironically, nobody even cares about the results; but we have to deal with the people running libwww-perl, or specially crafted JavaScript pages that resubmit form values hundreds of times. Since this isn’t online banking, we decided that using captcha wasn’t worth the effort, so the goal was to block excessive attempts.

The case for mod_evasive is pretty clear. In most cases, it’ll stop successive hits repeatedly sent to the same URL multiple times. Fifty hits enter, one hit leaves. The hit per second parameters are fully configurable. It also logs to syslog; so its behavior can be monitored.

The case against mod_evasive is scalability. Mod_evasive does not use shared memory between child processes. It also won’t work in a load balanced server pool unless the client IP is persistently tied to the same web server in the pool. For larger web server environments, a better solution should be implemented into the load balancing front-end. Finally, in some cases, mod_evasive may not be enough; because even though it still returns 403s, you’re still dealing with a hit and an open TCP socket connection on your server. If your infrastructure is under attack, mod_evasive will never replace firewall blocking or upstream filtering.

But, if your environment is relatively small, or if application abuse does not have a high impact, mod_evasive is a pretty good tool to have around.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #12: Proxy Log Fun - Proxy Logs vs Information Leakage

You probably know that web proxies (such as Squid, BlueCoat SG, BlueCoat Netcache and others) produce a lot of fun logs. Indeed, they are fun since they can be used for a whole range of things, from routine monitoring for AUP compliance to malware detection as well as possibly looking for the security scourge of 2007 - web client attacks.

Specifically, in this tip we will learn how proxy logs can be used for detection of file uploads and other outbound information transfers vie the web. First, think what is the legitimate use of file upload functionality for your web users. If web mail is allowed, then sending an attachment will include an upload. What else? The rest will be considered at least suspicious…

In addition to file uploads, some spyware application will also use similar methods to steal data. Looking for methods and content-type in combination with either known suspicious URL or user-agent (i.e. web client type) can often reveal spyware infections, actively collecting data. Admittedly, a well-written spyware can certainly fake the user-agent field so it is clearly not reliable, but still useful to add to our query above. Here are some of the criteria we will use to look for uploads in Squid and BlueCoat SG proxy logs:

• HTTP method (logged as “cs-method” by BlueCoat) = POST (as opposed to the usual GET, used to retrieve web content).
• For information uploads: content type (logged as “RS(content-type)” by BlueCoat) = anything but “html/text” (which is the type used for uploading web form contents) - especially try content types “application/octet-stream“, “application/msword“, “application/powerpoint“, “application/vnd.ms-excel“, “application/pdf” and a few others to look for common file uploads
• For spyware and application data transfers: user-agent set to anything but the common ones (i.e. not Mozilla, iTunes, LiveUpdate, etc) or even to “unknown.” One can also try user-agent containing your favorite messaging app (e.g. “MSN Messenger”, etc)

(if you feel adventurous, other interesting content-types to try are “application/x-javascript” and “text/javascript”)

Here are the examples of the above, including some “classics” (while spyware specimen are a bit dated, this method of detecting them via logs is relevant):

1. 1124376766.026 RELEASE -1 FFFFFFFF 4734C557F9315105CA6BE0FA56B94D55 200 1124276674 -1 -1 unknown -1/0 POST http://reports.hotbar.com/reports/hotbar/4.0/HbRpt.dll
2. 1124392388.975 RELEASE -1 FFFFFFFF 810FFBF233584C330353CF0A8C31F5D2 503 -1 -1 -1 unknown -1/813 POST http://log.cc.cometsystems.com/dss/cc.2_0_0.report_u
3. 2007-05-19 03:55:12 160 10.1.1.3 - - - OBSERVED “Spyware/Malware Sources;Spyware Effects;Web Advertisements” - 200 TCP_NC_MISS POST text/html;%20charset=utf-8 http bis.180solutions.com 80 /versionconfig.aspx ?did=5342&ver=1.0 aspx - 10.1.1.2 273 175 - - none - -
4. 2007-05-21 03:10:40 4 10.1.1.3 Joanna- authentication_redirect_to_virtual_host PROXIED “Search Engines/Portals” - 307 TCP_AUTH_REDIRECT POST - http storage.msn.com 80 /storageservice/schematizedstore.asmx - asmx “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; MSN Messenger 7.5.0324)” 10.1.1.2 791 2566 - - none - -

Here are some other signs that will make the above log entry extra-suspicious is:

• A dead giveaway: upload happens to a “known bad” URL (e.g containing “gator” and others above)
• Upload happens to an unresolved IP address (do a “whois” on it!)
• Uploads happens to a port not equal to 80 (i.e. the URL contains a port such as http://10.1.10.10:31337)
• Upload has confidential file name in the log entry (e.g. somebody dumb emailing a sensitive file to himself - as discussed here)

Overall, this log analysis method is good for casting a broad net to catch not just spyware-infected systems, but also unauthorized applications (e.g. method=POST and user-agent=iTunes), instant messaging (e.g. method=POST and then by user-agent, content or URL), simple forms of data theft and document handling policy violations (emailing files to self via web mail: method=POST and sensitive file name present in the entry; also content type set to popular file types) as well as other abuses of web access. As a result, proxy logs provide an extremely rich AND readily available source of data about threats that users face!

To top it off, one promising direction of future research is using web proxy logs to detect client-side exploits by malicious web servers (more on this in the near future!)

Possibly related posts:

• Anton Security Tip of the Day #6: The Other Web Log
• All my past security and logging tips

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

I recently had reason to spend a while working with Nessus on Windows XP (Service Pack 2). Usually, I use a Nessus Server running on Linux, either running locally if I am onsite, or one installed on our company infrastructure for scanning from the Internet. In fact, you read the documentation don’t you?, Tenable specifically recommends in the Nessus Installation Guide that you _not_ run Nessus on XP, and instead use a Windows Server product, such as Windows Server 2003.

The reason for this is that in Windows XP Service Pack 2, Microsoft introduced a number of Network Protection Technologies for mitigating the spread of malware. One of these limits the number of simultaneous incomplete outbound TCP connection attempts to 10, with additional attempts being queued and potentially dropped. This impacts the reliability of at least port scanning, and possibly other security checks.

Unfortunately the scenario I was working with required me to be running Nessus through a VPN client (never ideal), in reality requiring me to be on XP. Tenable does, however, have some recommendations for running Nessus as reliably as possible on XP:

• Max number of hosts: 10
• Max number of security checks: 4
• Max number of packets per second for port scan: 50

The maximum hosts/security checks setting is standard in all of the Nessus clients I’ve used, however the packets per second setting seems to only be available within the client shipped with the Windows Nessus server. If you, like me, are using the new NessusClient 3.0 beta for Windows, you need to make the following change to the Nessus server’s configuration to ensure that 50 is the default value:

• Go to the “config” directory in your Nessus server installation. By default this is C:\Program Files\Tenable\Nessus\config
• Open config.default.xml for editing - just use Notepad if you don’t have an XML editor
• Find the SYN Scan:Max number of packets per second for port scan node, and edit the value (the CDATA bit) from 500 to 50

This value should now be the default for all new scans.

This worked well for me, however needless to say that running a Nessus scan in VMWare (slowdown factor one), over a VPN link (slowdown factor two), over a transatlantic Internet connection (slowdown factor three), the scan took quite a while to complete…

Update: Ron Gula corrected me on this - this is available on the free registered feed.

A little while back I spotted this article on the Tenable Blog in reading my morning RSS feeds - Tenable have added a plugin with the ability to interrogate Windows machines for the wireless SSID that they are currently associated to. Why would this be handy? How about to identify clients on your network that are bypassing network controls through using the local Starbucks’ wireless network, and therefore providing a possible entry point back into your network.

This does of course have a few prerequisites:

• You need the Direct Feed (commercial) of Nessus plugins, or Security Center, to get this functionality. If you’re a security professional using Nessus as a core tool you of course have this, don’t you? Because then you get all sorts of useful stuff like SCADA plugins, and configuration/compliance auditing.
• You need to be doing a credentialed scan for the plugin to be able to use WMI to extract this information.

This should be able to give you a point in time view of whether hosts that you are scanning are connected to a wireless network when they are scanned. You can then match this against the list of known/authorised SSID’s to identify where clients are associated to unauthorised access points (i.e. the local Starbucks).

Does this solve the problem of identifying clients bridging to a wireless network? Well, no - it has a couple of weaknesses:

• It is at a point in time, so you only have the view of what wireless networks your clients connect to when you’re scanning them.
• This just identifies the SSID, not the access point itself (i.e. the access point’s MAC address), so it’s still possible it’s a rogue access point.

However, it is certainly handy to have this kind of functionality for those who don’t necessarily have a full blown wireless security solution in place.

As promised, I am following my Top 11 Reasons to Collect and Preserve Computer Logs with just as humorous and hopefully no less insightful ”Top 11 Reasons to Look at Your Logs.” 

1. The first reason is again disarmingly simple (is it, really? :-)). Read PCI DSS lately? Glanced at HIPAA? Suffer under FISMA? Yes, all of the above regulations say that you must not only have, but also review logs periodically.
2. Are your servers compromised now? How do you know if all your logs are stashed on a tape in a closet? Look at them! Now!
3. An incident happens. Really, who needs extra motivation to look at logs in such case? Using logs for incident response is a true ”no-brainer” (however, you need to be pretty “brainy” to actually analyze them in case of an incident)
4. Users - from a CEO to a janitor. You do have to know what they do on your IT systems! How? Read the logs! Everybody leaves tracks.
5. Systems log plenty of errors. Sometimes they are silly, sometimes - benign. However, often they mean that “stuff” is about to hit the fan. Periodic review of logs reveals them and saves the day.
6. Network slowed to a crawl?  Applications are slooow? Server is not … well, serving? :-) Where is the answer? In the logs, but you need to read them and understand them.
7. That policy you wrote a few months ago. Anybody following that? Anybody remembers that? Halloooo! Check the logs and you’d know.
8. By now you know that your auditor might ask for your logs. But do you know they might also check whether you looked at them? Do you? Review the logs and leave the record of this activity in the logs.
9. Change can be good. But then again, it may be the sign that your controls are lacking. Who changes what and when? From what and to what? Just review the logs.
10. Now, you hate looking at logs. You have too many (as if everybody else doesn’t…)! In this case, look at a specific subset of logs that you never saw before- NBS. Or just deploy log management that can do it for you.
11. Logs can help you predict the future (if you review, know and love them :-)). Don’t believe it? If you read them for long enough, you develop an ability to predict the future, albeit mostly future problems :-)

See also: Top 11 Reasons to Collect and Preserve Computer Logs. 

Coming soon: “Top 11 Reasons to Secure and Protect Your Logs”!

Following the new “tradition” of posting a security tip of the week (mentioned here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #11: But These Are OUR Logs!

A common and unfortunate situation that occurs when dealing with logs is not technical, but political: not being able to get the logs you need due to political, cultural, egotistic, or other “corporate” reasons. In this tip we will try to present a few situations and solutions for those trying to wrangle logs from whatever hostile (or ambivalent - sometimes worse!) entity at your organization and thus to break the siloed approach to log management.

So, here is the situation: a desktop system starts “behaving strangely” (as evidenced by network IDS logs, which are controlled by the security team) and security wants to take a peek at the system logs to determine how it was 0wned or infected (no centralized log collection takes place). However, the security team does not have administrator-level (or, sometimes, any) access to all desktops needed to grab security logs from a Windows PC: only the desktop division of IT department does. And they refuse! Why?

• what if the security team finds that the intrusion happened due to our negligence?
• what if “they” find something else wrong while looking for logs?
• what if they crash the system and leave “us” holding the bag?
• why should they mess with “our” systems? - we are perfectly capable of taking care of it ourselves? (not! -))
• <fill whatever reasons you’ve heard!>

What can you - the security analyst or manager - do?

• leverage your existing relationships and get the logs “informally” (obviously, doesn’t scale …)
• remind them of the company security policy (hopefully, written by you to support such cases!)
• [UPDATED] ask your internal auditors for help
• use whatever desktop logs you can get a hold of (example: client anti-virus logs; other security agent)
• report them to senior management (yours or theirs; of, if you report to the same boss, both yours and theirs)

As a side note, database administrators (DBAs) are even more famously resistant to providing log data.

Overall, the tips above might help; however, to resolve such control issues once and for all, the smart organization must deploy log management tools across the entire organization and then provide limited access to the logs to all the stakeholders on the “as needed” basis …

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Following the new tradition of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #10: Email Tracking Through Logs

Email tracking - oh, need I say more? :-) A nightmare for privacy fans - an “evil” weapon of lawyers and HR. Email tracking raises concerns that vary from a severe inability to do it all the way to having too much ability to do it. In this tip, we will focus on the following scenario: your boss says she just sent you an email; you never received it. What’s the story?

First we need to search our Sendmail server logs (this is that case where searching logs is appropriate) for the email address of the sender (unfortunately, we rarely can search the logs of the remote mail server which sent - or didn’t send! - the message in question). One thing to note before we start searching is to limit the time frame to the one close to the reported problem (let’s say to the same day).

So, we search for messages containing from=boss@examples.com (since we want the sender address) at January 11 (if you just search for boss@examples.com, the volume of unrelated results will be much higher) and find something like this:

Jan 11 11:45:49 ns1 sendmail[28022]: j0BGjkc28022: from=boss@example.com, size=376597, class=0, nrcpts=1, msgid=Pine.LNX.4.61.0501111145450.17081@boss.example.com, proto=SMTP, daemon=MTA, relay=esafe1.example.com[10.211.15.69] (may be forged)

Is this our message (we are not quite sure since we don’t see a recipient)? Why isn’t it here?! What do we do next? Well, this is where things get a little ugly (for those unfamiliar with Sendmail logs).

Next, we need to search the logs for this “mysterious” string “j0BGjkc28022” from the previous log record and find this:

Jan 11 11:46:00 ns1 sendmail[28025]: j0BGjkc28022: to=employee@example.com, delay=00:00:14, xdelay=00:00:11, mailer=local, pri=405901, dsn=2.0.0, stat=Sent

So, it looks like our mailserver have indeed sent the message, but what happened to it afterwards? Where did it go next? Given that “employee” has a mail account on the example.com system, most likely the message went into his mail spool (something like /var/spool/mail folder)

Next? If the employee uses local shell access to a mail server and Pine or other similar mail client to read mail (really unlikely, but possible - mostly at University systems), we need to see whether it exists in user’s local mail folders, such his /home/username/Mail folder or something similar.

But what if the employee uses POP3 or IMAP to read mail? In this case, we have more logs to go through … However, most POP3 server logs will not help us determine the fate of a specific message, but just to confirm that the user checked and/or downloaded mail. We will leave POP3 log fun for the next tip…

So, in today’s basic tip we covered one simple scenario of email tracking through logs.

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Somebody posted a message to a loganalysis list seeking help with analyzing a trillion log messages. Yes, you’ve heard it right - a trillion. Apart from some naive folks suggesting totally unsuitable vendor solutions, there was one smart post from Jose Nasario (here), which implied that the original poster will need to write some code himself. Why?

Here is why (see also my post to the list): assuming 1 trillions records of 200 bytes, which is a typical
PIX log message size (a bit optimistic, in fact), we are looking at roughly 180TB of uncompressed log data. And we need to analyze it (even if we are not exactly sure for what, hopefully the poster himself knows) … not just to store.

Thus, I hate (ehh, make it “have” :-)) to admit that Jose is probably right: writing purpose-specific code might be the only way out. About a year ago, there was a discussion titled “parsing logs ultra-fast inline” on firewall-wizards list about something very similar. We can look up some old posts by Marcus Ranum for useful tips on super-fast but purpose-specific log processing.

For example, here he suggests a few specific data structure to “handle truly ginormous amounts of log data quickly” and concludes that “this approach runs faster than hell on even low-end hardware and can crunch through a lot of logs extremely rapidly.” One of the follow-ups really hits the point that I am making here and in my post: “if you put some thought into figuring out what you want to get from your log analysis, you can do it at extremely high speeds.” A few more useful tips are added here.

So, nothing much we can do here - you are writing some code here, buddy :-) And, as far as tips are concerned, here is the “strategy” :-) to solve it:

1. figure out what you want to do

2. write the code to do it

3. run it and wait, wait, wait … possibly for a long time :-)

Indeed, there are many great general purpose log management solutions on the market. However, we all know that there is always that “ginormous” amount of data that calls for custom code, heavily optimized for the task at hand.

Ok, so I’m not completely sold yet. I still have a boatload of Perl code floating about, and for certain things I’m still writing *new* Perl code. However, I was coerced into using Python for a project I’m working on, and I have to say that I think Python is coming on to me.

I try to ignore the furtive glances, and those times when I could swear it’s actually winking at me. I don’t acknowledge the beguiling smiles and greetings I get from Python when I open my laptop. I just get down to the business of coding and pretend none of it ever happened. That pretending is getting harder by the day.

Here’s the thing. I’ve been doing all of my sysadmin scripting in perl, awk and shell (sometimes together) for a decade. After 10 years, Perl still doesn’t even say “hello” to me. It seems to stand ready to spit my own code all over me whenever I try to talk to it. And just when I’m about ready to call it my friend, just when I think I know it, it completely changes. Well, I’m tired of it. I’m tired of the schizophrenia. I’m tired of the attitude. I’m tired of feeling like a Perl n00b after using it for 10 years.

I’m leaving.

Of course, like lots of relationships, it’s complicated. Over the years, Perl and I have spawned offspring that aren’t going to just disappear because I decide I don’t like Perl anymore. I promise to care for them and keep them up to date.

But from this day forward, I’m going with Python in those places where I can. I *want* to feel confident with a language. I *want* to take advantage of code reuse, self-documenting code, and OO design principles. I *want* to have readable, concise code. I *want* to solve problems that are larger than the every day “please change my shell” requests. I want to build tools. I want to architect solutions. I want to solve some of the problems sysadmins face, but I had to solve my own big problem first: namely, being a self-hating Perl slinger who was never particularly comfortable with how Perl, at a very high level, works.

If you’re an admin using Python on a regular basis for your admin scripting, let me know how you think it compares to Perl for equivalent tasks. If you’re a Perl coder who has tried and *not* used Python for reasons besides the lack of curly braces, fill us in! If you’re a religious zealot for one language and have never used the other, feel free to move on!

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #9: But He “Wasn’t Logged!”

The idea for this tip originated when my presentation on log analysis was rejected by one of the high-profile security conferences on the grounds that “logs don’t matter since advanced attackers never leave traces in logs [or erase them before anybody can get to them] .” Indeed, some of my security friends of a more “offensive orientation” :-) have long developed this snobbish attitude about logs.

So, imagine a network that has fallen victim to a 0day-wielding “uber-haxor” :-), who kicked the door open (or snuck in), grabbed the crown jewels and took off like a bat out of hell :-) When, much too late as usual, the “good guys” rushed in to pick up the pieces, only there was seemingly nothing much to pick: the server logs were erased and the network IDS didn’t make a peep (and, no, Richard, NSM was not deployed). What do you do now?

So, let’s list some uncommon (and some common, but often untapped for the task at hand!) sources of log data and provide a few log analysis tips:

• firewall logs: boring they might be, but it is less likely that those are erased by the attacker (even though firewall logging can sometimes be jammed). And it is hardly possible to “not be logged” by the firewall (the analyst’s challenge is in analyzing the huge volume of data to find the attacker’s traces and to tell them from normal traffic). What is critical in making these logs useful is logging allowed connections, not only blocked ones.
• router logs and netflow records: same as firewalls logs - they are less likely to be erased, but huge log volumes make their use problematic. And, last but not least, these types of logging are more likely to be turned off completely (”routers should route, not log,” grumble many network types)
• application logs: was it a “legacy” network attack or something application-level? If the latter is true, there might be logs found in unusual places. Is there anything in the webserver logs? Websphere made a peep? MySAP recorded something? PHP app logged something fun?
• various client logs: in one old case FTP client logs were extremely helpful, even though the entire syslog directory was blown away and no remote logging was ongoing; look for other client logs (yeah, even a web browser history is kind of a log…) to look for things missed by the attacker
• other systems’ logs: was there anything that the attacker did that affected other systems? Might have they logged it? Even an attempted SSH connection or a scan from a compromised machine has a high chance of leaving traces elsewhere - cleaning all of such traces is a major challenge for an attacker (in any case, much harder than erasing logs and stopping logging on the compromised host)
• any remote or centralized logging: was any log saved from destruction by being copied to another system?

To conclude, while there is no “logwatch” pattern for “advanced attacks,” logs are still very useful in such circumstances if you prepare by setting up a broad scope of log collection (I suspect using a log management system will be your only choice as log volumes will be pretty bone-crashing) and then combing through the logs after the incident. And remember the less common sources of log data, such as database logs.

I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Sean Reifschneider has posted a post-mortem of the PyCon 2007 wireless networking installation he designed and managed. From my experiences at other community conferences, almost no venue knows what to expect beforehand. Good work, Sean!

Following the now old :-) “tradition” of posting a security tip of the appropriate time interval (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #8: What Just Changed?

Let’s close our eyes for a second and dive deep into the bizarre and menacing world of a Windows event log. As I mentioned before, massive Windows server log collection got a jump start in recent years due to wide availability of agentless Windows log collection tools, such as Project LASSO. (yes, many people think that agents suck event when they are useful - weird, isn’t it?)

Windows event logs, the “Big Three” of System, Security and Application as well as other logs, share a lot of contradicting properties: way too much detail in some areas and missing critical info in others, consistent and thoughtful design here and sheer stupidity there, nice structured data sometimes and confusing mumbo-jumbo in other cases. And the universe of the event log is never static, the whole thing flows and morphs with each Windows release and at time with each update. New event IDs are being created, changed and loaded with new roles and new info.

In this tip, we will look at some fun Windows log entries and explain their meaning for your organization as well as cover what you should do if you encounter them. Given that the realm of Windows event log is so huge, we will start from looking at events that indicate changes of different kinds, mostly configuration and user account. So, what just changed?

I. “Computer Account Deleted” or “User Account Deleted”: obviously, service or user account was deleted. Who did it? When? Why? Answer all the questions above and then you can go back to sleep - or to your incident response plan :-)

II. “Computer Account Created” or “User Account Created”: same thing - depending upon when? why? who? this event means nothing or something pretty ugly.

III. “Computer Account Changed” or “User Account Changed“: similarly, changes to accounts are reflected in the events containing this text. Account changes do include privilege level changes that are often of particular interest.

At this stage, it might be appropriate to ask: why aren’t we going by Windows event ID to identify the above events of interest, but instead choose to use the above text blurbs? Well, up to Vista, Windows event IDs often aren’t :-) Meaning that they don’t identify the event sufficiently. Sometimes, they are overloaded and the same ID applies to very different things. Sometimes, the opposite happens - same event, different IDs (e.g. a lot of login/logout stuff)

IV. “Policy Change”: might mean almost anything on a Windows system. Thus, we can’t really tell you much; you need to read the event to see what actually changed (if anything!)

V. “The system time was changed” might not matter that much, but if you are looking to use your logs as forensic evidence (i.e. use them in court) you might want to track all the time changes since they will affect the log timestamps on the server where time changed.

VI. “The following schema object was modified” oooh, don’t you love Active Directory! This indicates that some of the AD objects changed - fortunately, the object name will be in the same event.

Enough for today! Windows logging makes most everyone’s head hurt (unless you are Eric or Randy, I guess :-))

So, to conclude, make sure that you collect Windows event logs and analyze them on an ongoing basis, preferably using your log management system.

As I mentioned before, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Finally … the wait is over. In a few short seconds, my security predictions will be unleashed onto the world.

Unlike last year, I won’t muddy the waters with scores and ratings, but will just say it: predicting is easy if you go for the obvious trends that everybody sees (”hackers will hack, phishers will phish, spammers will spam, bot herders will bot herd, other criminals will, eh, do crimes!” and - yes! - both bad and good stuff will happen in ample quantities). Last year I erred on the side of caution and got every single prediction right. Promise I will be more aggressive - and thus more fun - this time :-)

I. Platforms: Vista will have no impact on the overall risk level of most organizations out there. Yes, some holes will certainly be plugged (and I even agree that “Vista is the most secure version ever”, just like every single one of its predecessors was - in its time), but others - possibly of types we don’t even know about - will crop up. Sorry, but secure platform =/= secure Internet (kinda like you wearing a Kevlar vest doesn’t lower crime in the neighborhood).

II. New technologies: no credible technology that can alone “solve” the problem of insider threat will emerge (many will try); the insider threat problem is just too broad, diverse and rich to be solved by a single technology or even a single vendor (corollary: if somebody is trying to sell you such a technology that claims to do exactly that on its own, then - well, you know what to do …)

III. Security market: we will see more than a few firesales and possibly total and miserable security vendor failures (wonna bet which legacy SIEM vendor will die first? :-)) There are way too many companies who sell some random and often irrelevant “protection” which sometimes doesn’t even work … at their own demo … when their CTO demos it … the third time …

IV. Risk management: a confusion about what is “risk management” will not subside this year. Business risk? Information risk? Risk as threat x vulnerability x asset? Risk as probability of loss? Arrrghh! - It goes on and on and on. No standard accepted definition of risk management in the field of infosec will emerge.

V. NAC: of course, no list of 2007 prediction is valid without mentioning knack :-) And you know what? NAC will shrink, NOT grow in importance this year! This is where the rubber meets the road and fish start to swim upstream :-) - this prediction started from me reading Richard’s piece “NAC is Fighting the Last War” which struck me like a Strength 15 Lighting Bolt. Indeed, narrowly defined NAC largely targets worm infections (and will thus lose relevance) while broadly defined NAC starts to sound like having a well-run network (which is as relevant today as it was in 1992 and probably 2012 as well). The Planet NAC is about to experience a premature eclipse :-)

VI. 0-days: 2006 was the year when this previously obscure term fell victim to malignant marketEErs. 2007 will see more of the same, no doubt. But what about the real 0-day-wielding attackers, poking jokes at the above “oh-day defenders”? Security research into new types of vulnerabilities will certainly continue and more types of previously “safe” (rather, “erroneously thought of as safe”) types of content will be used to attack applications. MPG with 0day? AVI with 0day? And, our old friends doc, xls, ppt and now PDF. On the other hand, a major 0-day worm still won’t happen.

VII. IP and ID theft, data loss: at the risk of sounding hilariously obvious, I would state that such incidents of ID theft (phishing, etc), broader intellectual property (IP) theft and loss will continue largely unabated. Will we, the security community, try to stop it? Of course, but nowhere near hard enough …

VIII. Compliance: but of course! Did you think I’d miss this bad boy? Mandatory regulatory initiatives that pack a bite or a punch, such as PCI, will continue to spread and thus grow in importance, while jokes like HIPAA will continue to languish, helping my # VII prediction come true with a bang … At the same time, I am undecided on the voluntary frameworks that you can choose to comply with (ISO17799/270001, COBIT, ITIL, etc) - will they take off like a rocketship or remain steadily interesting to some? Only time will tell.

IX. Security awareness: well, security awareness will … ah, come on, just laugh: bua-ha-ha-ha-haaa :-)

X. Finally, I would like to reiterate a few of the last year’s predictions that will still ring true this year. Client-side and application-level (especially, web application) vulnerabilities will still be outrunning the server-side and platform-level ones. Major wireless attacks and malware will still not destroy the world.

So, there you have it! Enjoy, comment, slam, compliment, etc! And remember - just as one cannot predict the threats of tomorrow today, one still won’t be able to do in 200X. And in 20XY :-) And - but I am going out on a limb here - in 2XYZ! :-)

All predictions for 2007 that I’ve seen are tagged here.

Here is a fun account of a [relatively] advanced attack against a high-profile site (a high-profile defacement mirror site, to be exact!)

Not surprisingly, web server logs play an important role in the investigation, in a situation which I highlighted in this tip. Specifically see these two records shown in the paper:

 - - [21/Dec/2006:23:23:15 +0200] "GET
/index2.php?act=img&img=ext_cache_94afbfb2f291e0bf253fcf222e9d238e_87b12a3d14f4b97bc1b3cb0ea59fc67a
HTTP/1.0" 404 454  "http://www.zone-h.org/index2.php?option=com_jce&no_html=1&task=plugin&plugin=..<>/<...<!--//..<////..<////..<////..<////images/stories/food/x
&file=defi1_eng.php.wmv&act=ls&d=/var/www/cache/&sort=0a"
 "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705)"

as well as

 - - [21/Dec/2006:23:23:59 +0200] "GET
/index2.php?option=com_jce&no_html=1&task=plugin&plugin=..<>/<...<!--//..
   <////..<////..<////..<////images/stories/food/x&file=defi1_eng.php.wmv
&act=ls&d=/var/www/cache/cacha/&sort=0a
 HTTP/1.0" 200 3411 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705)"
 212.138.64.176 - - [21/Dec/2006:23:25:03 +0200] "GET /cache/cacha/020.php
 HTTP/1.0" 200 4512 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.0.3705)"

(note the web server response codes in bold)

This is another fun log line from the incident account that has some lessons-learned value as well - it has a 200 code on a script that you (the web server admin) didn’t deploy (see bold)…

 - - [22/Dec/2006:01:05:15 +0200] "POST
 /cache/cacha/020.php?act=f&f=configuration.php&ft=edit&d=%2Fvar%2Fwww%2F
HTTP/1.0" 200 4781  "http://www.zone-h.org/cache/cacha/020.php?act=f&f=configuration.php&ft=edit&d=%2Fvar%2Fwww%2F"
"Mozilla/5.0 (Windows; U; Windows NT 5.1; ar; rv:1.8.0.9) Gecko/20061206
Firefox/1.5.0.9" 

So, what do we learn from this incident log analysis:

1. Look for weird commands (those containing “.<!–//..<” certainly qualify) with 200 HTTP response codes

2. Look for executable files you didn’t put on your server with 200 response codes

3. Don’t let your defacement mirror to be defaced :-)

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to go along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #6: The Other Web Log

We all know that Apache web server has its access_log log files while IIS has its w3c logs. When people think about web log analysis, they think about the above and only about the above logs, often calling them “the web server logs.” However, the other web log - error_log (in case of Apache) - contain a lot of fun and useful info as well. Today’s tip is to encourage the review and analysis of this treasure trough - pardon the idiom - of insight.

So, what goes into the error_log? Well, errors, duh! Why errors matter? ‘Cause errors often present the only indication that a) you are being 0wned or, worse, that b) you’ve been 0wned by the attackers. Additionally, apart from the obvious security usage mentioned above, errors matter since they - or rather, some of them - require actions by the user and thus knowing about them is of huge importance. What is even more striking is that many error messages present an interesting tradeoff - do only a little now to correct the reasons (or, “the root cause” as network management people would say) for the error, or do a whole lot later when the system crashes, gets hacked or “misbehaves” in some other truly spectacular way. Now, a grizzled BOFH would surely say “heh, but who would want to do that! surely, dealing with a dramatic world-ending catastrophe later is more fun that making a minor config fix now.” Well, I sincerely hope not all of my readers are such people :-)

Now, back to the Apache errors; we are going to show a few examples from a live phishing site, run by the attackers on a compromised server. We will start from the obvious - server restart, which actually happened as a result of an attack, in our case.

[Sun Mar 12 04:02:05 2006] [notice] SIGHUP received. Attempting to restart

Did YOU restart the server? No? Two choices then - someone else did (likely without permission!) or the server got restarted automatically for whatever strange reason. You certainly need to be at least somewhat concerned in both cases - thus: look for the above messages.

Here are a few more fun examples - the relevance of those for your business is left as an exercise for the readers, but all of the messages below

[Mon Mar 13 14:54:11 2006] [error] [client 10.10.10.10] Premature end of script headers: index.htm

Does “index.htm” sound like a script to you? It sure should not; and this error message indicates that somebody is trying to run it as a script - a clear indication of malicious behavior. The next message is also similar in this regard:

[Mon Mar 13 14:54:26 2006] [error] [client 10.10.10.10] attempt to invoke directory as script: /var/www/cgi-bin/tcpsupport/

and so is this one:

[Mon Mar 13 14:54:23 2006] [error] [client 10.10.10.10] (8)Exec format error: exec of ‘/var/www/cgi-bin/tcpsupport/main.htm’ failed

Other interesting things spotted on the same server -this one looks like an overflow attack attempt (and, no, the NIDS did not make a squeak about it)

[Thu Mar 19 07:16:11 2006] [error] [client 10.10.10.10] request failed: URI too long (longer than 8190)

So, to conclude, the tips is: when doing web server log analysis, make sure you look at the error logs as well as the access logs.

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #5: Sendmail Log Adventures

Today we will dig thru some Sendmail logs. Why? Heh, why not :-) On a more serious note, Sendmail (as well as Exchange, QMail, Postfix, etc) MTA logs hide a plethora of insights, useful for email security as well as email performance management and troubleshooting.

For our analysis we are using our Linux server running an open source version of Sendmail, but using any other Unix OS would be pretty much the same. Sendmail logs a dizzying array of messages to syslog. Typically on a healthy server,  90% or more of Sendmail log records describe email messages as they are being sent and received by the server. For example:

Jul 3 18:20:41 ns1 sendmail[25425]: i63MKWZ25425: from=<china9988@21cn.com>, size=0, class=0, nrcpts=0, proto=ESMTP, daemon=MTA, relay=[211.229.110.64]

Everybody knows that such messages are useful for email accounting. What is more interesting is that they can also be used for higher level pattern discovery or anomaly detection (which I sometimes call “log mining”) to look for things such as unusual count of email recipients, unusual message size, new high-volume email streams, etc. However, most of such log records are not too interesting individually (note that to trace an email, looking at just one record is not sufficient since the useful info is smeared over multiple sendmail log records). But what else is there? How do we dig our teeth into that elusive 10% which are different and potentially ominous? Well, one common approach is to look at rare messages; if such pool is still too  large - after all, a 10% of a billion is still a 100,000,000 -  one can then try to post-filter them with some “interesting” keywords, such as “error”, “failure”,  “abort”, “crash” and then looking at the related and nearby (by time) messages (it is left as an exercise for the reader as to not why do it in the opposite order…)

So, what does such approach yield in one particular case:

Nov 20 03:39:22 combo sendmail[2022]: rejecting connections on daemon MTA: load average: 14

Nov 11 08:56:20 combo sendmail[23863]: runqueue: Aborting queue run: load average too high

Oooh, our mailserver was overwhelmed at one point and mail might have been lost. Is that an attack, a spam wave or we just became popular? More digging is definitely required …  Check your own mail server for the presence of such log records and, upon finding them, be ready to handle complaints about missing emails.

Nov 11 08:54:49 combo sendmail[29820]: jABDsGrH029820: collect: premature EOM: Connection reset by combo.bkwexample.com

Hmm, what does that mean? Well, let’s take a position - and it won’t be a big stretch :-) - that we have no idea whatsoever. But the next message (notice the timestamp!) also shows up in our “rare messages” list

Nov 11 08:54:50 combo sendmail[29820]: jABDsGrH029820: SYSERR(root): collect: I/O error on connection from combo.bkwexample.com, from=edgardo_shapiro@cexample.net

Now it is a bit more clear. The email server experienced a critical failure in mid-mail and the email was corrupted and thus not received. Further analysis reveals a gap in logs after the above message that lasted for 20 minutes. Network or hardware failure has occurred.

So, today’s tip was about sendmail logs: now start looking for rare unusual messages using the approach we outlined. Specifically, watch for errors and failures in the set of rare messages and then investigate by reviewing the log records in the same timeframe. Also, look for gaps in logging, especially those gaps that occur after such rare messages.

Also, here is a link to my previous tip of the appropriate-time-interval (#1). Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

tags: security, tips, logs, log management, sendmail, email, chuvakin

So I’ve been taking an informal poll of the sysadmins I know to find out how people are managing the synchronization of files across a server farm. Looks like there are three popular ways of handling this, which I’ll list in no particular order:

First, there’s NFS. There are numerous places out there that have a central file server, and then the server farm mounts, say, /opt or /usr/local or something, and then there are lots of configuration files and stuff underneath those trees somewhere. The benefit of this method is that you can make a change in one place and have it take effect everywhere more or less immediately. The downside, as I see it, is something I call the “christmas light syndrome”: if the file server goes down, any services on any hosts relying on the mounted files become unavailable.

There’s another upside to the NFS scenario, which is that your server farms can mount config files read-only, which offers some protection should the machine be compromised in some way.

The second popular method is to use rsync. The upside to this method is that all files are local to the machine, so services on your hosts don’t depend on the availability of a file server. The downside is that generally there is some glue code and duct tape involved, which means you’re maintaining code to take care of all of this, which means there’s not really a standard procedure per se for handling file synchronization with rsync. In addition, you don’t have the benefit of having your config directories mounted read-only, which is just one less protective measure.

The third method is cfengine, which is still hanging on my list of things to make friends with. I tried using it during the version 1.x days, when it was quite a bit more difficult to use. I’m aware that the 2.x versions are much, much easier, more robust, supports RSA keys, and all that jazz, and I promise that once I get through the three projects currently on my plate, cfengine is number 4.

If you’re using other means of handling file synchronization, or you just wanna plug your favorite feature of cfengine, or have a cool rsync hack or something, please share!

One of the concerns system administrators might consider involves the upcoming release of Vista. You may not consider it a concern for a number of reasons, especially if you only focus on Linux and avoid Windows. At some point we generally have to deal with the Windows desktop.

Like previous versions of Windows, the older version goes off the shelves immediately. Microsoft will recall Windows XP. At that time, your choice becomes a desktop OS that will take some years to gain general adoption in business.

Windows 98SE, for example, has a significant user base today. It runs fairly well on Pentium III slot 1 450-550 Mhz processors. Most of those machines had maximum memory limitations of 384 MB. Ten years after the release of Win 98SE, it continues to run on those machines. And, many of those machines haven’t quit. Why the Intel 845 chip set and its slot 1 processor performed so well and for so long is a mystery, But it did and does.

What about the installed base of Windows XP Professional? Many companies will continue using it and if you need to use it for whatever reason, a legal copy will become a difficult acquisition,

So, why am I concerned about Windows desktops? In the US, the Linux desktop is not gaining market share. Some say that Apple’s Mac OS X hasn’t gained much either. In art and production departments, education and with media companies the Mac still has a solid following. But, the salesman on the go, the worker in the office and the executive writing his next report or creating his next presentation all use Windows. You’ll also find Windows in the small market where devices like the Treo 650 live in places like hospitals running diagnostic applications. Hey, I’ve seen them and they’re popular.

Linux continues to gain market share in the server arena and data center. But, it will not replace Windows servers completely. One example to consider, Cingular cell phones now come with an Exchange email client. Predictions have come true - the Windows device coupled with Exchange may become the Blackberry killer especially when the low price, easily replaceable cell phone runs Redmond’s mail client by default.

Recently, I had to run a simulated environment with Mac OS X 10.3, Windows 98SE & XP and Linux desktops. The Local network server ran Fedora Core 5 and Fedora Directory Server. The Linux desktop ran Ubuntu 6.06 because they needed to run on Pentium III slot 1 boxes. SUSE 10.1 and Fedora barely installed on the old hardware. That put a scare in my argument for replacing Win 98 with Linux. Ubuntu did install easily but couldn’t keep up with Win 98 on the same equipment.

The Mac ran well on the old PPCs. So, they aren’t going anywhere. The client needs to run software you won’t find optimum on anything Windows and not at all on Linux.

I usually auction off software I use for pilots. Once I’m finished with simulations, I typically put the hardware and software on eBay, But not this time, We’ll see more requests for simulations and the test environment will include a new entry called Vista. I’m betting that Vista adoption will take a long, long time.

If you have to do this sort of thing (and I think we’ll see an uptake in people wanting to test alternatives), then get yourself a copy of XP Pro now before they leave the shelves forever. I’m giving my trial copy a rest and going for the retail version.

Finally, I’m starting to tire of the comment clowns. While I like solid feedback from professionals, the last bean head who recommended that I start taking meds and fade off into the sunset put an end to the interactive mode. I found the guy at dcn.davis.ca.us. If you know him say hello for me and tell him he can find the type of people he wants to engage over at slashdot or digg. Those 14 year olds will welcome him with open arms.

Linux system administrators should consider getting their MCSE. What? That’s correct. You might also consider buying a Mac Mini desktop and practice with it at home. I’m serious, so take this recommendation to heart.

In the past, I have written about the crazy job market for Linux system administrators and help desk professionals. Hiring managers have problems with hiring pure Linux professionals. It might fall into the area of myth, but hiring managers believe those myhts. They believe Linux guys who say they don’t mind working with Windows will then turn around and leave within a short time frame. Secondly, Linux technologists with previous experience with Microsoft will find hiring managers leery of their Windows skills. Those hiring managers probably have a point.

Wanna eat? Here’s a email I wrote after a client wrote and said they would pay me for some Linux work I did:

That’s OK. I found a trash can with some cat food in it still in the
can. Though it was past the expiration date, it didn’t make me sick.
It was tasty too. I’m heading over to the shelter now. I have several
friends over there who are also contractors. We play dominoes
when we haven’t got work. Thank goodness we aren’t busy
at the same time cause only one terminal is in reserve for us at the
public library. But, I found a church where they let me use their
computer when the accountant goes to lunch or meets the minister at
some hotel for a conference where they can get away from the other
staff members and confessors.

I appreciate you keeping me up to date on the check thing. Wow. I’ll
get half a thousand within a week. I can go down to the thrift shop
and look for my fall wardrobe.

That was the 14th email in the thread. Tongue-in-cheek? Who knows?

So, here’s my recommendation: Go to a community college and take some flex courses and get your MCSE. That’s seven tests you have to take and you already know much of what they teach. You’ll need the jargon, of course.

Doing that will give your resume a fresh look and give hiring managers and recruiters a reason to look at your resume. Most Linux jobs require you to fit into a Microsoft infrastructure. It’s a fact many people I speak to will confirm.

Now, a note about comments. I’ve seen a lot of comments where people deny my observations. I don’t mind that if the person has a legitimate case. I’ve tracked down enough comment makers to find that they are not who and what they say. One so called Tech manager attends college in Australia and doesn’t have a job. Many comments come from the college crowd. Others have personal web pages and blogs and they do not have any experience in the areas they claim. I have emailed others.

So, read comments on sites with some skepticism. I learned to ignore many even though some have legitimacy.

Admittedly, this is again a repost from my other blog , but I am sure it would be useful to my O’Reilly readers.

Not the wormy “CodeRed”, but Red Alert, mind you.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #4: Code 200: Good or Bad?

Now, this is somewhat related to my previous tip (#3), but as it applies to web server logs, such as IIS Extended W3C (or other) logs or Apache access_log logs. Unlike, say, databases, all webserves come with logging enabled, so web server log review is a common task for web server admins as well as for security teams. But what do you look for in normal web access logs, apart from web site access statistics and visitor trends?

Obviously, observing various access failures is important. All the pesky 404s, 401s as well as 50X response codes.

Here are a couple of examples from Apache web server access_log logs:

61.52.47.251 - - [26/Sep/2004:21:10:22 -0400] “GET /MSADC/root.exe?/c+dir HTTP/1.0″ 404 286 “-” “-”

or

67.170.226.184 - - [10/May/2004:13:18:09 -0400] “GET /scripts/..%%35c../winnt/system32/cmd.exe?/c+dir HTTP/1.0″ 400 293 “-” “-”

Is this ominous or what? :) Nah, it is just my Linux + Apache honeynet responding to some good ole IIS 4.X attack. Truly nothing to worry about.

How about this one though?

68.49.152.86 - - [22/Apr/2004:11:56:49 -0400] “GET /cgi-bin/awstats.pl HTTP/1.0″ 200 2190 “-” “Mozilla/4.75 (Nikto/1.32 )”

Somebody using the web statistics CGI script? Benign, since the response code is 200 which means that the server served the page (or ran the script!) as expected? No, you are about to get 0wned thru an AWSTATS hole!

So, the gist of this tip is similar to the previous one: when monitoring web server logs, do not just focus on 404s, 401s, 50Xs and other “known bad” response codes; take a long hard look at the vanilla 200 response codes. “But how do I separate the “good” 200s from the “bad” 200s?” - you might ask! Well, that would be the subject of another tip in the future!

Just in case, here are the links to my previous tips: #3, #2, #1.

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

tags: security, tips, log analysis, log, web logs, web servers, log management

Admittedly, this is again a repost from my other blogs (see here and here), but I am sure it would be useful to my O’Reilly readers.

Following the new tradition of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #3: Watch For Access Failures AND Successes in Logs

Now, many a winter ago :-), people used to think that checking for access failures in logs is all they need to do to “stay secure.” Indeed, picking out failured access attempts seemed like a reasonable way of doing things. Similarly, people even considered firewall “connection denied” messages as more important than “connection allowed” log records (although this will be a subject of a separate tip - a whole paper, in fact - later)

So, what is more important? This

Sep 26 12:36:40 bridge sshd(pam_unix)[2128]: session opened for user anton by (uid=0)

or this:

Sep 25 14:31:24 ns1 sshd[19308]: Failed password for anton from 10.10.154.44 port 53452 ssh2

Let’s think about it: one log entry says that Unix security is doing its job and blocking a bad user (assuming no fat fingering and forgotten passwords for simplicity sake) from accessing the system, the other says that some kind of user now has access to your system.

To quote Doc from “Back to the Future” :-) : “Exactly!!!”

You do need to be aware of both; you can’t just focus on access failures while monitoring your logs. Make sure you review successes as well. And, yes, certain patterns, like a long string of failures followed by a success are even more fun to watch …

Also, here is a link to my previous tips of the appropriate-time-interval (#1) :-)

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

tags: log analysis, logs, security, tips, chuvakin

Technorati Profile

Admittedly, this is again a repost from my other blogs (see here and here), but I am sure it would be useful to my O’Reilly readers.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #2: Watch those Pesky Log Rotation Routines

What is the biggest threat to your Linux logs, which reside happily at their home in /var/log? Is it crackers? Blackhats? Evil BOFH sysadmins? Dumb users? Malware? No, its the log rotation routine!

For example, many Linux distros such as RedHat, Fedora, etc use logrotate tool to get rid of “unwanted” logs. Other distros use other log file rotation tools that accomplish the same. However, many ship with default settings that are not optimized for log analysis and long term retention.

Upon deploying a Linux box one needs to look at /etc/logrotate.com file as well as /etc/logrotate.d/ directory to make sure that you are happy with retention settings (and you shouldn’t be!)

One can safely increase the retention of most log files from a default of about 4 weeks (it differs by file type and distro) to at least 12 weeks (about 90 days, a common practice) or even more. While at it, one should also enable old log file compression since it is off by default (for whatever weird reason…) to save some disk space.

Note that if you are using a central log management system, this might not apply to you. But then again, in this case you are farther ahead of most system owners on the road to operational excellence.

Also, here is a link to my previous tip of the appropriate-time-interval (#1) :-)

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Admittedly, this is a repost from my other blogs (see here and here), but I am sure it would be useful to my O’Reilly readers.

Upon seeing folks giving security tips of the day on their blogs (like here, here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #1: Crank it Up!

Crank what up? Logging, of course! You’d be happy you did, I promise. The gist of this tip is that if you increase the level of logging on your system (even without looking at the resulting logs periodically…), you’d fare much better in case of an incident. And, by incident here I don’t mean that those evil hackers will 0wn you with a custom-coded 0day, but something much more common: a computer crash, spyware problem or system malfunction.

On Linux/Unix it will likely involve editing the /etc/syslog.conf and on Windows it will involve changing the Audit Policy.

Of course, you’d be commiting a mistake, if you only turn it on. But that’s a separate story.

I wanted to put a note here for myself, and for others who are looking for a way to get information about a particular executable running in a zone on their machine. While it is (to the best of my knowledge) not possible to do this from within the local zone itself, you can run dtrace from the global zone and specify the zone name and executable by using a logical AND in the predicate, like this:

dtrace -n ’syscall:::entry /zonename == “webserver” && execname == “httpd”/{ printf(”%S”, curpsinfo->pr_psargs); trace(pid) }’

This is a command I used to get some quick information about all of the httpd processes running on a web server that exists as a zone on a solaris 10 machine. Here’s some output:

0 6485 write:entry /var/local/httpd/bin/httpd -DSSL\0 12248
0 6779 llseek:entry /var/local/httpd/bin/httpd -DSSL\0 12248
0 6489 close:entry /var/local/httpd/bin/httpd -DSSL\0 12248
0 6789 pollsys:entry /var/local/httpd/bin/httpd -DSSL\0 12248

The command is run from the global zone. This isn’t exactly what I wanted — I wanted all of the arguments passed to all of the system calls, but I got the arguments passed to the httpd process itself at start time instead. I’m still chugging through the documentation, so if I find more cool stuff, I’ll put it here. Meanwhile, if you know how to do what I’m trying to do, or how to do some other cool stuff with DTrace, please share!!

Here are a few links about DTrace that I’m using to get more information. It’s plenty to get you started, but not a lot of info on how to do anything remotely advanced. Even coverage of ‘advanced’ features is pretty dumbed down compared to the crazy stuff I’m seeing in some of the example scripts. Nonetheless, this stuff *is* useful:

http://www.snpnet.com/sun_DTrace/dtrace.html
http://users.tpg.com.au/adsln4yb/dtrace.html
http://www.sun.com/bigadmin/content/dtrace/
http://blogs.sun.com/bmc

I’ve been administering production UNIX systems since 1998. I’ve been running Linux-and-only-Linux on my desktops and laptops for almost that same period of time. In 2003, after seeing an influx of Macs into our department, my boss asked if I’d keep a Mac around just so we could replicate any issues that were reported by our userbase to help them out. I found it perfectly usable, though at the time it was real work for me to get it to be usable for all of the things I’d need it to do at work, and I never was comfortable with it. Recently, however, I had the opportunity to get a new laptop through work, and I decided to give the new MacBook Pro a shot.

I really committed to making it my primary workhorse, shunning my nice dual-lcd-monitor linux box and a perfectly usable Gateway laptop (”the monster” - 17″ screen and a full numeric keypad!!). It has been about one month since the MacBook Pro showed up, and I haven’t had to use any other machine for about 2 weeks. It took a little bit of googling to find everything I needed, and there’s still one or two pieces of the pie missing, but overall, I think I can definitely call this machine “admin ready”.

So let’s have a look at what I’ve done to my MacBook to make it worthy, and maybe some friendly readers can help me fill in some as yet missing pieces!

Between November 2003 and May 2005, I was working on the rather mammoth task of evaluating my department’s environment, analyzing and auditing our infrastructure services, seeing if we could migrate from NIS to LDAP, and then evaluating LDAP server software, writing tools to perform the migration, writing tools to maintain data consistency between NIS and LDAP (for those things that couldn’t use LDAP yet), and writing tools to administer LDAP and integrate it into our environment.

Software evaluation at the time was fairly quick: I looked at eDirectory and SunOne, both of which (at the time) allowed you to store some obscene number of objects under a no-cost license. I chose OpenLDAP at this time because it was libre software with a pretty active support mailing list.

I spent the next year learning the intricacies of OpenLDAP: which back ends might benefit my deployment, how to configure that back end, why my distribution vendor chose not to use the recommended back end (forcing me to build from source), keeping up with the rather frequent upgrades (which the support list folks demand you do, lest you be heckled rather than supported), figuring out how Access Control Lists work, figuring out why some operations were so slow while others seemed blazing fast (this goes back to choice of back end and its configuration), and the list goes on. Over the course of a year, I probably upgraded OpenLDAP 3 times, upgraded the back end at least two times (one was the advent of Sleepycat BDB 4.0, if memory serves), and began to feel like there was no end to the tweaking. I feared I’d be pigeonholed into being solely an “LDAP Admnistrator” instead of a system administrator.

Then, on June 1, 2005, Fedora Directory Server (and Red Hat Directory Server) was released to the world. It wasn’t yet completely open source, but they announced that they were committed to open sourcing the bits that weren’t yet open in the coming months. I downloaded and installed the server, got on the support IRC channel, and imported all of our data in a couple of hours. By September, just three months later, it was in production, and I haven’t looked back.

Recently I decided for some reason to run my site (also, like the O’Reilly blogs, based on Moveable Type) through the W3C Validator to check to see if I was generating valid XHTML. If you’ve ever done this, you shouldn’t be surprised to find out that my site was anything but compliant. Why is this important? Well, it’s probably not going to break most modern browsers as they are quite tolerant of non-compliant code, but it’s bad coding - if your code is valid then you should be displayed well by any compliant browser or parser in use. Non-compliant code can also do bad stuff to dual use content, such as if where you’re using the web entries to provide RSS feeds for your site.

Some of the things that I wasn’t doing correctly included:

You can find many definitions and concepts about entrepreneurs and most involve innovation. In a formal way, you’ll see textbooks saying that they recognize opportunities and then orchestrate the people and resources to turn the advantage of an opportunity into a successful business venture.

Then you see additions to the descriptions that says an entrepreneur assumes the financial risk and management of going after the opportunity. I wouldn’t classify this group as entrepreneurs. Instead, I call them financial risk takers.

People who know have designated me an entrepreneur on a few occasions. I saw unexploited market niches and went after them. The first involved writing a microprocessor based accounting system for CPAs that allowed even sole proprietors to cut costs and do the work of ten staff members. That caught fire and had a nice exit strategy.

Others included automating the financial planning industry with a computerized system. Then we have the building of a Linux/UNIX clone of Outlook for Microsoft Exchange. In the course of events I learned to refine the process of orchestration and grow a second skin. I haven’t ventured out again since 2002 after three hostile takeovers.

My experience generally and in Entrepreneurs Anonymous says that if you orchestrate a niche and taking it to the next level involves the need for capital. The guys with the money typically kill the original entrepreneur. First they reduce the entrepreneur’s holdings and finally figure a way to push them out of the business usually through false accusations, litigation and withholding of promised rewards.

A Niche for the Brave Hearted

I consider myself a free software proponent. In my weltanschauung I also see the software in devices as something that manufacturers should open. But, they don’t.

An irony exists for the vocal free software crowd. It sits in their use of proprietary hardware to host their free software operating systems. Again, in my world view it’s like the irony of Ireland’s copying the government structure of the nation she most hated - incongruity between what might be expected and what actually occurs.

In my last article, I wrote about my commercial DVD player failing. As an update to that, I purchased a much smaller and more capable RCA in China brand replacement for $39. Feature wise it beat up my previous RCA box that cost multiples of $39. The new box took up 20 percent of the space of the previous one.

The new DVD player supports a multitude of proprietary formats including music and video we like in the free software world. That intrigued me. While I fiddled with it, I learned that it had an embedded OS and components that cost me more at my favorite computer stores or on-line. In other words, I couldn’t build that DVD player for $39 from off-the-shelf parts.

Going back to size, while compact, it also contained space. I would have designed the RCA in China DVD player differently and only used the platter and placed the components below it. But I can see why the manufacturer did not do that. Next to it on the same shelf, I saw a portable DVD player selling for $120 more and that was the cheap one.

That gave me the idea for a new business involving the use of an embedded OS and a set of proprietary audio and video formats. If free software advocates can live with Intel and VIA chip-sets, why couldn’t they live with multimedia on a chip?

The Changing World of Media

Recently, my wife made an astute observation about recorded music. She started off by discussing her vinyl record collection, then her cassette tapes (she missed the 8 track phase), then her CDs and now digital music. She found it somewhat funny that people invest in smaller and smaller delivery devices.

I wonder if Popsicle could increase the sales of its products by putting an iPod or other device in the center of their Fudsicle or Big Stick Cherry Pineapple Swirl. They could sell their Good Humor brand with a Blue Tooth enabled ear piece.

Anyone Interested?

As a kid, I coveted my grand dad’s McIntosh MA 230 Tube Preamp / Amplifier. I remember it had 60 watts RMS with 30 coming out of each side. Of course, it needed a huge piece of furniture to hold it and his Acoustic Research model AR XA turntable. I loved it. But, it came from his day.

Not too long ago, I found a 500 watt RMS preamp/amp that I picked up with my index finger and thumb that sold for $30. I just needed to add a volume control and a couple of other items to hook it up to my Cambridge speakers. Talk about irony.

Laugh as you will, hit this article with your vitriolic comments, etc. I get more validation from George Gershwin’s “They All Laughed”. I’m pretty sure you don’t know the song, since it comes from days before most of your grand parents grew up. It’s about that second skin I mentioned.

I see the opportunity for a work-around for the things Linux and FreeBSD need to compete with the Mac OS X and Windows desktops. If an entrepreneur(s) exists out there, consider music on a chip for free software platforms. It’s probably a money maker but you’ll need a little money to make it happen.

I’ve seen people describe LDAP as being a database enough times to get really annoyed, and finally, to blog about it. So here, in no particular order, are all of the reasons why LDAP is not a database (er, that I can think of):

* The “D” in “LDAP” stands for “directory”, not “database”
* The “P” in “LDAP” clearly indicates that LDAP is, in fact, a “protocol”.
* LDAP has no notion of rows, tables, or other database elements.
* LDAP has no notion of relational integrity (ie, foreign keys)
* SQL is useless when run against an LDAP server.
* LDAP data is a hierarchical collection of objects, not a linked collection of relations.
* There are no cascading deletes.
* You cannot write a stored procedure or trigger to help maintain LDAP data.
* Attributes of a given LDAP element can essentially be infinitely multi-valued without penalty.
* Individual attributes can be restricted in LDAP. You cannot do the equivalent (restricting access to a single field) in a database (to my knowledge. Post links if that’s wrong).
* LDAP, as a general rule, uses standardized schemas to describe objects. I have not seen a standardized database schema. Ever.
* LDAP objects and attributes are defined using ASN.1 syntax. Similar to SNMP (another protocol), and not similar to a database.
* Your email client is probably ready and willing to look at an LDAP directory to autopopulate its addressbook data. To my knowledge, the same cannot be said of a database.
* You can put “ldap” in your nsswitch.conf file. You can’t put “mysql”.
* Two words: “spin lock”

There are plenty of other examples to point to here. Also note that I’m not saying that things are technically impossible. You could probably develop standard data schemas for mysql, and then develop an nss_mysql module for your systems to use. However, this would not make LDAP any more similar to a database.

Recently, I had the pleasure of watching an article of mine show up on Slashdot, Digg.com, Newsforge and a myriad of other web sites at the same time. None of my colleagues had seen so much traffic on a Linux site before. Aside from the several million hits on our server, we had a quarter of a million unique visitors concentrated in a five-hour period.

When you see that kind of traffic, you don’t want the server to go down or you’ll miss new readers. In our situation, a reboot allowed the system to return to service for a few minutes, but then it locked up again. On a normal day, we used less than ten percent of our system resources, so we thought we had prepared for the hottest day of the year. Little did we know we would experience rolling blackouts.

I attempted to explain to our self-proclaimed system administrator that we needed a way to renice unnecessary processes automatically, restart those that failed and keep the server going at all costs. He just had no enterprise experience and insisted he knew the best way to manage the system. He used a RRD tool with a front-end. But when it came to handling processes, he insisted he would login with SSH and handled everything remotely.

This provides an example of hardheadedness and an inability to adapt and improvise. I know you do not know any one else like this, but I do. In fact, the majority of noise makers in the community refuse to use tools that execute in a web services environment.

You can characterize the modern business climate by friction, uncertainty, disorder, and rapid change. Each situation is a unique combination of shifting factors that cannot be controlled with precision or certainty. System administrators need to learn to adapt. Adaptability means shortening the time it takes to adjust to each new situation.

Two basic ways to adapt exist. First, if we have enough awareness to understand a situation in advance we can take preparatory action. Call this anticipation.

At other times we have to adapt to the situation on the spur of the moment without time for preparation. This is improvisation. To be fully adaptable, we must be able to do both.

We co-hosted our server at an ISP 250 miles from our base of operations. When the server went down, we had to call the ISP and have one of his service personnel run down to the server rack and power it back to a working situation.

From that point, our brilliant, know-it-all sys admin had to login into sshd. By the time the server powered back and started all the processes including sshd, the server locked up again. So what was next?

Insisting on using the command line, the sys admin had to walk a support technician through a series of steps to renice and shutdown PIDs without allowing the system to connect to the Internet.

Once he got the system dedicated to one purpose, he reconnected it to the Internet and though slow, the OS and web server stayed up.

Next, our sys admin had to stop several of the Content Management System processes to keep the server up and allow visitors to view a single article. Eventually, the server responded in a reasonable manner. But, consider all the trouble.

Sacreligion

I prefer using the command line myself. But some situations warrant something in addition. For example, when you manage dozens or even hundreds of servers, you need to poll and drill down into each server remotely and efficently.

One of the tools I like allows me to do the drilling. The developers call it Monit. Here’s the description from the web site:

Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses to much resources. You can use monit to monitor files, directories and devices for changes, such as timestamp changes, checksum changes or size changes. You can also monitor remote hosts; monit can ping a remote host and can check TCP/IP port connections and server protocols. Monit is controlled via an easy to use control file based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file and notifies you about error conditions and recovery status via customizable alert.

Monit allows you to view a server’s status through a web browser among other features. You can find the project at http://www.tildeslash.com/monit/. Additionally, Falko Timme has written a howto for installing and setting up Monit at http://howtoforge.com/server_monitoring_monit_munin.

Command line? Essential for any system administrator to know. But, consider adding adaptability to your skill set. It will pay off.

I discovered a stupid bug in Apple Mail today.

I have a SSH tunnel set up on my PowerBook back to a Linux machine at work so I can securely check my IMAP mail from open wireless networks. Today in class I found that the wireless network I’m on blocks outbound port 25, so I set up the SSH tunnel to take port 25 traffic back to my SMTP servers. This is a really cool trick, and you can do this all on a single SSH line:

ssh user@some.server.org -L2143:mail.server.org:143 -L2525:mail.server.org:25

This opens an SSH session to some.server.org, and then forwards your local port 2143 to mail.server.org:143, and likewise for port 25. You can chain as many of these ports as you need into a single SSH session.

So I set up the outbound server in Mail to “localhost:25″. No luck, Mail was having none of it. I could telnet to localhost 25 from commandline just fine. My IMAP connection is localhost:143 and that works in Mail. Puzzled, I tried 127:0.0.1:25 and THAT works.

How on earth did that happen? These preferences are in the same pane in the Mail Preferences. Someone had to specifically code Mail to allow “localhost” for IMAP, but not for SMTP.

Dumb.

The poll on “What do you do with system logs?” aims at learning “best practices” for system, network and security log storage and analysis.

Again, I will comment when the results trickle in.