Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #13: Into the Darkness … or The Ominous World of Unix Binary Audit Logs

In this tip, we will take a peek at one of the most esoteric areas of logging: Unix binary audit logs. Solaris BSM and Trusted Solaris auditing is the least unknown :-) example of it, even though other Unix vendors have similar auditing capabilities - see this for HP-UX Audit and this for IBM AIX audit. Linux kernel audit is also pretty much the same thing. If you look for information on ‘Solaris BSM audit logs’ , you’d find plenty of tips on how to enable such logging, a little on how to manage/rotate the log files, a bit on how to survive the resulting data deluge and ALMOST NOTHING on what to do with the log data, which is kinda sad :-) After looking at BSM logs for a while, I developed an opinion that nobody has ever looked at them on a regular basis :-)

So, let’s assume you enabled Solaris BSM kernel audit for user “root” and few other “interesting” users (there is no per-object logging in Solaris; other Unix’es do have it) via the following commonly recommended per-user configuration in /etc/security/audit_user:

root:lo,ad,fw:no

anton:all,-all:no

jsmith:all,-all:no

This config pretty much records all the actions by the users listed. Now, you have audit files growing like shrooms in you /var/audit. What good does it give us? First, we need to convert the binary audit files into text - something along the lines of

# auditreduce -A /var/audit/20071127193515.not_terminated.SunUltra10 | praudit -l > /tmp/sol_box_11272007

will do. Now what? In this tip we will learn how use the audit logs to see who is trying to copy sensitive files off the system.

First, who is connecting out - lets’s search the logs for ‘connect’ calls (if you are using LogLogic for it, use Index Search for this task; if not, grep will have to do, but be prepared to wait). A few recommended searches:

• “connect AND 172.16.10.*” or “connect AND NOT 172.16.10.*” (to look for connection to specific IPs or to the outside networks) or simply ‘connect AND username’

Here is an example found (with connect, IP and user in bold):

header,103,2,connect(2),,Tue Nov 27 11:36:46 PST 2007, + 193 msec,argument,1,0x4,so,socket,0x0002,0x0002,0x80d6,SunUltra10,0x0016,10.1.1.41,subject,root,anton,other,anton,other,29902,29720,0 1611 172.16.0.173,return,success,0

At this point we already know the user name of the user who run that connecting process since it will be in the results (you can also the user to search as I showed above).

Next, what are those connections - let’s try to uncover which programs actually connected (BSM logs don’t make that easy). Let’s search for process starts in the same time frame:

• “execve AND NOT ls AND NOT <whatever other commands you don’t care to see>” will give you a list of started programs.

Example:

header,124,2,execve(2),,Tue Nov 27 11:36:46 PST 2007, + 115 msec,path,/usr/bin/scp,attribute,100555,root,bin,136,1573,0,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,0

Notice that both records have the same timestamps. Sadly, time and parent process ID ( which is in our case 29720) is all that correlates them together.

Finally, what file was affected (i.e. copied off the system via scp in this case) - more digging is in order; we again use the process ID and time. The easiest is to search for a file name or browse all records around the same time frame (might be A LOT!):

• “*secret.zip* AND anton” will work; we can add the above process ID and look for “anton AND 29720″ (but expect a lot of data since this is a shell process ID)

For example:

header,135,2,open(2) - read,,Tue Nov 27 11:36:47 PST 2007, + 900 msec,path,/tmp/not-so-secret.zip.gz,attribute,100600,anton,other,0,32743959,18446744073709551615,subject,root,anton,other,anton,other,29901,29720,0 1611 172.16.0.173,return,success,4

What do we know now? This user connected to this system and MAYBE copied this file via, MAYBE via scp. How cool is that? (A: not cool at all, since we are not sure!)

To conclude, if you can avoid dealing with Solaris BSM logs, please do so :-) On a more serious note, you now know why these logs were called “the ugliest logs ever.”

Even more seriously (but still pretty humorously), these logs are a classic example of trees that make every effort to obscure the forest, because they record syscalls and not processes or user actions (and connect, execve and read are all logged separately). There are also many, many more idiosyncrasies (and, in fact, idiocies) where these come from :-)

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Time for another fun logging poll: What Do You Do With Collected Logs?

Vote here!

This is my Logging Poll #3, links to past polls:

• Poll#2 “Why Collect Logs?” (vote here, results so far, my analysis)
• Poll #1 “Which Logs Do You Collect?” (vote here, results so far here, analysis here)

UPDATE: analysis and results posted at here

While it could vaguely be considered system/application news, Nintendo announced a new version of “Photo Channel”, will be released in December. For those that own a Wii, Photo Channel is that icon that is mostly ignored, right next to “Wii Shop Channel”.

The update includes the comment that they’ll be dropping MP3 support for AAC support, and it has a few Nintendo users a little upset. The rumor was that the change was brought about due to licensing costs for the MP3 codec, but nobody knows for sure.

I’m one of the few people I know that’s made the switch from MP3s into AAC files, and burn CDs directly into AAC. I didn’t gain anything in quality from the MP3 to AAC conversion, but there was a slight savings in hard drive space. I still have my music, and it can still be played on the iPod, the Zune, or other players put out by Creative, SanDisk, Sony, etc, etc.

So my question is, is the MP3 file format really more popular than AAC; or does it only appear to be more popular because the term MP3 is slowly becoming a lost trademark in the same vein of Xerox and Coke?

AAC is superior to MP3, since it is the successor format; but do people actually use the extended features? Or is this a case where Betamax is technically superior, but for some reasons the markets prefer VHS?

The idea came from Jeremiah Grossman (here) when he described “The Best Web Application Vulnerability Scanner in the World” thus: “Within a few moments of pressing the scan button it’ll find every vulnerability, with zero false positives, generate a pretty looking report, and voila you’re compliant with GLBA, HIPAA, and PCI-DSS. Of course, we all know such a web application scanner is simply not possible to create for a variety of reasons.”

So, let’s imagine the idea log management application.

1. Logging configuration: the ideal log app will go and find all possible log sources (systems, devices, applications, etc) and then enable the right kind of logging on them according to a high level policy given to it (required: God-like powers)
2. Log collection: it will collect all the above logs securely (and without using any risky super-user access ) and with little to no impact to networks and systems (required: God-like powers)
3. Log storage: it can security store the above logs in the original format for as long as needed and in a manner allowing quick access to them - in both raw and summarized/enriched form (required: plenty of hardware)
4. Log analysis: this ideal application will be able to look at all kinds of logs, known to it and previously unseen, from standard and custom log sources, and tell the user what they need to know about their environment and based on their needs: what is broken? what is hacked? where? what is in violation of regulations/policies? what will break soon? who is doing this stuff? The analysis will power all of the following: automated actions, real-time notifications, long-term historical analysis as well as compliance relevance analysis (required: AI)
5. Information presentation: this tool will distill the above data, information and conclusions generated by the analytic components and present then in a manner consistent with the user’s role: from operator to analyst to engineer to executive. Interactive visual and drillable text-based data presentation across all log sources. The users can also customize the data presentation based on their wishes and job needs, as well as information perception styles (required: nothing more than a bunch of daring UI designers)
6. Automation: the ideal log management tool will be able to take limited automated actions to resolve discovered and confirmed issues as well as generate guidance to users so that they know what actions to take, when full-auto mode is not appropriate. The responses will range from full-auto actions to assisted actions (’click here to fix it’) to issuing detailed remediation guidance. The output will include a TODO-list of discovered items complete with actions suggested, ordered by priority (required: AI + some luck + some user stupidity :-))
7. Compliance: this tool can also be used directly by auditors to validate or prove compliance with relevant regulations by using regulation-specific content and all the collected data. The tool will also point at gaps in data collection as it applies to specific regulations that the user is interested in complying (required: God-like powers)

In other words, this magic black box will have crap shoveled from one side and will have answers to questions about the meaning of Life :-) coming out the other side…

What? :-) Am I nuts? Well, can I dream for a second? :-)

Following my now-famous Top 11 Reasons to Collect and Preserve Computer Logs and Top 11 Reasons to Look at Your Logs, here is the promised “Top 11 Reasons to Secure and Protect Your Logs”

1. Let’s review why you are reviewing logs. Will logs that might have been changed by somebody, somewhere, somehow still be useful for items 1-11 from here? No? Secure them!
2. Oooh, logs in court? Challenges abound! To respond to them, one needs to protect the logs so you can claim that they are both authentic and reliable.
3. A human error still beats an evil hacker as the main cause of IT problems. Are your logs safe from it? Available when needed? Protect them from crashes and other faults!
4. PCI DSS just says so: “Secure audit trails so they cannot be altered.” Wonna do it- or pay the fines?
5. Do you protect financial records? Identity info? Passwords? Some of it ends up in logs - thus making them more sensitive. Secure the C-I-A of logs!
6. Do you look at logs during incident investigation? Do you want them to be “true” or full of random (if creative…) cr*p, inserted by the guilty party? Secure the logs!
7. Think that “attacks vs logging” are theoretical? Think again. Are your logs safe or vulnerable? Is your logging tool 0wned?
8. Syslog + UDP = log injection. Are you protected (reliable TCP, confirmed delivery, encryption - SSH, SSL, VPN)?
9. Why change logs? No, really, why change logs? If you never change logs - and you never should - hash them right away after collection to make them immutable.
10. Logs are backed up on tape - who will see them? Well, whoever restores the tape, that’s who! Encrypt them to protect them from accidental and malicious disclosure if tape is lost.
11. Why log access to logs? Same reason why you had the logs in the first place - to review who did what. Who broke through and stole the logs? Who browsed them without permission? Only logs will tell - if you have them!

Overall, one need to strive for having no holes in log safeguards from log birth to analyst conclusion based on log information…

Possibly related posts:

• Top 11 Reasons to Look at Your Logs

• Top 11 Reasons to Collect and Preserve Computer Logs