O'Reilly Sysadmin

Anton Logging Tip of the Week #15: Fear and Loathing in Event 560 (and 562 and 567)

2008-05-08T22:29:03Z

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Logging Tip of the Day #15: Fear and Loathing in Event 567

This tip digs into a seemingly simple, but really VERY esoteric subject: monitoring file access and modification via a Windows event log. Now, some people - who never studied this subject - tend to have a very simplistic view of this: just enable Object Access auditing, then right-click on a file or directory, click Security->Advanced->Auditing and then pick what types of events will be logged and by what accessing entities (i.e. users or computers). OK, so this will produce some logs, that is for sure. But are they useful?

First, why are we doing this? We typically need to know the following when we audit file access in Windows (or any other OS for that matter) for security (monitoring and investigation) or compliance:

Time/date
Computer where it happened
User who touched the file
Application he used to access the file
File name + location (directory, share, etc)
Type of access (read, write, create, delete, etc)
Status (i.e. success or failure)

Can we get this from the above logs? No.

What? No!?! Really?

Yes, really. We can get some of the above, some of the time, not all of the above, all of the time. Here is an example, we are looking at event ID 560 (picture) and then at an extract from its description field.

Event:

Description (selected field):

Object Server: Security

Object Type: File

Object Name: C:\0\TestBed\simple_text_file.txt

Image File Name: C:\WINDOWS\system32\notepad.exe

Primary User Name: Anton

Primary Domain: XXXXXX

Accesses: READ_CONTROL

SYNCHRONIZE

ReadData (or ListDirectory)

WriteData (or AddFile)

AppendData (or AddSubdirectory or CreatePipeInstance)

ReadEA

WriteEA

ReadAttributes

WriteAttributes

WTH is that? Well, we know that the user ‘Anton’ has successfully read? wrote? changed attributes? did something? with a file named “C:\0\TestBed\simple_text_file.txt” using a program named “C:\WINDOWS\system32\notepad.exe.” That’s the best we can get, in this case! We may try to look at event IDs 562 and 567, but this missing information (i.e. the exact action performed) will not be added.

BTW, there will be a few more dozen (sometime hundreds!) of the 560s, 562s and 567s produced - all from just opening the text file in a notepad. The above event is notable for having BOTH “notepad” and “simple_text_file.txt” in the same event; others will have either of the two.

Anything else gets in the way? Yes, lots! MS Office will write to all files, even just opened for reading (with no user modifications to the content whatsoever), which will screw up your log monitoring efforts. If the file is on a share, more information will be missing (e.g. username might be).

So, how to use Windows event logs for file access tracking?

Enable logging (as described above)
Pick events 560 (most useful) and 562, 567 (useful too)
Look for fun filenames that might be touched by the users (have a list of files and users handy)
Figure out what programs were used to access them (this is called “Image File Name” in “WinLogSpeak”)
Ponder the ‘Accesses’ section of each event until your brain turns blue :-) or until you decide whether such access is authorized or not…

Overall, this is still very useful for file access monitoring, but the process is paaaaaainful.

BTW, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Technorati tags: security, tips, logging, log management

Poll #8: What additional information is most useful for log analysis?

2008-05-05T22:02:39Z

So, my next poll is up - and it is fun: Which of the types of information are most useful when trying to make sense of a log entry?

Vote here!

Past polls:

Poll #7 “What tools do you use for Windows Event Log collection?” (analysis)

Poll #6 “Which logs do you LOOK at?” (analysis)

Poll #5 “What are your top challenges with logs?” (analysis)

Poll #4 “Who looks at logs in your organization?” (analysis)

Poll #3 “What do you do with logs?” (analysis)

Poll #2 “Why collect logs?” (analysis)

Poll #1 “Which logs do you collect?” (analysis )

Filesystem Monitoring: You're Doing It Wrong

2008-05-02T15:05:46Z

This doesn’t look good, right?

Most open source monitoring tools do filesystem health checking by comparing the current percentage of used space against a set value. If it’s is 90% full, send out a warning page; if it’s 89%, send the all clear.

Notice that I said filesystem, and not actual disk. A single disk that’s 90% full can be a bad thing, because there are fewer free blocks available for writing, which leads to longer write times and file fragmentation. Not all filesystems are restricted to a single disk: there may be a back-end RAID solution, or the filesystem may be a shared filesystem served over NFS.

Unfortunately, you could be the receiver of flapping alert pages where a filesystem sits between 90% and 89%, but it still performs fine. Unlike a broken Ethernet cable, the resolution for a filesystem threshold may not be so easy. Sometimes there are files that can’t be deleted, or there may not be any additional storage to allocate. You may have a filesystem that sits at 91% full for months simply because a new disk shelf won’t arrive until the next budget cycle.

Everything comes down to disk blocks, even SAN and NAS solutions. That brings back the concern regarding fragmentation and performance. But what if your filesystem is a read-only OS image? Or what if it turns out 10% equates to 500 gigabytes on a huge disk appliance? If the filesystem is never being written to, or if the amount of writes equates to 0.001% of the entire filesystem, then where’s the fire?

What about the inverse? What if your filesystem never reaches 90% full? Can there still be problems?

In the above graph, nobody would have been paged by Nagios or other tools, because the filesystem never reached 90%. For the past few months it averaged 40% full, shot up to 75%, and then went back down. A newly released application was behaving incorrectly, and the issue was caught by the programmer. The next morning he stealthily re-released the application and corrected the issue. Nobody in systems administration noticed until the graph was checked in relation to another issue. If the programming error was never discovered, the filesystem would have filled up, probably at the most inconvenient time possible for a systems administrator.

I would like to recommend to people developing filesystem or disk monitoring solutions change their way of thinking about filesystem health. Hard limits on allocated space may still be required, but those warnings should be optional. Measuring fullness makes assumptions about block structure that may not be correct.

At the same time, the monitoring system should compare the standard deviation for the filesystem percentage over the past 24 hours, and compare it to the standard deviation for the past hour. Actually, you’d probably want to compare the first 23 hours out of 24, grab that standard deviation, and compare it to the deviation of the last hour.

If those two deviations aren’t close, then there could be radical changes made to your filesystem that need to be addressed. Maybe files are being added or deleted, either way, it may warrant an investigation. For large filesystems in the terrabyte/petabyte range, using the percentage value may not be granular enough, so you will need to work with the actual value of free kilobytes or blocks.

I take it back. This isn’t a recommendation to monitoring developers, this is a challenge. The first major open source monitoring guy that puts this solution together will have my undivided attention.

LVM and software raid, a powerful combination

2008-04-21T23:13:33Z

When I first started using LVM I got bit by a few bugs. It’s all part of being an early adopter. As a result I never really used it on production hardware. It wasn’t until about 2 years ago that I gave it another look. In a similar manner I never really thought much of software raid beyond a novelty. Much of that has changed now and I use them both on a regular basis for a number of reasons.

I’m in one situation right now where I have a filesystem that I did stupid things to and fsck.ext3 segfaults on it. Normally this would be a really bad thing (and it is) but because I’m using LVM, I can run many tests on the filesystem, in a snapshot, without harming the master filesystem (which has around 2.5T of data on it). In this case the bad blocks have already been written to and likely won’t be changing. So I can continue to use the share in production, while working with the fsck devs to figure out what is segfaulting. I can also take two snapshots at the same time. Run fsck.ext3 on one of them, and then compare the actual files to see which (if any) of them have changed or gotten corrupt.

This is just one case in a million where knowledge and use of lvm has proved immensely powerful. Another case (and the topic of this article) is a situation where the combination of lvm and software raid will allow me to convert a raid1 array into a raid5 array without any downtime on my critical apps. And for those of you that think you’ll take a performance hit when using software raid over hardware raid, I ask “which can calculate parody faster. a 486 processor in the card or one of the xeon processors in your box that isn’t on the card”. Go ahead, run the test yourself.

Anyway, to get started: in our environment we use virtualization heavily. One of our xen dom0’s has two raid1 arrays on it, it was a while ago when this machine was built and we can only assume that those two arrays were for two dedicated machines. But now we want to merge the arrays and just have one raid5. So the basic layout of our setup:

2 raid 0 arrays.
2 physical volumes.
2 volume groups.

At the end we want:

1 raid5 array
1 physical volume
1 volume group.

I should warn you that in prep for this we moved all critical apps to one of the volumegroups so essentially the other volume group is now empty so I’m treating it that way during my test, I’ll only be creating one volumegroup and array. To test this at home (as I’ll be doing in the examples) I assume you have a volume group with 400M free. I’ll be creating a logical volume as an analog for the “disks” in our production server.

First: create the “disks” (100M each)

# lvcreate -L 100M -n 1 VolGroup00
Logical volume “1″ created

# lvcreate -L 100M -n 2 VolGroup00
Logical volume “2″ create

# lvcreate -L 100M -n 3 VolGroup00
Logical volume “3″ created

# lvcreate -L 100M -n 4 VolGroup00
Logical volume “4″ created

Then create your raid array:
# mdadm -C /dev/md0 -a yes –level=1 –raid-devices=2 /dev/VolGroup00/1 /dev/VolGroup00/2
mdadm: array /dev/md0 started.

Next convert the array to a physical volume
# pvcreate /dev/md0
Physical volume “/dev/md0″ successfully created

Then create your group
# vgcreate testVG /dev/md0
Volume group “testVG” successfully created

# vgs
LV VG Attr LSize Origin Snap% Move Log Copy%
testFS testVG -wi-a- 96.00M

We’ve got a volume group with 96M free.

Then create your logical volume

# lvcreate -n testFS -L 96M testVG

# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
testFS testVG -wi-a- 96.00M

Next create a filesystem and stick some information on it:

# mkfs.ext3 /dev/testVG/testFS
mke2fs 1.40.4 (31-Dec-2007)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
24576 inodes, 98304 blocks
4915 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
12 block groups
8192 blocks per group, 8192 fragments per group
2048 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 24 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.

# mount /dev/testVG/testFS /mnt
# echo “Critical Data” > /mnt/ThisIsVaulable

At this point we’re in an analog to where I am in production. As I mentioned I moved everything critical off of one volumegroup already so I’ve essentially got a logical volume on top of a raid1 array with 2 free disks. At this point we can start moving stuff around:

# mdadm -C /dev/md1 –level=5 –raid-devices=2 /dev/VolGroup00/3 /dev/VolGroup00/4
# mdadm: array /dev/md1 started.

You can always check the status of your array in /proc/mdstat:

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid5 dm-11[2] dm-10[0]
102336 blocks level 5, 64k chunk, algorithm 2 [2/1] [U_]
[====>…………….] recovery = 21.0% (22408/102336) finish=0.0min speed=22408K/sec

md0 : active raid1 dm-9[1] dm-8[0]
102336 blocks [2/2] [UU]

Smarties will note that you have to have 3 disks to create a raid5 array. Technically you only need 2, but you don’t have redundancy. I’m basically running this array in a degraded mode.

After that finishes I’m going to steal one disk from the raid1 array. At this point I feel the need to mention that I have a solid backup of all of the data left on our soon to be gone raid1 array. If the disk it is on fails during the migration. The data is likely toast. The only way to protect against this is to add more disks to the machine which just isn’t feasible in my use case.

So lets fail a disk in the raid1 array and add it to the raid5 array:

# mdadm /dev/md0 –fail /dev/VolGroup00/2
mdadm: set /dev/VolGroup00/2 faulty in /dev/md0
# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid5 dm-11[1] dm-10[0]
102336 blocks level 5, 64k chunk, algorithm 2 [2/2] [UU]

md0 : active raid1 dm-9[2](F) dm-8[0]
102336 blocks [2/1] [U_]

Note the (F), its failed.

# mdadm /dev/md0 –remove /dev/VolGroup00/2
mdadm: hot removed /dev/VolGroup00/2

# cat /mnt/ThisIsVaulable
Critical Data

note we have not unmounted /mnt and will not do so.

# mdadm -G /dev/md1 –raid-devices=3 –backup-file=/tmp/backup
mdadm: Need to backup 128K of critical section..
mdadm: … critical section passed.
# mdadm /dev/md1 –add /dev/VolGroup00/2
mdadm: added /dev/VolGroup00/2

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid5 dm-9[3](S) dm-11[1] dm-10[0]
102336 blocks super 0.91 level 5, 64k chunk, algorithm 2 [3/2] [UU_]
[====>…………….] reshape = 24.0% (25088/102336) finish=0.4min speed=2787K/sec

md0 : active raid1 dm-8[0]
102336 blocks [2/1] [U_]

At this time it is reshaping the raid5 array and ultimately will add my new disk and rebuild the array. The array is online at that time and could be used.

So now we have two arrays, its time to add our raid5 array to the testVG volume group:

# pvcreate /dev/md1
Physical volume “/dev/md1″ successfully created

# vgs
VG #PV #LV #SN Attr VSize VFree
testVG 1 1 0 wz–n- 96.00M 0

# vgextend testVG /dev/md1
Volume group “testVG” successfully extended

# vgs
VG #PV #LV #SN Attr VSize VFree
testVG 2 1 0 wz–n- 292.00M 196.00M

notice the size of our volumegroup has changed. Now comes the magic. All of our valuable data is still on the raid1 (/dev/md0) raid array. To move it off of that disk, onto the new disk we use the pvmove command:

# pvmove -i 1 /dev/md0
/dev/md0: Moved: 4.2%
/dev/md0: Moved: 37.5%
/dev/md0: Moved: 70.8%
/dev/md0: Moved: 100.0%

Then to remove the disks from the testVG, use vgreduce:

# vgreduce testVG /dev/md0
Removed “/dev/md0″ from volume group “testVG”

At that point, the hard part is over, but we’re not done. Now we’re going to disable the /dev/md0 array and add the only remaining disk to /dev/md1.

# mdadm -S /dev/md0
mdadm: stopped /dev/md0

# mdadm -G /dev/md1 –raid-devices=4 –backup-file=/tmp/backup
mdadm: Need to backup 384K of critical section..
mdadm: … critical section passed.

# mdadm /dev/md1 –add /dev/VolGroup00/1
mdadm: added /dev/VolGroup00/1

# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4]
md1 : active raid5 dm-9[2] dm-11[1] dm-10[0]
204672 blocks super 0.91 level 5, 64k chunk, algorithm 2 [4/3] [UUU_]
[==>………………] reshape = 14.0% (15172/102336) finish=0.4min speed=3034K/sec

Again we’re reshaping then rebuilding the array, notice md0 is gone.

# pvs
PV VG Fmt Attr PSize PFree
/dev/md1 testVG lvm2 a- 196.00M 100.00M

# pvresize /dev/md1
Physical volume “/dev/md1″ changed

# pvs
PV VG Fmt Attr PSize PFree
/dev/md1 testVG lvm2 a- 296.00M 200.00M

# vgs
VG #PV #LV #SN Attr VSize VFree
testVG 1 1 0 wz–n- 296.00M 200.00M

Now we’ve got that additional 100M to use. remember, you lose a drive of usage when using raid0, so our total storage is only 3 of our 100M disks (for 300M of usable space) The 96M is already taken by the filesystem.

To use some of this space, first we need to extend the logical volume we created earlier (testFS). Note size before and after:
# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
testFS testVG -wi-a- 96.00M

# lvresize /dev/testVG/testFS -L +100M
Extending logical volume testFS to 196.00 MB
Logical volume testFS successfully resized

# lvs
LV VG Attr LSize Origin Snap% Move Log Copy%
testFS testVG -wi-ao 196.00M

next, resize the filesystem:

# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/testVG-testFS
93M 5.6M 83M 7% /mnt

# resize2fs /dev/testVG/testFS
resize2fs 1.40.4 (31-Dec-2007)
Filesystem at /dev/testVG/testFS is mounted on /mnt; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/testVG/testFS to 200704 (1k) blocks.
The filesystem on /dev/testVG/testFS is now 200704 blocks long.

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/testVG-testFS
190M 5.6M 175M 4% /mnt

# cat /mnt/ThisIsValuable
Critical Data

Done, no outage. It may be a few steps to do, but in my case its worth it to avoid having the downtime. This is a pretty specific use-case but there are plenty of other cases where this example will allow users to use these tools in a way that has little impact on the end users.

Virtualization: Kicking and screaming....

2008-04-03T13:30:48Z

I’m working with a product that includes this disclaimer in their support documentation:

“Virtual environments, such as VMWare (and others) are not recommended, and thus not supported.”

I can almost see their point. It’d be pretty daunting to gauge a benchmark if a customer described the running host as “1/13th of two dual core processors, 3.1 gigs of memory, and a 27 gigabyte filesystem disk”. True, that’s a pretty extreme situation, but I wouldn’t doubt it if there was the occasional bad provisioning by virtual system installers.

Anyone who implements virtualization is implicitly trusting the VM solution to do the right thing, and when we see the operating system up and running, we just assume everything works perfectly. But let’s be honest: almost every VM solution creates some overhead, so you’re missing out on a few resources. That loss shouldn’t amount to much, but it could mean a lot to an application. And while CPU and memory can be partitioned, device IO such as hard disks are a little sketchy.

To the developers of the above unnamed application, I know it’s going to be a big hassle, but five years from now, you’re not going to be able to avoid virtualization. Instead of the blanket disclaimers, increase your virtualization knowledge base, and create more test suites. Find out what works, what doesn’t, and why. It’s still okay to set guidelines on usage, but a wholesale avoidance of virtualization will hurt in the long run.

Is Hardware Still Commoditized?

2008-03-21T13:58:22Z

Last week I attended a virtualization seminar. I did not expect a lot from the event at first, but I was surprised by the qualities of the guest speakers. Both had strong backgrounds with VM environments, and they did a good job of explaining what it takes to migrate to VM.

One of the speakers made an interesting statement, saying that the hypervisor is now commoditized. The market for virtual solutions has gotten so big, it’s unavoidable. VMWare has ESX, Xen has their system, and Microsoft is coming out with Hyper-V. If everybody offers what is essentially the same thing, then how do these products stand out from one another?

Now your incentives for buying virtualization have changed. You don’t buy VMWare just because it offers virtualization; you buy VMWare because it has the best service, and the best hot migration features. You might buy Hyper-V because your familiar with Microsoft internal APIs and management tools. On top of virtualization, I’m not sure what else Xen has to offer, but there could be new features coming out from Citrix.

When I left the seminar, I started to re-evaluate hardware decisions that were made in the past. The nature of the beast has changed. Eight years ago, hardware decisions were taken for granted, because it too, was commoditized.

Everything runs on an x86, and everyone makes an x86, so the low price usually won out. Anything that the vendor offers on top of the low price might have clinched the deal. Better support, better service, free shipping? Whatever it took to sell a server and get it out the door. Hundreds of IT departments packed data centers full of tight 1ru servers. Virtualization has now made those servers worthless.

When a single server failed, it was no big deal. You probably had another one just like it running the same application. If that same server is now running multiple virtual hosts, then the service impact is higher. Two machines may now be fighting for access to the same mirrored local disks. What are the chances that they’re impacting each other?

If your server can only handle 2 running virtual hosts, then you cut hardware costs by 50%; but in order to win, your hardware savings still need to be higher than the support and licensing costs of your enterprise VM solution. A 2 to 1 hardware savings ratio isn’t good. it’s expected. In order to maximize your investment, you should aim for a 4 to 1 hardware savings ratio, maybe higher

Migrating to a VM environment does not mean building a VM solution into your servers; it means building your servers around a VM solution. If the hypervisor really is treated like a commodity, then the same can no longer be said about the hardware.

Anton Security Tip of the Week #14: More access_log Fun: What Are You Not GETting?

2008-03-12T21:45:29Z

Following the tradition of posting a tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #14: More access_log Fun: What Are You Not GETting?

In this tip, we will look at some bizarre artifacts that show up in web server access logs today. Here we have a production log from an Apache web server that is full of interesting (and sometimes ominous!) little mysteries that we will investigate in order to determine their impact on security and operational health of the site.

Logs do contain more mysteries than we have time, so we will focus on a few of them: specifically, unusual web request methods. Let’s see who is trying to POST or use some other method (OPTIONS, HEAD, PUT or something - see a list here) on our site, instead of just GET’ting the content (GET command is used by web browsers to retrieve the pages, while POST is used to upload content, press buttons, etc - at least in “web 1.0″ land - see earlier tip #12 where POST request was found in proxy logs)

Here is one little artifact that attracted my attention due to a POST request vs a web forum as well as a battery of slashes (which actually increases in subsequent request - of which there were many)

10.10.102.250 - - [12/Feb/2008:16:10:50 -0500] “POST /phpBB3////ucp.php?mode=register&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0″ 200 14397 http://www.example.com/phpBB3///ucp.php?mode=confirm&id=7640df05c7e24b7acf7a68800fe6dc59&type=1&sid=e5efaa77a777066c61f71808e9e57b19 “Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126″

… more…

10.10.102.250 - - [12/Feb/2008:16:12:29 -0500] “POST /phpBB3///////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19 HTTP/1.0″ 200 9355 “http://www.example.com/phpBB3//////////////ucp.php?mode=login&sid=e5efaa77a777066c61f71808e9e57b19″ “Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:1.2) Gecko/20021126″

This one really is a mystery; what do we know about it? The server responded to the request OK (code 200), so the POST actually happened. The first request was a request to register with a web discussion board and the second was a request to login. Multiple slashes are actually ignored by the web server, so why put them in the request (no answer)? Also, I think that the User-Agent is spoofed … do you know why? Finally, if I see something like that in my logs, I will definitely investigate it, primarily due to the fact that Apache responded with 200 OK code.

The next one is so classic it it dumb (and so dumb, it’s a classic :-))

10.10.123.226 - - [12/Feb/2008:03:46:54 -0800] “POST /_vti_bin/shtml.exe/_vti_rpc HTTP/1.1″ 404 - “-” “MSFrontPage/6.0″

10.10.123.226 - - [12/Feb/2008:03:46:55 -0800] “OPTIONS / HTTP/1.1″ 200 20210 “-” “Microsoft Data Access Internet Publishing Provider Protocol Discovery”

It is probably one of the ancient IIS attacks (check out this fun BlackHat preso on that, circa 2003) - why would someone probe for it now is beyond me. In any case, Apache on Linux and “*.exe” don’t mix :-)

The final log record is also fun:

10.10.101.222 - - [12/Feb/2008:15:33:22 -0800] “PUT /zk.txt HTTP/1.0″ 405 223 “-” “Microsoft Data Access Internet Publishing Provider DAV 1.1″

The above uses a PUT request which is pretty much deprecated now; the purpose of the above is clearly malicious. In fact, modern Apache shouldn’t even allow it, thus it responds with code 405 “Method Not Allowed.” Nothing to worry about (even though some poor critter got owned with that! BTW, if you follow that link, check out HTTP response code 201 - if you see it in your logs, run! :-))

Overall, if you see too many POSTs or too many “GET then POST” sequences from the same IP in rapid succession, investigate it since no legitimate access should produce such a pattern…

As further reading, I heartily recommend this paper: “Detecting Attacks on Web Applications from Log Files“

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

Technorati tags: logging, log management, tips, security

Poll #7: What tools do you use for Windows Event Log collection?

2008-03-07T20:54:17Z

My next fun logging poll is here - please vote! It is about tools for centralized collection of Windows Event Log from servers and other systems. One of the somewhat surprising discoveries from my previous poll was that few people look at Windows logs; this poll drills down into it.

UPDATE: the results and analysis posted here. Enjoy!

Past logging polls and their analysis:

Poll #6 “Which Logs Do You LOOK At?” (analysis)

Poll #5 “What are your top challenges with logs?” (analysis)

Poll #4 “Who looks at logs in your organization?” (analysis)

Poll #3 “What do you do with Logs?” (analysis)

Poll #2 “Why collect logs?” (analysis)

Poll #1 “Which logs do you collect?” (analysis )

Technorati tags: logs, logging, polls, log management

Bitten By The New Mac Bug

2008-03-03T14:23:22Z

I just bought a new MacBook. A co-worker of mine just bought a new MacBook Pro. To put it another way, we both bought new Apple Laptops….one week before the new models came out.

After Tuesday’s announcement I looked at the Apple website with a hint of despair. I missed out on a larger hard drive, a faster CPU, and more graphics memory. Catherine missed out on a faster CPU, more memory and storage, plus the holy grail of Multi-Touch. At the time of Apple’s announcement, my laptop was only five days old; and Catherine’s was six.

Both of our systems were bought in person at the same Apple store. We both knew what we wanted when we walked in so our sales encounters were short. Catherine’s sales rep was a short, clean cut guy, that looks like every Mac user walking down the street. Mine was a big, serious guy, who kind of looked like Dr. Artz. Both sales people were professional, courteous, and well informed on the products we asked about.

When the news of the new models hit Engadget, co-workers convinced us it wouldn’t hurt to go to the store and see what they would do for us. I didn’t have the laptop or receipt with me, but I figured I could just ask. During my lunch break I went out to that physical behemoth representing all that is good and capitalistic in this country; the Mall of America.

I found a manager and I explained my plight. To win over his sympathy, I made up a sob story about kids in an orphanage who only wanted the laptop so they could play a scratched up copy of Myst that somebody found in a dirty alley.

The manager explained my options: pay a restocking charge of 10% ($149), or get a refund of $200. The restocking charge was because I’d be returning an open/used laptop to Apple that they wouldn’t be able to sell. I told them I’d think about it and return the next day. Deep inside. I was kicking myself for selfishly taking my laptop out of the box and using it the very day I purchased it. I should have known better!

Since Catherine had the MacBook Pro her restocking charge would have been $250, but her refund would be $600. That’s a cost swing of $250 for me, and $850 for her. We both know we’re never going to be 100% ahead of the technology curve, but we didn’t expect to be hit with new models this quickly. To be fair, we’ll also admit that our gripes sound like a perfect candidate for White Whine.

Maybe the complaint is due to Apple’s notoriety in keeping new products secret. I can understand keeping the MacBook Air secret, because it’s a new product line. But why keep hardware upgrades secret? The reviews of the new models report that the units are faster, but there’s nothing really new or innovative. If the sales person had told us that new models was coming out, we both would have waited, and we would have been happier customers. Unfortunately, sales people at the Apple store have no idea when new models are coming, so they’re just as powerless as we are.

Both of us took the refund money and ran. We’re happy with what we have, but a little disappointed in how the product release cycles can make someone regret their purchase. The funny thing is, they’re both still good notebook computers. But the bleeding edge Mac culture (and to the PC culture as well) embraces newness, and shuns obsolescence.

Top 11 Reasons to Analyze Your Logs

2008-02-22T17:55:47Z

As promised, here is another “Top 11 Reasons” which is about log analysis. Don’t just read your logs (definitely don’t just collect them); analyze them. Why? Here are the reasons:

Seen an obscure log message lately? Me too - in fact, everybody have. How do you know what it means (and logs usually do mean something) without analysis? At the very least, you might need to bring additional context to know what some logs mean (example: IP address -> hostname -> server owner)
Logs often measure in gigabytes and soon will in terabytes; log volume grows all the time - it definitely passed the limit of what a human can read a long time ago, it then made simple filtering ‘what logs to read’ impossible as well: automated log analysis is the only choice.
Do you peruse your logs in real time? This is simply absurd! However, automated real-time analysis is entirely possible (and some logs do crave for your attention ASAP - e.g. major system failures, confirmed intrusions, etc)
Can you read multiple logs at the same time? Yes, kind of, if you print them out on multiple pages to correlate (yes, I’ve seen this done :-)). Is this efficient? God, no! Correlation across logs of different types is one of the most useful approaches to log analysis.
A lot of insight hides in “sparse” logs, logs where a single record barely matters, but a large aggregate does (e.g. from one “connection allowed” firewall log to a scan pattern). Thus, the only way to extract that insight from a pool of data is through algorithms that “condense” that collection of logs into usable knowledge (some say, visualization is the way to go)
Ever did a manual log baselining? This is where you read the logs for a while and learn which ones are normal for your environment. Wonna do it again? Thought so :-) Log baseline learning is a useful and simple log analysis technique, but humans can only do it for so much before burning out.
OK, let’s pick the important logs to review. Which ones are those? The right answer is “we don’t know, until we see them.” Thus, to even figure out which logs to read, you need automated analysis.
Log analysis for compliance? Why, yes! Compliance is NOT only about log storage (e.g. see PCI DSS). How to highlight compliance-relevant messages? How to see which messages will lead to a violation? How do you satisfy those “daily log review” requirements (again, see PCI DSS)? Through automated analysis, of course!
Logs allow you to profile your users, your data and your resources/assets. Really? Yes, really: such profiling can then tell you if those users behave in an unusual manner (in fact, the oldest log analysis systems worked like that). Such techniques may help reach the holy grail of log analysis: have the system automatically tell you what matters for you!
Ever tried to hire a log analysis expert? Those are few and far between. What if your junior analysts can suddenly analyze logs just as well? One log analysis system creator told me that his log data mining system enabled exactly that. Thus, saving a lot of money to his organization.
Finally, can you predict future with your logs? I hope so! Research on predictive analytics is ongoing, but you can only do it with automated analysis tools, not with just your head alone (no matter how big :-)) …

Past top 11 reasons:

Technorati tags: logging, log management, tips

Poll: What logs do you actually LOOK at?

2008-02-14T02:34:54Z

This is my 6th logging poll (vote here now!)- links to the previous five polls below.

This one is deceptively similar to the #1 below, but it is not. This poll is What logs do you actually LOOK at? and not Which Logs Do You Collect? In other words, are you a log packrat? Are you collecting and never using the log data? You are making a mistake, if you don’t.

Past polls:

Poll #5 “What are your top challenges with logs?” (analysis)

Poll #4 “Who looks at logs in your organization?” (analysis)

Poll #3 “What do you do with Logs?” (analysis)

Poll #2 “Why collect logs?” (analysis)

Poll #1 “Which logs do you collect?” (analysis )

UPDATE: analysis of this poll posted here. Enjoy!

Technorati tags: logging, logs, polls, log management

Private Podcast Feeds

2008-02-08T16:58:26Z

A podcasting friend of mine ran into the problem of always having to send the new episodes to his co-hosts for review. Once everyone agreed that it was okay, the show was put live on the RSS feed. Their method of distributing a raw mp3 file? Email.

If only there was a way to distribute the file electronically, without the overhead of email, and yet still get the file automatically once it’s ready. How about RSS?

ITunes and other feed aggregators have the feature of handling RSS feeds that are protected by HTTP authentication. When you download the feed, your client will prompt you for a username and password before downloading the RSS XML.

My friend’s podcast now has two RSS feeds:

http://www.example.com/feeds/public/podcast.xml
http://www.example.com/feeds/private/podcast.xml

The first URL is what’s submitted to all of the podcast directories. The second one is strictly for preview purposes. All of the responsible parties for the show subscribe to the private feed. This allows them to test new episodes, and verify that the RSS <item> content for the episode is correct.

Once everyone has agreed that the episode is ready, the RSS tags for the episode are copied over to the public feed XML file. Now outside users can see the episode and download it.

This is pretty good for most situations, but there is still one risk: HTTP URLs can contain authentication encoding in them, like so…

http://username:password@www.example.com/feeds/private/podcast.xml

Avoid using this convention in your bookmarks, or feed entires. If that URL were to be copied to an outside data source, there’s a chance that it could get into the wild. When that happens, outsiders may end up listening to your private, not production ready, podcast.

To reduce the chances of that happening, ITunes won’t list a podcast in their directory if the URL contains an embedded username and password. They won’t even list a podcast if the server makes a request for HTTP authentication.

Operations Mantras

2008-02-04T22:15:08Z

Years ago I visited Danga back before the Six Apart acquisition, when the company had its headquarters a couple of miles away and when Brad lived a block and a half down the street. Brad showed me some of their management tools — almost all home-grown.

I mention that because today I stumbled on Dormando’s [crappy] Operations Mantras. Dormando works for Six Apart, and he has the same philosophy I see in Brad. Relentless automation and merciless monitoring are the two secrets of efficient and effective system administration and operations management.

I only wish that someone had handed me this list of mantras when I started as a system administrator in the ’90s. (Puppet and Xen would have been nice too.)

What is the cost of SMS?

2008-01-29T18:21:00Z

Sam wrote a blog post about the cost of SMS messages. I admire the effort, but I’m not in 100% agreement with his conclusion.

I think the goal the author was trying to make is that SMS messages are overpriced, and consumers should be outraged. To support his arguments, he compares a price per byte breakdown between a SMS message, an email message, and a printed document with thousands of characters (whether binary, hexadecimal or base64, I couldn’t tell you). Unfortunately, the comparisons seem a little weak, and a real cost breakdown between these two technologies is not fair.

Let’s take a look at 3 points the author tries to make.

1. What is the value (and the cost) of an SMS message?

A SMS message typically originates from one user’s cell phone and arrives at another user’s cell phone. People use them for quick messages that either don’t need an immediate reply, or do not require the receiver to have 100% of their attention on their cell phone. Everyone has sent one of these at one time or another: “I’ll be late for lunch”, “OMG! TTYL?”, or “I never want to see your cheating face again”.

You don’t need to boot up a PC, and you have a higher expectation that someone will notice the message quicker, because almost everyone carries their cell phone with them. These aspects of SMS are features that carry a dollar value. Twenty cents may seem high per message when the technology cost is almost zero, but technology is not the only expense by a carrier.

First and foremost, people are needed to maintain the service. That includes systems administrators, engineers, and customer support personnel. Those costs need to be in balance with the total number of SMS messages that pass through a network. Every carrier out there has already measured their own cost per SMS message, and that includes transport, personnel costs, and billing costs. If SMS isn’t profitable, people could lose their jobs, or the per message price could go up.

2. Ah, but SMS shouldn’t have any cost, because the infrastructure is already in place, right?

Not exactly.

A per unit SMS charge could mean that you’re actually paying less for your base service. If every customer is expected to send 25 messages a month, they could reduce the base price to be competitive, knowing they’ll recover that cost with the add ons. If you send under 25 messages, the carrier hopes that the emotional teenager down the street will send 50 messages to cover your slack.

Alternatively, a carrier could reduce operating costs by offering unlimited messages for a base price, like $10 for unlimited messages; but that creates the risk that consumers would just forgo SMS altogether.

3. Apples and Oranges (Is email cheaper than SMS?)

The other aspect of this article that confuses me is the comparison between SMS and email. To put this in a better perspective, I’ll make my own argument.

If I were to fly from Minneapolis to Denver, the price of a one way ticket would be $290.23 with a flight time of approximately 2.5 hours. If I were to drive the exact same trip, my gas cost would be $180 (assuming that I can get 350 miles for every full tank of gas), and I would get there within 17 hours.

With either solution, I end up in Denver. Why does the airline in this case feel justified in charging such a higher cost? And if you think about it, that plane is going to Denver anyways, so I should be able to just ride it for free.

My comparison is a fail of epic proportions, because both methods of transportation have different operating costs. Airplanes cost more than cars, automobile gasoline has a tax for maintaining roads, flight attendants need paychecks. In both cases I’m paying different amounts of money, with different service expectations, but getting the exact same result.

The comparison between email and SMS isn’t fair because the author admits that there is no per email message cost. No ISP would ever want to deal with billing per email message, because the tracking of incoming and outgoing messages would only increase the price. You can use your bandwidth for web surfing, email, online games, or anything. SMS messaging does not offer these features.

Next, let’s discuss the idea of sending a single MP3 file (much less 2,560) over the SMS protocol. This is totally unreasonable action due to the size limits of SMS messages. SMS was never designed to transfer files, so why compare a file transfer? Most cell phone offer other methods, such as Bluetooth, or dedicated data networks for sending files. And while I’m at it, email isn’t the best protocol for file transfers either. If I had 2,560 individual emails of 4gb each, I would be looking at one mbox file of 10gb. Managing that mailbox would kill most mail clients, and probably a couple of IMAP servers as well.

Finally, the author is incensed that the person receiving the SMS message may also have to pay a surcharge. Unfortunately, he fails to point out that the recipient of the email message will very likely have an ISP charge as well.

Conclusion

If I’m coming across as harsh, it’s not my intention. This analysis is simply a counterpoint to the claim that text messages are expensive. Yes, they do cost a consumer money, and they probably make a profit for the carriers. I do not think this means SMS messages are bad, or exploitative.

The best way for a consumer to determine the cost of SMS messaging is to see what benefits the service gives you. If they save time, improve communication, or reduce confusion, there’s a value to that. If SMS does not do any of these, then you have the option of not using it.