Now, this is one of’em philosophical posts, but with direct relevance to system administration, security and audit, thus I am reposting here (original and some discussion of it that occured elsewhere). After all, I do have to justify the “Ph” in my Ph.D., right? :-) At the same time, this post will have an unmistakable stench of a rant :-) for some of my readers.

Recently, I was involved in some fun discussions on storage security. And, in most cases, you store “stuff” to let others access it, not just for archival or - gasp!- compliance purposes. One of the storage vendors I talked to recently mentioned that every year they’ve been in business (since early 90s), they have to add one or more audit features to their information access solution to increase the level of details, performance of their audit logging or whatever other audit related feature.

My response was: “What? You didn’t build them from the very beginning?” And then I thought: why provide access without audit logging?

No, really, why have it?! Disks are cheap, bandwidth is affordable, CPUs are powerful: why provide access to any information without having an ability (at least) to log each and every successful and failed access?

Before some of you label me “a privacy Nazi”, I have to disclose that I am somewhat of a fan of Scott McNealy’s saying “You have no privacy. Get over it.” Having access audit info is useful in so many cases, that not doing it becomes inexcusable and, frankly, stupid. Some of the many uses for such information are:

• Operational troubleshooting: knowing who failed to access the info and why
• Policy audit: who accessed what, with or without authorization?
• Regulatory compliance: legal requirement to have audit data is there to stay
• Incident response: what info got stolen and by whom?
• Information access trending and performance optimization: are we providing quick and reliable access to information?

So, what about privacy? Privacy is defined (in Wikipedia, where else) as “ability of an individual or group to keep their lives and personal affairs out of public view, or to control the flow of information about themselves.” We see two completely different things here: keeping the info out of public view and controlling the info about you. The former is clearly reasonable and possible. How about the second? To be honest, it sounds like a sheer idiocy to me, because you do not control it and never did. You’ve got to a) become invisible and b) stay home all the time :-) for a fair shot - albeit not a certainty! - at controlling the info about yourself. I can still talk about you - and thus control the information flow about you - by saying “ah, that invisible guy that stays home all the time!” :-)

So, what is the connection between the above definition and my call for “no access without logging”? Logging is NOT a privacy risk; inappropriate use for collected data is. Before you object by invoking the infamous “guns don’t kill people; gaping holes in vital organs do” :-) I have to say that the above privacy definition is about access to information about people, not about the existence of said information. And, yes, Virginia, there IS a difference!

Similarly, nowadays many folks are appalled when they see stuff like this (”Fresh calls for ISP data retention laws. US attorney general cranks up the volume.”), but it actually - gasp! - seems reasonable to me, in light of the above. Admittedly, if your bandwidth is so huge that you cannot log and retain, you might be able avoid logging or at least avoid long term log retention, but that is a different story altogether.

Another thing that is tied to this is the whole “privacy vs security” debate which never made quite sense to me - until now. This is indeed the area where those who want to have logs for security and other uses will clash with those who don’t trust controls on the collected log data and would prefer for such data to never get created in the first place. But that would be a subject of a follow-up post later….

So, have doing security and especially log analysis for whatever number of years gone to my head? Or am I onto a critical trend here? Comment away!!!

tags: security, storage, access, audit, privacy

Admittedly, this is again a repost from my other blog , but I am sure it would be useful to my O’Reilly readers.

Not the wormy “CodeRed”, but Red Alert, mind you.

Following the new “tradition” of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #4: Code 200: Good or Bad?

Now, this is somewhat related to my previous tip (#3), but as it applies to web server logs, such as IIS Extended W3C (or other) logs or Apache access_log logs. Unlike, say, databases, all webserves come with logging enabled, so web server log review is a common task for web server admins as well as for security teams. But what do you look for in normal web access logs, apart from web site access statistics and visitor trends?

Obviously, observing various access failures is important. All the pesky 404s, 401s as well as 50X response codes.

Here are a couple of examples from Apache web server access_log logs:

61.52.47.251 - - [26/Sep/2004:21:10:22 -0400] “GET /MSADC/root.exe?/c+dir HTTP/1.0″ 404 286 “-” “-”

or

67.170.226.184 - - [10/May/2004:13:18:09 -0400] “GET /scripts/..%%35c../winnt/system32/cmd.exe?/c+dir HTTP/1.0″ 400 293 “-” “-”

Is this ominous or what? :) Nah, it is just my Linux + Apache honeynet responding to some good ole IIS 4.X attack. Truly nothing to worry about.

How about this one though?

68.49.152.86 - - [22/Apr/2004:11:56:49 -0400] “GET /cgi-bin/awstats.pl HTTP/1.0″ 200 2190 “-” “Mozilla/4.75 (Nikto/1.32 )”

Somebody using the web statistics CGI script? Benign, since the response code is 200 which means that the server served the page (or ran the script!) as expected? No, you are about to get 0wned thru an AWSTATS hole!

So, the gist of this tip is similar to the previous one: when monitoring web server logs, do not just focus on 404s, 401s, 50Xs and other “known bad” response codes; take a long hard look at the vanilla 200 response codes. “But how do I separate the “good” 200s from the “bad” 200s?” - you might ask! Well, that would be the subject of another tip in the future!

Just in case, here are the links to my previous tips: #3, #2, #1.

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

tags: security, tips, log analysis, log, web logs, web servers, log management

About once a week, a ticket works its way down the grapevine. User cannot upload a file by FTP. It’s always a different user, but the problem is the same. I don’t think it’s the frequency of the problem that’s frustrating, it’s tracking the source of the problem.

What are their firewall settings? What are our firewall settings? Are they coming across the VPN? Is it an active or passive data connection? Is their FTP script sending the wrong password again? Are they hitting the proftpd server or the wu-ftpd server? Hey, this FTP daemon log doesn’t report any connection details!

In the end, there’s always a different root cause. Maybe it’s a NAT translation failure, or a load balancer that needs to be rebooted. Next week it will be a totally different scenario, and a totally different solution.

FTP is an antiquated protocol, designed to address early shortcomings in the IP protocol over 25 years ago. In my humble opinion, it’s time to give up and let this protocol die. There are other protocols out there that can do the job just as well, if not better. Here are the top contenders:

SCP/SSH. Secure authentication, and data encryption to boot. SCP is available on almost every platform out there. There are even a few decent SCP GUI clients out there. The only downside is there aren’t too many tools that can script SCP uploads on a Windows host.

SVN/CVS/Other. Okay, these protocols are a little restrictive. They allow for file uploads and downloads, but their main function is versioning control. Neither of them will be helpful if you’re trying to perform simple file transfer operations outside of a centralized repository.

Jabber. Not a popular protocol, but something to keep in mind. It would be incredibly easy to set up peer to peer file exchanges between client/servers or client/client environments.

HTTP. This should be the obvious choice; I know it’s my personal favorite. HTTP has authentication mechanisms built into the protocol, SSL is available, download recovery is possible, and servers can design fancy HTML interfaces for uploads and downloads through a web page. Plus, almost every programming and scripting language out there has a HTTP client library to facilitate scripted actions.

Probably the only thing that doesn’t work well is HTTP upload through a web browser. They will upload files, but web clients like Firefox (and others) don’t bother to report upload statistics like bytes sent or time remaining. Anyone uploading files a file is stuck in feedback limbo. Is my file being sent? The blue E is just spinning.

If somebody out there develops an extension or program to make HTTP uploads easier, they will have my immediate gratitude. I’ll email the program’s webpage to every remote user I have, with a side note saying that they will no longer need to send me their entire firewall configuration to debug problems in the future.

At some point, you’re going to be disappointed in somebody that provides you with products or services. Well, maybe you might be one of the lucky ones who never has to deal with that. The odds, however, are in my favor.

I have a hardware supplier that I truly appreciate. Their equipment is top notch, reliable, and it makes my job incredibly easy. And I’ve also had great support from that company…. until last week. For the past seven days, I dealt with random reboots, unanswered tickets, and parts that were never shipped. Because of this lapse in service, I am writing a letter of complaint.

In my case, I’m not about to dump vendor X for replacement Y. I have six years of good reliability and support behind me, while this is just one bad incident. My goal is to make sure my concerns are addressed, and that the company makes steps in preventing this in the future. I’m disappointed, but I still have a degree of faith in this vendor. Ironically, I did talk to one service technician who was very reliable, and did a great job in helping me. One person did a great job, but my overall feelings were that the support system had failed.

Once my ordeal was finished, I spent about five hours putting together a letter of complaint (If you’re chromatic, you know it takes me forever to edit my own stuff). From what I have learned, and studied from my old business writing textbook, I’ll give you some advice on how to write a complaint letter.

1. Make sure you’re addressing the right person. Talk to your VAR, or your salesperson. They will usually point you in the right direction since you are a valued customer (Unless of course, you’re complaining about your VAR or salesperson). If you’re not communicating with the right people, your letter may be totally ineffective. Don’t assume that writing directly to the CEO of a company is going to give you more attention.

2. Detail everything. Try to explain the situation as accurately as possible. Provide dates, names or references when applicable. The details will act as your own retelling of events, and may provide a huge contrast compared to any internal notes or memos on the incident.

3. Explain the basics in the first 2-3 paragraphs. You might have a long winded explanation of what happened, but try to make sure your person knows the reason you are trying to communicate with him or her. You need to literally capture the attention of the person you are writing to. If your vendor’s product was determined to be the root cause of the halon discharge, that should probably be mentioned at the beginning.

4. Explain why you feel things went bad. The vendor might see things differently from your point of view. From their perspective, they may only see a single service ticket in a queue that was closed within 24 hours. They may not understand the details of what happened during that time, or how this problem has affected your business. If you’re dealing with trouble tickets, also keep in mind that they may have internal notes that you might not see if you also have access to the ticket.

5. Give praise when they did something right. If you had great support, or any positive experience, compliment them on it. While it is important to know what a company has done wrong, it is also important to know what they did right.

6. Venting frustration is not the same as all out anger. If you’re intent on maintaining a relationship with this vendor, maintain a sense of professionalism. Your needs will not be addressed if the addressee of your letter thinks your a jerk. If anything, the relationship will probably go downhill.

It’s safe to say that when someone receives a box full of rattling bits of a circuit board with a note, “Enclosed is your crappy raid controller”, that you have no desire to further pursue a relationship with this vendor.

7. Don’t forget that you are representing yourself and your company. Written correspondence is still held in high regard over email. Depending on what you write, your letter might be framed, it might be quoted in the company newsletter, or it might be tacked up to a bulletin board in the break room with red pen marks correcting grammer and spelling errors.

Always show your letter to a higher up if possible, and don’t be afraid to let somebody else proofread it.

Admittedly, this is again a repost from my other blogs (see here and here), but I am sure it would be useful to my O’Reilly readers.

Following the new tradition of posting a security tip of the week (mentioned here, here ; SANS jumped in as well), I decided to follow along and join the initiative. One of the bloggers called it “pay it forward” to the community.

So, Anton Security Tip of the Day #3: Watch For Access Failures AND Successes in Logs

Now, many a winter ago :-), people used to think that checking for access failures in logs is all they need to do to “stay secure.” Indeed, picking out failured access attempts seemed like a reasonable way of doing things. Similarly, people even considered firewall “connection denied” messages as more important than “connection allowed” log records (although this will be a subject of a separate tip - a whole paper, in fact - later)

So, what is more important? This

Sep 26 12:36:40 bridge sshd(pam_unix)[2128]: session opened for user anton by (uid=0)

or this:

Sep 25 14:31:24 ns1 sshd[19308]: Failed password for anton from 10.10.154.44 port 53452 ssh2

Let’s think about it: one log entry says that Unix security is doing its job and blocking a bad user (assuming no fat fingering and forgotten passwords for simplicity sake) from accessing the system, the other says that some kind of user now has access to your system.

To quote Doc from “Back to the Future” :-) : “Exactly!!!”

You do need to be aware of both; you can’t just focus on access failures while monitoring your logs. Make sure you review successes as well. And, yes, certain patterns, like a long string of failures followed by a success are even more fun to watch …

Also, here is a link to my previous tips of the appropriate-time-interval (#1) :-)

Also, I am tagging all the tips on my del.icio.us feed. Here is the link: All Security Tips of the Day.

tags: log analysis, logs, security, tips, chuvakin

Technorati Profile