Related link: http://techrepublic.com.com/2300-10879_11-5896894-1.html

Whoever redid the cabling in this server room should attach a copy of the final slide to their resume. If I was hiring a server room engineer, I’d only have to take one look at the photo and I’d immediately offer him the job.

Cabling is an artform. You need good tools, supplies, patience, and above all, discipline. You can’t just wire up a new server room and leave it at that. Because sooner or later the quality starts to fade. Somebody will use the wrong color cable, or they’ll run a patch in between cabinets for a temporary job. Then in a year, what started out as a nice setup has turned into an unmanagable mess.

It looks like the guy who took on this project had the luxury of being able to take every server down at once and just rewire from scratch. Not too many people can do that, which is understandable. Once a server is in production, it’s hard to take it down unless you have long maintenance windows. And fixing the wiring of racked servers on a 1 by 1 basis is not a simple task.

The great thing is, I agree with every point that camarodave made when he described the work he did. The following is a reinteration of some of his ideas, with my own personal experience mixed in.

Get rid of the cable management arms

The management arm of a Dell 2650 adds seven inches to the overall length of your server, and all that ends up doing is blocking airflow.

But, you may ask, what if I need to fully extend a running server?

The only reason you should ever pull out a server from the cabinet is if you’re doing maintenance. And if you’re doing maintenance, then there’s no need for the server to be cabled. It’s my experience that when a cabinet is fully stacked with servers, cable management arms actually make it more difficult to pull out equipment.

Keep a well assorted stock of cable lengths and colors

At my first job, I had about 10 different category five cable colors that were assigned by Ethernet, Serial, T1, digital phone, POTS line, etc, etc. At my second job, the standard was only two colors: Ethernet and OOB Management (serial or KVM).

Unless you’re very good with a crimper, buy a large stock of cables in multiple lengths. If you can handle the cost, break them up in 1ft increments (3ft, 4ft, 5ft, etc). After you wire 2 or 3 identical cabinets, you’ll get an idea of what your inventory stock should look like.

It’s unlikely that you’ll ever need cables shorter than 3 feet. Keep in mind, you’ll always need up to 19 inches in length to go horizontally across the patch panel; maybe more depending on where the network ports are located on the server.

Design your perfect cabinet (or rack) and establish a standard

Build out the perfect rack or cabinet. Make sure you have enough power outlets running down the sides, and create a standard layout for patch panels and cable management panels. The goal is to make sure you can accomodate the power, network, and cooling needs so you can put the most computing power into your enclosure.

A common problem with some cabinets is that one resource suddenly becomes overutilized or unavailable. For example, all of the outlets in a power strip may be fully populated, but the cabinet is only half full. To correct this, power connections are migrated to a strip in a different cabinet, or strips might be chained together.

The goal is to make every enclosure configuration identical. If you can afford it, throw in more power outlets and network ports than you will initially need; you’ll save more money in the long run. Since server hardware is getting more powerful in smaller form factors, you could find yourself trading in all of those 3.5 inch servers for 1.75 inch servers. You’re saving space, but your power and network requirements end up doubling.

In an unrelated post, Richard Bejtlich stated on his blog that “Tor servers will have to run inline filters to police this sort of activity.”

This issue troubled me for a while. Somebody smart :-) told me some time ago that Tor license and legal FAQ actually prohibits such monitoring and (?) filtering. Specifically, it says:

“Q: Should I snoop on the plaintext that exits through my Tor server?
A: No. You are technically capable of monitoring or logging plaintext that exits your node if you modify the Tor source code or install additional software to enable such snooping. However, Tor server operators in the U.S. can create legal and possibly even criminal liability for themselves under state or federal wiretap laws if they affirmatively monitor, log, or disclose Tor users’ communications, while non-U.S. operators may be subject to similar laws. Do not examine the contents of anyone’s communications without first talking to a lawyer.”

My response that was that the above goes against common sense, but I was told that law and common sense have nothing to do wich each other…

Ideas? Discussion?

Here is an interesting doc, pertaining to summarize current ISP operational security practices. It even has a neat section on logging practices:

“2.7. Logging Considerations

Although logging is part of all the previous sections, it is
important enough to be covered as a separate item. The main issues
revolve around what gets logged, how long are logs kept and what
mechanisms are used to secure the logged information while it is in
transit and while it is stored.”

The weird part is that the document advocates “exception logging”, rather than a full audit logging of network connections. Is that because those ISP usually have huge network pipes? Or is there some legal requirements to not have discoverable data on connectivity?

Related link: http://oedipus.rubyforge.org

A friend and old colleague of mine has been working on a tool for penetration testing and exploitation of web applications for quite a while now and he has recently released it publicly. It is called Oedipus, and it is available on Rubyforge here.

The reporting functionality is pretty basic at the moment, but the testing framework seems pretty robust. The entire thing is written in Ruby, and is designed on a modular plugin basis to enable easy maintenance and extension. I’ll probably be covering more on it sometime soon… he is working on a GUI, so I am looking forward to that too.

Falko Timme will be thirty this year. He’s possibly one of the most popular mentors for people wanting to become Linux system administrators on the Internet. His step-by-step tutorials have gained a large readership.

He was born in Celle, Germany. He studied industrial engineering in Dresden and Braunschweig and speaks and writes perfect English, even though it’s not his first language.

I have interviewed a lot of people over the years. I don’t remember an unpleasant one. But this interview with Falko stands out in my mind as the best. I hope you enjoy spending time with him. He’s a remarkable person.

On Digg.com

TA: Falko, your tutorials show up on just about every Linux system
administration search I do these days. When did you start writing
them?

Falko: I started the “Perfect Setup” tutorials in 2003 as part of the 42go
ISP-Manager documentation. At projektfarm GmbH in Lüneburg, we had developed the Linux web server control panel called “42go ISP-Manager”, and though we had an installation manual for it, we got lots of support requests from people that had no idea how to install the base system for 42go.

At that
point I decided “Ok, let’s write a tutorial that covers every single step of
the system installation and configuration so that people can simply copy and
paste the commands from the tutorial so that 42go runs on these systems out
of the box.”

First, we put those tutorials (we started with Debian Woody) on
projektfarm.com and also on my personal web site falkotimme.com. We
noticed that we got a lots of page views from our tutorials. We
even had visits from people that weren’t interested in 42go at all, so we thought that there’s a need for the kind of tutorial we wrote.

We created HowtoForge in 2005 as a source for high-quality tutorials, and it’s open for everyone. Anyone can contribute. I don’t test every tutorial that gets submitted, but I test my own tutorials to verify that they don’t contain errors or that I don’t omit
small but important details.

TA: When did you start working with Linux and how did you get started?

Falko: I started in 1998. I wanted to host my small search engine and
web sites for friends so I bought a Cobalt RaQ2. That was my first Linux
server, and I barely had Linux knowledge then.

It had a nice web frontend
that you could use for basic things, but soon I realized that if I want to
do more sophisticated things with this server, I had to dive in into the
Linux world and get my feet wet on the command line. Many sleepless nights
followed…

TA: I notice you answer questions on a lot of forums. Do you ever sleep?
Where can people go to ask you questions?

Falko: Don’t worry, Tom. I get enough sleep - and I don’t dream of little penguins…

I’m active on the HowtoForge forum, so people can ask me questions there.

TA: Tell us a little about projektfarm.de -what is it exactly and when did
you get involved?

Falko: projektfarm is a company in Lüneburg, Germany that focuses on software for
Linux web servers like ISPConfig. We offer ISPConfig which is free, the 42go ISP-Manager, and the 42go SPAM-Filter. We also offer professional Linux services for companies and individuals.

When a company or individual has problems needing support with Linux
or if they need help with Linux system administration, then they can contact us.
Our job involves providing help and we actually enjoy doing it.

TA: OK. And you got involved with this company how?

Falko: I met the founder of projektfarm GmbH, Till Brehm, in 2001, and we soon realized that we both had a Cobalt RaQ and weren’t satisfied with the
control panel. Neither of us saw any alternatives at the time. Some projects were on the market but they were immature.

So we said, ” Let’s write our own.” That’s a typical Linux person’s response. So we gave birth to
the 42go ISP-Manager. I was fully involved with projektfarm at that time.

TA: I had several Cobalt boxes myself. I thought they were insecure. But now, I have ISPConfig running on my server at home and it’s tight and hard. That must have been an objective in your development.

Falko: That’s correct. Till wrote to you and explained that.

TA: He wrote and said “My server was fully patched and got hacked twice because Cobalt had not released all patches available from Red Hat. These experiences lead to the decision that the base linux system of an ISPConfig server should be kept up to date with the packages from the underlying Linux
distributon without breaking ISPConfig”.

Falko: That allows you can keep your server up to date with the most current patches for SSH, BIND, proftpd, etc.

TA: I want to change the subject a little. We hear a lot about Linux in Germany. And with your country as the second largest economy in the world, just how is the climate in Germany for Linux?

Falko:It’s very friendly, especially on the server market. If you want to
rent a dedicated server here, you’ll find many people who offer Linux. Other providers offer other operating systems, but in the ISP hosting market other operating systems are vanishing.

Also many communities have switched their computers to Linux. I’m sure you know about Munich. Even the servers of the German Parliament run under Linux. SuSE was a German company until Novell bought them.

Linux is widely accepted on servers here, and if you search for Linux topics
on Google, I’m sure you’ll find many German web sites, forums, mailing lists and so on that deal with Linux.

TA: And of course, Howtoforge is one of those sites. OK. Let’s get back to your business. You’re on the ISPConfig team. Tell us about the project and your role.

Falko: ISPConfig emerged from projektfarm’s 42go ISP-Manager. We didn’t sell as many copies as we thought we would. The price was moderate, EUR 189,- per copy.

We wanted to sell more and so we analyzed our business. We believe we didn’t sell as many as we should have for two reasons:

First. we are good programmers, but we are no marketing gurus. Secondly,
the name, 42go ISP-Manager, didn’t connect with people. It just didn’t stick as a brand.

So, we decided to fork a branch of the 42go ISP-Manager. We chose a new name that people could remember. We chose ISPConfig. We also
decided to make it free and put it under a BSD license which, generally
speaking, says that you can take it and do whatever you want, which means real
freedom for the user.

ISPConfig has all the features of 42go and a lot of new features that you don’t find in 42go. It isn’t a limited version of 42go. I think it’s likely that we stop 42go and merge it with ISPConfig and just continue the ISPConfig development.

We made this software free because we thought it would get the
attention it deserved. We changed our business model to sell
support and services instead of software.

TA: That makes so much sense. People have to like what you’ve done.

Falko: With a large user base, you will find users who also want professional support and want to pay for it. Also, many companies and government units require support contracts. So, we see a market of people who will want and need to pay for support.

This business model works better than the previous one. And as an open source project we offer free support on the ISPConfig forum. We have a growing
and active community there.

TA: And the project itself?

Falko: Right now ISPConfig has a development team of 15 people, most of them concentrate on translations. ISPConfig is available in English, German, Spanish, French, Italian, Dutch, Polish, and Swedish. The core team members are Till and me. Till’s focus is mainly the web frontend, mine is more
on the backend that automates configuration. The borders
between these two areas float.

TA: Are you active on any other projects? Or is this enough?

Falko: I am involved in other projects. I just released MyDNSConfig (), a web-based
control panel for the MyDNS name server. This is now a standalone
application. And it will be part of ISPConfig’s next iteration.

TA: On your web site you tell people that they can write you about Linux
support issues, then you say you would accept a small contribution.
You’re personally offering your service for free. How can you do that?

Falko: Basically, I’m trying to help people in the HowtoForge forum. This is a free, growing and very active forum with a very friendly atmosphere where
everyone is welcome - newbies as well as Linux experts.

We manage to find solutions for lots of problems there, but sometimes it’s
hard to guess what causes a problem. If they like,
they can then contact me and ask for my support. I will even log in to their
systems and fix the problem. But this kind of support can’t be
free.

TA:Your tutorials are all on HowtoForge.com. They are also on your web
site. Tell us more about HowtoForge.

Falko: When we had our first tutorials on projektfarm.com and falkotimme.com, we
realized that we got lots of page views because of the tutorials. The
feedback was also very positive. So Till and I decided to create HowtoForge
in April 2005. We put all our existing
tutorials on HowtoForge, and we post our new tutorials exclusively on
there.

HowtoForge is open to everyone and any one can contribute Linux
tutorials. We want to make it a source of high-quality Linux tutorials. I
really think there’s a need for this because very often when I read Linux
tutorials and try to follow them, I do not succeed.

Typical documentation does not describe the exact steps you have to take. I mean I see things like “… then patch the kernel”. But how do I patch the kernel? What are the exact steps?

In other situations people forget to mention some small but important details. I think it’s the usual case. Even I have problems following existing tutorials. So, I wonder how a
newbie or even an experienced system administrator is supposed to do follow the existing documentation and become happy with Linux?

On HowtoForge we publish tutorials that describe every single step.
As a reader you can simply copy and paste the commands to your shell, and
you’re done. This gets the people started, and if they are newbies they can
at least play around with a working system in order to understand how it’s
working instead of pulling out their hair.

If there are questions or
problems, people can go to the HowtoForge forum.

TA: Your tutorials are in English.

Falko: We decided to publish only tutorials written in English on HowtoForge
because English is the language that most people dealing with computers
understand.

Often, when I search on Google for a solution to some Linux
problem, I find pages in Spanish or Portuguese or Polish, etc. and I think
“Damn, they seem to have the solution, but I don’t understand a single
word…” We could have decided to write the tutorials in German, but that
would have been unfair for most people.

TA: Do you have any books planned for the future? If so, what are you
going to write?

Falko: I haven’t planned any books yet. Maybe a compilation of my best tutorials,
but I don’t know yet.

TA: Well, you can write one with me any time you want. And I hope you will let us read more about you in the future. Thank you.

Related link: http://eusecwest.com

The second and last day of EUSecWest has been and gone. It turned out to be a fun experience, with a lot of valuable and interesting information shared by the speakers, and a lot of interesting folks met at the conferece. Here are my notes from the main speakers today (I haven’t included the lightning talks or vendors because I was busy drinking beer by that stage…):

style="font-weight: bold;">Shreeraj Shah
from net-square
talked about web application attacks and defences.  He
introduced
and demonstrated a number of tools he has written for the enumeration
of information from the MSN Search engine, as well as some cool tools
for web services testing and penetration, including:

• MSNPawn - discovery and enumeration of information about
  HTTP
  hosts (including discovering running hosts by the server IP address)
  from querying using the MSN Search web service
• MSNKnight - for building a profile about the site, by
  acting as a local proxy
• wsPawn - for footprinting web services
• wsKnight - for interacting with the web service using a
  WSDL file
• wsAudit - for performing attack fuzzing on web services

style="font-weight: bold;">Justin Clarke
(me) from Ernst &
Young talked about automating web application assessment and
exploitation.  The talk seemed to go down fairly well.
 I demonstrated some of the tools that were written for href="http://www.oreilly.com/catalog/networkst">Network
Security Tools, as well as one tool (SQLBrute) that is
available from my site.
 I also completely forgot to demonstrate one small tool
(IEnterceptor)… whoops.

Andy Davis
from Information Risk
Management talked about ColdFusion security.  They
have been doing a lot of research on version 7, 6.1 and 6.0 of CF, and
talked about some of the issues (especially in the admin interface)
that can be leveraged for nefarious purposes.  Some of the
issues they found haven’t been fixed yet (in the services etc that ship
with CF), so we can look forward to more once Adobe release the fixes.

Tim Hurman
from Pentest Limited
talked about the security over personal ARM devices, such as common
PDAs.  This covered some similar ground to Barnaby Jack’s talk
yesterday, with the differences that Tim was using JTAG to debug IPAQ’s
and the like, and went on to demo an “always on” vulnerability in (I think) the
vCal parsing via Bluetooth OBEX file transfer on a (I think) HP 5xxx IPAQ running Windows Mobile
2003.  The exploit was a nice Window showing “0wn3d”.
 Tim mentioned how this type of issue could be used to
formulate an “airborne virus” that you could pick up from an infected
device, which would attack your desktop PC when in the sync cradle, and
attack other mobile devices via Bluetooth when not attached.
 Nice :-)

Raffael Marty
from ArcSight
talked about visual security event analysis using the href="http://afterglow.sourceforge.net/">Afterglow
toolset.  Raff went through a number of visualisation
examples, and these did look very useful for this type of application.
 I will definitely be having a look into these sometime soon.

style="font-weight: bold;">Michael Boman
from KPMG Singapore
talked about network security monitoring theory and practice, and also
the SGUIL
network monitoring console.  This looked pretty useful, and a
possible alternative to some of the (expensive) commercial consoles
that are becoming more available.

Jim DeLeskie from href="http://www.vsnlinternational.com/">Teleglobe
& Danny McPherson from href="http://www.arbornetworks.com">Arbor Networks,
talked about securing the infrastructure from the point of view of the
service provider.  This was pretty interesting to me as well,
especially when talking about the provider techniques and limitations
when responding (or not) to DDOS attacks.

Andrea Barisani
from Inverse Path
(and the Gentoo
team) talked about the Gentoo rsync server compromise that happened in
December 2003 (of a core portage rsync server), the detection of the
compromise, analysis of what happened (including identification of the
flaw in rsync), and the coordination of working with the rsync
developers in fixing the flaw.  Very informative.

A very fun paper on bots (a type of malware, in case you did not know). Two quotes of note are below:

About the botnet owner: “The young hacker doesn’t have much sympathy for his victims. ‘All those people in my botnet, right, if I don’t use them, they’re just gonna eventually get caught up in someone else’s net, so it might as well be mine,’ 0×80 says. ‘I mean, most of these people I infect are so stupid they really ain’t got no business being on [the Internet] in the first place.’”

About one of the victims: “He eventually opted to buy a new PC rather than spend the time and money to repair the infected one. ‘It just made more sense for me to get a new $300 Dell that came with a free monitor that was better than the one I had,’ he says.”

Related link: http://eusecwest.com

Here are my notes from day one of the first annual EUSecWest/core06
security conference in London:

van Hauser
from The Hacker’s Choice
talked about IPv6 vulnerabilities, including the
differences, and similarities of performing certain types of exploits
in IPv4 and IPv6, and some of the new issues raised by the introduction
of IPv6. The
presentation can be downloaded from here.

Some of the interesting highlights from my point of view were:

• IPv6 is not in widespread use right now in Europe or
  the US
  (it is in Japan and South Korea).  A lot of the people using
  IPv6 (especially over IPv4) at the moment are Blackhats
• A lot of the IPv4 reconnaissance techniques (such as
  ping
  sweeping) are not possible on IPv6 due to the number of IP’s in
  the space - DNS is going to become key in identifying systems
• Once we can compromise a “public” server (i.e. one
  we can
  find through DNS) we can use IPv6 multicast to find systems on that
  subnet
• OSPFv3 relies on IPSec…. so we’d better be
  running it,
  or use another routing protocol
• vh has written a toolset (library?) for testing IPv6,
  and
  created a whole host of tools as part of the IPv6 attack suite:
• alive6 - for “alive” checking of hosts.
   If you
  are on the
  local subnet.  Ping sweeping the Internet for live hosts is
  not
  going to be feasible in an IPv6 world
• parasite6 - for Man-in-the-middle attacks,
  leveraging
  weaknesses in Neighbour Discovery protocol (the IPv6 replacement for
  ARP)
• dos-new-ipv6 - for preventing people from joining
  the
  network by claiming all IPs are already taken
• fake_router6 - for Man-in-the-middle attacks using
  Router
  Advertisements
• smurf6 - local subnet DoS using multicast
• rsmurf6 - remote DoS for broken Linux IPv6 implementation
• redir6 - route implanting using “secure” ICMPv6 redirects
• toobig6 - reduce client’s MTU
• vh tested various IPv6 implementations (Windows XP
  SP2, Linux 2.6,
  OpenBSD, FreeBSD 5.3), and found that all of the implementations were
  vulnerable to one or more issues
• Source routing may make a
  comeback, as all of the OS’s tested passed on source routed packets,
  and these can be disguised by putting a fragmentation header in front
  of the routing packet, hence passing routers.

The next talk was by Barnaby
Jack (from eEye
Digital Security)
about exploiting embedded
systems.  This one was pretty interesting, and he was using a
standard D-Link DI-604 broadband router as the exploit target.
 Here are some notes:

• A lot of embedded systems use ARM architectures, and
  real time OS’s such as ThreadX
• By using a JTAG emulator, and with some soldering,
  you can
  hook up a debugger to the device and start debugging the firmware as it
  runs
• Because of the watchdog timer, the device may need
  a patch to prevent the watchdog from resetting the device
• Modifying firmware involves figuring out how it is
  encoded and checksummed by the vendor - from debugging the web server
  code
• The demo was pretty impressive
• It used a LAN-side 0day bug in a function
  in
  the uPnP code for the router - the exploit removed the admin password,
  and enabled the external WAN side admin port
• Uploaded modified firmware with a “payload” that
  modified
  every .exe file downloaded through the router (popped up a command
  prompt with “OWNED” on it)

Javier
Burroni and Carlos
Sarraute from CORE
Security gave a quite interesting talk on using
statistical methods and neural networks to more accurately determine
operating system versions from the network.  They talked about
a couple of
different tools:

• An improved DCE-RPC endpoint mapper, which much more
  reliably determined what Windows version and service pack was running
• An OS detection tool using the nmap OS signature
  database,
  designed to reduce false positives, and “improbable” results that you
  can get with nmap
• first neural network to determine whether it is a
  “relevant” OS or not (i.e. one we have exploits for)
• second neural network to determine what
  OS family the system is - i.e. Linux, Windows, OpenBSD, FreeBSD,
  NetBSD, Solaris
• neural networks for each OS family to determine
  what version is running.

Nguyen
Anh Quynh from Keio
University in Japan (he is involved
with
development of Xen) gave a talk on next generation honeypot technology
using Xen.

He talked about the weaknesses with Sebek in a honeynet environment -
specifically that there are several ways to detect that you are in a
honeypot, from detecting the presence of Sebek, to detecting the
traffic that it sends on the network.

He then talked about “Xebek”, which uses Xen to run honeypot systems as
virtual machines (”user domains”) within Xen, with the Xebek server
running on the “Domain 0″ (host/first VM) on the machine.
 Apparently this, patching the system calls in the kernel, and
using shared memory to share data with the server will fix most of the
weaknesses with Sebek.

There was a spirited discussion in the Q&A about how detectable
this approach will be versus the approach of Sebek.  Quynh did
note that they are not attempting to hide the fact that the
system
in running in Xen, on the assumption that enough production systems
will be running Xen that it will not be suspicious.

Frederick
Raynal from the EADS
Corporate Research Centre
talked about the (ab)use of crypto.  This included discussion
of a
number of scenarios where crypto, humans, and trust could be
abused, such as:

• breaking the crypto (mis)used in the SuckIt rootkit
  versions 1 and 2 to take over a network of systems with SuckIt installed
• using the trust relations, and information stored in
  a
  user’s directory to theorise the possibility of a SSH worm.
   This
  included noting the use of existing sessions and other methods of
  capturing or leveraging existing access
• looked at methods/possibilities for preventing the
  analysis
  of a binary through the use of crypto, including differing situations
  over keys and decryption
• looked at scenarios for performing Man-in-the-Middle
  attacks against Skype.

Cesar
Cerrudo from Argeniss
talked about Windows local shellcode injection.  The technique
he
was demonstrating was based on the use of LPC named ports.
 You
can see these things using Process Explorer from href="http://www.sysinternals.com">SysInternals.
 He went through the process of connecting to a process’s LPC
port
(including some of the problems, like how to find the name of the
port), create a shared memory section, and use the shared section to
put shellcode into the process’s memory space.  This also
returns
you a pointer to where the shellcode resides, making exploit even
easier, more portable, and Windows version independant.

Cesar then went through and demonstrated his exploits for MS05-12 and
MS05-40 that leverage this approach.  

Andrew
Cushman from href="http://www.microsoft.com/security">Microsoft
came and talked about the work that MS is putting in on the security
front over their products.  Andrew went through a lot of the
efforts that are going into Vista, as well as an interesting talk about
the Microsoft response to the WMF bug.

By centralizing a ticket tracking system on the Internet you can create a virtual 24 by 7 Linux support business. You’ll want to use a combination of VoIP and a global clearing provider for taking payments. A system like this costs pennies to start and has the ability to scale rapidly.

The Need for a Friendlier Service Model

In the open source software market, you have to provide excellent service and provide easy access to your support personnel. Business people who believe they can built a viable company on unique software alone will fail. Unfortunately, not many successful service models exist and even consultants in this area have a difficult time managing a service-oriented function.

The best models I have seen use automation to cut costs. But those systems tend to annoy clients. Promising to provide a living, breathing person on the end of a phone call sounds great in a TV Commercial but rarely solves the problem. Have you run into a friendly service provider lately?

To give you an example, I had a lousy experience with a VoIP provider. I tried AT&T CallVantage for several months last year in combination with Charter Communications high speed cable. I discovered that both companies outsourced so many of their services that I could never get the VoIP system to work properly. I had to move back to copper to get reliable service and to bring the costs in line.

My Case Study

Three years ago, I noticed a significant trend of declining sales in the IBM business partner segment of a business I managed. I looked at every indicator I could find and nothing stood out as a problem. While tracking the cause of the sales decline, I noticed that my entire customer service department had go to lunch at the same time and left the phones unmanned for over a hour and a half.

I inquired and found out that in spite of policies to the contrary, the service personnel had followed this practice for several weeks. So, with their desks vacated, I logged on to one of their workstations and discovered the team was 85 days behind answering service requests. I could hardly believe it.

I went back to my programming department and discussed the problem with our head engineer. I asked him to research ticket tracking systems and come up with a high quality solution. Within a day, he came back to me and showed me a system called Request Tracker (RT) by Best Practical Solutions. I had worked with several enterprise level trouble ticket systems at major enterprises such as Gateway and Ericsson, but I had never seen a product of RT’s quality.

My head engineer informed me that the product was free and open source. I felt dazzled. How could someone give this product away I asked myself.

I called a meeting of the development team and posed the problem. We decided to write every client in the old queue. I explained in an email that we had started auditing the service department and found significant delays. I wanted to know if the customer had received help from our company and had they solved their problems.

We used RT to do the mailing and it automatically assigned each customer incident a tracking number. The feedback created some intense situations and many customers felt betrayed. But using RT and splitting up the work load so that each engineer covered a three hour segment, we caught up with the service requests in less than ten days.

Within a week, our sales started to climb back to the levels of the previous quarter. After we caught up, we agreed to continue following the protocol of having engineers devote time to service requests. Our goal was to respond immediately with a ticket number which RT did automatically and to communicate with the customer within 30 minutes.

If I only Knew

Two years before I discovered the issues with my support center, we had built one of the first pay-per-incident Linux call centers. We built the business from scratch, stayed independent and finally became an outsourcer for a major Linux distribution. Initially, we received referrals from Red Hat and Caldera.

I had little problem finding Linux people to man the phones. A call center named Stream had lost a major contract and laid off many highly trained call center employees. As we grew, facilities and infrastructure kept us from scaling.

Red Hat and Caldera saw our business growing and began call centers of their own. Corel had abandoned their Linux business and we had difficulties working with Mandrakesoft. So, we abandoned the call center business and devoted our energies tobecoming a Linux ISV. We wrote Linux software which worked with Outlook and MS Exchange.

After seeing how well we managed our service department with RT, I realized that we could have used it to keep the costs down and to provide better service than we had previously. I wrote another business plan for a Linux support division with RT at the hub. The startup costs were insignificant and I found lots of people familiar with Linux in the Information technology field looking for work from California to Germany.

The model called for setting up a self-service web site with a significant database of FAQs. We also wrote howtos and built both backports and packages that the original Linux distributions refused to offer. The pilot focused on Sun’s JDS Linux distribution.

The success of the web site convinced me to set up a pay-per-incident call center. Ready to launch, Sun decided to abandon its Linux desktop. Prior to that event we had some interesting opportunities.

One of the top five IT Consulting firms asked us to sell them a bundle of pre-paid incidents. We also had similar discussions with resellers and integrators. Because of our low cost structure, we could sell incidents and the channel people could resell them at a significant profit.

A Low Cost Start-up with Significant Potential

Two developments in IT provided a new model for a service oriented business. The first involved the use of VoIP using Skype and Asterisk . The second involved qualified Linux support people available globally.

In discussing the business plan with friends and acquaintances around the globe, I discovered a significant appetite among Linux people who wanted to contract and cover request queues. For example, I found people who excelled at Debian, Red Hat, SUSE, etc. Some excelled at DNS, databases, mail and other specialties. Some were good generalists.

Some of the people who contacted me wanted extra work and some wanted to simply work at home. The key to creating a low cost global communications system involved setting up Asterisk servers in locales where the contractors worked and then centralizing and exchange using one of the free VoIP services like Skype.

Using caching techniques, the request queues fill up and people manning RT can grab a new request when the next engineer becomes available. If an engineer grabs a request but can’t service it, then he or she can pass it to an engineer who can. I’ve seen this work particularly well using RT.

My first call center produced opportunities over and above initial level one calls. In a majority of circumstances, the level one call turned into multiple incidents and/or projects. In fact, two case studies I wrote for Macmillan were the result of rollouts of Linux in two enterprises. We got those contacts because the businesses started using our call center, then our level two and level three technicians. They asked for us to answer tenders and we won the business.

So many people around the globe use Linux and desired to work in a Linux company that we had no scarcity of people wanting to join the business. I plotted out the locales where we could have placed Asterisk servers that could feed into an exchange that our telecommunication costs looked small. In addition many of the tickets closed could feed a knowledge base available from Practical Solutions that attached to RT.

Is this a Feasible Business?

When Sun backed out of the Linux desktop business, I felt that we wasted a year developing our pilot. We also lost the funding opportunities available by selling incidents in advance. Having walked over burning coals in my previous encounter with VCs, I had no desire to work with them again.

I doubt many people would want to work 90 hours a week for three years and have some person tell you to surrender your stock, sign a five year contract at the end of which you could earn back nine percent of your previous holdings. In addition, those people went to the other members of the firm asking them if they would move to another city while failing to inform you.

While I chose to delay a move to into another venture, I still see the value of Request Tracker as the hub of a service business coupled with VoIP technology. This model allows you to centralize your support in a web portal while using a distributed work force. Any software provider wanting to offer support to end-users, channel partners or resellers can use this combination to provide high quality service. Just make sure you man RT and get to your customers quickly.

What Else Do You Need?

By centralizing a ticket tracking system on the Internet and publicizing the business opportunity, Linux people should be able to organize a 24 by 7 support business. You’ll want to operate it like a project and share the revenues. You should also be able to work with Asterisk to build a global VoIP exchange while clearing payments. I hope you will look into this as Linux is growing and users need support and are willing to pay you for it.

Related link: http://eusecwest.com

The first annual EUSecWest conference (from the organisers of PacSec and CanSecWest) kicks off in London tomorrow. I’ll be there, speaking on Tuesday, and blogging some detail about each of the talks.

These are the talks that have been accepted:

• van Hauser THC / n.runs GmbH
  Attacking the IPv6 protocol suite
• Javier Burroni & Carlos Sarraute - Core Security Technologies
  Analyzing OS fingerprints using Neural Networks and Statistical Machinery
• Nguyen Anh Quynh - Keio University
  XEBEK: A Next Generation Honeypot Monitoring System
• Fred Raynal - EADS
  Malicious Crypto
• Cesar Cerrudo - Argeniss
  Windows Local Shellcode Injection
• Andrew Cushman - Microsoft
  Microsoft Security Fundamentals
• Shreeraj Shah - Net Square
  Advanced Web Hacking - Attacks & Defense
• Justin Clarke - Ernst & Young LLP
  Practical Automated Web Application Attack Techniques
• Andy Davis - IRM PLC
  ColdFusion Security
• Tim Hurman - Pentest Ltd.
  ARMed Combat: The Fight For Personal Security
• Raffael Marty - ArcSight
  A Visual Approach to Security Event Management
• Michael Boman - KPMG Singapore
  Network Security Monitoring: Theory and Practice
• Jim DeLeskie & Danny McPherson - Teleglobe, Arbor Networks
  Protecting the Infrastructure
• Andrea Barisani - Inverse Path
  Lessons in Open Source Security: The Tale of a 0-Day Incident

More details here tomorrow :-)

When someone writes an article for a publication like Lxer, they need to use an expository style. Expository articles expose or reveal a subject through facts, reason and argument. When you use an expository style you deal with an event, concept, or idea using facts and examples and not opinions.

When you write an expository piece it does not have to be dry and boring. Your observations and experience often form the basis of your finding in the first place. Using your experience humanizes your writing and makes for an interesting read.

The Format

An article contains a title or heading, lead, conclusion, a body of paragraphs and headings and an ending. Most articles run between 400 and 800 words. Some articles contain 1500 words or about four pages of writing on a word processor.

Consider the heading of your article a meta-phrase. When you write a heading try to summarize your point in three words. That may be impossible but if you can do it, you will catch a readers attention. Sometimes, you will want to use a working title until you finish your article. Afterwards, you will find writing the title much easier.

When you write a lead, remember that it summarizes the topic while saying to the reader that some conflict exists. You tease out something about the drama so the reader can see value in reading the article. Your have competition for the reader’s time so let them know right away what they will get if they read your piece. Try to stay with three sentences in your lead.

Write your conclusion at the top of your article body after the lead. For example, you might start your article with a paragraph like this:

After interviewing the participants Forrestor used in the survey, we found discrepancies in their conclusions. When asked, 78 percent of the people surveyed said they used Linux in mission critical applications instead of 34 percent. In addition, the original surveyor biased the outcome with his or her introductory remarks.

You might consider starting an article with a conclusion odd. It does serve an important function. It tells the reader that facts, reasoning and investigation have yielded a result that affects them.

Next you begin your body of paragraphs and headings. Start with the work you did. Tell the reader that you investigated something. For example, you might simply write:

During the past six weeks, we investigated an Enron partnership still running in the Bahamas. The tip came from a former attorney formally connected with the partnership. We found the original partnership filing documents in Austin and gained access to the partnership bank accounts. We tracked down the payments and verified them with the Secretary of State.

I don’t expect you to have found such a tip, but the above example gives you an idea of how to describe your work. You could also say that a friend of yours told you about a new database. You downloaded it and installed it on an instance of Debian Sarge 3.1 r2. You ran a series of tests, created tables from MySQL and benchmarked performance of both databases.

Now you have informed the reader what you did. You can proceed to write the body of paragraphs and headings. This will explain your finds and allow you to make a case.

Expository paragraphs usually have three sentences. On occasion you can have more than three sentence. You would only add additional sentences if you needed to support your paragraph’s topic further.

The previous paragraph demonstrates my point. We call the first sentence the topic. The other two sentences support the topic sentence.

Notice that the first sentence declared the topic that expository paragraphs usually have three sentences. Then we wrote something about the topic sentence. The support sentences gave information about when the assertion in the topic sentence might differ.

You have already given your conclusion and stated the work you did. That should provide ample subject matter to continue writing your article. Start with the most interesting and/or important information first.

I like to add drama or tension into an article. You can accomplish this by providing information in small chucks. You want to provide a way to get to “who done it”.

When you follow the who-done-it approach you simply eliminate one possibility at a time. You start with the plethora of possibilities. You then say why one possible or another doesn’t answer the question you raise.

This differs from a simple news item. In a news item you simply announce an event. You can do so in a few sentences and usually two to three paragraphs.

If news leads to a bigger story then you have the basis of an article. An article allows you to explain something that no one has solved before. Your article should solve or nearly solve that something.

I investigated Microsoft’s political activities for nearly three years after I encountered their lobbying effort in Texas. I helped introduce an open source bill. Then I saw the dirty tricks for the first time.

I found ample information about Microsoft’s attempt to influence the case against them in Federal Court. It didn’t seem to make sense. I continued to collect the information I found hoping it work make sense.

I saw a small news article about a former Preston Gates lobbyist who had something to do with Tom DeLay’s potential ethics hearing in Congress. I searched for information on that that lobbyist and discovered that Preston Gates had paid an American Express bill for part of one of DeLay’s trips trips to Scotland. That allowed me to put the information I had gathered into a cohesive argument and gave me a scoop on Jack Abramoff.

Once I had a conclusion, I wrote an article and presented my facts. The facts lead to a way of reasoning and an argument. When comments surface that said Microsoft acted like any other corporation I had a counter argument.

Other corporations make contributions and lobby Congress. At the time the events occurred, Microsoft faced a breakup of the company. They had a different incentive for their intense lobbying effort.

When you structure the body of your article look at the kinds of thinking that exists in your readership. Will you article help them interpret the facts? If so, then start writing.

When you have eliminated the possibilities of who-done-it, you can begin the ending. You will find the ending much like the conclusion at the start of your article. You want to write an ending using less formal language and a clever statement.

Writing for a publication like Lxer requires an expository style. If you understand the guts of that style it can offer you a fun experience. Now, please go off and write something for me.

Last fall I wrote an article entitled “Critical Shortage of Linux Talent Slowing Adoption”. The article used a parody, a spoof about human resource management. I wrote:

Most human resource people believe Linux is an air conditioner company. They get confused between the term Linux and Lennox. So, HR recruiters define their job profiles like this:

Linux programmer needed by enterprise. Skills required:

REFRIGERANT METERING DEVICE CALIBRATION
LEAK TESTING
LIQUID & SUCTION LINE SERVICE VALVES KNOWLEDGE
START-UP
CHARGING FOR TXV SYSTEMS

Five to ten years of relevant training and master plumbers’ license required. Will accept equivalent for H1B applicants. Microsoft Certifications a plus.

The article title has become an urban myth and from the comments I have read about it, most people took the title to heart and never read the article. So, let’s set the record staright.

Summary of OSX/Oompa-A from AmbrosiaSoftware.

Summary of OSX/Oompa-A from Sophos.

Summary of OSX/Ooma-A from Symantec

Each report describes the basics of Oompa-A, yet they all reach different conclusions about what it is. Actually, Sophos has no idea what to call it, so they’re saying it’s both a worm and a virus.

I’m leaning a little more towards the trojan side myself. Oompa isn’t capable of doing anything without a form of user intervention, and that includes its propegation to other users.

A trojan isn’t just defined by a user clicking on an icon, it’s the art of deceiving the end user. Nobody would have excitedly clicked on their personal 401k statement the same way they would have clicked on potential screen shots of OSX Leopard.

I won’t deny the fact that it propegates, but maybe that is just a sign that the old malware defintions of virus, worm, trojan, and logic bomb aren’t suitable for pigeon-holeing entire programs. Most malware nowadays uses techniques from all four in order to infect as many hosts as possible.

Whether a program is a worm or a virus is not really important. What end users need to understand are the infection vectors, the propegation methods, the sustainability of the program, the threat level, and the potential damage. Any additional hype on top of that is usually just added on to sell you anti-virus (or anti-trojan?) software.

Maybe the time has come to stop using terms like virus or worm, and just create one cool sounding word for all forms of harmful software.

Here is a fun poll that I want people to answer: Inline Network Intrusion Prevention poll: “What is the worst thing your inline Network Intrusion Prevention system can do? ”

Treat this as a puzzle right now, I will explain why I am asking it when I get a semplance of a representative set (maybe 50-100 votes).

Hello, readers. With great pleasure I introduce the revamped and reworked O’Reilly Network Sysadmin site. We’ve revised the site display to feature the knowledge, opinions, and wisdom of our expert webloggers as well as to give you better access to newer, fresher information.

Our goal is to update the site several times a week with new postings from our webloggers as well as original articles and links to useful information elsewhere.

We’re still in transition putting all of the pieces together (and gathering varied articles from several years of the O’Reilly Network to present in a meaningful and useful way), but we’ll have everything up and running in the next couple of weeks. In the meantime, please feel free to let us know what we’re doing right, where we can improve, which projects and authors to watch and to recruit in the comments section here, or by mailing me directly at chromatic@oreilly.com.

Thanks for reading!

Sorry for cross-blogging :-), but its a fun bit so I figured I’d post it here too (I want to test the new platform as well). In this blog post I discuss what was one of the important lessons of CME-24 (you do know what it is, don’t you?)

Related link: http://chuvakin.blogspot.com/2006/02/cme-24-rampage.html

Sorry for cross-blogging :-), but its a fun bit so I figured I’d post it here too. In this blog post I discuss what was one of the important lessons of CME-24 (you do know what it is, don’t you?)

Related link: http://blogs.zdnet.com/Murphy/?p=525

I love those “security mistakes” papers (I’ve written a few myself) and here is a fun one specifically on Unix. “The four most common Unix security mistakes” by Paul Murphy covers “four worst security strategies affecting Unix deployment in business and government.”

Here they are:

#1: Using Windows to administer Unix
#2: Abandoning minimalism for convenience
#3: Failing to practice preventative management
#4: Focusing where the risk isn’t

You can also see my comments about it here.

Related link: http://lxer.com/module/newswire/view/53395/index.html

Ubuntu hasn’t backported Openoffice.org 2.0 or FireFox 1.5 and it’s an annoyance. This article tells you how to do it yourself.

My days of trying and writing reviews for every Linux distribution have slowed. I’m currently using Ubuntu 5.10 and I’m fairly happy with it. Spending most of my days as editor-in-chief of a popular news site, writing a system administrative book under contract and keeping up with a free lance writing career doesn’t give me much time to play these days.

I had already installed FireFox 1.5 but it wouldn’t upgrade. We have several computers here and I noticed how quickly SUSE 10 got their upgrade out. So, I knew someone had to know something I didn’t.

I went to the Ubuntu Wiki and found an article by Ben Scholl. Ben had updated it yesterday. So, I landed on the site at the best time. His howto called FirefoxNewVersion really helped. It entails downloading the FireFox tar.gz from Mozilla.

The main thing I got out of the article was how to enable FireFox’s own updating to work. Ben wrote: “This is the only way to get update notification working, but doing this has security implications in a multi-user environment.” I didn’t have that concern and I wanted to get 1.501 which supposedly has the memory leak fixed. I wound up reinstalling the new Firefox from Mozilla and by just following Ben’s instructions, got it working fine.

Next, OOo 2.

The Ubuntu folks say: “The latest versions of K/Ubuntu came with very, very late betas of OpenOffice.org 2, but since then OOo came out of beta, but K/Ubuntu hasn’t been updated”.

That’s fine. Scott Granneman wrote a nice Howto in his blog. I want to thank him personally for doing that.

Scott wrote:

Fortunately, in open source fashion, one Ubuntu user has compiled OOo2 & made it available for users to download. I’m using it, & things work just fine.

Click the link above or following his simple instructions:

To upgrade your OOo2, you need to edit your /etc/apt/sources.list file, AKA your repository list if you use Synaptic. I use the command line, so I first ran this:

sudo vim /etc/apt/sources.list

Then I added this line:

deb http://people.ubuntu.com/~doko/OOo2 ./

I saved, closed the file, & then ran the following commands:

sudo apt-get update
sudo apt-get upgrade

APT wanted to update OOo2, so I told it to go ahead. A bit later, I had the latest & greatest OOo2 running on my Kubuntu machine. Beautiful.

There you have it. I’ve noticed a significant increase in performance with the browser. I’m glad I upgraded to the new version. Also, I can tell a difference in the performance of my word processor. I just seems to run smoother.

I generally say if it ain’t broke don’t fix it. But in this case, I’d prefer Ubuntu to put these two applications in their update manager and be done with it.

I really shouldn’t complain. The price was right eh.