A buddy over at my favorite small ISP called me the other day: "Our T1
is running really, really slow!" I tried a
traceroute from the
outside world. While I hit the backbone's border router in 50
milliseconds, my packets took over 2 seconds to cross the T1 and hit
the ISP's router.
The network monitor showed that his T1 was actually doing just fine. After all, it was pushing about 1.544 megabits out and about 20 megabits in -- excellent throughput for a circuit that was running at capacity. The traffic had been averaging about half a T1 just the day before -- something had obviously changed.
Because every web server on the network is monitored, it was trivial to track down the offender. One of the web servers accounted for all that traffic. Five minutes with WebTrends gave him everything we needed, and we checked the site in question.
Nope, not porn. No MP3s. None of the usual suspects, just a small company with the typical half-dozen page site, a guestbook CGI, and a banner ad.
The banner ad was consuming a full megabit a second. The ultra-high-end FreeBSD server -- Celeron 433-MHz processor, 128 Mbytes of RAM, 20-Gbyte IDE hard disk -- wasn't even noticing the strain, but the T1 was buckling and every other site was slower than Microsoft admitting wrongdoing in the antitrust suit.
These days, a small Web company stays in business by maintaining a high level of customer service. My friend called the user. The client appreciated advance warning of the outrageous bandwidth charge he was going to get, and asked that we help trim the bandwidth used to a reasonable level.
And herein lies the problem. How do you control bandwidth usage? One thought rattled at the back of my brain, but I pushed it aside in favor of something that wouldn't touch the kernel.
We first checked in Apache itself. Apache has a module to control
You can easily install it just like any other port.
mod_throttle configuration is very simple. First, edit
to include the
mod_throttle configuration file.
throttle.conf file starts off by telling Apache to load the
throttle module, and to add it to the system. Reading the
mod_throttle documentation, we quickly arrive at a basic setup.
LoadModule throttle_module libexec/apache/mod_throttle.so AddModule mod_throttle.c ThrottleUser websiteusername speed 300 60 m
This worked, but was rather jerky. Pages would sometimes load quickly, sometimes slowly, and frequently be flat-out denied. It would be better to enable a smooth bandwidth restriction -- say, hook up this web site with a 128-Kbps bottleneck and let Apache behave normally. I've done this before by connecting an entire server to the network via PPP over a serial cable, but you can't disconnect one virtual domain from the rest of the server.
Also in Big Scary Daemons:
It's quite possible that
mod_throttle could have been tweaked to
perform better. I also suspect that on a large installation, the law
of large numbers would have taken over. Circuit bandwidth would not
have been irregular, because every throttled site would be behaving
Also, this Apache daemon was already heavily patched --
php4. I'm no Apache guru, and adding yet another
module into the mix isn't in my comfort zone. The simpler we keep the
systems, the better.
The next thought was to try shaping the traffic. The ISP has a Cisco router, after all. Ciscos can do this sort of thing. That's what you buy that expensive service contract for.
It turned out that we could have done this exactly as we liked on the Cisco -- if we upgraded to a more recent IOS. And added memory. Oh, you can't add more memory to a Cisco 2501? Would you like to speak to a salesman?
This is when that niggling thought in the back of my head stood up and screamed, "Dummynet!"
Dummynet was originally invented to simulate poor or lossy links, so
the author could test network protocols in bad conditions. However, it also allows you to control bandwidth usage. One example the developer uses
on his web page is using
dummynet to simulate an ADSL link to the moon! We don't want to
throttle our client that badly, but it's nice to know it's possible.
Articles about IPFW
One nice thing about
dummynet is that it works on arbitrary ports, IP
addresses, and protocols. If you want to restrict the bandwidth usage
of IPSec tunnels, Sendmail, or anything, you can do it.
Dummynet is part of IPFW. You can control all sorts of packet flows with
dummynet. One of the most
basic controls is the amount of traffic through a given IP address.
The IPFW integration was my problem. I'm loathe to muck with the kernel of a working production system. It's just not a good idea. Things have only gone wrong a couple of times, but those failures have been pretty spectacular. New kernel problems might only take a couple of hours to fix, but ISPs don't get downtime. Customers don't like outage explanations like "rebuilding the kernel." Their eyes glaze over, then and they find another ISP that doesn't use those "kernel" thingies.
You must have IPFW in your kernel to use
dummynet. I added the
following configuration to the system kernel.
options IPFIREWALL options IPFIREWALL_VERBOSE options DUMMYNET options IPFIREWALL_DEFAULT_TO_ACCEPT
Technically I didn't need the
IPFIREWALL_VERBOSE statement, but I like
logs. They give me a warm fuzzy feeling. I've been known to keep
logs for 30 months at a time (generally when I thought I was
rotating them daily, but was actually rotating them once a month).
Because we're not using IPFW for packet filtering, but for bandwidth control, I'm using the default accept mode. If you're doing packet filtering, you probably don't want this.
We can use two different types of
dummynet filters -- pipes and queues.
For this example, we'll use pipes. We'll look at queues some other
Dummynet expects to have some outside agency forward packets to it.
IPFW includes this functionality. For one
dummynet operation, you
need at least two IPFW configuration statements. If you aren't
familiar with IPFW commands, check out the FreeBSD Basics IPFW
dummynet, like all IPFW subsystems, with the
command. First, redirect the packets from the IP address of the site
(192.168.1.88) to a
ipfw add 100 pipe 1 ip from 192.168.1.88 to any
The syntax is pretty straightforward. We're adding a rule, number
100, and giving it a pipe. Packets that match
ip from 192.168.1.88
to any -- packets from this web site -- will be sent to pipe
Our pipe configuration is really very basic, too.
ipfw pipe 1 config bw 128Kbit/s
The pipe is created and labeled. The
config statement tells IPFW
that this is a configuration statement. The
bw means that this is a
bandwidth control and 128 Kbps is the bandwidth allowed.
The most disconcerting thing was when I checked the IPFW rule list.
# ipfw list 00100 pipe 1 ip from 192.168.1.88 to any 65535 allow ip from any to any #
Where's my pipe?
It turns out that the pipes are stored in a separate list. To view
your pipes, do
ipfw pipe list.
# ipfw pipe list
00001: 128.000 Kbit/s 0 ms 50 sl. 1 queues (1 buckets) droptail
mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
0 tcp 192.168.1.88/80 184.108.40.206/2415 128050681 35518324182 0 0 50486587 #
Wow, impressive! We can see where traffic is coming from, where it's going to, and how many packets have been passed and dropped.
Our rules covered the straightforward
dummynet configuration. There's
many more options that we'll look at another time. If you're
interested in experimenting with network congestion,
dummynet is your
I put this rule in and the bandwidth instantly dropped. My friend went away happy, his client didn't have to pay for a T1 for a month, and I got back to changing the rodent cages, watering the fish, and cleaning the garage.
One of the nice things about FreeBSD is that you get your software for free. You can use terribly inexpensive hardware. The one irremediable expense is bandwidth. FreeBSD lets you control your bandwidth usage as well, keeping your costs in check.
Michael W. Lucas
Read more Big Scary Daemons columns.
Return to the BSD DevCenter.
Copyright © 2009 O'Reilly Media, Inc.