All day today, we’ve noticed some strange behavior on the wireless network at OS X con. Every so often, clients would get a “Connection Refused” when trying to go to a web page, or would be kicked off of iChat spontaneously, only to be reconnected a moment later. Hitting Reload would usually work, but would sometimes show another “Connection Refused”, and then just as mysteriously go away. Strangely, ssh connections seemed to work just fine, and never got dropped. Many other people (beyond the usual wireless ultra geeks) also reported similar problems.
We finally got tired of putting up with this state of affairs, and took a look at network traffic directly with tcpdump. What we found was very interesting. (Lines below are broken and highlighted for readability).
root@jojo~# tcpdump -eni en1 host 192.168.2.234 22:51:09.343951 0:30:65:aa:bb:cc 0:a0:c9:00:11:22 0800 74: 192.168.2.234.52774 > 216.92.122.183.80: S 4010194416:4010194416(0) win 32768(DF)
Here we see a client with the MAC address 0:30:65:aa:bb:cc and IP address 192.168.2.234 make an initial SYN setup request to a web page at 216.92.122.183, via the router with the MAC address 0:a0:c9:00:11:22.
22:51:09.344882 0:30:65:42:23:ff 0:30:65:aa:bb:cc 0800 54: 216.92.122.183.80 > 192.168.2.234.52774: R 0:0(0) ack 4010194417 win 0 (DF) 22:51:09.345665 0:30:65:42:23:ff 0:30:65:aa:bb:cc 0800 54: 216.92.122.183.80 > 192.168.2.234.52774: R 0:0(0) ack 1 win 0 (DF)
What’s this? Almost immediately (about .8ms later), a different host entirely (0:30:65:42:23:ff) sends two TCP RESET packets back to the client machine. How rude! Of course, the client machine aborts the current connection attempt (as a “Connection Reset by Peer”) and forgets about it.
22:51:09.417171 0:a0:c9:00:11:22 0:30:65:aa:bb:cc 0800 74: 216.92.122.183.80 > 192.168.2.234.52774: S 1491529510:1491529510(0) ack 4010194417 win 65535
Next we see the router (at 0:a0:c9:00:11:22) returning with the SYN ACK from the original web page…
22:51:09.418044 0:30:65:aa:bb:cc 0:a0:c9:00:11:22 0800 54: 192.168.2.234.52774 > 216.92.122.183.80: R 4010194417:4010194417(0) win 0
…which of course the client naturally refuses (sending a RESET back to the web site), since it has already received a RESET from the misbehaving peer!
22:51:09.863614 0:30:65:aa:bb:cc 0:a0:c9:00:11:22 0800 74: 192.168.2.234.52775 > 216.92.122.183.80: S 4246974705:4246974705(0) win 32768(DF)
And finally, since the connection didn’t go through the first time, the client retries, and the cycle repeats again, and again, and again… Until the client’s browser finally gives up a few seconds later.
The rude machine that sent the gratuitous RESETs above (0:30:65:42:23:ff) was actually my own laptop. But by logging traffic for a while, we found that arbitrary hosts on the wireless segment were also sending RESETs. What was the common variable on all of these machines? What we have been able to determine is that if any host on a wireless network is both running the Jaguar Firewall and running a program that throws the AirPort into promiscuous mode (like tcpdump, ngrep, etherpeg, or other network monitoring tool) then that machine will send arbitrary TCP RESETs for every packet that it sees on the wireless, even if it wasn’t destined for itself. Likely, this is because something in the firewall code sees the packets as a local destination (as the card is in promiscuous mode), even though it’s not really a local destination. This also explains why ssh connections were unaffected: most people have an exception for ssh in their firewall rules, and so packets destined for port 22 (the ssh port) wouldn’t ever get matched, and so wouldn’t get rejected.
This is an exceedingly easy thing to do, especially at this conference (where people like me are working with firewalls, monitoring tools, and wireless networks, and there are also many active wireless clients!) It is possible that this behavior would manifest itself on a wired network as well, if all of the clients involved were connected to a network hub (but not a switch). As wireless APs necessarily act as a hub, every client can see the traffic of every other, and hence can send responses to packets that weren’t destined for them. Unfortunately, we don’t have a hub to test with here at the conference.
Turning off firewalling immediately eliminates the problem, and turning it back on recreates it reliably. This is very odd behavior, as filtered packets should normally be dropped on the floor (and we should certainly not automatically send RESETs to addresses that aren’t involved with a locally bound address).
As Cliff Skolnick (who instigated tracking down the above strangeness) also points out, there is even more dementia when OS X 10.2 is set up as a router. As part of a nifty hack he’s presenting that enables a Bluetooth enabled Palm to use the network through his Titanium and over its wireless network to get to the Internet, he needs to enable packet forwarding:
root@jojo~# sysctl -w net.inet.ip.forwarding=1
Now, since he’s using a Titanium, he isn’t using the internal AirPort card (as the wireless range is, well, not what it could be) but instead is using an external PCMCIA card for his network connection. As soon as routing is enabled and promiscuous mode is turned on (say, by simply running tcpdump), it suddenly attempts to send ICMP redirects on every ICMP packet it sees on the wireless segment, redirecting them to the router it was destined for in the first place. This generates a huge amount of inadvertent traffic, with very little effort:
root@caligula:~# ping 192.168.2.234 PING 192.168.2.234 (192.168.2.234): 56 data bytes 64 bytes from 192.168.2.234: icmp_seq=0 ttl=64 time=4.916 ms 64 bytes from 192.168.2.234: icmp_seq=0 ttl=64 time=7.294 ms (DUP!) 64 bytes from 192.168.2.234: icmp_seq=1 ttl=64 time=12.44 ms 64 bytes from 192.168.2.234: icmp_seq=1 ttl=64 time=15.105 ms (DUP!) 64 bytes from 192.168.2.234: icmp_seq=2 ttl=64 time=9.754 ms 64 bytes from 192.168.2.234: icmp_seq=1 ttl=64 time=1020.15 ms (DUP!) 64 bytes from 192.168.2.234: icmp_seq=2 ttl=64 time=22.165 ms (DUP!) 64 bytes from 192.168.2.234: icmp_seq=1 ttl=64 time=1026.62 ms (DUP!) 64 bytes from 192.168.2.234: icmp_seq=1 ttl=64 time=1029.02 ms (DUP!) 64 bytes from 192.168.2.234: icmp_seq=1 ttl=64 time=1032.94 ms (DUP!) 64 bytes from 192.168.2.234: icmp_seq=3 ttl=64 time=6.755 ms 64 bytes from 192.168.2.234: icmp_seq=3 ttl=64 time=23.359 ms (DUP!) 64 bytes from 192.168.2.234: icmp_seq=2 ttl=64 time=1216.67 ms (DUP!)
Quitting tcpdump immediately makes this problem go away. This is very unexpected network behavior, and certainly bears more examination…
So, to sum up:
- If you’re running promiscuous mode tools, do not run a firewall in OS X 10.2
- If you’re acting as a router… DON’T run promiscuous mode utilities!
- If you’re at OS X con and are running Etherpeg (or ethereal or ngrep or ettercap or ntop any other utility that throws the card into promiscuous mode), and are running routing or firewalling, we will hunt you down and make you knock it off. ;)
Note that the actual MAC and IP addresses have been changed to protect the innocent. Thanks to Cliff Skolnick and the bunch of people hanging out on the mezzanine that helped get to the bottom of this! If you read this, and you’re at the conference, help spread the word to any other curious Etherpeg aficionados who may have their Firewall turned on…
Have you seen this strange behavior on Jaguar?
I have seen this strange behaviour
on wired ethernet. I thought it was a bug with chimera, but I've also had problems with my mail client (getting a wrong password error that mysteriously fixes itself). I'll try running the utilities mentioned here to see if I can find a similar pattern.
I'd be very interested in knowing the following things:
1) does the problem manifest itself when using Brickhouse instead of Apple's Firewall preference pane?
2) are you running both of these products (Brickhouse and Apple's Firewall) at the same time?
yes, I have seen this on a wired network as well
First, big thanks for the explanation
Some weeks ago we had a really big problem with our internet connection (adsl- bell sympatico, business 3mb) on our network. The connection went down very intermittently
We added a whole bunch of widows machines to the network recently, so I thought that was that.
I run some pings to our server (off-site) and a tcpdump and what can I tell, nobody was able to receive, or send mail, nor to browse the web, except my machine.
frantic calls to techhsupport at sympatico, resetting the router, resetting the modem didn't help much.
at one point I exited tcpdump and guess what, mail came in, mail went out, the web was accessible again.
Roberto
Rude Firewall behavior
Hi,
I had a similar problem. I have a G4 wired to a
router, an Airport that just routes packets to RF,
a sleeping power book using wireless and PPOE DSL on the
other side of the router. I was getting connection
refused and just like you, and had to try twice to access
a site. Turing off the Firewall fixed this
also. As far as I can see I was not running any
ethernet utilities just two browsers (Cyberdog and Mozilla)
--jim
oh, yeah, I've seen it
I posted about this on the Apple support forums a few days ago, and I just added a link to this article to prove I'm not crazy.
-j
Underlying flaw in BSD stack
I've seen this on wired networks as well.
The real issue is an underlying flaw in the BSD stack when a device is set to promiscuous mode.
In Open Transport (Mentat/TCP), each client of a network device (Data Link Provider) registers or binds with that device to indicate which packets it wants to receive. The Local IP stack should only want to see IP packets addressed to an IP interface, but the BSD stack has no such concept so when a device is set to promiscuous mode for some other tool, the resident TCP/IP stack sees everything and responds according to its rules by forwarding packets that are not addressed to it, sending ICMP redirects, TCP Resets, etc.
A possible work around is to configure ipfw
to reject IP datagrams that are not addressed to a local IP interface.
- Peter Sichel
www.sustworks.com
X 10.1.5, Airport in Pismo to WAP; Windows on Ethernet; both to Router, Asante DSL -- same thing
Same problem's been bedviling our home setup -- Alcatel DSL modem to a Linksys router; thence by ethernet to a Win laptop and by wireless to the Airport card in the Pismo.
It's happened forever, since we set up a year ago; usually just try again; when I'm using the wireless, sometimes it crashes Mozilla or hangs OSX for my Pismo; my wife's generic Windows laptop on Ethernet is more stubborn and doesn't die.
It's not Jaguar, or not only Jaguar.
Seen simular things
Especially with 10.1, I saw very evil things when I put an iface in promiscuous mode. RSTs, destunreach, etc.
Basicly, Mac OS X _should_ be dropping packets that are not to its MAC address (or a broadcast, multicast, or anycast address it is on) and not doing any IP processing on them. Instead, it goes ahead and does IP processing, which is wrong, wrong, wrong, wrong, and kills networks.
[Wonder how hard the patch to the darwin kernel is to fix this...]
Resets in Jaguar Stack
I've run into this as well (spent a day trying to figure out why our Sun boxes had all of a sudden stopped accepting connections). Had the netowrk down to the two main servers and a dell/linux box and my Pismo. Looked at the MAC addresses of the resets and noticed that although the IP was from the intended target the MAC was wrong, it was me!
Simple fix (but you have to do it each time you boot Jaguar) is to delete rule firewall 12180 (this is the offending one) if you are running anything promiscuous. (like snort which was my problem).
Have a similar problem with Airport, the "reset any setup" rule causes internet sharing to fail when the firewall is on. That fix is to insert a rule specifically allowing setup's over the Airport device (en1).
cheers.
carl.
Firewall hiccups even without promiscuous mode
There also seem to be conditions where the Jaguar firewall drops packets in a completely erratic manner. Don't know if this is related to the problem described here.
There's a thread in the BrickHouse support forum:
http://brianhill.dyndns.org/phpBB/viewtopic.php?topic=142&forum=2&4
I have the same problem at home
I have an iBook and a Titanium that have the same problem. I have an old Airport base station. My browser is Mozilla. If I have the firewall turned on for the Titanium, my iBook gives me the "connection refused" and it stops when I turn it off, or if I put the Titanium to sleep. This does not happen if the firewall is turned on for the iBook however.
I don't think it's Jaguar either...
I've seen similar problems, and they predate jaguar.
I'm using SSH port forwarding to connect between my home network, and a machine at work.
The connection keeps going down inexplicably with a "connection reset by peer" message. On my home network there is an airport, and some other ethernet based machines, connected by a plain vanilla hub. These go through an Alcatel firewall/dsl router.
At the other end, a similar setup - an open port in the firewall through to an internal network containing all sorts of stuff, including an airport base station.
It seems to make no difference whether I'm using my actual airport machine, or a machine on ethernet, to initiate the connection at home.
Other people in the company, using the same connection method and connecting to the same server, but coming in from a machine running Windows, don't seem to have any trouble. This leads me to think that something must be wrong at my end, but I can't figure out if it's the router, the airport, or the configuration of my macs. I'm not running a firewall on any of my mac's (unless there's something running on the airport), as they are all behind the Alcatel Firewall/Router.