hm. the DoS looks like a Syn flood.

| | Comments (1)
I have no idea who the attackers were (other than the heaviest attackers were sending packets from china) or why the host was attacked. However, I do have packet dumps, and it looks like a simple syn flood. The attacker either has a large botnet, or is spoofing the source IPs, which makes it much less simple.

Now, this was my first DoS attack, so I was completely unprepared. I was figuring 'eh, I've got a 100Mbps commit. Don't worry about it' I didn't even have bandwidth monitoring setup. Midway through the attack I setup a SPAN port on my cisco and started capturing packets on a spare linux box.

I've spent a bunch of time going through tcpdump output with perl, essentially duplicating some of the really basic functionality of something like pmacct, and doing it badly. But I'm learning (and learning how to use pmacct) and even with my silly perl regexes and my tcpdump output, I am seeing some rather interesting patterns to the data. over a 5 hour period, I got 102755167 packets. 26149105 packets were tcp packets with a data payload of zero (which is how tcp connects look, and I believe how syn floods would also look.)

Here is a sampling of the tcpdump output: 15:37:09.233702 IP > S 2892936907:28929369\ 07(0) win 65535 15:37:09.233734 IP > S 1941102781:19411027\ 81(0) win 65535 15:37:09.764316 IP > S 2691934649:26919346\ 49(0) win 65535 15:37:10.295979 IP > S 2875418850:28754188\ 50(0) win 65535 15:37:10.825192 IP > S 2935806847:29358068\ 47(0) win 65535 15:37:11.354255 IP > S 1707919298:17079192\ 98(0) win 65535 15:37:11.885067 IP > S 955913454:955913454\ (0) win 65535 15:37:12.175995 IP > S 2892936907:28929369\ 07(0) win 65535 15:37:12.176011 IP > S 1941102781:19411027\ 81(0) win 65535 15:37:12.416718 IP > S 3541068542:35410685\ 42(0) win 65535

I also got a boatload of pings. (If I had my monitoring equipment up, and if I was paged when this started, I believe I could have killed it at my border pretty easily. Of course, if the pps kill upstream routers, that doesn't help, but if I have tools to blackhole IPs upstream, well, then it does help.) 20153524 'ICMP echo' packets, almost all 'length 72'

Also, from what I'm seeing, at least megs/sec, I was doing OK; the problem was that the routers couldn't handle the PPS. Now, obviously, it's completely irrational to think that just 'cause I'm buying 100Mbps of bandwidth that I can take that 100Mbps in the smallest packets I can send. (what's a tcp packet with no data? 64 bytes? yeah. that's a lot of packets) but it's not something I'd have thought to monitor before, as I'm usually charged on megs/sec.

I need to get something so I can announce null-routed /32s to my upstreams I honestly don't know what the standard procedure is, but I'd really prefer to have something I could do programatically, I mean, it's my IP space we are talking about blackholing, so it shouldn't require supervision

I believe that If I am doing my own BGP the best way to do it would be to have my upstreams configure their bgp routers to accept /32 routes from me, then I just announce my /22 or whatever as usual, then announce the /32s I need to kill with a null route. They will travel as far upstream as the routers are configured to accept /32s from me, which could be far enough that it becomes no longer our problem. That would be programatic and under my control.

of course, I'd rather null-route the source of the attack, but that becomes pretty difficult in cases like this where src IPs are either spoofed or coming from a large botnet.

That's the other thing; in the case of a syn flood like this, assuming that my upstreams could handle it without charging massive overages, (or without their routers falling over) I would put up a OpenBSD box with synproxy, and protect my customer. the question there becomes 'how many packets is too many'

I mean, I can 'finish the job' for the attackers, and kick off my customer, but that seems pretty unfair to me. (well, and the guy is paying me, after all. I like getting paid.)

I got lucky, though, the vast majority of my customers are on another link with another provider; only 18 Xen customers were taken down, and one co-lo customer. This was a 'cheap lesson' compared to what could have been. If I was this unprepared and someone hit my main uplink like this, I'd be in much worse shape. I mean, I've got some serious egg on my face, and I've seriously damaged the good reputation I've been building for the last two years (Yeah, I've been doing this for 4 years, but, well, until 2 years ago, i had serious reliability issues, that were fixed by the move to new hardware and internal disk.) Also the target of the DoS is a customer of a fairly large co-location customer. so it wasn't that cheap. But it certainly could have been a lot worse.

(all 18 xen customers are pretty new, and they all get a free month or a refund.)


I might be able to help you out with you BGP setup. The best tactic here would be using BGP Communities to tag the routes. If you want to talk more in detail, send me an email. (hopefully your comment administration has it in there.. not sure how moveable type works.. if not, visit my website and drop me a line. )

Leave a comment