luke: February 2014 Archives

I think we're almost done here.   ETA 20 minutes.

Okay, we're back.  as an explanation, chime has had unknown hardware issues that has been causing it to reboot.  I took the hard drives and put them in a newer HP DL160.  so you are now on an intel cpu and not a amd.  but I will give you more ram, as that server has 72gib vs. 32 on the old chime. 

Note, this is bad, even though the hardware is faster/better because I moved from amd to intel CPUs.  If you have a custom compiled kernel, this could cause you issues.  (if you are using a distro kernel, it should just work as it did before, except maybe a bit faster.)

Now, I need to catch my flight to scale.

reboot of crock and apples

| | Comments (0)
they are both running old mcp55 chipsets that have issues with hot-swapping drives;  I had to reboot them to replace a drive in each.  

An email went out to the users several hours back.  I should have given you more notice (and I should have given notice here)   sorry. 

Anyhow, it's done.  Everyone ought to be back up now, and the drives are rebuilding.  

another outage, likely unrelated

| | Comments (0)
So after the outgoing attack was squashed, there was a massive incoming attack.   The target of that attack has now been blackholed.   

I don't know if the two are related.   The outgoing attack was a compromised host;  I know the person and expect an incident report sometime tomorrow.    But the incoming attack, was it a response?  I don't know.   It was directed at a different IP, so probably not.

note how it seems that just as I had the outgoing attack whipped, I got hit by the incoming attack.     The incoming attack was made more difficult to deal with by the fact that I don't, at the moment, have an out of band link.  This is... bad, as it made getting in to figure out what happened really hard.   

network outage report

| | Comments (0)
We were hit by a DoS attack last night;  or, rather, we were the source of a DoS attack last night, which caused intermittent packet loss.   The traffic was to and from Chinese IP addresses, so clearly, someone was doing it the old fashioned way and spoofing a source address.

Now, most of my Xen hosts do egress filtering.   But turns out I didn't have egress filtering enabled on my router.   My router is a debian box running quagga, so I whip up a few firewall rules.      Of course, I'm a good 10 minutes out from 55 s. market, so I put in a 'sleep 100' then a flush while testing the rules.  Everything looks good during this time period, so I do it again with a 'sleep 1000' 

Turns out my testing was faulty, and the network was down hard for approximately 1000 seconds.  

Yeah.  I feel like an asshole.

I will be getting help before enabling the rules again.

About this Archive

This page is a archive of recent entries written by luke in February 2014.

luke: January 2014 is the previous archive.

luke: March 2014 is the next archive.

Find recent content on the main index or look in the archives to find all content.