February 2014 Archives

I think we're almost done here.   ETA 20 minutes.

Okay, we're back.  as an explanation, chime has had unknown hardware issues that has been causing it to reboot.  I took the hard drives and put them in a newer HP DL160.  so you are now on an intel cpu and not a amd.  but I will give you more ram, as that server has 72gib vs. 32 on the old chime. 

Note, this is bad, even though the hardware is faster/better because I moved from amd to intel CPUs.  If you have a custom compiled kernel, this could cause you issues.  (if you are using a distro kernel, it should just work as it did before, except maybe a bit faster.)

Now, I need to catch my flight to scale.

reboot of crock and apples

| | Comments (0)
they are both running old mcp55 chipsets that have issues with hot-swapping drives;  I had to reboot them to replace a drive in each.  

An email went out to the users several hours back.  I should have given you more notice (and I should have given notice here)   sorry. 

Anyhow, it's done.  Everyone ought to be back up now, and the drives are rebuilding.  

another outage, likely unrelated

| | Comments (0)
So after the outgoing attack was squashed, there was a massive incoming attack.   The target of that attack has now been blackholed.   

I don't know if the two are related.   The outgoing attack was a compromised host;  I know the person and expect an incident report sometime tomorrow.    But the incoming attack, was it a response?  I don't know.   It was directed at a different IP, so probably not. 


note how it seems that just as I had the outgoing attack whipped, I got hit by the incoming attack.     The incoming attack was made more difficult to deal with by the fact that I don't, at the moment, have an out of band link.  This is... bad, as it made getting in to figure out what happened really hard.   

network outage report

| | Comments (0)
We were hit by a DoS attack last night;  or, rather, we were the source of a DoS attack last night, which caused intermittent packet loss.   The traffic was to and from Chinese IP addresses, so clearly, someone was doing it the old fashioned way and spoofing a source address.

Now, most of my Xen hosts do egress filtering.   But turns out I didn't have egress filtering enabled on my router.   My router is a debian box running quagga, so I whip up a few firewall rules.      Of course, I'm a good 10 minutes out from 55 s. market, so I put in a 'sleep 100' then a flush while testing the rules.  Everything looks good during this time period, so I do it again with a 'sleep 1000' 

Turns out my testing was faulty, and the network was down hard for approximately 1000 seconds.  

Yeah.  I feel like an asshole.

I will be getting help before enabling the rules again.

About this Archive

This page is an archive of entries from February 2014 listed from newest to oldest.

January 2014 is the previous archive.

March 2014 is the next archive.

Find recent content on the main index or look in the archives to find all content.