network problems at SVTIX

| | Comments (6)

We've been working on it for 8 hours. will update when we know more.

Edit: looks like it /may/ be a Ddos. I mean, there /is/ a DDos of one of my customers, and that /may/ be why the router can't stand up. I asked my upstream to blackhole the target IP (I hate finishing the job for the attacker, but at the moment it's my only choice.) If that fixes it, we should be up within 1/2 hour. If not, then I will probably need to replace my router, a process that will probably take closer to 5 hours.

Yes, in fact it was a DDos. null routing the target at my upstream solved the problem.

I do want to make a personal apology for taking more than 8 hours to figure this out (and it was nick who deserves credit for finally figuring it out, not me) - I can explain some of it by the fact that the problem happened about the time I normally go to sleep, and the symptoms /looked/ a lot like the mac address conflict I had quite some time ago. But still, I should have figured this out in a half an hour. There's really no excuse.

As per policy, all effected customers will get 1 month credit. (I probably won't get the credits sent out for a few days, but you /will/ get them) - this will be painful, but not fatal.

6 Comments

Please don't send me my credit..

The DDOS attack was outside of your control, you guys clearly prioritised fixing it (from what I saw on IRC, you guys wasted your entire Sunday in a datacentre, and you gave us updates throughout the day on IRC) and my node isn't hosting much at the moment anyway.

I've never had a problem with Prgmr, and had I hosted anything important, the fact is, you guys are so cheap that I would have purchased a second node in another data-centre for redundancy. Any admin (with skills, I'm pretty-average and even I know) who needs high-availability would surely do this already, so I doubt many Prgmr node-owners would be really affected. I'm also sure though that Google could wait a few hours to discover there was no new content on my site ;)

The support/service I get is much more valuable than the server cost (when I wanted native IPV6 support for instance, Nick walked me through in painstaking detail on setting it up, I think I was probably the first, and when I needed an updated Ubuntu image, I was given one within a few hours).

Sorry, but you guys still deserve my money, and I don't believe 8hrs of downtime (for a problem outside your control) is worth a refund. Consider it a donation towards the 32bit-for-council-support(and council anti-reconnect patch for SSH XEN Menu)/new-prgmr-NFS-storage-service/add-a-bandwidth-monitor/rapid-response-to-rediculous-hackernews-complaints fund. Or just spend it on booze, seeing as your Sunday was ruined :P

I agree with Andrew Luecke, please do not credit my account for something you guys couldn't control.

thank you for your kind words; It has been a really rough past few days.

Thanks guys, you never fail to impress me with your professionalism and quality of service!

No harm done here. I don't need a refund either.

I don't get it, why do you keep your 99.5% SLA. We all know that the service is cheap. Personnaly for $115/y I'm not expecting a perfect support ect ...

Leave a comment

About this Entry

This page contains a single entry by luke published on June 20, 2010 7:20 AM.

sorry about the early past-due notices was the previous entry in this blog.

robe network driver is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.