svtix upstream outage

| | Comments (1)
The fiber optic line card for our provider EGI Hosting's connection at SVTIX is apparently still having problems. EGI is working it out with XO, who owns the line. Here is what they said:
We apologize for any inconvenience this might have caused.
There seems to be a problem with one of the XO devices that one of our transport lines into SVTIX is connected to.
XO moved our port to another line card yesterday as it was rebooting itself.
The new line card we were allocated today had frozen and needed a restart and thus is the reason for today's outage.
We have been working very closely with XO in the past two days and they think that today's issue is not related to yesterday's issue. They think that a card reload was need in order to push all the recent configurations to the router and they say believe we should not be seeing this anymore. We have however asked XO to escalate today's event to their tier 3 support for further investigation.

This is the most up-to-date information we have on the situation. We certainly hope that XO have truly fixed the issue.
They should be getting back to us with a final confirmation later today or early tomorrow.

Please be advised that these issues are affecting every carrier in SVTIX that is connected through this XO device, it is not isolated to EGI's network only.
We will update this post again when we have more information.


We received further correspondence from EGI:

As of approximately 4:10pm today, XO has informed EGI that it finished replacing the bad line card in their SVTIX device.
EGI deeply apologizes for any inconvenience today’s outage may have caused.
We understand that such events may raise concerns in regards to the stability of the San Jose network.

EGI is routing bandwidth from various data centers into the facility and is relying on XO fiber for transport between data centers.
Yesterday, The line card on the XO device that is coming in from Fremont was failing by constantly rebooting itself.
EGI immediately opened a ticket with XO asking them to investigate the issue.
Upon investigation of the matter, XO moved EGI’s port to another line card and it was working fine for almost a full day.

Today the second line card started failing as well causing downtime for any carrier that is using that specific XO device.
After further investigation, XO dispatched a tech to the site and replaced the second line card in the device. Today’s outage took longer than expected as it was initially thought that the cause for the outage is due to a fiber between the two remote point.
XO have now confirmed that things should stable after replacing the second line card and that they will be closely monitoring the situation.
EGI will keep monitoring the issue closely and will send out an RFO as we receive it from XO.

Please be advised that today’s outage affected any carrier that is using the same XO device in SVTIX. This issue was not isolated to EGI’s network only.

Leave a comment

About this Entry

This page contains a single entry by nick published on September 14, 2010 2:26 PM.

NetBSD with Xen 4.0 (and Gentoo!) was the previous entry in this blog.

a discussion on the SLA is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.