Fluid Hosting Forums  
HomeWeb HostingSupportContact UsAbout UsForums |

Go Back   Fluid Hosting Forums > Customer's Forums > Maintenance Notifications > Unscheduled Maintenance / Critical Alerts

Reply
 
Thread Tools Display Modes
Old 09-19-2011, 02:54 PM   #1
FH-John
Fluid Hosting Tech
 
FH-John's Avatar
 
Join Date: Feb 2003
Location: Washington
Posts: 4,177
Network Outage

Currently we're experiencing a network outage. We're working to quickly identify and resolve the problem.
__________________
John T. Yocum
"An empty stomach is not a good political adviser." -- Albert Einstein
"There never was a good war or a bad peace." -- Benjamin Franklin
FH-John is offline   Reply With Quote
Old 09-19-2011, 04:08 PM   #2
FH-Andrew
Fluid Hosting Staff
 
Join Date: Jul 2011
Location: United States
Posts: 46
We are currently still experiencing a network outage, we are working as quickly as possible to resolve the issue. Any updates will be posted here.
FH-Andrew is offline   Reply With Quote
Old 09-19-2011, 05:19 PM   #3
FH-John
Fluid Hosting Tech
 
FH-John's Avatar
 
Join Date: Feb 2003
Location: Washington
Posts: 4,177
The network is back up, it's been up for about 18 minutes globally. Others saw things come up sooner due to BGP convergence and damping delays. An RFO (Reason for Outage) will be sent once all information is gathered, and reviewed.
__________________
John T. Yocum
"An empty stomach is not a good political adviser." -- Albert Einstein
"There never was a good war or a bad peace." -- Benjamin Franklin
FH-John is offline   Reply With Quote
Old 09-19-2011, 05:35 PM   #4
FH-John
Fluid Hosting Tech
 
FH-John's Avatar
 
Join Date: Feb 2003
Location: Washington
Posts: 4,177
We've been in constant contact with a number of groups, datacenter support, upstream NOC, etc. Our upstream has identified a BGP bug which can cause a crash in some models of Juniper routers. They are going to be apply a filtering policy on their end to block the BGP update type that results in a crash. This change will cause a brief under 5 minute outage when they reset the BGP session.
__________________
John T. Yocum
"An empty stomach is not a good political adviser." -- Albert Einstein
"There never was a good war or a bad peace." -- Benjamin Franklin
FH-John is offline   Reply With Quote
Old 09-20-2011, 12:56 PM   #5
FH-John
Fluid Hosting Tech
 
FH-John's Avatar
 
Join Date: Feb 2003
Location: Washington
Posts: 4,177
The following notice has been sent out to customers:

Quote:
Dear Valued Customer,

At approximately 2:40PM ET, our network experienced an outage that effected our external connectivity for approximately 2 hours 20 minutes.

Upon receiving automated outage notification and verifying the outage, our staff immediately took action. Calls were placed to dispatch site technicions, along with contact made with upstream network operations staff.

At 2:50PM ET we made initial contact with Internap's network operations team which determined that our fiber crossconnect was up and running. In addition, the traffic level they had seen before the outage did not indicate any kind of denial of service attack. Their staff did identify that our BGP routing session (protocol used to distribute routing data within and between internet providers) was down. At this point, a ticket was opened to track the issue on their end.

At 3:10PM ET a datacenter technician was tasked to check the hardware on our end, and reboot the router as it appeared to have crashed. The router was rebooted, which took about 10 minutes to fully come up. At this point, we started to see some packet flow, and ping response. However, this was short lived as the traffic flow stopped after about 30 seconds. At this point, we prepared to replace the router with a hot spare, so an in house technician was routed to the location.

At 4:25PM ET our in house tech arrived, we noticed the network had begun to come up, but was still behaving badly. A remote console session was established with the router to determine what was going on. The router logs had shown that it was forcibly rebooted (which we had done) and things were returning to normal. We cleared the BGP damping table to speed up global reachability. While performing a diagnostic on the live router, the spare was prepared for a swap in.

At 4:27PM ET while we were preparing the new router, an email arrived from Internap asking for details on our router model, crash logs, etc. This was odd at first, as we hadn't indicated to them that we were using Juniper, nor had we updated them regarding the outage yet. After a brief exchange of emails with some senior network engineering folks, they revealed that they were aware of a bug in Juniper's BGP code that could cause a crash in some models of Juniper routers, as several of their other customers had been effected by this issue recently. Further they determined that the source of the crash was an invalid BGP update being distributed from another network, and that route update propagated the internet causing crashes within numerous routers. Since they had determined the source of the BGP update, their team setup filtering at their network level to prevent the issue.

In short, our network border routers crashed due to a BGP bug within Juniper's network operating system which was triggered by an invalid BGP update that originated from a remote network. The invalid BGP update caused routers to crash at a number of different networks, and for several of Internap's customers. Internap's network engineering team has placed filters on their end to block this bug, and advised us to contact Juniper's Technical Assistance Center.

Regards,
John Yocum
IT Manager
Fluid Hosting
__________________
John T. Yocum
"An empty stomach is not a good political adviser." -- Albert Einstein
"There never was a good war or a bad peace." -- Benjamin Franklin
FH-John is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 10:37 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright© Fluid Hosting WebHosting Services