Hurricane Sandy

FH-John · 10-29-2012, 03:15 PM

The following notification has been sent out:

Quote:

Dear Valued Customer,

As you may be aware, Hurricane Sandy is located in the northeast United States and is expected to make landfall today. However, rest assured our services will continue to function normally during the storm.

We have received confirmation from Equinix (our datacenter provider) that their redundant power and cooling systems have been tested to proper operating conditions, and that they will maintain on-site personnel through the storm. In addition, our network connectivity is fed via diverse fiber-optic paths to ensure continued operation in the event of cable damage. Finally, our support staff will continue to be available 24/7 during the storm, as our staff are in multiple locations. However, some inquiries may take longer to process in the event we have a temporary reduction in staff due to weather conditions.

If you have any questions, please submit an inquiry via your control panel's ticket system.

Regards,
Management
Fluid Hosting, LLC

FH-John · 10-29-2012, 09:51 PM

We have received notification from Equinix, that their NY2 facility has switched to generator power. The facility has sufficient fuel to run on generator for a long period of time, with additional fuel supplies and in place should they be needed.

FH-John · 10-29-2012, 10:20 PM

We have received notice that Equinix NY4 is now on generator power. The switch over was done smoothly, and no equipment in the facility was impacted.

FH-John · 10-30-2012, 04:16 AM

Due to sporadic power failures in the NYC area, some Internap POPs in Manhattan have lost power or experienced connectivity issues. This is resulting in route flaps, and reconvergence. Due to these failures you may notice sub-optimal routing, as networks route around failures in the region.

However, we are dual-homed with Above as well. They are currently operating normally in the region.

FH-John · 10-30-2012, 11:17 AM

Routing performance has returned to normal.

FH-Dave · 10-30-2012, 11:53 AM

We received updates from Equinix that all streets around NY2 were flooded. While the datacenter integrity was not compromised by the flood, NY2 was not accessible.

The flood seems to have cleared as of now. We are still on generator power both at NY2 and NY4. Equinix has enough fuel to last for another 2.5 days before refueling is needed.

FH-John · 10-30-2012, 01:51 PM

NY4 is currently not accessible due to flooded roads. In addition, a roof leak has been located at the building. However, the leak has no effect on us. Equinix staff are working to get the leak repaired.

gtvault · 10-30-2012, 02:47 PM

Thanks for keeping us up to date. I hope everyone affected by Sandy is faring as best as can be expected.

FH-John · 10-30-2012, 04:37 PM

We're seeing sub-optimal routing heading into the NYC area. You may notice reduced site performance due to these routing issues. Unfortunately, there isn't anything we can do as the routing problems are beyond our network.

FH-John · 10-30-2012, 05:17 PM

The streets around NY4 are no longer flooded.

FH-John · 10-31-2012, 12:09 PM

Equinix NY2 is scheduled for a fuel delivery today, as the facility only has another days worth of fuel on-site.

FH-Dave · 11-01-2012, 12:03 PM

Both NY2 and NY4 are still on generator. Fuel has been delivered and the generators are at 90% capacity.

FH-John · 11-01-2012, 07:00 PM

NY2 and NY4 continue to run on generator, and are scheduled for another fuel delivery tonight.

FH-John · 11-02-2012, 03:44 PM

Equinix has informed us that they expect utility power to be restored sometime this weekend.

FH-Dave · 11-05-2012, 12:40 AM

Just a quick update. As of Saturday evening, utility power has been restored on both NY2 and NY4. While on generator, NY2 and NY4 have never maintained less than 50% generator fuel capacity.

So in summary: there was some sub-optimal routing due Internap outages as their POPs in lower Manhattan were heavily affected by the Hurricane. But due to our redundant BGP network design, our network was never completely out. And while both NY2 and NY4 lost their utility power feeds, we experienced no power outage. Equinix's generators took over the power as it should and fuel deliveries were always maintained at regular intervals.

Our thoughts are with many other people that are affected by this Hurricane.

10-29-2012, 09:51 PM	#2
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	We have received notification from Equinix, that their NY2 facility has switched to generator power. The facility has sufficient fuel to run on generator for a long period of time, with additional fuel supplies and in place should they be needed. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

10-29-2012, 10:20 PM	#3
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	We have received notice that Equinix NY4 is now on generator power. The switch over was done smoothly, and no equipment in the facility was impacted. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

10-30-2012, 04:16 AM	#4
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	Due to sporadic power failures in the NYC area, some Internap POPs in Manhattan have lost power or experienced connectivity issues. This is resulting in route flaps, and reconvergence. Due to these failures you may notice sub-optimal routing, as networks route around failures in the region. However, we are dual-homed with Above as well. They are currently operating normally in the region. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

10-30-2012, 11:17 AM	#5
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	Routing performance has returned to normal. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

10-30-2012, 11:53 AM	#6
FH-Dave Fluid Hosting Staff Join Date: Feb 2003 Posts: 3,302	We received updates from Equinix that all streets around NY2 were flooded. While the datacenter integrity was not compromised by the flood, NY2 was not accessible. The flood seems to have cleared as of now. We are still on generator power both at NY2 and NY4. Equinix has enough fuel to last for another 2.5 days before refueling is needed. __________________ Do to others what you would want others do to you.

10-30-2012, 01:51 PM	#7
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	NY4 is currently not accessible due to flooded roads. In addition, a roof leak has been located at the building. However, the leak has no effect on us. Equinix staff are working to get the leak repaired. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

10-30-2012, 02:47 PM	#8
gtvault Registered User Join Date: Jan 2005 Posts: 20	Thanks for keeping us up to date. I hope everyone affected by Sandy is faring as best as can be expected.

10-30-2012, 04:37 PM	#9
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	We're seeing sub-optimal routing heading into the NYC area. You may notice reduced site performance due to these routing issues. Unfortunately, there isn't anything we can do as the routing problems are beyond our network. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

10-30-2012, 05:17 PM	#10
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	The streets around NY4 are no longer flooded. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

10-31-2012, 12:09 PM	#11
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	Equinix NY2 is scheduled for a fuel delivery today, as the facility only has another days worth of fuel on-site. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

11-01-2012, 12:03 PM	#12
FH-Dave Fluid Hosting Staff Join Date: Feb 2003 Posts: 3,302	Both NY2 and NY4 are still on generator. Fuel has been delivered and the generators are at 90% capacity. __________________ Do to others what you would want others do to you.

11-01-2012, 07:00 PM	#13
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	NY2 and NY4 continue to run on generator, and are scheduled for another fuel delivery tonight. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

11-02-2012, 03:44 PM	#14
FH-John Fluid Hosting Tech Join Date: Feb 2003 Location: Washington Posts: 4,177	Equinix has informed us that they expect utility power to be restored sometime this weekend. __________________ John T. Yocum "An empty stomach is not a good political adviser." -- Albert Einstein "There never was a good war or a bad peace." -- Benjamin Franklin

11-05-2012, 12:40 AM	#15
FH-Dave Fluid Hosting Staff Join Date: Feb 2003 Posts: 3,302	Just a quick update. As of Saturday evening, utility power has been restored on both NY2 and NY4. While on generator, NY2 and NY4 have never maintained less than 50% generator fuel capacity. So in summary: there was some sub-optimal routing due Internap outages as their POPs in lower Manhattan were heavily affected by the Hurricane. But due to our redundant BGP network design, our network was never completely out. And while both NY2 and NY4 lost their utility power feeds, we experienced no power outage. Equinix's generators took over the power as it should and fuel deliveries were always maintained at regular intervals. Our thoughts are with many other people that are affected by this Hurricane. __________________ Do to others what you would want others do to you.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode