Fluid Hosting Forums  
HomeWeb HostingSupportContact UsAbout UsForums |

Go Back   Fluid Hosting Forums > Customer's Forums > Maintenance Notifications > Unscheduled Maintenance / Critical Alerts

Reply
 
Thread Tools Display Modes
Old 02-05-2017, 03:44 PM   #1
FH-Vlad
Senior Member
 
Join Date: Mar 2006
Posts: 402
(Shared) All servers

We seem to have a switch issue on our cloud infrastructure causing a bunch of our shared hosting servers to be unreachable. We are aware of the issue and are currently working on it. A staff has been dispatched to the data center. We shall continue to update you here. Please accept our sincerest apology for this unexpected service interruption.

Last edited by FH-Vlad; 02-05-2017 at 03:53 PM.
FH-Vlad is offline   Reply With Quote
Old 02-05-2017, 06:30 PM   #2
FH-Dave
Fluid Hosting Staff
 
Join Date: Feb 2003
Posts: 3,302
We are still working on this issue. This is one strange issue, only affecting three servers on our cloud infrastructure. The three servers host quite a bunch of our shared hosting services. These three servers are connected to two public network switches for redundancy reason. One of the switch dies, traffic should have failed over to the other switch, but it did not. We are checking for possible misconfiguration isse on these servers at this moment.

Will continue to update everybody.
__________________
Do to others what you would want others do to you.
FH-Dave is offline   Reply With Quote
Old 02-05-2017, 07:50 PM   #3
FH-Dave
Fluid Hosting Staff
 
Join Date: Feb 2003
Posts: 3,302
Nothing new, we are still working on it. Still trying to solve some network configuration issues.
__________________
Do to others what you would want others do to you.
FH-Dave is offline   Reply With Quote
Old 02-06-2017, 12:17 AM   #4
FH-Dave
Fluid Hosting Staff
 
Join Date: Feb 2003
Posts: 3,302
Quick update: we are able to solve one of the issues with networking, but still working on the rest. Note: the servers have been up all this time, data integrity is intact. All of your data is safe. Only the network configuration is having issue.
__________________
Do to others what you would want others do to you.
FH-Dave is offline   Reply With Quote
Old 02-06-2017, 12:29 AM   #5
FH-Dave
Fluid Hosting Staff
 
Join Date: Feb 2003
Posts: 3,302
Quick update: we are able to solve one of the issues with networking, but still working on the rest. Note: the servers have been up all this time, data integrity is intact. Only the network configuration is having issue.
__________________
Do to others what you would want others do to you.
FH-Dave is offline   Reply With Quote
Old 02-06-2017, 03:35 AM   #6
FH-Dave
Fluid Hosting Staff
 
Join Date: Feb 2003
Posts: 3,302
We have found a way around, and are now starting all servers/services up. All services should resume fully shortly.
__________________
Do to others what you would want others do to you.
FH-Dave is offline   Reply With Quote
Old 02-06-2017, 03:50 AM   #7
FH-Dave
Fluid Hosting Staff
 
Join Date: Feb 2003
Posts: 3,302
cp.fluidhosting.com is up, but it seems the license file may get corrupted. We are working on it.
cp.nozonenet.com is up and is working fine.

Please check your website/database/mail and should you have any issue, please contact us at live support (fluidhoting.com).

So far everything looks stable. We will continue to monitor this.
__________________
Do to others what you would want others do to you.
FH-Dave is offline   Reply With Quote
Old 02-07-2017, 12:03 AM   #8
FH-Dave
Fluid Hosting Staff
 
Join Date: Feb 2003
Posts: 3,302
The following notification has been sent to HSphere customers:

Quote:
Suject: [Fluid Hosting Notification] Unplanned and Extended Downtime: Sunday, February 5th, 2017.

Dear Valued Customer,

You are receiving this notification because your service may be affected by the recent incidence occurring on Sunday, February 5th. Please note that this incidence affects most of our shared hosting customers, shared resellers, managed dedicated customers, and a couple unmanaged dedicated customers only. Below are the timeline of the incidence.

2:55 PM Eastern Time (GMT -05:00)
Our Citrix XenServer cloud infrastructure experienced issues which caused major service interruption to our shared hosting (fluidhosting and nozone HSphere) clusters as well as our own website, fluidhosting.com. Our senior system admin was paged.

3:30-4:45 PM
The cause of the issue was identified. The secondary public network switch had stopped responding. Since this cloud infrastructure was designed for redundancy, we were puzzled as to the reason the traffic did not fail over to the primary public network switch. Our other cloud infrastructures, based on AppLogic (also hosting few fluidhosting HSphere VMs), were not affected by this incidence and traffic did fail over to the other public switch as designed. Around this time we also realized that our VPN server, also hosted on this cloud infrastructure, was affected. For security reason, this VPN server is setup as gateway to remote access of our equipment. Thus, our remote system admins were no longer have the ability to access our network switches at this time. A staff was paged and dispatched to the data center. We posted update on our community forums, forums.fluidhosting.com.

6:00 – 11:30 PM
Visual inspection at the data center confirmed that the secondary public switch had died due to power supply failure. Attempts to make our system admin regain remote access to the nodes had failed. Further attempts to make network traffic fail over on the primary network switch had also failed. At this time, we decided that we would stop focusing on the cloud infrastructure and instead bring our VPN server up.

11:30 PM – 03:30 AM (Monday, February 6th)
VPN server was restored. We continued to fight some network configuration anomaly. But finally, we were able to gain remote access to all of our switches and cloud servers.

3:30 AM – 5:00 AM
We are resuming traffic to cloud servers and started to bring each individual virtual machines hosted on this infrastructure up one by one. Services started to resume at different time during this period. We continue to work on isolated issues. Overall, the issue had been fixed.

With that, we would like to take this opportunity to address few things. First, the Citrix XenServer cloud infrastructure had been very and distinctively reliable over the past years. This, unfortunately, has made us complacent. We failed to do periodic redundancy test of our Citrix XenServer cloud infrastructure. In addition, we trust our cloud infrastructure so much that we not only host your sites on it, we even host our website, our mail server, and, moreover, our VPN on it. As a result, this issue did not only caught us by surprise and off guard, the fact that service interruption to our website our VPN server had made the matter much worse. We were unable to communicated with our customers during this time. You were left out of the dark and for that reason, we truly apologize. For your future reference, we have a community forums (forums.fluidhosting.com) as well as secondary ticket system (https://support.fluidhosting.com/) that you may use in the rare event like this. Please accept our sincerest apology for this unexpected and extended service interruption and for all inconveniences this must have caused you. We are looking for ways to improve our services to you.

Second, at this moment we are setting up a new cloud infrastructure based on Virtuozzo7. Behind this technology is Virtuozzo’s proven Cloud Server and Distributed Cloud Storage. We are planning to rejuvenate our cloud infrastructure by migrating and consolidating all of our aging Citrix XenServer and AppLogic cloud infrastructure to this new Virtuozzo based cloud infrastructure.

Third, in addition, we are also migrating away from HSphere to Plesk. As you may be aware, new development on HSphere has practically ceased. This puts us in a difficult situation. Our HSphere cluster were no longer able to support newer Operating System and, as time goes, newer applications. Plesk, however, continues to be actively developed. As a result, we believe Plesk will offer much more to our customers.

We have been testing both Virtuozzo and Plesk for the past few months and will be rolling these new platforms within the next few weeks. By the end of the second quarter of this year we expect to finish migration of all shared hosting and managed dedicated customers to this new and exciting platform. Please expect further follow ups form us within the next few weeks with regards to this important matter.

Should you have any further questions and concerns, please do not hesitate to contact us by opening a ticket from your control panel.

Sincerely,
System Administrators
Fluid Hosting LLC
__________________
Do to others what you would want others do to you.
FH-Dave is offline   Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -5. The time now is 06:23 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Copyright© Fluid Hosting WebHosting Services