Scheduled Maintenance: Servers Migration from Internap/Boston to Equinix/Secaucus

[FH-Dave]

The following maintenance has been scheduled for all shared hosting customers. Email notification has been sent to all shared hosting customers as well.


Dear Valued Customers,

Since June of 2004, Fluid Hosting has been building up and operating a secondary presence in another data center, in addition to the one we have in Internap Boston. This secondary presence is located within Equinix IBX data center in Secaucus, NJ (just 15 minutes off Manhattan, NYC). Equinix Secaucus data center is a tier-one carrier neutral data center, a $300 Million facility, boasting a total facility space of 200,000 square feet. Being a carrier neutral, Equinix gives us the freedom and flexibility to choose our network providers. Within this facility, we currently own a private cage and run our own BGP network using our own IP address space as assigned directly by the American Registry of Internet Numbers (ARIN).

Since June of 2004, we have been operating our presence in two separate data centers. The overhead of operating both presences has proven itself to be too high and inefficient to be indefinitely sustained. We believe that the time has come for us to start reconciling the two separate operations into one. This will allow us to concentrate all of our resources into sustaining only one single operation which will then further enable us to serve our customers better. Over the last eighteen months, we have been putting a considerable amount of resources, both financial and human resources, into building up and operating this secondary presence. We are happy to announce that our presence in Equinix/Secaucus has been growing to become our primary presence. Thus, we are planning to integrate our presence in Internap/Boston into Equinix/Secaucus.

This integration will imply a migration process for all customers in our Internap/Boston facility into our Equinix/Secaucus facility. We have been planning for this migration for the last few months and we believe we are ready to perform the migration within the next few days. Currently we have another set of servers in the Equinix/Secaucus facility to mirror the servers in the Internap/Boston facility. This will allow us to shorten the migration time and minimize all downtime by only migrating customers’ data rather than migrating the physical servers from one data center to another. The mail servers, the control panel, and one database server (db2/mysql2/pgsql2) are already located within Equinix/Secaucus data center. Thus, during this migration, no mail services will be affected. Web services (both UNIX and Windows) will be affected for all websites/customers currently residing in Internap/Boston. Furthermore, this migration will involve a change of IPs as we will be required to renumber all IPs to our own assigned IP space.

There will be some downtime involve. However, we will minimize all possible downtime. In short, the migration will involve two crucial steps:
1. Synchronizing all data between the source servers (in Internap/Boston) to the target servers (in Equinix/Secaucus). This will be done one day prior to the migration day. On the specified migration time, we will stop all services and synch all servers again for the last time. This will ensure that the target servers will be an exact mirror of the source servers.
2. Redirecting traffic from the source servers to the target servers. This step is important as the new servers will be on new IP address space. So, before DNS propagation is finished, we will need to redirect traffic that comes to the old IP address space to the new IP address space.

This migration process will be assisted by control panel developers, PSOFT. And the time slot that PSOFT has given us for this migration is this coming Thursday, December 8th. The migration will commence at 8 AM GMT (3 AM Eastern Time). We estimate the migration will around 5-8 hours to complete. We will work closely with PSOFT to ensure the smooth migration process.

For your information, the following services will be migrated:

UNIX servers:
web1.fluidhosting.com. Old IP: 66.150.196.125. New IP: 204.14.90.21
web2.fluidhosting.com. Old IP: 66.150.201.81. New IP: 204.14.90.22
web3.fluidhosting.com. Old IP: 66.150.196.130. New IP: 204.14.90.23
db1.fluidhosting.com. Old IP: 66.150.201.91. New IP: 204.14.91.11

Windows servers:
win1.fluidhosting.com. Old IP: 66.150.196.66. New IP: 204.14.91.21
win2.fluidhosting.com. Old IP: 66.150.196.200. New IP: 204.14.91.22
mssql1.fluidhosting.com. Old IP: 66.150.196.67. New IP: 204.14.91.16

Please note that for the web servers (web1, web2, web3, win1, and win2) the server IP is also the IP use by all domains using shared IP. The new shared IP will be the new IP listed above. If your domain(s) or sub-domain(s) use a dedicated IP, your IP will also change. As you may have to update your name servers, please contact support[at]fluidhosting.com if you need to know what your new dedicated IP will be. If you are using our name servers (ns1.fluidhosting.com, ns2.fluidhosting.com, and ns3.fluidhosting.com), there should be nothing you need to do.

We are aware that this migration is as important for you as it is for us. Therefore, to ensure the smooth migration process, the following methods of communication will be maintained.
1. You can reach us through email via support[at]fluidhosting.com.
2. We will open our support phone line 6 hours early, starting at 3AM Eastern Time (8AM GMT) on December 8th. This phone line will remain open until the end of our phone support hours coverage at 9PM Eastern Time or until the migration is done, whichever is later. To use our phone support, please dial (860) 656-6191 and select option #2.
3. You can also reach us at our IRC support channel. The IRC server is at support.fluidhosting.com and the channel name is #support.
4. We will also update the migration process in our community forum at http://forums.fluidhosting.com/showthread.php?p=14110.

Should you have any further questions and/or concerns, please do not hesitate to contact us at support[at]fluidhosting.com.

Sincerely,
System Administrators
Fluid Hosting, LLC

[FH-Dave]

I have created a thread for discussion around/about this migration process. The thread is located at http://forums.fluidhosting.com/showthread.php?p=14112 .

[FH-Dave]

Further information pertaining to the migration tommorow has been sent to all shared hosting customers. Below is the email sent:


Note: Please do not reply to this email directly as your email may not be attended to. Instead, please email us at support[at]fluidhosting.com should you have any further questions and/or concerns.

Dear Valued Customers,

We have received further information from PSOFT regarding the execution details of the migration process tomorrow, Thursday, December 8th, 2005. Please read the following important information carefully. Note: this notification only affects customers who are being migrated from Internap/Boston to Equinix/Secaucus. If your account already resides within our Equinix/Secaucus data center, then you can ignore this email notification.

As of few hours ago, we have started synchronizing the old and new servers. On the migration day, another synchronization process will be done to make sure that all file changes made between today and tomorrow will be copied over to the new server. The last synchronization will be brief as we expect a lot fewer files/data to be copied over. Once this last synchronization is completed, we will be ready to move into the second phase of the migration.

For Unix web servers: we will forward all traffic to port 80 and 443 (http and https traffic) from the old IPs to the new IPs. We will stop FTP and SSH service on the old web servers to prevent modification of files/data on the old web servers. You can update your files on the new web servers using the File Manager from your control panel. If you need to use FTP, please make sure that you are connecting to the new web server using the new IPs. During DNS propagation, request to the old web servers will be directly redirected to the new web servers.

For Windows web servers: we will not redirect traffic from the old servers to the new servers. Instead, we will let both web servers to serve requests. We will stop FTP service on the old web servers to prevent modification of files/data on the old web servers. You can update your files on the new web servers using the File Manager from your control panel. If you need to use FTP, please make sure that you are connecting to the new web server using the new IPs. During DNS propagation, some request will end up in the old web servers and some will end up in the new web servers. Thus keep in mind that during the DNS migration, you will have two copies of your web files, in the old web servers and in the new web servers. Furthermore, keep in mind that you can only update/modify your files in the new web servers. Thus, please consider doing updates/modification to your files after DNS has fully propagated. Please give at least 24 hours for a full DNS propagation. After migration has been done, we will also be synchronizing the old web servers to the new web servers at a regular interval of time to ensure that modification done to the new web server will also be copied to the old web server.

Due to the limited amount of time, PSOFT may have to postpone the migration of all the database servers--mysql1, pgsql1, and mssql1--until the next day, Friday. This means that once all web services have been migrated, you will be connecting to the web servers remotely for at the very most 24 hours duration. During this 24 hour period of time, you may receive additional latency for connecting to the remote database server. This additional latency should be minimal and may not be noticeable for most purposes.

For more discussion about this migration, please visit our community forums at http://forums.fluidhosting.com/showthread.php?t=2584. Should you have any further questions and/or concerns, please do not hesitate to contact us at support[at]fluidhosting.com.

Sincerely,
System Administrator
Fluid Hosting, LLC

[FH-Dave]

Updates:

Synchronization on the Unix web servers is done. We are now migrating the config files. Soon we will do IP migration. No downtime as of yet.

Synchronization on the Windows web servers has been started since around 3 AM. I have not heard anything back from PSOFT regarding the windows server. But no downtime on the windows server as of yet (that pertains to the migration).

Control Panel service has been stopped as part of the migration.

[FH-Dave]

Because CP needs to be taken offline, our helpdesk will not work. We have forwarded all support[at]fluidhosting.com to our secondary helpdesk system. If you have sent us tickets in the past 1.5 hours and have not gotten reply, please resend your email again. Thank you!

[FH-Dave]

IP Migration on Unix web servers is done. FTP service on the old web servers are being stopped. We will do one final last sync. Your websites should be up and running fine.

IP Migration on Windows web servers are not yet done. We are still migrating and synching contents between the old and new windows servers. We will keep everybody updated.

[jvreeland]

IP Migration on Windows web servers has begun, and with this migration CP has once again been taken offline.

We will update this thread as we get more info.

[FH-Dave]

The following email notification has been sent to all shared hosting customers.


Dear Valued Customers,

This email notification is to update you about the status of the web servers migration from Internap/Boston facility to Equinix/Secaucus facility performed on Dec 8th, 2005.

UNIX migration was completed on Dec 8th at around 12:15 PM Eastern Time (GMT -05:00). By this time, all IPs from the old server have been redirected/forwarded to the new servers. There was no downtime involved during this whole process. Some customers reported that their website index page was changed to the HSphere default utility page. This was quickly resolved soon after. There were some other specific issues on a small number of customers, which were all resolved right away.

Windows migration was not completed until few hours later. By around 5 PM Eastern Time (GMT -05:00), IP migration on windows servers was completed. Some number of customers also reported that their default website index page had been replaced by the HSphere default utility page. This was resolved as quickly as possible as we had to go through each domains/sub-domains folder to remove the default utility page.

In all, no websites had suffered a total service interruption (e.g. websites unreachable) during this migration process. Some anomaly was found after the migration (e.g. the reappearance of HSphere default utility page on customer websites folder, some script issues, etc), but these problems were all handled quickly, much shorter than the estimated 5-8 hours service interruption.

Please check your websites, your applications, and your scripts. Please make sure that everything works as expected on the new servers. Should you discovered any problems/anomalies, please contact support[at]fluidhosting.com immediately.

I would also like to take this opportunity to remind everybody that the database servers (mysql1, pgsql1, and mssql1) are yet to be migrated. The migration of these database servers will commence in another 3-4 hours, on December 9th, at around 4-5 AM Eastern Time (GMT -05:00). We will initially synchronize the new database servers to the old servers. Then we will stop the database server, and do one last synchronization to ensure that all data on the new web server is the most updated on. After the final synchronization is done, IP migration and redirection will take place. We estimate a small amount of downtime, around 30-60 minutes, for the final synch to happen. As before, efforts will be made to minimize all downtime.

Should you have any further questions and/or concerns, please do not hesitate to contact us at support[at]fluidhosting.com.

Sincerely,
Dave Tong
Fluid Hosting, LLC


Note: database servers (mysql1, pgsql1, and mssql1) are still to be migrated within the next few hours.

[FH-Dave]

mysql1 and pgsql1 will start to be synched to the new servers within the next 15-30 minutes. After which, both mysql1 and pgsql1 will be stopped for the last synchronization. PSOFT estimated the downtime to only be in minutes.

I am still waiting to hear about the status of mssql1 migration.

[FH-Dave]

Update: initial synchronization for mysql1 and pgsql1 services is done. All services are still up and running at this point. We will stop CP service and do IP migration momentarily. After this is done, we will stop the mysql1 and pgsql1 service and do final synchronization, which should be done within 5-10 minutes. Once done, the mysql1 and pgsql1 database server migration will be completed.

[FH-Dave]

mysql1 and pgsql1 migration has been completed. Please report if you see any problem.

[FH-Dave]

mssql1 migration has been completed as well. This concludes the scheduled migration. The migraion has been performed smoothly and extended downtime has been avoided. All customers and all services are now being served on our Equinix/Secaucus data center.

Please check your websites and make sure that everything is running correctly. Please report any problems/anomalies you see to support@fluidhosting.com.

An official update notification email will be sent to all customers shortly.

[FH-Dave]

The following post-migration notification email has been sent to all shared hosting customers.


Dear Valued Customers,

I am happy to announce that the migration of all shared hosting customers and all shared hosting services from Internap/Boston to Equinix/Secaucus has been completed. Yesterday all web services in Internap/Boston were successfully migrated. And today, the remaining database services in Internap/Boston has also been successfully migrated. Please check your websites, scripts, and any application you have running and make sure that everything work as expected. Please contact us at support[at]fluidhosting.com immediately should you discover any problems.

I am also happy to announce that the total downtime, service interruption, or any anomalies/problems resulting or expected as being part of this migration process have been kept to a minimum and is much better than we initially estimated. I would like to thank PSOFT team (the control panel developers) in assisting us with this migration. I would also like to thank our technical support staffs (Jeff Jones and Jeff Vreeland), our network administrator (Joseph West), and our system administrators (Haryono Tafianoto and John Yocum) for their hard work to ensure this migration be performed as smoothly as possible and all problems be resolved as quickly as humanly possible. All efforts have ben made as to minimze any impact this migration may have caused you, especially during this holiday season.

At this moment, all shared hosting customers and all services (web, database, and mail) are being served in our Equinix/Secaucus data center, where we have been expanding at a steady rate in the last 12 months. We will soon have all of our resources be concentrated in the Equinix/Secaucus facility and by doing so, we believe we can take our service to a higher level. And we are looking forward into this.

Thank you for your support and understanding with this migration process, despite the short notification time given. And should you have any questions/concerns regarding this migration process and the whole Equinix/Secaucus facility in general, feel free to contact me at dave[at]fluidhosting.com. I will be happy to answer any questions/concerns you have.

Thank you and have a nice hosting experience in our new facility!

Sinerely,
Dave Tong
Owner/Member
Fluid Hosting, LLC