Service Alert on WIN2

[FH-Dave]

At this moment we seem to experience some sort of DOS attach to one of our windows server, WIN2. We are working to normalize the situation. The notification below has been sent to all customers.


Dear Customer,

We are currently experiencing a large inbound and outbound traffic to and from one of our windows web server, WIN2. The unusual traffic sustain at about 7 Mbps in both direction (in/out). Websites hosted on this server may experience itermittent problem. Only customers hosted on WIN2 is affected at this moment. If your website IP address falls in the range of 66.150.196.200-66.150.196.254, then you may be affected. All other servers/services are working normally.

We apologize for this itermittent problem. At this moment, we are working to normalize traffic from/to WIN2. When needed, we will also work with our upstream provider, Internap. In our attempt to resolve this problem, we may be required to reboot the server several times and/or to stop IIS/other services at WIN2.

Please do not reply to this email directly. Should you have any further questions/concerns, please do not hesitate to contact us at support[at]fluidhosting.com.

Sincerely,
Administrator
Fluid Hosting, LLC

[FH-Dave]

We are still working on this issue. There is a much added complication in resolving this problem. First, this seems to be a DDOS (Distributed Denial Of Service) attack, with IP origins vary greatly from one to another, thus making it practically impossible to block any particular IP. Second, the IP being targeted is the shared IP. We can not just null-route this IP since doing so will affect hundreds of web sites hosted on this IP.

We will keep working on this issue.

[FH-Hary]

A follow up e-mail sent to our customers last night.


Dear Customer,

This is a continuation on service alert sent earlier today. At this moment, we are still experiencing a high traffic going to/coming from WIN2. The traffic is still sustained at around 6 Mbps on each way. This seems to be a DDOS (Distributed Denial Of Service) Attack on the IIS running on WIN2, causing some itermittent problems with IIS. We have attempted to resolve this problem, however the complication has arisen due to the fact that the attack is a distributed attack and thus, we can not simply block an offending IP. Furthermore, the attack is being targetted on the shared IP (66.150.196.200) and we simply can not unbind this IP due to the many other websites sharing this IP.

IIS is not completely down although on ocassions, it can cause problem serving web pages. Due to the extreme emergency to resolve this problem, we plan to schedule a maintenance work that will start on 10:00 PM Eastern Time (GMT -04:00). During this maintenance window, we will stop all services on WIN2 to allow us install and configure a software-based firewall on WIN2. No ETA on how long this work will take. We expect no more than 2 hours.

As mentioned before, only websites hosted at WIN2 are affected. No other servers/services are being affected at this moment.

To those affected, we sincerely apologize for all the problems and inconveniences. We will keep working on this issue and will continue updating this situation through our community forums at http://forums.fluidhosting.com/showthread.php?t=1296 . Please note that you will need to register on our forums in order to see the customer-only forums.

Should you have any further questions/concerns, please do not hesitate to contact us at support@fluidhosting.com.

Sincerely,
Administrator
Fluid Hosting, LLC

[FH-Hary]

Another follow up to our customer. As of now, the issue still persists, however, we are keep working to block suspicious IPs with a lot of packets.


Dear Hary,

I would like to update you on the situation on WIN2. Right after we sent out the last notification, Internap has come to offer their assistance. Just before Internap stepped in, we were getting more than 12000 pps (packets per seconds) directed towards 66.150.196.200. This attack was even higher at peak times. Within the next 15-30 minutes, Internap was able to block some of the offending IPs that as targetted our server and reduced the packets to 66.150.196.200 down to roughly 2000-3000 pps. Currently, the traffic has subsided down from an of ~7 Mbps average down to ~ 2.5 Mbps.

I would like to specifically thank Chip Gywn from Internap, who is also our valued shared hosting customer. As soon as he received the email notification we sent out to all customers, he had made an initiative to contact us and offer his assistance.

We will keep monitoring the traffic/packets going to WIN2. If necessary we will still continue with the scheduled maintenance to install/configure a new software firewall on WIN2. In the near future, we will be investing in a hardware based firewall solution. An update to this will follow in the near future.

Should you still experience any troubles with your websites or if you have questions/concerns, please do not hesitate to contact us at support[at]fluidhosting.com. Please note, however, we may still go ahead with our scheduled maintenance on WIN2. During this time, your websites
may not be available.

Sincerely,
Administrator
Fluid Hosting, LLC

[FH-Dave]

WIN2 is being rebooted. Should be up shortly.

A thread has been opened for further discussions on this issue. Thread is located at http://forums.fluidhosting.com/showthread.php?t=1300