Tuesday
The HELM server is currently on the receiving end of distributed denial-of-service (DDoS) attack.
Watch this short video, which explains what a DDoS attack is and why it's so damaging
Here's the latest update from Robert, the system administrator, this morning:
Having blocked most of the attacks throughout the night we managed to remove it from most of the network but the attack is still ongoing on two IPs 62.197.38.70 and 62.197.38.199 – the shared hosting IPs.
Both of these IPs are blackholed at the moment as the DDOS team is struggling to scrub the traffic successfully due to the complexity of this attack. Everything else apart from the above two IPs should be performing as expected but service may be slightly slower due to the work on the routers and switches. Our senior network engineer is still working with the DDOS team to customize the filters/rules to block the last of the attack and we hope to update you soon to say its been completed.
More informally, Robert has described this as "the mother of all attacks", so it really is an exceptional situation.
If your website is offline right now, it's because your domain is on the first of those two IP addresses: 62.197.38.70
Email has been working fine throughout this period - it's only website traffic that is currently affected.
This incident is not connected to the ongoing migration of services away from the HELM server, which has already begun and will continue between now and the end of 2014 when the HELM server will be decommissioned.
Update from the system administrator at 3.00pm:
Unfortunately this server is still under heavy attack and we have to block the IP completely while it is being mitigated so that it does not affect the rest of the network.
We then switch it back on to see if traffic is at an acceptable level. If not, we have to disable again and block more IP addresses.
I'm afraid this is an ongoing process as attackers used compromised computers of innocent people to do the attacks, and as you block one IP address, another pops up to take its place.
Update 6pm
We are still blocking the main shared server IP to protect the sites from these malicious attacks whilst we do more work on our firewalls. This afternoon was another attack from a different source (Korea) which has been targeting only our firewall. The high availability configuration we run is working but the load its coming under is huge due to the size of the attack. The firewalls are back online now and we are seeking to install a second pair which should allow us to spread and clean the traffic and open up the shared IP addresses again, the rest of the network and services are live and functioning.Wednesday 9.25am
The NTT predictive service we joined the network to last night is also removing over 80% of the traffic to let through the clean traffic, but 13k packets per second DDOS attacks are still getting through which is knocking out the server connections every time we bring the main shared server IPs back up. We have 3 engineers from our side and 4 from NTT all working on finding a resolution asap. Im afraid that all we can say.
Affected websites have now been restored, but the admins are still working on overall performance so please expect the service to continue to improve throughout the day.
Latest update from Robert on the front line this morning at 8.30am:
We are totally on top of it now, found the signature of the attack and blocking is still ongoing. We had to restrict packet sizes through our core switches to stop them in the end but this is meaning that some sites still aren't loading before timing out. Shouldn't be much longer im told whilst we remove these rules. 36+ hours straight, we are all definitely feeling it now...Thursday 6.10pm
The server has continued to suffer problems due to the ongoing attacks and some clients' sites are unfortunately almost completely inaccessible. Here's the latest from Robert, the system administrator.
We have had to block another IP on the shared server which came under attack this morning. As we started to remove the scraping rules to allow things like FTP and let the network be accessible for more locations it hit us again, this time with a different signature.As a result of the ongoing problems, we've accelerated the migration of domains away from the server, which was due to happen progressively at a relatively sedate pace over the next four weeks. Watch this space for further updates if you haven't already heard from me in person.
Saturday November 29th 1.35pm
It's been a heck of a week and there has been a great deal of disruption to various services at different times, but as a result of migrating websites away from HELM, along with the ongoing efforts of the system administrators, the vast majority of services have now been restored. But the attack has not stopped and so work to mitigate its effects is continuing. Again, contact me directly if you have any questions and in the meantime keep watching this space.
