12th December 2021

Core Network Amsterdam Incident - Amsterdam AMS01

We are seeing a major outage on our AMS-01 location.

Our Team is investigating the issue with multiple suppliers as multiple redundant services seem to be down as well.

Update 15:27 - The Datacenter reports a large scale (presumably) power outage.

Update 16:10 - Our DC provider has given us the following update: "Our local field engineer has confirmed a power issue on site. Power specialists are currently dispatching on site"

Update 18:02 - Our DC provider has given us the following update: "Please note that power on site has been partially restored and hence, some services have restored as well Based on the info gathered from the field, we expect that most if not all the services are restored by 18h30 CET"

Update: 18:55 - Some links remain down, we are conducting remote hands operations to bring back those links.

Update 20:17 - RFO from the DC indicates a power grid issue, generators started but the switchover to the running generators was unsuccessful due to an issue with the transfer switch system.

Update 20:49 - Our Carriers and DC provider are still facing issues. The DC provider is "still rebooting and changing stuff around" according to the Datacenter.

Update 21:37 - We are currently partially re-routing traffic over our backup upstream and over Frankfurt. In addition, we are waiting on remote hands for switch troubleshooting.

Update: 22:50 - Switch troubleshooting has been completed. Most Transits and Peerings are restored.

Update: 23:48 - We see Amsterdam operational again. We are waiting on a reply from one Carrier whose Session is still down. Customers who still see issues with their services, please open a ticket with us.