Data center uplink not reachable
Incident Report for Flying Circus
Postmortem

On Tuesday, 2019-5-21, our uplink in the RZOB data center was offline for about 12 minutes, between 13:40 to 13:52. We’re sorry for the downtime.

We have been in touch with our data center and they explained that an erroneous configuration took down the uplinks of several customers, including the Flying Circus. They noticed the error quickly and reversed the configuration.

For the time being we trust in our data center’s ability to deliver highly reliable services as they have demonstrated continuously over the last years as their ability to evolve based on experiences like this week.

We’re also staying in touch with DC personnel to review why the routing redundancies that are in place have not helped avoiding this outage and whether further improvements can be made.

Posted May 24, 2019 - 08:33 CEST

Resolved
We haven't seen any further interruptions and are closing the issue now. We're expecting an analysis from our data center soon and will update the incident then.
Posted May 21, 2019 - 15:33 CEST
Update
Connectivity to RZOB is back and services are available again. We're reviewing any aftermath based on the tickets we received and our monitoring. We'll keep this incident open for another 30 minutes just in case that anything should happen again.
Posted May 21, 2019 - 13:55 CEST
Monitoring
The data center has restored the uplink. We have not yet been informed about the cause.
Posted May 21, 2019 - 13:53 CEST
Investigating
The RZOB data center is currently not reachable. We are in contact with the data center to resolve the issue.
Posted May 21, 2019 - 13:48 CEST
This incident affected: RZOB (production) (Network and Internet uplink).