Limited availability of backups and backup reinitialization needed

Incident Report for Flying Circus

Resolved

Fresh full backups are running fine. We're closing the incident now and will follow up with a review later.

Posted Mar 28, 2017 - 20:35 CEST

Monitoring

We have finished all current restore requests. We have reinitialized the affected backup server and are running fresh full backups for all affected VMs. We'll inform affected customers about their VMs personally.

Posted Mar 28, 2017 - 17:20 CEST

Identified

The underlying filesystem of one of our backup servers has experienced a fatal error and has become unwritable. We currently have read-only access to those backups and are finishing current restore operations that were requested today.

After finishing those, we have to re-initialize the volume and start recreating fresh full backups from scratch. We will contact individual VM owners about which VMs were affected later today.

Posted Mar 28, 2017 - 16:26 CEST