Read-only /tmp filesystems on some VMs
Incident Report for Flying Circus
Resolved
The bug in our Qemu instrumentation code has been fixed and all VMs are back to normal now.
Posted Sep 24, 2014 - 14:59 CEST
Monitoring
We have made a bugfix and are currently rolling out an improved version of our Qemu instrumentation code. The affected VMs will be rebooted soon.
Posted Sep 24, 2014 - 14:15 CEST
Identified
The Qemu instrumentation code which has been rolled out today contains a bug that is triggered by VM with long host names. The /tmp filesystem has been corrupted on approximately 10 VMs after the instrumentation code has started an unsolicited filesystem reinitialization for /tmp. Root filesystems (production data) are not affected. We are currently trying to reproduce this condition in a test environment and developing a bugfix.
Posted Sep 24, 2014 - 13:52 CEST
Investigating
We are currently getting reports that some VMs have their /tmp filesystems remount r/o due to a filesystem error. We are currently investigating what is going on.
Posted Sep 24, 2014 - 13:35 CEST