The bug in our Qemu instrumentation code has been fixed and all VMs are back to normal now.
Posted Sep 24, 2014 - 14:59 CEST
Monitoring
We have made a bugfix and are currently rolling out an improved version of our Qemu instrumentation code. The affected VMs will be rebooted soon.
Posted Sep 24, 2014 - 14:15 CEST
Identified
The Qemu instrumentation code which has been rolled out today contains a bug that is triggered by VM with long host names. The /tmp filesystem has been corrupted on approximately 10 VMs after the instrumentation code has started an unsolicited filesystem reinitialization for /tmp. Root filesystems (production data) are not affected. We are currently trying to reproduce this condition in a test environment and developing a bugfix.
Posted Sep 24, 2014 - 13:52 CEST
Investigating
We are currently getting reports that some VMs have their /tmp filesystems remount r/o due to a filesystem error. We are currently investigating what is going on.