On 2023-05-05 we had an incident that began with a faulty virtualisation host. During the recovery of the affected virtual machines we accidentally caused data corruption. Due to an operator error we misdiagnosed the situation of the failed host and caused the VMs to run on multiple hosts at the same time, which in turn resulted in corrupted file systems.
Overall, we found a compounding effect of issues that (each of them being reasonably small) in combination made the situation hard to understand and hard to respond to by our operators.
A regular maintenance for our my.flyingcircus.io system had been scheduled for 2023-05-05 06:45 CEST. This roll-out of a new software release was completed successfully by 08:18. While it is unrelated to the actual data corruption incident at a technical level, this software upgrade raised the general level of background noise in our monitoring system. This in turn might have influenced our operators' interpretations of certain monitoring data and influenced their responses.
Diagnosis: By 08:08, an operator got alarmed about an unresponsive virtualisation host - which in turn caused all VMs running there to also become unreachable. From that point on, several colleagues started triaging the issue together.
Our initial assessment of the issue was that the host had crashed, which has happened a few times in the recent past. This diagnosis was based on the observation that the host became unreachable both on our server-to-server network as well as the out of band management network. Incidentally all the monitoring of the host indicated it being completely offline. (The last conclusion later turned out to be faulty as the monitoring would show all checks as "out of date" as the connection to the monitoring system was offline.)
Triage: Our standard procedure for faulty hosts is to restart the affected VMs on other hosts. Our management system provides high level operations for this task and can cause hosts to be evacuated with a single click. However, to restart the VMs we had to manually break their locks. Those locks provide safety to avoid running multiple instances of a VM on different hosts, as that can cause damage to the data integrity of the underlying file systems. Breaking locks is a valid strategy in the situation of a crashed host as the VMs can not properly unlock their disk images any longer. Respectively, we broke the locks and the VMs were then able to start again on their new hosts.
With the VMs running again we turned back to diagnosing the original issue with the broken KVM host and tried accessing it on the remaining two interfaces (out of the 4 physical network interfaces in total). MGM and SRV had broken down (and are both located on the same physical network card). However, the dedicated storage interface (STO) was still available. When trying to log in using that interface we realised that the host had not actually crashed, but was still running. Which in turn meant, that the VMs on the host were also still running: the evacuation never reached the host as it relies on the SRV network. However, due to us breaking the locks and then starting the VMs on other hosts, we suddenly had 2 instances of every VM running at the same time. They concurrently wrote data to the same file system thus corrupting it in most cases.
Within seconds after realising our misstep we completely shut down the faulty host to limit any further damage.
When we reported the incident on our status page at 10:00 CEST, this immediate issue was already fully understood and resolved.
The short period of concurrent VM writes caused several cases of data corruption. While the VMs on the broken host were disconnected from the internet and thus not receiving any incoming traffic workload, some components like the system journal continued their periodical write operations and thus corrupting the file system.
Immediately after resolving the incident's root cause, we started fixing these data corruptions. On Friday, we concentrated on restoring the service of obviously broken VMs, i.e. where either the VM itself or a core service failed to start successfully. In most cases, repairing the XFS file system of the VM was successful without affecting customer data, and restored the availability of the VM service.
Some particular VMs were corrupted beyond what a file system repair could handle and had to be restored from their latest backup state. These backup restores were coordinated with every customer individually. As a result, data written to the VM in the meantime between backup and restore has been lost.
VMs not holding any meaningful local state have been routinely restored from their last backup as a precaution.
After services ran smoothly we started to perform routine filesystem verifications on Monday of all other VMs that did not exhibit specific symptoms.
Looking back after the fact it's easy to see that our incident response was not perfect. We identified a number of issues, apart from human error (which can and will always be an issue that can't be fully eliminated) where we see our processes and tools not having been helpful in this situation.
We intend to prevent similar problems from recurring again by taking the following counter measures:
We explicitly apologise for all inconveniences we caused by this incident. Let us know if you have any further questions by sending us an email to support@flyingcircus.io or through any other regular channel you are in contact with our team.