Resolved -
Between 00:15 and 00:26 our storage was unable to requests for a number of VMs hosted in the HDD class storage pool.
This happened after a regularly scheduled maintenance when two storage daemons did not cleanly reconnect to the cluster and caused requests to be stuck. We have seen this behaviour in the past but considered it resolved after having implemented workarounds. This has been proven untrue and we're now immediately disabling automatic maintenances until a further insights indicate a solution.
May 21, 00:00 CEST
Resolved -
During a routine update of our Configuration Management Database (CMDB), the service‑discovery mechanism broke. Service discovery is used, for example, to tell a virtual machine which NFS server it should mount.
The CMDB schema was altered in a way that was technically backward‑compatible, but the semantic meaning of several records changed. The existing service‑discovery software continued to run the previous version, interpreting the updated records under the old schema. As a result, it returned incorrect service endpoints.
We will review the update processes within our information security management system to prevent this kind of failure the future.
May 19, 09:40 CEST
Monitoring -
A fix has been implemented and we are monitoring the results.
May 19, 09:14 CEST
Update -
We are continuing to work on a fix for this issue.
May 19, 09:05 CEST
Identified -
Machines were not able to discover services causing outages.
May 19, 08:58 CEST
Completed -
The scheduled maintenance has been completed.
May 18, 23:00 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 18, 21:00 CEST
Scheduled -
Release 2026_018 is ready and will be rolled out during the specified timeframe.
See the Changelog for information about the specific changes and impact of the release.
Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.
Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.
May 15, 14:04 CEST
Completed -
We have managed to partially complete the maintenance activities planned for this evening, however due to unforeseen difficulties some changes could not be finished during the allocated time. These changes will be rescheduled for a later date.
May 12, 20:54 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 12, 19:00 CEST
Scheduled -
We will be performing maintenance on our routers in both RZOB, our production datacentre, and in WHQ, our backup and staging datacentre. We do not expect any interruptions to services but will be monitoring for any unexpected impact.
May 11, 15:49 CEST
Completed -
The scheduled maintenance has been completed.
May 11, 23:00 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 11, 21:00 CEST
Scheduled -
Release 2026_017 is ready and will be rolled out during the specified timeframe.
See the Changelog for information about the specific changes and impact of the release.
Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.
Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.
May 7, 21:31 CEST
Completed -
The scheduled maintenance has been completed.
May 7, 21:30 CEST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 7, 21:00 CEST
Scheduled -
Security releases for Nix and for the Linux kernel have been published which affect our NixOS platform CI/CD server ("Hydra") that generates our Linux distribution and provides Nix channel tarballs. There will be at least one downtime of 10 minutes in this period while we apply updates, which can affect system rebuilds ("fc-manage"). Deployments might fail because of this.
The performance of running applications is not affected and package downloads are handled by our S3 object storage independently of Hydra.
May 7, 11:47 CEST