Resolved -
The situation has stabilized since we switched to direct authoritative resolution. We are in touch with Quad9 and will implement further measures at a later time.
Dec 17, 11:43 CET
Monitoring -
Starting at around 3:15 CET we have seen increased errors on external DNS lookups.
Queries to our preferred upstream DNS provider started timing out. The issue improved at around 5:30 CET but has still been latently showing errors.
We have switched external DNS lookups away from Quad9, using direct authoritative access as a temporary countermeasure.
Dec 17, 08:39 CET
Completed -
The scheduled maintenance has been completed.
Dec 16, 22:00 CET
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 16, 21:00 CET
Scheduled -
Release 2024_035 is ready and will be rolled out during the specified timeframe.
See the Changelog for information about the specific changes and impact of the release.
Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.
Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.
Dec 16, 14:21 CET
Completed -
The scheduled maintenance has been completed.
Dec 12, 22:15 CET
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 12, 21:15 CET
Scheduled -
We are working on our NixOS platform CI/CD server ("Hydra") that generates our Linux distribution and provides NixOS channel tarballs. There may be a downtime of 10 minutes in this period which can affect system rebuilds ("fc-manage"). Deployments might fail because of this.
The performance of running applications is not affected and package downloads are handled by our S3 object storage independently of Hydra.
Dec 11, 13:58 CET