NixOS: rebuilds temporarily deactivated

Incident Report for Flying Circus

Resolved

We have re-enabled access to the central build server again. NixOS-based deployments/updates requiring "fc-manage" are now able to work again.

Posted Mar 10, 2017 - 11:27 CET

Monitoring

We discovered (and fixed) a bug in our NixOS build server that manages which releases are visible to customers. This caused some customers to be provided with configuration that was intended for their staging environments. We have fixed the bug but disabled the access to our build server so that all machines are currently pinned to their version.

This will affect deployments calling out to 'fc-manage'. We are currently reviewing all affected machines to decide for a strategy how to re-enable our build server but avoid further service outages on the affected machines.

Posted Mar 09, 2017 - 15:27 CET