Flying Circus
All Systems Operational
VM servers   Operational
VM storage cluster   Operational
Network and Internet uplink   ? Operational
Central services   ? Operational
Related external services Operational
Bitbucket Git via HTTPS   Operational
Bitbucket Mercurial via HTTPS   Operational
Bitbucket SSH   Operational
GitHub   Operational
pypi.python.org   Operational
Fastly Europe (FRA)   Operational
Fastly Europe (AMS)   Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Dec 15, 2017
Completed - The scheduled maintenance has been completed.
Dec 15, 00:00 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 14, 21:00 CET
Scheduled - Release 2017_031 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2017/r031.html for information about the specific changes.
Dec 13, 15:25 CET
Dec 13, 2017
Completed - The scheduled maintenance has been completed.
Dec 13, 00:00 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 12, 21:00 CET
Scheduled - Release 2017_030 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2017/r030.html for information about the specific changes.
Dec 11, 17:13 CET
Dec 11, 2017

No incidents reported.

Dec 10, 2017

No incidents reported.

Dec 9, 2017

No incidents reported.

Dec 8, 2017

No incidents reported.

Dec 7, 2017

No incidents reported.

Dec 6, 2017
Resolved - The situation has resolved. The overall issue was a "thundering herd" effect of many (NixOS) machines unexpectedly pulling updates from our build server.

Earlier today we updated our build server software. Unexpectedly this resulted in an instantaneous rebuild of our NixOS platform (not a real release: nothing actually changed but some files that included the build version number were replaced). This resulted in many machines pulling those updates at the same time and starting to update those files on their disks. This resulted in an overload situation of the storage cluster that lasted from around 16:45 to around 17:20 CET.

We're sorry for the inconvenience. As it happens we discussed general improvements for our management utilities earlier today that will reduce the impact of those "thundering herd" moments.
Dec 6, 17:36 CET
Investigating - We're currently seeing higher response times from customer services and high IO latency in our storage cluster. We're investigating the situation.
Dec 6, 16:55 CET
Completed - The scheduled maintenance has been completed.
Dec 6, 00:00 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 5, 21:00 CET
Scheduled - Release 2017_029 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2017/r029.html for information about the specific changes.
Dec 4, 11:55 CET
Dec 4, 2017
Completed - The upgrade was performed successfully. The downtime of the customer portal was around 10 minutes.
Dec 4, 21:23 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Dec 4, 21:00 CET
Scheduled - Tonight we will upgrade our customer portal infrastructure (my.flyingcircus.io) which will cause a few minutes of downtime. Customer services itself will not be affected by this. However, this may result in spurious alerts for any monitoring based on the portal's status pages.
Dec 4, 14:27 CET
Dec 3, 2017

No incidents reported.

Dec 2, 2017

No incidents reported.

Dec 1, 2017

No incidents reported.