Flying Circus
All Systems Operational

About This Site

This is the global, public status page of the Flying Circus Internet Operations GmbH


Privacy Policy

VM servers Operational
90 days ago
99.98 % uptime
Today
VM storage cluster Operational
90 days ago
100.0 % uptime
Today
VM backup ? Operational
90 days ago
100.0 % uptime
Today
Network and Internet uplink ? Operational
90 days ago
100.0 % uptime
Today
Central services ? Operational
90 days ago
99.96 % uptime
Today
my.flyingcircus.io ? Operational
90 days ago
100.0 % uptime
Today
gitlab.flyingcircus.io ? Operational
90 days ago
100.0 % uptime
Today
Related external services Operational
Bitbucket Git via HTTPS Operational
Bitbucket Mercurial via HTTPS Operational
Bitbucket SSH Operational
GitHub Operational
Fastly Europe (FRA) Operational
Fastly Europe (AMS) Operational
Python Infrastructure pypy.org Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Scheduled Maintenance
Rolling out release 2023_012 Jun 5, 2023 21:00 - Jun 6, 2023 00:00 CEST
Release 2023_012 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

Posted on Jun 02, 2023 - 12:12 CEST
Past Incidents
Jun 2, 2023

No incidents reported today.

Jun 1, 2023
Completed - The scheduled maintenance has been completed.
Jun 1, 18:01 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jun 1, 10:00 CEST
Scheduled - We are starting to roll out our platform based on NixOS 23.05

This process will take off by switching all staging/non-production machines currently running fc-22.11-staging to fc-23.05-staging and then make necessary adjustments to prepare the roll out for production environments at a later date.

May 27, 23:41 CEST
May 31, 2023
Completed - The scheduled maintenance has been completed.
May 31, 00:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 30, 21:00 CEST
Scheduled - Release 2023_011 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

May 25, 22:21 CEST
May 30, 2023
May 29, 2023

No incidents reported.

May 28, 2023

No incidents reported.

May 27, 2023

No incidents reported.

May 26, 2023

No incidents reported.

May 25, 2023

No incidents reported.

May 24, 2023

No incidents reported.

May 23, 2023

No incidents reported.

May 22, 2023
Completed - The S3 object gateways have been restarted.

Further analysis revealed that the issue has arisen due to a logic error in our migration script. The script was dutifully tested in the last weeks but has not caught a specific ordering issue that prohibited the last round of flushes in our production environment.

As we temporarily re-enabled the S3 gateways to limit downtime this may have caused your application to have seen incomplete data in the buckets (likely to have seen them as errors where objects where not found). Data that was written in this period was consolidated by us.

May 22, 22:51 CEST
Update - The copy is still in progress as we had to work around a few inconsistencies that stopped the copy process. It's currently running smoothly and should finish in the next minutes. After wrapping up we will double check the results and re-enable the S3 gateways.
May 22, 22:34 CEST
Update - We have analyzed the situation and decided to go forward copying the remaining objects manually as we did not detect any data mismatches. We expect this to happen within the next 15 minutes and will update you further after that.
May 22, 22:18 CEST
Update - The pool had migrated almost all of the roughly 230 million objects in our pool in the last week. Unfortunately, finishing up the last 11k objects is currently giving unexpected errors and diagnosis is taking longer than expected. We currently need to keep the S3 gateways offline for the diagnosis and will update you here in at latest 30 minutes - hopefully with our new course of action.
May 22, 21:47 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 22, 21:00 CEST
Scheduled - This is the second part of the previously announced storage migration for increased data durability.

To finish the migration we need to take the object gateways offline. Ideally this will take only a few minutes, but copying the last data may also take longer and thus we schedule a 1 hour maintenance.

May 11, 10:31 CEST
May 21, 2023

No incidents reported.

May 20, 2023

No incidents reported.

May 19, 2023

No incidents reported.