Flying Circus

All Systems Operational

About This Site

This is the global, public status page of the Flying Circus Internet Operations GmbH


Privacy Policy

RZOB (production) Operational
90 days ago
99.96 % uptime
Today
VM servers Operational
90 days ago
99.99 % uptime
Today
VM storage cluster Operational
90 days ago
99.9 % uptime
Today
VM backup ? Operational
90 days ago
100.0 % uptime
Today
Network and Internet uplink ? Operational
90 days ago
99.97 % uptime
Today
Central services ? Operational
90 days ago
99.92 % uptime
Today
my.flyingcircus.io ? Operational
90 days ago
99.99 % uptime
Today
gitlab.flyingcircus.io ? Operational
90 days ago
100.0 % uptime
Today
Emergency Channels Operational
90 days ago
100.0 % uptime
Today
Hotline ? Operational
90 days ago
100.0 % uptime
Today
Support emails ? Operational
90 days ago
100.0 % uptime
Today
RZHAL (backup and development data center) Operational
90 days ago
100.0 % uptime
Today
VM servers Operational
90 days ago
100.0 % uptime
Today
VM storage cluster Operational
90 days ago
100.0 % uptime
Today
VM backup Operational
90 days ago
100.0 % uptime
Today
Network and Internet uplink Operational
90 days ago
100.0 % uptime
Today
Related external services Operational
Bitbucket Git via HTTPS Operational
Bitbucket Mercurial via HTTPS Operational
Bitbucket SSH Operational
GitHub Operational
Fastly Europe (FRA) Operational
Fastly Europe (AMS) Operational
Python Infrastructure pypy.org Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

Scheduled Maintenance

Rolling out release 2025_012 Apr 28, 2025 21:00-22:00 CEST

Release 2025_012 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

Posted on Apr 25, 2025 - 09:56 CEST
Apr 25, 2025

No incidents reported today.

Apr 24, 2025

No incidents reported.

Apr 23, 2025
Completed - The scheduled maintenance has been completed.
Apr 23, 00:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 22, 22:00 CEST
Scheduled - KAMP, our primary datacenter and upstream connectivity provider, will be performing router maintenance during this time. We do not expect any interruptions to services, but will be monitoring the situation for any unexpected impact.
Apr 17, 10:26 CEST
Apr 22, 2025
Resolved - Our fix has been rolled out over the last hours and has been stable.

We've removed our workarounds from early this morning and our telemetry shows no performance impacts after removing the workarounds, thus confirming our fix.

Please note: one of the workarounds was to disable processing of configuration management events. A number of events (like changes to VM resources, maintenance mails, some internal DNS records) have backed up over the day and have been processed in the last 15 minutes.

Apr 22, 17:44 CEST
Update - We have implemented a long-term fix for tonight's issue and are preparing to roll it out in our production cluster over the next hours.

This will happen in a staggered fashion, host by host, and will be transparent for our customers. We will disable the the short-term workarounds that we implemented earlier today when the rollout shows that our long-term fix is holding up.

Apr 22, 12:15 CEST
Update - We are implementing and testing a structural fix for the storage management software component.
Apr 22, 06:15 CEST
Update - We are continuing to work on a fix for this issue.
Apr 22, 06:14 CEST
Update - All VMs are back online. We are verifying the individual services now.
Apr 22, 06:13 CEST
Update - There are still some VMs offline. We are working on it.
Apr 22, 05:53 CEST
Update - The storage server traffic is back to normal now. We now have a look a the affected VMs and services.
Apr 22, 05:04 CEST
Update - We are still in the process of preventing the calls.
Apr 22, 04:53 CEST
Identified - An expensive metadata call seems to be the issue. We are about to prevent the specific call.
Apr 22, 04:28 CEST
Update - We are continuing to investigate this issue.
Apr 22, 04:10 CEST
Investigating - A single OSD ("Disk") shows unusual data transfer behaviour causing VMs which use the OSD to slow down significantly.
Apr 22, 03:59 CEST
Apr 21, 2025
Completed - The scheduled maintenance has been completed.
Apr 21, 22:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Apr 21, 21:00 CEST
Scheduled - Release 2025_011 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

Apr 17, 11:02 CEST
Apr 20, 2025

No incidents reported.

Apr 19, 2025

No incidents reported.

Apr 18, 2025

No incidents reported.

Apr 17, 2025
Completed - The scheduled maintenance has been completed.
Apr 17, 11:02 CEST
Scheduled - Release 2025_009 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

Apr 3, 14:12 CEST
Completed - The scheduled maintenance has been completed.
Apr 17, 11:01 CEST
Scheduled - Release 2025_010 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

Apr 10, 17:46 CEST
Apr 16, 2025

No incidents reported.

Apr 15, 2025

No incidents reported.

Apr 14, 2025

No incidents reported.

Apr 13, 2025

No incidents reported.

Apr 12, 2025

No incidents reported.

Apr 11, 2025

No incidents reported.