Flying Circus

All Systems Operational

About This Site

This is the global, public status page of the Flying Circus Internet Operations GmbH


Privacy Policy

RZOB (production) Operational
90 days ago
99.97 % uptime
Today
VM servers Operational
90 days ago
100.0 % uptime
Today
VM storage cluster Operational
90 days ago
99.9 % uptime
Today
VM backup ? Operational
90 days ago
100.0 % uptime
Today
Network and Internet uplink ? Operational
90 days ago
99.98 % uptime
Today
Central services ? Operational
90 days ago
100.0 % uptime
Today
my.flyingcircus.io ? Operational
90 days ago
100.0 % uptime
Today
gitlab.flyingcircus.io ? Operational
90 days ago
100.0 % uptime
Today
Emergency Channels Operational
90 days ago
100.0 % uptime
Today
Hotline ? Operational
90 days ago
100.0 % uptime
Today
Support emails ? Operational
90 days ago
100.0 % uptime
Today
RZHAL (backup and development data center) Operational
90 days ago
100.0 % uptime
Today
VM servers Operational
90 days ago
100.0 % uptime
Today
VM storage cluster Operational
90 days ago
100.0 % uptime
Today
VM backup Operational
90 days ago
100.0 % uptime
Today
Network and Internet uplink Operational
90 days ago
100.0 % uptime
Today
Related external services Operational
Bitbucket Git via HTTPS Operational
Bitbucket Mercurial via HTTPS Operational
Bitbucket SSH Operational
GitHub Operational
Fastly Europe (FRA) Operational
Fastly Europe (AMS) Operational
Python Infrastructure pypy.org Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

Scheduled Maintenance

Rolling out release 2025_025 Jul 21, 2025 21:00-22:00 CEST

Release 2025_025 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

Posted on Jul 17, 2025 - 17:34 CEST
Jul 19, 2025

No incidents reported today.

Jul 18, 2025

No incidents reported.

Jul 17, 2025

No incidents reported.

Jul 16, 2025

No incidents reported.

Jul 15, 2025

No incidents reported.

Jul 14, 2025
Completed - The scheduled maintenance has been completed.
Jul 14, 22:30 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 14, 22:00 CEST
Scheduled - We will upgrade our own Gitlab instance to a new platform and software version. This requires some manual migration steps causing brief downtimes.

The Gitlab instances we host for particular customers as a managed service are not affected.

Jul 14, 12:05 CEST
Completed - The scheduled maintenance has been completed.
Jul 14, 22:01 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 14, 21:00 CEST
Scheduled - Release 2025_024 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

Jul 10, 22:58 CEST
Jul 13, 2025

No incidents reported.

Jul 12, 2025

No incidents reported.

Jul 11, 2025

No incidents reported.

Jul 10, 2025

No incidents reported.

Jul 9, 2025
Postmortem - Read details
Jul 11, 09:50 CEST
Resolved - We have solved the issue which manifested sporadically in two ways:

1. DNS queries that required our border routers to retrieve data from the internet timed out
2. Traffic from VMs to the internet that required NAT timed out

The issue was caused by a new transfer network that was introduced in the maintenance earlier today. It was generally carrying traffic to our data center, but traffic destined for addresses on the network itself was lost.

Our routers dynamically used transfer net addresses when sending traffic to the outside world (i.e. for DNS and NAT as described above). To resolve the issue we have changed our router configs to use source addresses from our networks which are properly routed.

We've tested the configuration in the redundant configuration using fail-overs and have seen the issues on affected machines subside.

We're sorry for the outage.

Jul 9, 00:07 CEST
Update - We have achieved a semi-stable routing state again, where connectivity has mainly been restored.
DNS resolution within RZOB is currently not stable.

Jul 8, 23:16 CEST
Investigating - Unfortunately, our triaging attempt caused additional connectivity issues, now also via IPv6. These issues only affect some machines, but the underlying cause is not identified yet.
Jul 8, 23:10 CEST
Identified - Due to the changes in the announced RZOB network uplink maintenance, the RZOB data centre lost some of its IPv4 connectivity.

We are triaging this by reverting the routing changes to the state before the maintenance right now to restore IPv4 connectivity.

Note: The revert itself will also briefly interrupt the data centre connectivity again.

Jul 8, 23:05 CEST
Jul 8, 2025
Completed - We have completed this maintenance.
Jul 8, 21:46 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 8, 21:00 CEST
Scheduled - We will be performing maintenance on the internet uplink in RZOB, our production datacentre, in cooperation with our provider in order to complete a pending internal migration. We will be monitoring the situation for any unexpected impact to customer services, in order to respond promptly should problems arise.
Jul 1, 08:45 CEST
Jul 7, 2025
Completed - The scheduled maintenance has been completed.
Jul 7, 22:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 7, 21:00 CEST
Scheduled - Release 2025_023 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.

Jul 3, 18:45 CEST
Jul 6, 2025

No incidents reported.

Jul 5, 2025

No incidents reported.