Flying Circus

All Systems Operational

About This Site

This is the global, public status page of the Flying Circus Internet Operations GmbH


Privacy Policy

Uptime over the past 90 days. View historical uptime.
RZOB (production) Operational
90 days ago
99.99 % uptime
Today
VM servers Operational
90 days ago
100.0 % uptime
Today
VM storage cluster Operational
90 days ago
99.99 % uptime
Today
VM backup ? Operational
90 days ago
100.0 % uptime
Today
Network and Internet uplink ? Operational
90 days ago
99.98 % uptime
Today
Central services ? Operational
90 days ago
100.0 % uptime
Today
my.flyingcircus.io ? Operational
90 days ago
100.0 % uptime
Today
gitlab.flyingcircus.io ? Operational
90 days ago
100.0 % uptime
Today
Emergency Channels Operational
90 days ago
100.0 % uptime
Today
Hotline ? Operational
90 days ago
100.0 % uptime
Today
Support emails ? Operational
90 days ago
100.0 % uptime
Today
RZHAL (backup and development data center) Operational
90 days ago
100.0 % uptime
Today
VM servers Operational
90 days ago
100.0 % uptime
Today
VM storage cluster Operational
90 days ago
100.0 % uptime
Today
VM backup Operational
90 days ago
100.0 % uptime
Today
Network and Internet uplink Operational
90 days ago
100.0 % uptime
Today
Related external services Operational
Bitbucket Git via HTTPS Operational
Bitbucket Mercurial via HTTPS Operational
Bitbucket SSH Operational
GitHub Operational
Fastly Europe (FRA) Operational
Fastly Europe (AMS) Operational
Python Infrastructure pypy.org Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

Scheduled Maintenance

Rolling out release 2025_028 Aug 11, 2025 21:00-22:00 CEST

Release 2025_028 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.
Posted on Aug 08, 2025 - 08:29 CEST

NixOS CI/CD server ("Hydra") maintenance Aug 12, 2025 21:00-22:00 CEST

We are working on our NixOS platform CI/CD server ("Hydra") that generates our Linux distribution and provides Nix channel tarballs. There will be at least one downtime of 10 minutes in this period which can affect system rebuilds ("fc-manage"). Deployments might fail because of this.

The performance of running applications is not affected and package downloads are handled by our S3 object storage independently of Hydra.
Posted on Aug 06, 2025 - 13:37 CEST
Aug 9, 2025

No incidents reported today.

Aug 8, 2025

No incidents reported.

Aug 7, 2025

No incidents reported.

Aug 6, 2025
Storage cluster issues in RZOB
Resolved - The issue we've experienced yesterday has resurfaced today. We recovered the cluster within 14 mnutes and things where operational again.

We've traced the trigger for the issue to a potential race condition in the boot process of our Ceph servers and have disabled further automated maintenances that can trigger reboots until we have solved the issue.
Aug 6, 21:13 CEST
Aug 5, 2025
Storage cluster issues in RZOB
Resolved - This incident has been resolved.
Aug 5, 19:35 CEST
Update - We found a stuck storage daemon which Ceph did not properly remove from the cluster. Deactivating the stuck daemon unblocked the traffic immediately.

The direct impact was limited to VMs running in the SSD-class pool. As those typically are used for databases this had more widespread impact, especially on complex services leveraging clusters of VMs even if only one of VMs was affected.
Aug 5, 19:35 CEST
Monitoring - We've identified the issue and have taken action to resolve the problem. We're continuing to assess the situation.
Aug 5, 19:27 CEST
Investigating - We're seeing issues related to the Ceph storage cluster in RZOB.
Aug 5, 19:22 CEST
Aug 4, 2025
Rolling out release 2025_027
Completed - The scheduled maintenance has been completed.
Aug 4, 22:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Aug 4, 21:00 CEST
Scheduled - Release 2025_027 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.
Jul 31, 16:02 CEST
Aug 3, 2025

No incidents reported.

Aug 2, 2025

No incidents reported.

Aug 1, 2025

No incidents reported.

Jul 31, 2025

No incidents reported.

Jul 30, 2025
Supplementary Matrix chat server maintenance
Completed - The scheduled maintenance has been completed.
Jul 30, 22:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 30, 21:00 CEST
Scheduled - We will be performing maintenance on our internal staff Matrix chat server (matrix.flyingcircus.io), in order to complete the work started yesterday evening (2025-07-29). We do not expect the availability of the chat server to be affected during this maintenance.

Customer Matrix chat servers hosted as managed services are not affected by this maintenance.

If an emergency occurs with a project which has an SLA during this maintenance, our on-call support staff can be contacted through the emergency telephone hotline or by emailing the emergency support email, as documented on the "Emergency support sheet" page for the associated resource group in my.flyingcircus.io.
Jul 29, 22:35 CEST
Jul 29, 2025
Matrix chat server maintenance
Completed - We have successfully completed the maintenance for our customer Matrix server (customermatrix.flyingcircus.io), however due to issues related to DNS caching and propagation delay we were not able to fully finish the maintenance for our staff Matrix server (matrix.flyingcircus.io) as originally intended. We will complete this work on our staff Matrix server tomorrow evening after the affected DNS records have expired from intermediate resolver caches.
Jul 29, 22:29 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 29, 21:00 CEST
Scheduled - We will be performing maintenance on our internal Matrix chat servers, including our staff server (matrix.flyingcircus.io) and the server we host for customer use (customermatrix.flyingcircus.io). We do not expect the availability of the chat servers to be affected during this maintenance, but we will be monitoring for any unexpected impact.

Customer Matrix chat servers hosted as managed services are not affected by this maintenance.

If an emergency occurs with a project which has an SLA during this maintenance, our on-call support staff can be contacted through the emergency telephone hotline or by emailing the emergency support email, as documented on the "Emergency support sheet" page for the associated resource group in my.flyingcircus.io.
Jul 28, 16:12 CEST
Jul 28, 2025
Rolling out release 2025_026
Completed - The scheduled maintenance has been completed.
Jul 28, 22:00 CEST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Jul 28, 21:00 CEST
Scheduled - Release 2025_026 is ready and will be rolled out during the specified timeframe.

See the Changelog for information about the specific changes and impact of the release.

Note: releases themselves do not cause downtimes. If a change within a release requires downtime then every VM that is affected will schedule a maintenance period according to your projects' preferences and notify you with specific details about the downtime.

Check the "Impact" section of the changelog (linked above) to see potential causes of downtimes in a release.
Jul 24, 16:38 CEST
Jul 27, 2025

No incidents reported.

Jul 26, 2025

No incidents reported.

← Incident History Powered by Atlassian Statuspage