Flying Circus
All Systems Operational

About This Site

This is the global, public status page of the Flying Circus Internet Operations GmbH.

Privacy & Imprint

VM servers   Operational
90 days ago
100.0 % uptime
Today
VM storage cluster   Operational
90 days ago
100.0 % uptime
Today
VM backup   ? Operational
90 days ago
100.0 % uptime
Today
Network and Internet uplink   ? Operational
90 days ago
100.0 % uptime
Today
Central services   ? Operational
90 days ago
100.0 % uptime
Today
Related external services Operational
Bitbucket Git via HTTPS   Operational
Bitbucket Mercurial via HTTPS   Operational
Bitbucket SSH   Operational
GitHub   Operational
Fastly Europe (FRA)   Operational
Fastly Europe (AMS)   Operational
Python Infrastructure pypy.org   Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Sample IMAP server
Fetching
Sample SSH server
Fetching
Past Incidents
Nov 16, 2018
Completed - The scheduled maintenance has been completed.
Nov 16, 00:00 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 15, 21:00 CET
Scheduled - Release 2018_034 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2018/r034.html for information about the specific changes.
Nov 14, 13:45 CET
Nov 15, 2018
Completed - The scheduled maintenance has been completed.
Nov 15, 22:00 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 15, 19:00 CET
Scheduled - Our data center has informed us about upcoming maintenance: its routing infrastructure needs to be updated. During this maintenance, brief connectivity interruptions may occur.

Data center staff will be there all the time to solve service problems immediately. We will also monitor our infrastructure closely during this maintenance and take appropriate measures when problems occur.
Nov 12, 10:17 CET
Nov 14, 2018

No incidents reported.

Nov 13, 2018
Completed - The scheduled maintenance has been completed.
Nov 13, 00:00 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 12, 21:00 CET
Scheduled - Release 2018_033 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2018/r033.html for information about the specific changes.
Nov 9, 17:27 CET
Nov 12, 2018
Completed - The scheduled maintenance has been completed.
Nov 12, 13:45 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 12, 11:45 CET
Scheduled - Components in the rack's power supplies must be replaced due to hardware failure. We expect no service interruptions.
Nov 12, 11:33 CET
Nov 11, 2018

No incidents reported.

Nov 10, 2018

No incidents reported.

Nov 9, 2018
Resolved - Apparently the cause was a misdirection of UDP packets intended for a virtual machine that was crashed and stuck in a non-functional state. This caused (after ARP expiry) that a lot of traffic that was directed to this machine (~30MBit/s) to be flooded to our whole network which in turn was amplified on KVM hosts through their network bridges into the VMs. This two-fold amplification caused a very high amount of CPU time being spend in the kernel, specifically on hosts with many virtual machines.

Restarting the VM caused the traffic flooding to stop. We are still analyzing the specific cause of the flooding to construct preventative measures in the future.
Nov 9, 15:36 CET
Monitoring - The situation has resolved after we rebooted a suspiciously acting VM.

We noticed a potential packet loop that caused at least one KVM server to spend almost all CPU time on dealing with the extreme packet flow and also suppressed regular traffic.

We're still investigating the cause.
Nov 9, 14:56 CET
Investigating - We are seeing parts of the network showing high latency causing monitoring alerts and service degradation. We are currently investigating the issue.
Nov 9, 14:40 CET
Nov 8, 2018

No incidents reported.

Nov 7, 2018

No incidents reported.

Nov 6, 2018
Completed - The scheduled maintenance has been completed.
Nov 6, 00:00 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 5, 21:00 CET
Scheduled - Release 2018_032 has not been rolled out yesterday due to an operator error. We apologize for that. We will redo that on Monday 2018-11-05, starting at 21:00 CET.

See http://flyingcircus.io/doc/reference/changes/2018/r032.html for information about the specific changes.
Nov 2, 14:11 CET
Nov 4, 2018

No incidents reported.

Nov 3, 2018

No incidents reported.

Nov 2, 2018
Completed - The scheduled maintenance has been completed.
Nov 2, 00:01 CET
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 1, 21:00 CET
Scheduled - Release 2018_032 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2018/r032.html for information about the specific changes.
Oct 30, 16:00 CET