Flying Circus Status

Flying Circus

All Systems Operational

About This Site

This is the global, public status page of the Flying Circus Internet Operations GmbH.

Privacy & Imprint

Uptime over the past 90 days. View historical uptime.

VM servers Operational

90 days ago

100.0 % uptime

Today

VM storage cluster Operational

90 days ago

100.0 % uptime

Today

VM backup Operational

90 days ago

100.0 % uptime

Today

Network and Internet uplink Operational

90 days ago

100.0 % uptime

Today

Central services Operational

90 days ago

100.0 % uptime

Today

Related external services Operational

Bitbucket Git via HTTPS Operational

Bitbucket Mercurial via HTTPS Operational

Bitbucket SSH Operational

GitHub Operational

Fastly Europe (FRA) Operational

Fastly Europe (AMS) Operational

Python Infrastructure pypy.org Operational

Operational

Degraded Performance

Partial Outage

Major Outage

Maintenance

System Metrics Month Week Day

Sample IMAP server

Fetching

Sample SSH server

Fetching

Past Incidents

Nov 16, 2018

Rolling out release 2018_034

Completed - The scheduled maintenance has been completed.
Nov 16, 00:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 15, 21:00 CET

Scheduled - Release 2018_034 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2018/r034.html for information about the specific changes.
Nov 14, 13:45 CET

Nov 15, 2018

Data center maintenance

Completed - The scheduled maintenance has been completed.
Nov 15, 22:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 15, 19:00 CET

Scheduled - Our data center has informed us about upcoming maintenance: its routing infrastructure needs to be updated. During this maintenance, brief connectivity interruptions may occur.

Data center staff will be there all the time to solve service problems immediately. We will also monitor our infrastructure closely during this maintenance and take appropriate measures when problems occur.
Nov 12, 10:17 CET

Nov 14, 2018

No incidents reported.

Nov 13, 2018

Rolling out release 2018_033

Completed - The scheduled maintenance has been completed.
Nov 13, 00:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 12, 21:00 CET

Scheduled - Release 2018_033 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2018/r033.html for information about the specific changes.
Nov 9, 17:27 CET

Nov 12, 2018

Maintenance work on power supply

Completed - The scheduled maintenance has been completed.
Nov 12, 13:45 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 12, 11:45 CET

Scheduled - Components in the rack's power supplies must be replaced due to hardware failure. We expect no service interruptions.
Nov 12, 11:33 CET

Nov 11, 2018

No incidents reported.

Nov 10, 2018

No incidents reported.

Nov 9, 2018

High internal latency - service degradation

Resolved - Apparently the cause was a misdirection of UDP packets intended for a virtual machine that was crashed and stuck in a non-functional state. This caused (after ARP expiry) that a lot of traffic that was directed to this machine (~30MBit/s) to be flooded to our whole network which in turn was amplified on KVM hosts through their network bridges into the VMs. This two-fold amplification caused a very high amount of CPU time being spend in the kernel, specifically on hosts with many virtual machines.

Restarting the VM caused the traffic flooding to stop. We are still analyzing the specific cause of the flooding to construct preventative measures in the future.
Nov 9, 15:36 CET

Monitoring - The situation has resolved after we rebooted a suspiciously acting VM.

We noticed a potential packet loop that caused at least one KVM server to spend almost all CPU time on dealing with the extreme packet flow and also suppressed regular traffic.

We're still investigating the cause.
Nov 9, 14:56 CET

Investigating - We are seeing parts of the network showing high latency causing monitoring alerts and service degradation. We are currently investigating the issue.
Nov 9, 14:40 CET

Nov 8, 2018

No incidents reported.

Nov 7, 2018

No incidents reported.

Nov 6, 2018

Rolling out release 2018_032

Completed - The scheduled maintenance has been completed.
Nov 6, 00:00 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 5, 21:00 CET

Scheduled - Release 2018_032 has not been rolled out yesterday due to an operator error. We apologize for that. We will redo that on Monday 2018-11-05, starting at 21:00 CET.

See http://flyingcircus.io/doc/reference/changes/2018/r032.html for information about the specific changes.
Nov 2, 14:11 CET

Nov 4, 2018

No incidents reported.

Nov 3, 2018

No incidents reported.

Nov 2, 2018

Rolling out release 2018_032

Completed - The scheduled maintenance has been completed.
Nov 2, 00:01 CET

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 1, 21:00 CET

Scheduled - Release 2018_032 is ready and will be rolled out during the specified timeframe.

See http://flyingcircus.io/doc/reference/changes/2018/r032.html for information about the specific changes.
Oct 30, 16:00 CET