Data center upgrade
Scheduled Maintenance Report for Flying Circus
Completed
The scheduled maintenance has been completed.
Posted 20 days ago. Jun 01, 2018 - 00:43 CEST
Update
We have finished our data center migration. Despite us having prepared thoroughly we've hit a couple of issues (which we'll likely write a blog post about later) that caused intermittent downtimes. Public facing downtime was in total less than 20 minutes. Some backend services had issues recovering autonomously with status page downtimes of usually less than 45 minutes (two extreme cases showed 3 hours).

Almost all of our expected tasks have been implemented but we'll need to perform two additional network maintenances in the next weeks as some of the network components were not delivered in time by our vendor. Nevertheless, our network has already been upgraded with a 40g backbone infrastructure and some configuration adjustments caused a pleasant performance boost in our storage cluster.

Thanks for your patience and your business - and have a good night!
Posted 20 days ago. Jun 01, 2018 - 00:42 CEST
Update
We have removed the faulty switch from the cluster and relocated all affected servers to a new switch.
Posted 21 days ago. May 31, 2018 - 10:02 CEST
Update
One of our switches is behaving erratically when plugging cables. We're suspecting an electric grounding problem or capacitor issue. We're extracting the switch from our cluster and are moving affected servers over to another switch which will take around 10-15 minutes.
Posted 21 days ago. May 31, 2018 - 09:46 CEST
Update
Network issues resolved, storage is working again
Posted 23 days ago. May 29, 2018 - 12:05 CEST
Update
Currently seeing random network failures in the storage network which lead to flaky I/O. We are orking on the problem.
Posted 23 days ago. May 29, 2018 - 11:50 CEST
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted 23 days ago. May 29, 2018 - 09:00 CEST
Scheduled
We are consolidating our hardware and racks in the data center and will perform a longer period of maintenance over multiple days. We have prepared thoroughly for the migration to avoid any downtimes and will use this opportunity to further improve our network.

You can read more about this in our blog: https://blog.flyingcircus.io/2018/05/17/2018-05-29-2018-05-31-major-data-center-upgrade/
Posted about 1 month ago. May 17, 2018 - 16:38 CEST
This scheduled maintenance affected: VM servers, VM storage cluster, VM backup, Network and Internet uplink, and Central services.