Major storage software upgrade (Ceph)
Scheduled Maintenance Report for Flying Circus
Completed
The cluster has finished rebalancing and we've thus completed our maintenance.

We've seen a couple of performance-related notifications during the rebalancing. We're keeping an eye on cluster performance as the cluster has now achieved its steady state and will follow up separately if necessary.
Posted Feb 03, 2019 - 09:45 CET
Update
We've managed to complete most of the tasks of our update by now. There was with very little service impact.

The cluster is currently performing a large rebalancing of data which will take until tomorrow. Everything is operational so far and performance is well within the usual expected ranges.

If you should notice anything strange, let us know and we'll be happy to take a look.
Posted Feb 02, 2019 - 18:20 CET
In progress
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Feb 02, 2019 - 08:01 CET
Update
We are postponing our storage upgrade due to a last minute issue. We noticed intermittent severe performance limitations with application impact on our development and staging clusters and were not able to solve them.

This issue has the potential to cause major performance degradation or repeated intermittent loss of availability for customer applications and we decided to take the time to solve this issue before attempting to update.

Our predicted new window for the Ceph upgrade will be February 2nd/3rd and we'll keep you updated until then.
Posted Jan 11, 2019 - 17:42 CET
Scheduled
We have prepared a major version upgrade for our storage cluster software (Ceph).

Our preparation shows that we'll be able to roll out the update without downtime but may incur periods with reduced IO performance. We will also need to perform a data center wide live migration of all VMs to activate the new client software in our hypervisors.

The update has been tested, fixed all issues we found and evaluated it for multiple months. We have also performed multiple upgrades already ensuring suitability for production and reliability of our processes. Still, such a major update may incur issues in a production environment that may have not manifested before. We will be available all the time and monitor the systems closely to respond to any adverse affects as quickly as possible.
Posted Dec 17, 2018 - 09:58 CET
This scheduled maintenance affected: RZOB (production) (VM storage cluster).