VM server instabilities

Incident Report for Flying Circus

Resolved

This incident has been resolved.

Posted Sep 14, 2019 - 07:36 CEST

Monitoring

The downgrade was successfully finished yesterday evening. No more servers had been crashed/rebooted. The overall situation is stable now.

Posted Sep 13, 2019 - 14:43 CEST

Update

We are in the process of downgrading most VM servers now. We will leave some upgraded for further investigation, but will only run selected VMs there.

Posted Sep 12, 2019 - 15:00 CEST

Update

We will be rolling back a kernel update which was rolled out with release 2019_020.

Meanwhile now other VM server has been rebooted.

Posted Sep 11, 2019 - 17:29 CEST

Investigating

We are experiencing occasional (about once a day) VM server reboots. VMs running on a rebooting server are rebooting too, so application availability might be affected.

Posted Sep 11, 2019 - 11:57 CEST

This incident affected: RZOB (production) (VM servers).