Low network performance in recently updated VMs
Incident Report for Flying Circus
Resolved
The kernel downgrad has been verified in our lab and is currently being deployed in staging. We have cancelled all scheduled maintenance for 24.05 machines that were due before 22:00 CEST tonight so those will receive new schedules. The production release is expected to be ready around 16:00 CEST later today.

Some non-production VMs might still be experiencing slow network connections - those will be updated soon according to the staging maintenance cycle of the respective VMs.
Posted Sep 03, 2024 - 12:54 CEST
Update
We have reviewed the overall situation and have found that a number of our internal services are affected and that a few machines from a single customer have received the update but are not experiencing the issue in a noticable magnitude.

To prevent the issue from manifesting for our customers on the 24.05 platform we are

- postponing the currently scheduled maintenances for upgrades
- downgrading the kernel to a known good release
- prepare a hotfix release for later today for postponed maintenances to pick this up
Posted Sep 03, 2024 - 10:11 CEST
Identified
We have identified an issue that affects network performance. At the moment we see some internal networking scenarios being affected. The issue is caused by a Linux kernel regression that was introduced with the kernel update in our most recent platform release 2024_026.

We are currently deciding for a course of action and will give an update in an hour.

We do not currently see any more widespread issue affecting customer service availability.
Posted Sep 03, 2024 - 09:45 CEST
Update
We are continuing to investigate this issue.
Posted Sep 03, 2024 - 09:41 CEST
Investigating
Downloading from the gitlab docker registry is very slow right now. We are investigating.
Posted Sep 03, 2024 - 08:31 CEST
This incident affected: RZOB (production) (Network and Internet uplink), gitlab.flyingcircus.io, and RZHAL (backup and development data center) (Network and Internet uplink).