The kernel downgrad has been verified in our lab and is currently being deployed in staging. We have cancelled all scheduled maintenance for 24.05 machines that were due before 22:00 CEST tonight so those will receive new schedules. The production release is expected to be ready around 16:00 CEST later today.
Some non-production VMs might still be experiencing slow network connections - those will be updated soon according to the staging maintenance cycle of the respective VMs.
Posted Sep 03, 2024 - 12:54 CEST
Update
We have reviewed the overall situation and have found that a number of our internal services are affected and that a few machines from a single customer have received the update but are not experiencing the issue in a noticable magnitude.
To prevent the issue from manifesting for our customers on the 24.05 platform we are
- postponing the currently scheduled maintenances for upgrades - downgrading the kernel to a known good release - prepare a hotfix release for later today for postponed maintenances to pick this up
Posted Sep 03, 2024 - 10:11 CEST
Identified
We have identified an issue that affects network performance. At the moment we see some internal networking scenarios being affected. The issue is caused by a Linux kernel regression that was introduced with the kernel update in our most recent platform release 2024_026.
We are currently deciding for a course of action and will give an update in an hour.
We do not currently see any more widespread issue affecting customer service availability.
Posted Sep 03, 2024 - 09:45 CEST
Update
We are continuing to investigate this issue.
Posted Sep 03, 2024 - 09:41 CEST
Investigating
Downloading from the gitlab docker registry is very slow right now. We are investigating.
Posted Sep 03, 2024 - 08:31 CEST
This incident affected: RZOB (production) (Network and Internet uplink), gitlab.flyingcircus.io, and RZHAL (backup and development data center) (Network and Internet uplink).