We are wrapping up things from our side for today.
Here's our final report:
* Going through the application logs looking for anything related to SSL issues uncovered a handful of customer applications affected by third party servers providing the X3 chain and using OpenSSL 1.0 locally and we sent out advisories accordingly. * Our 15.09 platform has been updated to include the X1 certificate.
Feel free to contact us if you experience any further issues.
We'd like to thank you for your patience. Overall we are grateful that only very few high impact issues arose which could be fixed quickly. However, we are annoyed that we were not able to predict this better and will perform a review later.
Posted Oct 01, 2021 - 15:08 CEST
Update
A quick status update after this morning:
* Our Gentoo platform turned out to already have the current X1 certificate available in the trust store. Here compatibility will be relevant when acting as clients using OpenSSL 1.0 talking to servers that use certificates with the Let's Encrypt default chain. * NixOS 15.09 is currently going through our build and testing pipeline and will be released in the next hour or so. * We updated our batou_ext library to quickly allow choosing the Let's Encrypt short chain in the future if the need arises. * Up until now we only had to fix a small number of customer applications that showed signs of problems.
We will soon roll out the 15.09 platform update and will continue to verify customer applications during the remainder of the day.
Let us know if you encounter any issues!
Posted Oct 01, 2021 - 13:42 CEST
Identified
We are seeing compatibility issues for some applications with the new Let's Encrypt X1 certificate chain.
Customer services are generally available, however, there are some issues with older certificate trust stores and compatibility of the new root certificate chains with old SSL client software (e.g. OpenSSL before 1.1).
For today we are planning the following activities to ensure that your applications continue to run properly or are recovered from malfunctions as quickly as possible:
* Provide Let's Encrypt certificates based on the short chain for the relevant infrastructure components to ensure compatibility with OpenSSL 1.0 based systems. * Provide updated trust stores for our older platforms (Gentoo and NixOS 15.09). * Raising awareness with our ops team to quickly pick up compatibility issues that customers experience and provide advice as well as solutions depending on the situation. * Review all application logs and scan for SSL issues that our monitoring did not pick up otherwise. * Review customer deployments that might use outdated OpenSSL versions due to pinned versions.
Please contact us through the usual channels if you experience any issues related to our services or your applications.
We are sorry that we did not predict those compatibility issues and their impact early enough.