blog.erincandescent.net
On finding bugs in Ceph(X)
On the 28th August 2026, the Ceph project released fixes for CVE-2025-30156 and CVE-2026-39944. I first reported these to the project on the 20th December 2024; they were independently rediscovered in the meantime.
These are high risk bugs that are also _extremely_ difficult to remediate, and require coordination across a lot of projects (not just Ceph itself, but the Linux kernel, Rook, and other related projects). They also required implementing methods of doing cluster key rotations. On the one hand, I understand why it took so long to fix these; on the other hand, it is still disappointing that the resolution timeline took over 18 months. Nontheless, my thanks to the Ceph team for their handling of this.
These were human findings. Today, I expect you could throw an LLM at the Ceph codebase and these sorts of bugs would fall out quite quickly, but that wasn't the case in 2024.
## Background
Ceph RGW has functionality which integrates with OpenStacks KeyStone, and also emulates the AWS STS API. I was investigating how these worked, looking at the possibility of doing an integration. As I was doing this, I noticed that the same `CryptoKeyHandler` class was used for to hold keys for both HMAC-SHA256 MACs and AES-CBC encryption.
This piqued my curiosity; mixing keys with different algorithms like this is a code smell. In the end, the same keys are never reused with both, so while this is suboptimal design it isn't a security problem as written. However, this lead me to look at the AES-CBC encryption code, where I discovered that fixed keys were reused with hardcoded nonces.
AES-CBC is not an authenticated-encryption mode (meaning it does not protect against message alteration, though it does hinder it), and also degrades in security significantly when nonces are reused. As I discovered, this use of AES-CBC was the _only_ protection applied to messages protected with CephX, and to RGW session tokens.
As the CLYSO researchers who also found this bug demonstrated, it is possible to elevate privileges on vulnerable RGW installs with simple bitflips.
This is sadly not a complicated issue to find; any junior cryptographic engineer should have several klaxons going off in their head at the sight of a hardcoded nonce without good justification. I was not searching through the Ceph codebase for a critical security vulnerability; I just happened to stumble upon it.
## Takeaways
Ceph is an old project; the design of CephX likely predates the wide dissemination of the AEAD construct. We should pay attention to other codebases of similar vintage which also sometimes had to hand-roll their cryptographic protocols out of necessity.
