Cloudanix VulnDB

CVE-2026-17507

High · 7.5

CVSS

7.5

EPSS

0.4%

Published

Oct 2, 2026

CISA due date

—

Cloudanix analysis

Next patch cycle SSVC: Track

A programming mistake lets one person inside a secure group chat send a tiny message that crashes the entire group's software, stopping everyone from communicating until the program is fixed or restarted.

Exploitation
Remote Unauthenticated
Attacker needs
None
Complexity
Low
Impact
Denial Of Service
Component
Library
Class
Improper Input Validation
Remediation
Upgrade
Effort
Medium
Patch risk
Low

What it actually does

Signed integer handling of RFC 9420 uint32 leaf_index in org.bouncycastle.mls allows negative decoded values to bypass membership checks in GroupKeySet.SecretTree.hasLeaf and Group.validateRemove, enabling unbounded NodeIndex.parent() traversal in LeafIndex.directPath that exhausts heap memory via a single in-group PrivateMessage.

Why it matters to the business

Any participant in an MLS group can trigger a denial of service that crashes the JVM for every other member, disrupting encrypted group communications until the affected service is restarted or the library is upgraded.

How an attack unfolds

An authenticated group member crafts a minimal PrivateMessage whose SenderData carries a leaf_index with the high bit set. The signed comparison accepts the out-of-range index, after which directPath arithmetic generates an ever-growing list of nodes without reaching the tree root, consuming all available heap and terminating the process for the entire group.

How to fix it

Upgrade Bouncy Castle bc-java to version 1.86 or later, which replaces direct signed comparisons with Integer.toUnsignedLong checks. Verify the updated jar is on the classpath of all MLS-using applications and restart affected services.

How to confirm the fix worked

Confirm the presence of bcprov-jdk*.jar version 1.86+, run the MLS interop test vectors that exercise full-range leaf indices, and verify that messages with high-bit leaf_index values are now rejected with an out-of-range error rather than accepted.

Typical attack path

  1. Join or be added to a target MLS group using Bouncy Castle MLS
  2. Send a single small PrivateMessage containing a leaf_index with MSB set
  3. Trigger the signed comparison that accepts the forged index
  4. Cause unbounded growth inside LeafIndex.directPath until heap exhaustion

How to detect it

Search application logs for repeated MLS PrivateMessage processing followed by OutOfMemoryError or sudden JVM termination; correlate with group membership events that show no corresponding membership change before the crash.

What your security team should do

  • Immediately inventory all deployments that ship bc-java and schedule upgrade to 1.86
  • Prioritize services that expose MLS endpoints or participate in multi-party groups
  • Add dependency version checks to CI pipelines to prevent regression to pre-1.86 builds
  • Monitor heap usage and JVM health for MLS processes until patches are applied

The record

In Bouncy Castle for Java before 1.86, the MLS implementation (org.bouncycastle.mls) holds RFC 9420's uint32 leaf_index in a signed int, so a wire value with the top bit set decodes to a negative number. That is a legitimate encoding rather than malformed input, and it must still decode, since the MLS interop test vectors round-trip the full range. GroupKeySet.SecretTree.hasLeaf and Group.validateRemove compared the decoded value directly against the tree's leaf count, and a signed comparison treats any negative int as less than a positive bound, so an out-of-range sender passed the membership check. In the hasLeaf case the SenderData of an unprotected PrivateMessage could then drive LeafIndex.directPath through NodeIndex.parent() arithmetic that never reaches the tree root, growing the resulting node list without bound until the JVM exhausted its heap. A single small message from any current group member could therefore deny service to every other member of the group. Both comparisons now interpret the value as unsigned via Integer.toUnsignedLong, rejecting an out-of-range sender however it was encoded; well-formed leaf indices are unaffected.

Affected packages

Ecosystem Package Introduced Fixed

Related vulnerabilities