CVE-2023-6237

What it actually does

EVP_PKEY_public_check() on an overly large prime RSA modulus performs a slow primality test that can consume excessive CPU time, enabling denial of service when the key originates from an untrusted source. The OpenSSL SSL/TLS stack itself is unaffected; only direct callers of the function and the openssl pkey -pubin -check command are exposed.

Why it matters to the business

Services that validate RSA public keys supplied by external users may experience prolonged unresponsiveness or resource exhaustion, degrading availability until the process completes or is terminated. The issue has been public for over two years with stable low exploitation probability and no known ransomware or threat-actor campaigns.

How an attack unfolds

An attacker generates an RSA public key whose modulus is a single very large prime instead of a product of two primes, then supplies this key to any application that calls EVP_PKEY_public_check() on untrusted input. The function begins the expensive composite-number test and blocks for an extended period, starving worker threads or triggering watchdog timeouts.

How to fix it

Upgrade OpenSSL to the commit 0b0f7abfb37350794a4b8960fafc292cd5d1b84d or later; rebuild and redeploy all dependent applications and containers. Disable or restrict use of the openssl pkey -pubin -check command on untrusted files until patched. No configuration workaround exists.

How to confirm the fix worked

Confirm the installed OpenSSL library reports a build containing commit 0b0f7abfb37350794a4b8960fafc292cd5d1b84d, or run a timing test that submits a known large-prime RSA key and verifies the check completes within expected bounds. Monitor build pipelines for the updated source.

Typical attack path

1. Craft an RSA public key containing an excessively large prime modulus
2. Deliver the key to an application that invokes EVP_PKEY_public_check() on untrusted data
3. Trigger the primality test inside the function
4. Sustain high CPU usage until the test finishes or the process is killed

How to detect it

Review application and system logs for unusually long execution times or high CPU attributed to RSA key validation routines; correlate with inbound untrusted public-key material. No specific post-exploitation artifacts are produced.

What your security team should do

• Apply the fixed commit to all OpenSSL instances within standard change windows
• Inventory internal code and third-party libraries that directly call EVP_PKEY_public_check() on external keys
• Add timeouts or resource limits around any remaining unpatched key-validation paths
• Update dependency scanners to flag versions prior to the referenced commit