Cloudanix VulnDB

CVE-2024-53169

None

CVSS

—

EPSS

0.5%

Published

Dec 27, 2024

CISA due date

—

Cloudanix analysis

Monitor only SSVC: Track

A flaw in Linux storage handling can cause computers to crash unexpectedly when shutting down certain network storage connections.

Exploitation
Other
Attacker needs
Local User
Complexity
Moderate
Impact
Denial Of Service
Component
OS kernel
Class
Race Condition
Remediation
Upgrade
Effort
Medium
Patch risk
Low

What it actually does

A race condition in the NVMe fabrics keep-alive handling during controller shutdown allows concurrent access to a deleted admin queue, resulting in a kernel crash.

Why it matters to the business

Systems using NVMe over Fabrics may experience unexpected kernel panics during controller shutdown operations, leading to service interruptions and potential data loss in storage environments. The issue affects a wide range of Linux kernel versions and impacts the stability of enterprise storage systems.

How an attack unfolds

An attacker or a misbehaving process triggers the shutdown of an NVMe fabric controller while a keep-alive request is being processed, causing the keep-alive work to access resources that have been freed by the shutdown path, resulting in a crash.

How to fix it

Upgrade to a Linux kernel version that includes commit 98f7e32f20d28ec452afb208f9cffc08448a2652 which moves nvme_stop_keep_alive() to nvme_remove_admin_tag_set().

How to confirm the fix worked

Verify the presence of the fix commit in the kernel source and test controller shutdown operations under load to ensure no crashes occur.

Typical attack path

  1. Initiate NVMe fabric controller shutdown
  2. Ensure keep-alive timer fires during shutdown window
  3. Trigger concurrent queue dispatch operations
  4. Observe kernel panic due to invalid queue access

How to detect it

Look for kernel panic logs containing nvme_keep_alive_work and __blk_mq_sched_dispatch_requests, or autoremove_wake_function in call traces during shutdown.

  • nvme_keep_alive_work
  • __blk_mq_sched_dispatch_requests
  • autoremove_wake_function

What your security team should do

  • Apply the kernel patch or update to fixed version immediately
  • Monitor for kernel crashes related to nvme_keep_alive_work
  • Review NVMe fabrics configurations for unnecessary keep-alive usage

The record

In the Linux kernel, the following vulnerability has been resolved: nvme-fabrics: fix kernel crash while shutting down controller The nvme keep-alive operation, which executes at a periodic interval, could potentially sneak in while shutting down a fabric controller. This may lead to a race between the fabric controller admin queue destroy code path (invoked while shutting down controller) and hw/hctx queue dispatcher called from the nvme keep-alive async request queuing operation. This race could lead to the kernel crash shown below: Call Trace: autoremove_wake_function+0x0/0xbc (unreliable) __blk_mq_sched_dispatch_requests+0x114/0x24c blk_mq_sched_dispatch_requests+0x44/0x84 blk_mq_run_hw_queue+0x140/0x220 nvme_keep_alive_work+0xc8/0x19c [nvme_core] process_one_work+0x200/0x4e0 worker_thread+0x340/0x504 kthread+0x138/0x140 start_kernel_thread+0x14/0x18 While shutting down fabric controller, if nvme keep-alive request sneaks in then it would be flushed off. The nvme_keep_alive_end_io function is then invoked to handle the end of the keep-alive operation which decrements the admin->q_usage_counter and assuming this is the last/only request in the admin queue then the admin->q_usage_counter becomes zero. If that happens then blk-mq destroy queue operation (blk_mq_destroy_ queue()) which could be potentially running simultaneously on another cpu (as this is the controller shutdown code path) would forward progress and deletes the admin queue. So, now from this point onward we are not supposed to access the admin queue resources. However the issue here's that the nvme keep-alive thread running hw/hctx queue dispatch operation hasn't yet finished its work and so it could still potentially access the admin queue resource while the admin queue had been already deleted and that causes the above crash. The above kernel crash is regression caused due to changes implemented in commit a54a93d0e359 ("nvme: move stopping keep-alive into nvme_uninit_ctrl()"). Ideally we should stop keep-alive before destroyin g the admin queue and freeing the admin tagset so that it wouldn't sneak in during the shutdown operation. However we removed the keep alive stop operation from the beginning of the controller shutdown code path in commit a54a93d0e359 ("nvme: move stopping keep-alive into nvme_uninit_ctrl()") and added it under nvme_uninit_ctrl() which executes very late in the shutdown code path after the admin queue is destroyed and its tagset is removed. So this change created the possibility of keep-alive sneaking in and interfering with the shutdown operation and causing observed kernel crash. To fix the observed crash, we decided to move nvme_stop_keep_alive() from nvme_uninit_ctrl() to nvme_remove_admin_tag_set(). This change would ensure that we don't forward progress and delete the admin queue until the keep- alive operation is finished (if it's in-flight) or cancelled and that would help contain the race condition explained above and hence avoid the crash. Moving nvme_stop_keep_alive() to nvme_remove_admin_tag_set() instead of adding nvme_stop_keep_alive() to the beginning of the controller shutdown code path in nvme_stop_ctrl(), as was the case earlier before commit a54a93d0e359 ("nvme: move stopping keep-alive into nvme_uninit_ctrl()"), would help save one callsite of nvme_stop_keep_alive().

Affected packages

Ecosystem Package Introduced Fixed
Debian:13 linux 0 6.12.3-1
Debian:14 linux 0 6.12.3-1
Linux Kernel 6.11.0 6.11.11
SUSE:Linux Enterprise High Availability Extension 15 SP6 kernel-default 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-ESPOS kernel-64kb 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-ESPOS kernel-default 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-ESPOS kernel-default-base 0 5.14.21-150500.55.94.1.150500.6.43.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-ESPOS kernel-docs 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-ESPOS kernel-obs-build 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-ESPOS kernel-source 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-ESPOS kernel-syms 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-LTSS kernel-64kb 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-LTSS kernel-default 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-LTSS kernel-default-base 0 5.14.21-150500.55.94.1.150500.6.43.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-LTSS kernel-docs 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-LTSS kernel-obs-build 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-LTSS kernel-source 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise High Performance Computing 15 SP5-LTSS kernel-syms 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Live Patching 15 SP5 kernel-default 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Live Patching 15 SP5 kernel-livepatch-SLE15-SP5_Update_23 0 1-150500.11.3.1
SUSE:Linux Enterprise Live Patching 15 SP6 kernel-default 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Live Patching 15 SP6 kernel-livepatch-SLE15-SP6-RT_Update_7 0 1-150600.1.3.1
SUSE:Linux Enterprise Live Patching 15 SP6 kernel-livepatch-SLE15-SP6_Update_7 0 1-150600.13.3.1
SUSE:Linux Enterprise Micro 5.5 kernel-default 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Micro 5.5 kernel-default-base 0 5.14.21-150500.55.94.1.150500.6.43.1
SUSE:Linux Enterprise Micro 5.5 kernel-rt 0 5.14.21-150500.13.85.1
SUSE:Linux Enterprise Micro 5.5 kernel-source 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Micro 5.5 kernel-source-rt 0 5.14.21-150500.13.85.1
SUSE:Linux Enterprise Module for Basesystem 15 SP6 kernel-64kb 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Basesystem 15 SP6 kernel-default 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Basesystem 15 SP6 kernel-default-base 0 6.4.0-150600.23.33.1.150600.12.14.1
SUSE:Linux Enterprise Module for Basesystem 15 SP6 kernel-source 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Basesystem 15 SP6 kernel-zfcpdump 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Confidential Computing Technical Preview 15 SP6 kernel-coco 0 6.4.0-15061.12.coco15sp6.1
SUSE:Linux Enterprise Module for Confidential Computing Technical Preview 15 SP6 kernel-coco_debug 0 6.4.0-15061.12.coco15sp6.1
SUSE:Linux Enterprise Module for Confidential Computing Technical Preview 15 SP6 kernel-source-coco 0 6.4.0-15061.12.coco15sp6.1
SUSE:Linux Enterprise Module for Confidential Computing Technical Preview 15 SP6 kernel-syms-coco 0 6.4.0-15061.12.coco15sp6.1
SUSE:Linux Enterprise Module for Development Tools 15 SP6 kernel-docs 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Development Tools 15 SP6 kernel-obs-build 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Development Tools 15 SP6 kernel-source 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Development Tools 15 SP6 kernel-syms 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Legacy 15 SP6 kernel-default 0 6.4.0-150600.23.33.1
SUSE:Linux Enterprise Module for Public Cloud 15 SP6 kernel-azure 0 6.4.0-150600.8.23.1
SUSE:Linux Enterprise Module for Public Cloud 15 SP6 kernel-source-azure 0 6.4.0-150600.8.23.1
SUSE:Linux Enterprise Module for Public Cloud 15 SP6 kernel-syms-azure 0 6.4.0-150600.8.23.1
SUSE:Linux Enterprise Server 15 SP5-LTSS kernel-64kb 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server 15 SP5-LTSS kernel-default 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server 15 SP5-LTSS kernel-default-base 0 5.14.21-150500.55.94.1.150500.6.43.1
SUSE:Linux Enterprise Server 15 SP5-LTSS kernel-docs 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server 15 SP5-LTSS kernel-obs-build 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server 15 SP5-LTSS kernel-source 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server 15 SP5-LTSS kernel-syms 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server 15 SP5-LTSS kernel-zfcpdump 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server for SAP Applications 15 SP5 kernel-default 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server for SAP Applications 15 SP5 kernel-default-base 0 5.14.21-150500.55.94.1.150500.6.43.1
SUSE:Linux Enterprise Server for SAP Applications 15 SP5 kernel-docs 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server for SAP Applications 15 SP5 kernel-obs-build 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server for SAP Applications 15 SP5 kernel-source 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Server for SAP Applications 15 SP5 kernel-syms 0 5.14.21-150500.55.94.1
SUSE:Linux Enterprise Workstation Extension 15 SP6 kernel-default 0 6.4.0-150600.23.33.1
SUSE:Real Time Module 15 SP6 kernel-rt 0 6.4.0-150600.10.23.1
SUSE:Real Time Module 15 SP6 kernel-rt_debug 0 6.4.0-150600.10.23.1
SUSE:Real Time Module 15 SP6 kernel-source-rt 0 6.4.0-150600.10.23.1
SUSE:Real Time Module 15 SP6 kernel-syms-rt 0 6.4.0-150600.10.23.1
Ubuntu:14.04:LTS linux 0 3.11.0-12.19
Ubuntu:14.04:LTS linux-aws 0 4.4.0-1002.2
Ubuntu:14.04:LTS linux-azure 0 4.15.0-1023.24~14.04.1
Ubuntu:14.04:LTS linux-lts-xenial 0 4.4.0-13.29~14.04.1
Ubuntu:16.04:LTS linux 0 4.4.0-2.16
Ubuntu:16.04:LTS linux-aws 0 4.4.0-1001.10
Ubuntu:16.04:LTS linux-aws-hwe 0 4.15.0-1030.31~16.04.1
Ubuntu:16.04:LTS linux-azure 0 4.11.0-1009.9
Ubuntu:16.04:LTS linux-gcp 0 4.10.0-1004.4
Ubuntu:16.04:LTS linux-hwe 0 4.8.0-39.42~16.04.1
Ubuntu:16.04:LTS linux-hwe-edge 0 No fix
Ubuntu:16.04:LTS linux-kvm 0 4.4.0-1004.9
Ubuntu:16.04:LTS linux-oracle 0 4.15.0-1007.9~16.04.1
Ubuntu:18.04:LTS linux 0 4.13.0-16.19
Ubuntu:18.04:LTS linux-aws 0 4.15.0-1001.1
Ubuntu:18.04:LTS linux-aws-5.0 0 No fix
Ubuntu:18.04:LTS linux-aws-5.3 0 No fix
Ubuntu:18.04:LTS linux-aws-5.4 0 5.4.0-1018.18~18.04.1
Ubuntu:18.04:LTS linux-azure 0 No fix
Ubuntu:18.04:LTS linux-azure-4.15 0 4.15.0-1082.92
Ubuntu:18.04:LTS linux-azure-5.3 0 No fix
Ubuntu:18.04:LTS linux-azure-5.4 0 5.4.0-1020.20~18.04.1
Ubuntu:18.04:LTS linux-azure-edge 0 No fix
Ubuntu:18.04:LTS linux-gcp 0 No fix
Ubuntu:18.04:LTS linux-gcp-4.15 0 4.15.0-1071.81
Ubuntu:18.04:LTS linux-gcp-5.3 0 No fix
Ubuntu:18.04:LTS linux-gcp-5.4 0 5.4.0-1019.19~18.04.2
Ubuntu:18.04:LTS linux-gke-4.15 0 No fix
Ubuntu:18.04:LTS linux-gke-5.4 0 No fix
Ubuntu:18.04:LTS linux-gkeop-5.4 0 No fix
Ubuntu:18.04:LTS linux-hwe 0 No fix
Ubuntu:18.04:LTS linux-hwe-5.4 0 5.4.0-37.41~18.04.1
Ubuntu:18.04:LTS linux-hwe-edge 0 No fix
Ubuntu:18.04:LTS linux-ibm-5.4 0 5.4.0-1010.11~18.04.2
Ubuntu:18.04:LTS linux-kvm 0 4.15.0-1002.2
Ubuntu:18.04:LTS linux-oem 0 No fix
Ubuntu:18.04:LTS linux-oracle 0 4.15.0-1007.9
Ubuntu:18.04:LTS linux-oracle-5.0 0 No fix
Ubuntu:18.04:LTS linux-oracle-5.3 0 No fix
Ubuntu:18.04:LTS linux-oracle-5.4 0 5.4.0-1019.19~18.04.1
Ubuntu:18.04:LTS linux-raspi-5.4 0 5.4.0-1013.13~18.04.1
Ubuntu:20.04:LTS linux 0 5.4.0-9.12
Ubuntu:20.04:LTS linux-aws 0 5.4.0-1005.5
Ubuntu:20.04:LTS linux-aws-5.11 0 No fix
Ubuntu:20.04:LTS linux-aws-5.13 0 No fix
Ubuntu:20.04:LTS linux-aws-5.15 0 5.15.0-1014.18~20.04.1
Ubuntu:20.04:LTS linux-aws-5.8 0 No fix
Ubuntu:20.04:LTS linux-azure 0 5.4.0-1006.6
Ubuntu:20.04:LTS linux-azure-5.11 0 No fix
Ubuntu:20.04:LTS linux-azure-5.13 0 No fix
Ubuntu:20.04:LTS linux-azure-5.15 0 5.15.0-1007.8~20.04.1
Ubuntu:20.04:LTS linux-azure-5.8 0 No fix
Ubuntu:20.04:LTS linux-azure-fde 0 No fix
Ubuntu:20.04:LTS linux-azure-fde-5.15 0 No fix
Ubuntu:20.04:LTS linux-bluefield 0 5.4.0-1007.10
Ubuntu:20.04:LTS linux-gcp 0 5.4.0-1005.5
Ubuntu:20.04:LTS linux-gcp-5.11 0 No fix
Ubuntu:20.04:LTS linux-gcp-5.13 0 No fix
Ubuntu:20.04:LTS linux-gcp-5.15 0 5.15.0-1006.9~20.04.1
Ubuntu:20.04:LTS linux-gcp-5.8 0 No fix
Ubuntu:20.04:LTS linux-gke 0 No fix
Ubuntu:20.04:LTS linux-gke-5.15 0 No fix
Ubuntu:20.04:LTS linux-gkeop 0 No fix
Ubuntu:20.04:LTS linux-gkeop-5.15 0 No fix
Ubuntu:20.04:LTS linux-hwe-5.11 0 No fix
Ubuntu:20.04:LTS linux-hwe-5.13 0 No fix
Ubuntu:20.04:LTS linux-hwe-5.15 0 5.15.0-33.34~20.04.1
Ubuntu:20.04:LTS linux-hwe-5.8 0 No fix
Ubuntu:20.04:LTS linux-ibm 0 5.4.0-1003.4
Ubuntu:20.04:LTS linux-ibm-5.15 0 5.15.0-1033.36~20.04.1
Ubuntu:20.04:LTS linux-intel-5.13 0 No fix
Ubuntu:20.04:LTS linux-intel-iotg-5.15 0 5.15.0-1003.5~20.04.1
Ubuntu:20.04:LTS linux-iot 0 5.4.0-1001.3
Ubuntu:20.04:LTS linux-kvm 0 5.4.0-1004.4
Ubuntu:20.04:LTS linux-lowlatency-hwe-5.15 0 5.15.0-33.34~20.04.1
Ubuntu:20.04:LTS linux-oem-5.10 0 No fix
Ubuntu:20.04:LTS linux-oem-5.13 0 No fix
Ubuntu:20.04:LTS linux-oem-5.14 0 No fix
Ubuntu:20.04:LTS linux-oem-5.6 0 No fix
Ubuntu:20.04:LTS linux-oracle 0 5.4.0-1005.5
Ubuntu:20.04:LTS linux-oracle-5.11 0 No fix
Ubuntu:20.04:LTS linux-oracle-5.13 0 No fix
Ubuntu:20.04:LTS linux-oracle-5.15 0 5.15.0-1007.9~20.04.1
Ubuntu:20.04:LTS linux-oracle-5.8 0 No fix
Ubuntu:20.04:LTS linux-raspi 0 5.4.0-1007.7
Ubuntu:20.04:LTS linux-raspi2 0 No fix
Ubuntu:20.04:LTS linux-riscv 0 No fix
Ubuntu:20.04:LTS linux-riscv-5.11 0 No fix
Ubuntu:20.04:LTS linux-riscv-5.15 0 5.15.0-1015.17~20.04.1
Ubuntu:20.04:LTS linux-riscv-5.8 0 No fix
Ubuntu:20.04:LTS linux-xilinx-zynqmp 0 5.4.0-1020.24
Ubuntu:22.04:LTS linux 0 5.13.0-19.19
Ubuntu:22.04:LTS linux-allwinner-5.19 0 No fix
Ubuntu:22.04:LTS linux-aws 0 5.13.0-1005.6
Ubuntu:22.04:LTS linux-aws-5.19 0 No fix
Ubuntu:22.04:LTS linux-aws-6.2 0 No fix
Ubuntu:22.04:LTS linux-aws-6.5 0 No fix
Ubuntu:22.04:LTS linux-aws-6.8 0 No fix
Ubuntu:22.04:LTS linux-aws-6.8 0 6.8.0-1027.29~22.04.1
Ubuntu:22.04:LTS linux-azure 0 5.13.0-1006.7
Ubuntu:22.04:LTS linux-azure-5.19 0 No fix
Ubuntu:22.04:LTS linux-azure-6.2 0 No fix
Ubuntu:22.04:LTS linux-azure-6.5 0 No fix
Ubuntu:22.04:LTS linux-azure-6.8 0 No fix
Ubuntu:22.04:LTS linux-azure-6.8 0 6.8.0-1027.32~22.04.1
Ubuntu:22.04:LTS linux-azure-fde 0 No fix
Ubuntu:22.04:LTS linux-azure-fde-5.19 0 No fix
Ubuntu:22.04:LTS linux-azure-fde-6.2 0 No fix
Ubuntu:22.04:LTS linux-gcp 0 5.13.0-1005.6
Ubuntu:22.04:LTS linux-gcp-5.19 0 No fix
Ubuntu:22.04:LTS linux-gcp-6.2 0 No fix
Ubuntu:22.04:LTS linux-gcp-6.5 0 No fix
Ubuntu:22.04:LTS linux-gcp-6.8 0 No fix
Ubuntu:22.04:LTS linux-gcp-6.8 0 6.8.0-1028.30~22.04.1
Ubuntu:22.04:LTS linux-gke 0 5.15.0-1002.2
Ubuntu:22.04:LTS linux-gkeop 0 5.15.0-1001.2
Ubuntu:22.04:LTS linux-hwe-5.19 0 No fix
Ubuntu:22.04:LTS linux-hwe-6.2 0 No fix
Ubuntu:22.04:LTS linux-hwe-6.5 0 No fix
Ubuntu:22.04:LTS linux-hwe-6.8 0 No fix
Ubuntu:22.04:LTS linux-hwe-6.8 0 6.8.0-58.60~22.04.1
Ubuntu:22.04:LTS linux-ibm 0 5.15.0-1002.2
Ubuntu:22.04:LTS linux-ibm-6.8 0 6.8.0-1024.24~22.04.1
Ubuntu:22.04:LTS linux-intel-iot-realtime 0 No fix
Ubuntu:22.04:LTS linux-intel-iot-realtime 0 5.15.0-1021.26
Ubuntu:22.04:LTS linux-intel-iotg 0 5.15.0-1004.6
Ubuntu:22.04:LTS linux-kvm 0 5.13.0-1004.4
Ubuntu:22.04:LTS linux-lowlatency 0 5.15.0-22.22
Ubuntu:22.04:LTS linux-lowlatency-hwe-5.19 0 No fix
Ubuntu:22.04:LTS linux-lowlatency-hwe-6.2 0 No fix
Ubuntu:22.04:LTS linux-lowlatency-hwe-6.5 0 No fix
Ubuntu:22.04:LTS linux-lowlatency-hwe-6.8 0 No fix
Ubuntu:22.04:LTS linux-lowlatency-hwe-6.8 0 6.8.0-58.60.1~22.04.1
Ubuntu:22.04:LTS linux-nvidia 0 5.15.0-1005.5
Ubuntu:22.04:LTS linux-nvidia-6.2 0 No fix
Ubuntu:22.04:LTS linux-nvidia-6.5 0 No fix

Related vulnerabilities