> ## Documentation Index > Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Prefer Using Secrets Files Over Secrets Environment Variables ### More Info: Kubernetes supports mounting secrets as data volumes or as environment variables. Minimize the use of environment variable secrets. ### Risk Level High ### Address Security ### Compliance Standards * APRA CPS 234 (Australia) * AWS Startup Security Baseline * BSI C5 (Germany) * Brazil LGPD * CCPA / CPRA (California) * CIS Critical Security Controls v8 * CIS EKS * CMMC 2.0 * CSA Cloud Controls Matrix v4 * DPDPA * Digital Operational Resilience Act (EU) * Essential 8 * ISO/IEC 27017 * ISO/IEC 27018 * ISO/IEC 27701 * KSA PDPL * MAS Technology Risk Management (Singapore) * MITRE ATT\&CK (Cloud) * NIST SP 800-171 * NYDFS 23 NYCRR 500 * SWIFT Customer Security Controls Framework * Sarbanes-Oxley IT General Controls * UK NCSC Cyber Assessment Framework ### Triage and Remediation ### Remediation 1. **List all workloads using secrets as environment variables** * Run on: any machine with `kubectl` access. ```bash theme={null} kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}' ``` 2. **Inspect a specific workload’s env-based secret usage** * Pick one namespace/kind/name from step 1 and inspect its manifest: ```bash theme={null} # Example for a Deployment kubectl -n NAMESPACE get deploy DEPLOYMENT_NAME -o yaml > /tmp/deploy-NAMESPACE-DEPLOYMENT_NAME.yaml ``` * In the downloaded YAML, locate `env:` entries that use `valueFrom.secretKeyRef`. 3. **Refactor the manifest to mount secrets as files instead of env vars** * Edit the saved manifest file on your local machine: * Under `spec.template.spec`, add or update a `volumes` entry referencing the secret: ```yaml theme={null} spec: template: spec: volumes: - name: app-secret secret: secretName: SECRET_NAME ``` * For each container that currently uses `env.valueFrom.secretKeyRef`, remove those `env` entries and mount the secret as a volume: ```yaml theme={null} containers: - name: app-container volumeMounts: - name: app-secret mountPath: /var/run/secrets/app readOnly: true ``` * Ensure the application code/config is updated to read the secret from the mounted file path (for example, `/var/run/secrets/app/KEY_NAME`). This requires application changes outside of Kubernetes and must be coordinated with the application owner. 4. **Apply the updated manifest to the cluster** * Run on: any machine with `kubectl` access. ```bash theme={null} kubectl apply -f /tmp/deploy-NAMESPACE-DEPLOYMENT_NAME.yaml ``` * Repeat steps 2–4 for each resource reported in step 1 (`Deployment`, `StatefulSet`, `DaemonSet`, `Job`, `CronJob`, etc.), adapting `kind` and filenames as needed. 5. **Optionally remove any remaining env-based secret refs programmatically (review required)** * To review all manifests currently using `secretKeyRef` before final cleanup: ```bash theme={null} kubectl get all --all-namespaces -o yaml | grep -n "secretKeyRef" -n ``` * Manually confirm each usage is safe to remove and that the application reads from files; then adjust the corresponding manifests as in steps 2–4. Do not delete `env` entries without confirming application behavior. 6. **Verify no secrets are used as environment variables** * Run on: any machine with `kubectl` access. ```bash theme={null} result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}') if [ -z "$result" ]; then echo "NO_SECRETS_AS_ENV_VARS" else echo "SECRETS_AS_ENV_VARS_FOUND: $result" fi ``` On any machine with kubectl access: 1. Identify workloads using `secretKeyRef` (same as audit) ```bash theme={null} kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}' ``` 2. For each listed object, switch from env-var secrets to volume-mounted secrets. Example: convert a Deployment using `env.valueFrom.secretKeyRef` to use a Secret volume. **Before (env-var based):** ```bash theme={null} kubectl get deployment my-app -n my-namespace -o yaml > /tmp/my-app-deployment.yaml ``` Edit `/tmp/my-app-deployment.yaml` and replace: ```yaml theme={null} env: - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password ``` with volume-based access: ```yaml theme={null} volumeMounts: - name: db-secret-vol mountPath: /var/run/secrets/db readOnly: true env: - name: DB_PASSWORD_FILE value: /var/run/secrets/db/password volumes: - name: db-secret-vol secret: secretName: db-secret ``` Make sure `volumes:` is under `spec.template.spec` and `volumeMounts:` is under the container spec. Adjust `mountPath`, `name`, and `secretName` as needed. Your application code must read from the file path (e.g., `/var/run/secrets/db/password`) instead of the environment variable. 3. Apply the updated manifest (this will roll the workload) ```bash theme={null} kubectl apply -f /tmp/my-app-deployment.yaml ``` Repeat step 2–3 for each Deployment/StatefulSet/DaemonSet/Pod reported in step 1. 4. Verification (no remaining `secretKeyRef` in running workloads) ```bash theme={null} result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}') if [ -z "$result" ]; then echo "NO_SECRETS_AS_ENV_VARS" else echo "SECRETS_AS_ENV_VARS_FOUND: $result" fi ``` ```bash theme={null} #!/usr/bin/env bash # # Automation: Prefer using secrets as files over secrets as environment variables # # Scope: any machine with kubectl access to the cluster # # IMPORTANT: # - This script CANNOT safely rewrite application code or pod specs automatically. # - It is designed to: # * Detect all workloads using secretKeyRef in env or envFrom # * Generate patched manifest templates that mount the same secrets as volumes # under /var/run/secrets//, with comments guiding manual spec changes # * Allow you to manually adjust container commands/code to read from files # - It is safe to re-run; it overwrites only its own output directory and re-runs audits. # # You must manually: # - Update application code/images to read from the mounted files instead of env vars # - Remove the env/secretKeyRef entries once applications are updated and tested # # Requirements: # - bash, kubectl, jq, yq (v4+), and mkdir/cat/sed # - Cluster context configured (kubectl works without extra flags) # # NOTE: # - There is no deterministic, fully automated fix because the benchmark requires # application-level changes. This script prepares and guides those changes. set -euo pipefail OUT_DIR="./secret-env-to-file-plans" TIMESTAMP="$(date -u +%Y%m%dT%H%M%SZ)" mkdir -p "${OUT_DIR}" echo "=== [1/4] Detecting workloads using secrets as environment variables ===" # Reuse the authoritative audit logic, but also collect JSON for processing. AUDIT_RAW_FILE="${OUT_DIR}/audit_raw_${TIMESTAMP}.txt" AUDIT_JSON_FILE="${OUT_DIR}/audit_workloads_${TIMESTAMP}.json" result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}') || true echo "${result}" > "${AUDIT_RAW_FILE}" if [ -z "${result}" ]; then echo "NO_SECRETS_AS_ENV_VARS" echo "No workloads using secretKeyRef in env were found. Nothing to prepare." exit 0 fi echo "SECRETS_AS_ENV_VARS_FOUND:" cat "${AUDIT_RAW_FILE}" # Collect full JSON of affected resources (idempotent snapshot) # We deliberately limit to objects that 'get all' may include, but cover common controllers. kubectl get all --all-namespaces -o json > "${AUDIT_JSON_FILE}" echo "Raw audit list saved to: ${AUDIT_RAW_FILE}" echo "Full JSON snapshot saved to: ${AUDIT_JSON_FILE}" echo "=== [2/4] Generating migration plans with volume-mounted secrets ===" PLAN_DIR="${OUT_DIR}/plans_${TIMESTAMP}" mkdir -p "${PLAN_DIR}" # Function to generate a plan for a single named object generate_plan_for_object() { local ns="$1" local kind="$2" local name="$3" local safe_ns safe_kind safe_name safe_ns="${ns//\//_}" safe_kind="${kind//\//_}" safe_name="${name//\//_}" local out_file="${PLAN_DIR}/${safe_ns}__${safe_kind}__${safe_name}.yaml" echo " - Processing ${ns} ${kind} ${name}" # Fetch live object as YAML if ! kubectl get "${kind}" "${name}" -n "${ns}" -o yaml > "${out_file}.orig" 2>/dev/null; then echo " ! Skipping (unable to fetch YAML for ${ns}/${kind}/${name})" rm -f "${out_file}.orig" return fi # Copy original to working file cp "${out_file}.orig" "${out_file}" # Add comments at the top of the file to guide manual changes { echo "# =============================================================" echo "# PLAN for ${ns} ${kind} ${name}" echo "# Generated: ${TIMESTAMP}" echo "#" echo "# This file is a COPY of the live object and is NOT applied automatically." echo "# Actions you should take:" echo "# 1) For each container using env.secretKeyRef or envFrom.secretRef," echo "# this plan adds a volume and volumeMount for the referenced Secret" echo "# under: /var/run/secrets//" echo "# 2) Modify container code/command/args to read secrets from these files" echo "# instead of from environment variables." echo "# 3) Once the application works with files, REMOVE the corresponding" echo "# env and envFrom.secretRef entries, then apply this manifest:" echo "# kubectl apply -f ${out_file}" echo "# 4) After rollout, re-run the verification step (see end of script)." echo "#" echo "# NOTE: This plan does not alter the live cluster until you apply it." echo "# =============================================================" echo "" } | cat - "${out_file}" > "${out_file}.tmp" && mv "${out_file}.tmp" "${out_file}" # Using yq to append volumes and volumeMounts for each secretRef used in env or envFrom. # # Strategy: # - For each spec.template.spec.containers[*]: # * Collect unique secret names from: # - env[].valueFrom.secretKeyRef.name # - envFrom[].secretRef.name # * For each unique secret name: # - Ensure a pod spec volume named "secret-" exists: # spec.template.spec.volumes += { name: "secret-", secret: { secretName: } } # - Ensure the container has a volumeMount: # mountPath: "/var/run/secrets/" # name: "secret-" # # The keys will be present as files inside that directory. # # Add a helper annotation to mark that a plan was generated yq -i ' .metadata.annotations."secret-env-to-file-plan/generated" = "'"${TIMESTAMP}"'"' \ "${out_file}" # Process top-level pod specs and template-based specs # Handle objects with spec.template.spec (Deployments, DaemonSets, etc.) yq -i ' (.. | select(has("template") and .template.spec? != null) | .template.spec) |= ( . as $spec | ($spec.containers // []) as $cts | ($cts | to_entries | map( . as $ci | ( $ci.value.env // [] | map(select(.valueFrom.secretKeyRef.name != null) | .valueFrom.secretKeyRef.name) ) + ( $ci.value.envFrom // [] | map(select(.secretRef.name != null) | .secretRef.name) ) | unique | . as $secretNames | # ensure volumes ($spec.volumes // []) as $vols | $secretNames as $sns | ( $sns | reduce .[] as $s ( $vols; if any(.[]; .name == ("secret-" + $s)) then . else . + [{ name: ("secret-" + $s), secret: { secretName: $s } }] end ) ) as $newVols | # ensure volumeMounts per container ($ci.value.volumeMounts // []) as $vms | $sns as $sns2 | ( $sns2 | reduce .[] as $s2 ( $vms; if any(.[]; .name == ("secret-" + $s2)) then . else . + [{ name: ("secret-" + $s2), mountPath: ("/var/run/secrets/" + $s2) }] end ) ) as $newVms | # write back mounts to this container entry $spec.containers[$ci.key].volumeMounts = $newVms )) | # write back updated volumes .volumes = ( ($spec.volumes // []) + ([] | .) # no-op for structure ) | .volumes = ($spec | .volumes? // []) ) ' "${out_file}" # Handle Pod objects (without template) similarly yq -i ' (select(.kind == "Pod") | .spec) |= ( . as $spec | ($spec.containers // []) as $cts | ($cts | to_entries | map( . as $ci | ( $ci.value.env // [] | map(select(.valueFrom.secretKeyRef.name != null) | .valueFrom.secretKeyRef.name) ) + ( $ci.value.envFrom // [] | map(select(.secretRef.name != null) | .secretRef.name) ) | unique | . as $secretNames | ($spec.volumes // []) as $vols | $secretNames as $sns | ( $sns | reduce .[] as $s ( $vols; if any(.[]; .name == ("secret-" + $s)) then . else . + [{ name: ("secret-" + $s), secret: { secretName: $s } }] end ) ) as $newVols | ($ci.value.volumeMounts // []) as $vms | $sns as $sns2 | ( $sns2 | reduce .[] as $s2 ( $vms; if any(.[]; .name == ("secret-" + $s2)) then . else . + [{ name: ("secret-" + $s2), mountPath: ("/var/run/secrets/" + $s2) }] end ) ) as $newVms | $spec.containers[$ci.key].volumeMounts = $newVms )) | .volumes = ( ($spec.volumes // []) + ([] | .) ) | .volumes = ($spec | .volumes? // []) ) ' "${out_file}" echo " Plan written to: ${out_file}" echo " Original saved as: ${out_file}.orig" } # Iterate over the audit results and generate plans while read -r ns kind name; do [ -z "${ns}" ] && continue generate_plan_for_object "${ns}" "${kind}" "${name}" done <<< "${result}" echo "=== [3/4] Manual steps required (read carefully) ===" cat <<'EOF' For each generated PLAN file under the directory printed above: 1) Open the plan file and review added volume and volumeMount entries. - Secrets referenced by env.secretKeyRef/envFrom.secretRef are now: mounted under /var/run/secrets/ in each affected container. 2) Update application behavior to read secrets from files: - If using command/args: change them to read from file paths. - Prefer updating application code/images so that: - They load secrets from the mounted file paths. - They no longer depend on the corresponding environment variables. 3) Remove secretKeyRef-based environment variables from the plan: - Delete specific env entries that use valueFrom.secretKeyRef. - Delete envFrom entries that use secretRef. - Be careful not to remove non-secret env vars. 4) Apply and roll out the modified manifests: - kubectl apply -f - Monitor rollout and logs to confirm correct behavior. 5) Once all workloads have been migrated and are stable, re-run this script to confirm that no env.secretKeyRef usages remain. EOF echo "Plans directory: ${PLAN_DIR}" echo "=== [4/4] Verification (post-migration) ===" echo "When you believe you have fully migrated workloads to file-based secrets," echo "run the following verification command on any machine with kubectl access:" cat <<'EOF' result=$(kubectl get all --all-namespaces -o jsonpath='{range .items[?(@..secretKeyRef)]}{.metadata.namespace} {.kind} {.metadata.name}{"\n"}{end}') if [ -z "$result" ]; then echo "NO_SECRETS_AS_ENV_VARS" else echo "SECRETS_AS_ENV_VARS_FOUND: $result" fi EOF echo "This script has completed generating migration plans and guidance." ``` ### Additional Reading: * [https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets](https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets)