Fix: Kubernetes ConfigMap Changes Not Reflected in Running Pods

The Error

You update a ConfigMap but running pods still use the old configuration:

kubectl edit configmap my-app-config
# Edit and save — but the running app still reads old values

kubectl exec my-pod -- cat /etc/config/app.properties
# Shows OLD values — not the updated ones

Or environment variables from the ConfigMap are not updated:

kubectl exec my-pod -- printenv DATABASE_URL
# Shows the old DATABASE_URL even after updating the ConfigMap

Or the application restarted inside the pod but still reads old values from the mounted file.

Why This Happens

ConfigMap propagation is governed by two completely different code paths inside Kubernetes, and the symptom you see depends on which path is in play. The first is env injection: when a container is created, the kubelet reads the ConfigMap and copies values into the container’s process environment block. That copy is permanent for the life of the process; updating the ConfigMap after the pod starts has no effect on the environment because the kernel does not let you mutate environ from outside the process. The only way to refresh an env-driven config is to recreate the container, which means a pod restart.

The second path is volume projection: the kubelet syncs ConfigMap data into a directory on the host (a tmpfs mount under /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~configmap/). When the ConfigMap changes, kubelet’s syncLoop rewrites the tmpfs files using an atomic symlink swap. The default sync frequency on the kubelet is syncFrequency: 1m, plus the watch propagation delay, plus the cache TTL — so changes typically land in the pod within 60–90 seconds. The files update on disk, but the application still must re-read them. Most config libraries cache on first read and never look again, which is why “ConfigMap is updated but the app still uses old values” is so common.

There is also a third, rarer path: the subPath mount. When you mount a single key of a ConfigMap into a specific filename via subPath, the file is treated as a static initialisation: Kubernetes does not update it on subsequent syncs. This is by design — the symlink swap that powers atomic volume updates cannot work through a subPath. Many teams discover this only after wiring up Reloader and watching it restart pods that then read identical, stale files because of the subPath.

• Environment variables (envFrom/env) — injected at pod creation time. Updating the ConfigMap does not update environment variables in running pods. A pod restart is required.
• Volume mounts — ConfigMap data mounted as files is updated automatically by kubelet, but with a delay (typically 1–2 minutes). The files on disk update, but the application must re-read them — most apps only read config files on startup.
• Immutable ConfigMaps — if immutable: true is set, the ConfigMap cannot be edited at all.
• Kubelet sync period — the default kubelet --sync-frequency for ConfigMap volumes is 60 seconds plus propagation time. Changes are not instant.

Version History That Changes the Failure Mode

Kubernetes has added several ConfigMap-related features over the years, and the behavior you see depends on the cluster version. Check with kubectl version and kubectl get nodes -o wide to see what is installed:

• Kubernetes 1.13 (December 2018) — Default syncFrequency for kubelet ConfigMap watches established at 1 minute. Before 1.13 the propagation behavior was inconsistent across kubelet builds.
• Kubernetes 1.18 (March 2020) — Immutable ConfigMaps and Secrets reached beta (immutable: true field). Marking a ConfigMap immutable lets the kubelet skip the watch on it entirely, which improves API server load on large clusters. Immutable ConfigMaps cannot be edited — you must create a new one.
• Kubernetes 1.19 (August 2020) — Immutable ConfigMaps reached GA. The ConfigMapAndSecretChangeDetectionStrategy kubelet option became stable.
• Kubernetes 1.21 (April 2021) — Changed the default behavior so that pods mounting a ConfigMap as a volume see updates automatically via atomic symlink swaps. This was already the case in most distributions but became the documented default.
• Kubernetes 1.22 (August 2021) — Reworked the projected volume support so multiple sources (ConfigMaps + Secrets + downward API) can share a mount point cleanly.
• Kubernetes 1.24 (May 2022) — Removed the legacy dockershim container runtime. If a node still ran dockershim with a custom sync configuration, the upgrade reset the sync defaults — sometimes pushing propagation delay back to the default 60s when it had been tuned to 10s.
• Kubernetes 1.27 (April 2023) — Introduced in-place pod vertical resize (resize subresource), which can update some pod fields without recreation. ConfigMap-driven env vars are still not part of this — env injection remains a one-shot at creation time.
• Kubernetes 1.30 (April 2024) — Promoted MultipleHugePageSizes and other volume-related stability work.
• Stakater Reloader — Community controller. v1.0 (2020) handled ConfigMap and Secret watches. v1.2 (2023) added support for Rollout (Argo) resources. The annotation reloader.stakater.com/auto: "true" triggers a rolling restart on any referenced ConfigMap or Secret change.

If you upgraded from a 1.20-or-earlier cluster, double-check the kubelet syncFrequency and configMapAndSecretChangeDetectionStrategy settings on each node — they reset to defaults on some managed-Kubernetes upgrades. EKS, GKE, and AKS each handle this slightly differently and an upgrade can silently change propagation behavior.

Fix 1: Restart Pods to Pick Up ConfigMap Changes

For environment variable-based ConfigMaps, a pod restart is always required:

# Rolling restart — replaces pods one by one (zero downtime)
kubectl rollout restart deployment/my-app

# Watch the rollout
kubectl rollout status deployment/my-app

# Verify the new pod has the updated value
kubectl exec deployment/my-app -- printenv MY_CONFIG_VALUE

Restart a specific pod (it will be recreated by the Deployment):

kubectl delete pod my-app-abc123-xyz789
# Deployment controller creates a new pod with current ConfigMap values

Patch the Deployment to force a restart without changing the spec:

# Add/update a timestamp annotation — triggers rolling restart
kubectl patch deployment my-app \
  -p '{"spec":{"template":{"metadata":{"annotations":{"kubectl.kubernetes.io/restartedAt":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}}}}}'

Fix 2: Use Volume Mounts for Live Config Updates

Pods that consume ConfigMaps via volume mounts receive file updates automatically (within ~1–2 minutes). But the application must re-read the file:

ConfigMap as a volume:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
        - name: my-app
          image: my-app:latest
          volumeMounts:
            - name: config-volume
              mountPath: /etc/config     # Files appear here
              readOnly: true
      volumes:
        - name: config-volume
          configMap:
            name: my-app-config         # ConfigMap name

# configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config
data:
  app.properties: |
    database.url=jdbc:postgresql://postgres:5432/mydb
    cache.ttl=300
  feature-flags.json: |
    {"dark_mode": true, "beta_feature": false}

Application must watch for file changes:

# Python — watch config file for changes
import json
import time
import os
from pathlib import Path

CONFIG_PATH = Path('/etc/config/feature-flags.json')

def load_config():
    with open(CONFIG_PATH) as f:
        return json.load(f)

# Option A — reload on each request (simple, slight overhead)
def is_feature_enabled(feature: str) -> bool:
    config = load_config()
    return config.get(feature, False)

# Option B — poll for file changes
class ConfigWatcher:
    def __init__(self, path: Path):
        self.path = path
        self._config = load_config()
        self._mtime = path.stat().st_mtime

    def get(self, key, default=None):
        current_mtime = self.path.stat().st_mtime
        if current_mtime != self._mtime:
            self._config = load_config()
            self._mtime = current_mtime
        return self._config.get(key, default)

config = ConfigWatcher(CONFIG_PATH)

// Node.js — watch for file changes with fs.watch
const fs = require('fs');
const path = '/etc/config/app.json';

let config = JSON.parse(fs.readFileSync(path, 'utf8'));

fs.watch(path, (event) => {
  if (event === 'change') {
    try {
      config = JSON.parse(fs.readFileSync(path, 'utf8'));
      console.log('Config reloaded');
    } catch (err) {
      console.error('Failed to reload config:', err);
    }
  }
});

Note: Kubernetes updates ConfigMap volumes atomically using symlinks. The actual file path is a symlink that points to a new directory when updated. Using fs.watch on the symlink target may not fire. Watch the directory or use a polling approach instead.

Fix 3: Automate Restarts with Reloader

Install Stakater Reloader — it watches for ConfigMap and Secret changes and automatically triggers rolling restarts:

# Install with Helm
helm repo add stakater https://stakater.github.io/stakater-charts
helm install reloader stakater/reloader -n kube-system

Annotate your Deployment to enable automatic restarts:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    # Restart when any ConfigMap or Secret changes
    reloader.stakater.com/auto: "true"

    # Or restart only when specific ConfigMaps change
    configmap.reloader.stakater.com/reload: "my-app-config,my-other-config"

    # Or restart only when specific Secrets change
    secret.reloader.stakater.com/reload: "my-app-secret"

Now whenever my-app-config is updated, Reloader automatically triggers a kubectl rollout restart — no manual intervention needed.

Fix 4: Trigger Restarts via CI/CD After ConfigMap Updates

In a CI/CD pipeline, always restart after updating a ConfigMap:

# GitHub Actions example
- name: Update ConfigMap
  run: |
    kubectl create configmap my-app-config \
      --from-file=app.properties=./config/app.properties \
      --dry-run=client -o yaml | kubectl apply -f -

- name: Restart deployment to pick up new config
  run: |
    kubectl rollout restart deployment/my-app
    kubectl rollout status deployment/my-app --timeout=300s

Using kustomize for ConfigMap management:

# kustomization.yaml
resources:
  - deployment.yaml

configMapGenerator:
  - name: my-app-config
    files:
      - config/app.properties
    options:
      disableNameSuffixHash: false  # Adds a hash suffix — forces pod restart when content changes

When disableNameSuffixHash: false (default), kustomize generates a new ConfigMap name (e.g., my-app-config-abc123) whenever the content changes. The Deployment references the new name, triggering an automatic rolling restart.

Fix 5: Use Projected Volumes for Multiple Sources

Combine ConfigMaps and Secrets into a single mount point:

spec:
  containers:
    - name: my-app
      volumeMounts:
        - name: combined-config
          mountPath: /etc/config
  volumes:
    - name: combined-config
      projected:
        sources:
          - configMap:
              name: app-config
          - secret:
              name: app-secrets
          - configMap:
              name: feature-flags
              items:
                - key: flags.json
                  path: feature-flags.json   # Custom filename in the mount

Fix 6: Verify ConfigMap Is Mounted Correctly

# Check the ConfigMap exists and has the right data
kubectl get configmap my-app-config -o yaml

# Verify what files are mounted in the pod
kubectl exec my-pod -- ls -la /etc/config/

# Check the actual file content inside the pod
kubectl exec my-pod -- cat /etc/config/app.properties

# Check if the symlink is updated (Kubernetes uses symlinks for atomic updates)
kubectl exec my-pod -- ls -la /etc/config/
# Look for: ..data -> ..2024_01_15_10_30_00.12345 (timestamp changes on update)

# Force kubelet to sync (for debugging — not for production)
kubectl exec my-pod -- kill -HUP 1  # Send SIGHUP to PID 1 — triggers graceful reload in some apps

Check the kubelet sync period on a node:

# SSH into a node and check kubelet config
sudo cat /var/lib/kubelet/config.yaml | grep -i sync
# syncFrequency: 1m0s  ← Default 1 minute

Fix 7: Use Immutable ConfigMaps for Safety

For config that should not change at runtime, mark the ConfigMap as immutable:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-app-config-v2
immutable: true    # Cannot be modified — must create a new ConfigMap
data:
  DATABASE_URL: "postgres://db:5432/mydb"

Immutable ConfigMaps provide:

• Protection against accidental changes
• Better kubelet performance (kubelet doesn’t watch immutable ConfigMaps for changes)
• Forced version control via naming (config-v1, config-v2)

To update an immutable ConfigMap, create a new one and update the Deployment to reference it:

kubectl create configmap my-app-config-v3 \
  --from-literal=DATABASE_URL=postgres://db-new:5432/mydb

kubectl set env deployment/my-app --from=configmap/my-app-config-v3
kubectl rollout status deployment/my-app

Still Not Working?

Check the pod’s actual environment variables vs ConfigMap:

# What the ConfigMap currently contains
kubectl get configmap my-app-config -o jsonpath='{.data}'

# What the running pod actually has
kubectl exec my-pod -- env | sort

# If they differ, the pod predates the ConfigMap update — restart it
kubectl rollout restart deployment/my-app

Check kubelet logs on the node for volume sync errors:

# Find which node the pod is on
kubectl get pod my-pod -o jsonpath='{.spec.nodeName}'

# SSH into the node and check kubelet logs
journalctl -u kubelet | grep -i "configmap\|volume\|sync" | tail -50

Verify RBAC allows the kubelet to read the ConfigMap. In locked-down clusters, the kubelet’s service account may not have permission to read ConfigMaps in your namespace.

Check whether the mount uses subPath. A ConfigMap key mounted via subPath is never updated by the kubelet — the symlink-swap mechanism cannot work through subPath. If your volumeMounts entry has subPath: app.properties, the file is effectively immutable for the life of the pod. Either drop the subPath and mount the entire ConfigMap into a directory, or accept that updates require a pod recreation.

Check whether the watch propagation is blocked. On a cluster under load, the kube-apiserver watch can fall behind, which delays kubelet’s view of the new ConfigMap. Run kubectl get --watch configmap my-app-config from a control plane node and compare the timestamp to the actual edit time. If the lag is more than a few seconds, the apiserver is the bottleneck — check apiserver_request_duration_seconds in Prometheus.

Check whether a sidecar or init container has cached the old value. Some sidecar patterns (Vault Agent, Consul Template, Envoy) re-render config from a ConfigMap on init and never re-read. The main container sees fresh files but the sidecar serves stale data. Either restart the pod to refresh the sidecar or configure the sidecar to watch its own source for changes.

For related Kubernetes issues, see Fix: Kubernetes CrashLoopBackOff, Fix: Kubernetes Ingress Not Working, Fix: Kubernetes ImagePullBackOff, and Fix: Kubernetes OOMKilled.