Skip to main content
The External Secrets Enterprise Edition is product suite is a premium product. It requires a specific subscription. Contact us for more information.
This guide helps you diagnose and resolve common problems encountered when using the ESI Pod Webhook.

​
Webhook Not Triggering or Pods Not Mutated

Symptoms:
  • Pods are created without any injected init/sidecar containers or environment variables/secret files, despite having the correct annotations.
Checks & Solutions:
  1. Verify MutatingWebhookConfiguration:
    • Ensure the MutatingWebhookConfiguration for esi-pod-webhook exists: 
      kubectl get mutatingwebhookconfiguration esi-pod-webhook
    • Inspect its configuration: 
      kubectl describe mutatingwebhookconfiguration esi-pod-webhook
    • clientConfig.service: Verify name, namespace, and port point to the correct ESI Pod Webhook service (usually esi-pod-webhook-service in secretless-system on port 443, which maps to the webhook’s target port, e.g., 8443).
    • rules.operations: Should include CREATE and UPDATE for pods.
    • namespaceSelector / objectSelector: Ensure these selectors are not unintentionally excluding the namespace or pods you are targeting. If they are too restrictive, the API server won’t send admission requests to the webhook for those pods.
    • failurePolicy: Usually Fail for critical webhooks, but could be Ignore. If Ignore, errors during webhook invocation might not be immediately obvious.
    • caBundle: This should be populated by cert-manager (or manually if not using cert-manager) and match the CA of the certificate used by the webhook service.
  2. Webhook Pod Logs:
    • Check the logs of the ESI Pod Webhook pod(s) for any errors during its startup or when processing requests: 
      kubectl logs -n secretless-system -l app=esi-pod-webhook
    • Look for errors related to TLS, authentication, or processing admission reviews.
  3. Pod Annotations:
    • Double-check that your target pod has the necessary annotations, especially secretless.externalsecrets.com/externalsecret and either secretless.externalsecrets.com/env-vars: "true" or secretless.externalsecrets.com/file-secrets: "true".
    • Ensure there are no typos in annotation keys or values.
  4. skip Annotation:
    • Make sure the target pod doesn’t have secretless.externalsecrets.com/skip: "true" if you expect it to be processed.

​
TLS Errors

Symptoms:
  • Webhook pod logs show TLS handshake errors.
  • The API server logs (if accessible) might indicate failures calling the webhook due to TLS issues.
  • MutatingWebhookConfiguration caBundle might be missing or incorrect.
Checks & Solutions:
  1. cert-manager Status:
    • Ensure cert-manager pods are running and healthy in their namespace (usually cert-manager): 
      kubectl get pods -n cert-manager
  2. Webhook Certificate Resource:
    • Check the status of the Certificate resource created for the webhook (e.g., esi-pod-webhook-cert in secretless-system): 
      kubectl describe certificate esi-pod-webhook-cert -n secretless-system
    • Look for Status.Conditions like Ready: True. If not ready, check events on the Certificate and related CertificateRequest or Order resources for errors from your Issuer or ClusterIssuer.
  3. Webhook Pod Mounts:
    • Ensure the webhook deployment correctly mounts the TLS secret (created by cert-manager) into the paths specified by --tls-cert and --tls-key arguments (e.g., /etc/webhook/certs/tls.crt, /etc/webhook/certs/tls.key).

​
Secrets Not Injected (Webhook Triggers, but Secrets Missing)

Symptoms:
  • Init or sidecar containers are added to the pod, but the application doesn’t receive the expected environment variables or the /secrets/secrets.json file is missing/empty.
Checks & Solutions:
  1. Webhook Pod Logs:
    • These are crucial. They might indicate issues fetching the ExternalSecret, problems with esi-cli, or other processing errors. 
      kubectl logs -n secretless-system -l app=esi-pod-webhook
  2. Init Container Logs (for env-vars: "true"):
    • If an init container (secretless-init by default) was injected, check its logs. This is where esi-cli runs. 
      kubectl logs <your-pod-name> -n <your-namespace> -c secretless-init
    • Look for errors from esi-cli related to fetching the ExternalSecret, connecting to a federated server, or issues with the specified secret store.
  3. Sidecar Container Logs (for file-secrets: "true"):
    • If a sidecar container (secretless-sidecar by default) was injected, check its logs. 
      kubectl logs <your-pod-name> -n <your-namespace> -c secretless-sidecar
    • This will show esi-cli daemon mode activity, including attempts to fetch and write secrets.
  4. Inspect Shared Volume (for file-secrets: "true"):
    • Exec into your application container and check the contents of the /secrets directory: 
      kubectl exec <your-pod-name> -n <your-namespace> -c <your-application-container-name> -- ls -la /secrets
kubectl exec <your-pod-name> -n <your-namespace> -c <your-application-container-name> -- cat /secrets/secrets.json
  5. ExternalSecret Status:
    • Ensure the ExternalSecret resource specified in the pod’s annotation (secretless.externalsecrets.com/externalsecret) exists in the same namespace as the pod.
    • Check its status: 
      kubectl describe externalsecret <your-externalsecret-name> -n <your-namespace>
    • It should have a Status.Condition of type Ready with Status: True. If not, troubleshoot the ExternalSecret itself (e.g., issues with its SecretStore, permissions, or the external provider).
  6. RBAC Permissions for Webhook’s Service Account:
    • Verify the ClusterRole bound to the webhook’s ServiceAccount (e.g., esi-pod-webhook-sa in secretless-system) has sufficient permissions to:
      • get, list, watch ExternalSecret resources cluster-wide (or at least in namespaces it needs to operate on, though cluster-wide is common for webhooks).
      • get, list, watch SecretStore and ClusterSecretStore resources.
      • Potentially other permissions if using advanced esi-cli features that interact with other Kubernetes resources.
  7. esi-cli Image Issues:
    • If using custom images for --init-image or --sidecar-image in the webhook deployment, ensure these images are correctly built, contain a compatible esi-cli, and are accessible from your Kubernetes nodes (check image pull errors on the pod if image-pull-secrets are needed but not configured via annotation).
By systematically checking these areas, you can usually pinpoint the cause of issues with the ESI Pod Webhook.