Secrets store CSI driver vs external secrets in a nutshel

• Intro
• Architecture
• Usage
• Main differences
• Secret changes management

Intro

I think Secret is one of the most critical resource in Kubernetes because, as you know, it contains sensitive data that must be secured and managed based on projecs/companies requirements.

Delegates secret contents to external specialized tools like AWS Secret Manager, Hashicorp Vault, etc is the first step to start taking the governance of them in a professional way. In this blog post I will try to put on the table the main two solutions in the "market":

• External Secret Operator (ESO)
• Secrets Store CSI Driver
  I already wrote a CSI deep dive in an other article, I'll suggest to read it before going on to better understand from where comes from and how it works.

How Kubernetes CSI driver works

Explanation of how CSI driver works in Kubernetes in order to be ready to troubleshoot!

YuBaYuri Bacciarini

I'll try to explain the architecture, how to to use, main differences between them and a though about secret changes/rotation management.

Architecture

External Secret Operator (ESO)

External Secret Operator acts as a top layer of the kubernetes kind=Secret resource. Mainly, though ESO, we can totally delegates secrets' content to an external vault while continue to use secrets as before. ESO, once configured using its owned CRDs, will automatically generate the original kind:secret resources for us.

Some supported external vaults are: AWS Secrets Manager, Azure Key Vaul, GCloud Secret Manager, Hashicorp Vault, IBM Secrets Manager, Oracle Vault and many others (full list here https://external-secrets.io/main/provider/aws-secrets-manager/)

ESO has mainly 3 CRDs objects:

• SecretStore (namespaced, defined one time. It will allow ExternalSecret objects to use a specific external provider)
• ClusterSecretStore (as secret store but cluster wide)
• ExternalSecret (it will define the secret template using a specific SecretStore/ClusterSecretStore, automatically ESO will create the real kind:secret based on this template)

SecretStore Architecture

ClusterSecretStore Architecture

Secrets Store CSI Driver

The Kubernetes Secrets Store CSI (Container Storage Interface) Driver is a Kubernetes volume plugin that allows you to securely retrieve secrets from an external secret management system and mount them as files into your Kubernetes pods. It enables you to integrate Kubernetes with various secret management systems, such as Azure Key Vault, HashiCorp Vault, or AWS Secrets Manager (full list here https://kubernetes-csi.github.io/docs/drivers.html).

More details can be found in the following blog post where I focused on CSI driver in Kubernetes.

How Kubernetes CSI driver works

Explanation of how CSI driver works in Kubernetes in order to be ready to troubleshoot!

YuBaYuri Bacciarini

Usage

External Secrets Operator

SecretStore

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

ExternalSecret
once defined you can easily use "example" directly as a regular secret.
Also, you can easly check with k get secrets that a new secret named example was automatically created by ESO.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: secret-to-be-created
    creationPolicy: Owner
  data:
  - secretKey: secret-key-to-be-managed
    remoteRef:
      key: provider-key
      version: provider-key-version
      property: provider-key-property
  dataFrom:
  - extract:
      key: remote-key-in-the-provider

Secrets Store CSI Driver

SecretProviderClass
Take note that secretObjects is optional and can be useful when we need to use it as a Kubernetes regular secret, otherwise by default, secret defined with Secrets store CSI driver could be only mounted as a pod volume.
See the next two examples.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: azure-sync
spec:
  provider: azure
  secretObjects:                                 # [OPTIONAL] SecretObject defines the desired state of synced K8s secret objects
  - secretName: foosecret
    type: Opaque
    labels:
      environment: "test"
    data:
    - objectName: secretalias                    # name of the mounted content to sync. this could be the object name or object alias
      key: username
  parameters:
    usePodIdentity: "true"
    keyvaultName: "$KEYVAULT_NAME"               # the name of the KeyVault
    objects: |
      array:
        - |
          objectName: $SECRET_NAME
          objectType: secret                     # object types: secret, key or cert
          objectAlias: secretalias
          objectVersion: $SECRET_VERSION         # [OPTIONAL] object versions, default to latest if empty
        - |
          objectName: $KEY_NAME
          objectType: key
          objectVersion: $KEY_VERSION
    tenantId: "tid"                             # the tenant ID of the KeyVault

Than you can use it as a secret or directly as a pod volume.

Example of using as tipical volume (like a PVC)

kind: Pod
apiVersion: v1
metadata:
  name: secrets-store-inline
spec:
  containers:
    - name: busybox
      image: registry.k8s.io/e2e-test-images/busybox:1.29
      command:
      - "/bin/sleep"
      - "10000"
      volumeMounts:
      - name: secrets-store01-inline
        mountPath: "/mnt/secrets-store"
        readOnly: true
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-sync"

Or using as a regular secret

spec:
  containers:
  - image: registry.k8s.io/e2e-test-images/busybox:1.29
    name: busybox
    command:
    - "/bin/sleep"
    - "10000"
    env:
    - name: SECRET_USERNAME
      valueFrom:
        secretKeyRef:
          name: foosecret
          key: username

Main differences

Secrets Store CSI Driver and External Secrets Operator

Putting aside the version, both mainly have the same features but there are some things to take in mind before the adoption:

• ⚠️ Backward compatibility
  TL;DR
  ✖️ Secrets Store CSI driver  
  ✅ ESO
  ---
  If you are thinking to do a migration from homemade managed k8s secrets to an external vault you have of course to deal with backward compatibility.
  Unfortunataly Secrets Store CSI driver is not backward compatible.
  It means that you need to change your applications yaml (deployments, statefulesets, etc) as below (ex. https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-driver#set-an-environment-variable-to-reference-kubernetes-secrets)

kind: Pod
apiVersion: v1
metadata:
  name: busybox-secrets-store-inline
spec:
  containers:
	 ....
      env:
      - name: SECRET_USERNAME
        valueFrom:
          secretKeyRef:
            name: foosecret
            key: username
  volumes:
    - name: secrets-store01-inline
      csi:
        driver: secrets-store.csi.k8s.io <---
        readOnly: true
        volumeAttributes:
          secretProviderClass: "azure-sync"

✅ Using instead ESO you are totally backword compatible (your lovely yaml can remain the same)

• imagePullSecrets

imagePullSecrets is commonly used to pull images from authenticated container registries.
While using ESO you will not have problems, the same could NOT happen while using CSI Driver for its design.

Some deep details in the following discussions
https://github.com/aws/secrets-store-csi-driver-provider-aws/issues/16 https://www.reddit.com/r/kubernetes/comments/yuot5z/secrets_csi_and_imagepullsecrets/ https://github.com/Azure/secrets-store-csi-driver-provider-azure/discussions/707

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
spec:
  containers:
  - name: private-reg-container
    image: <your-private-image>
  imagePullSecrets:
  - name: regcred <----

• While using Secrets Store CSI Driver, the secrets will only sync once you start a pod mounting the secrets. Solely relying on the syncing with Kubernetes secrets feature thus does not work. When all the pods consuming the secret are deleted, the Kubernetes secret is also deleted.
  This means that there's a strict correlation between them and above all we are not able to read the content of a secret if not mounted to a pod 😞😞😞
  This last thing not happen when using external secret solution because ESO works mainly as a kind:secret generator starting from the kind: ExternalSecret CRD template. So using ESO you are able to read the content of your kind:secret whenever you want.
• The Secrets Store CSI Driver daemonset runs as root in a privileged pod.
  The provider plugins are also required to run as root (though privileged should not be necessary). This is because the provider plugin must create a unix domain socket in a hostPath for the driver to connect to.
  More details here https://secrets-store-csi-driver.sigs.k8s.io/concepts.html#security
• ESO could be also used in other use cases outside Kubernetes, for example to manage Jenkins credentials https://external-secrets.io/main/examples/jenkins-kubernetes-credentials/
  I personally never tried yet but it could be an opportunity to understand a technology and use widely also outside Kubernetes.

Secret changes management

In both solutions, it's essential to take in mind how to manage secrets changes on the external secret provider, for example on an automatic or manual secret rotation.
A more real example is when a TLS certificate is updated (usually every year). This activity usually takes place on the external secret provider (ex. Hashicorp vault), than must be managed also in every place it is used.

Most common places in which it could be used is inside some Kubernetes components like ingress, service mesh configuration and so on. So we have to manage also this kind of refresh from "users" side.

There is no an unique answer to this challange, it depends on the use cases and their requirements.

For sure we need to know that ESO has a related  refreshInterval config

RefreshInterval is the amount of time before the values are read again from the SecretStore provider Valid time units are “ns”, “us” (or “µs”), “ms”, “s”, “m”, “h” May be set to zero to fetch and create it once. Defaults to 1h.

On the other hand, Secrets Store CSI Driver guarantees that will periodically update pod's mount content and the Kubernetes Secrets but it does not allow any configurations regarding this behaviour (for example change frequency).

Finally take in mind that, for both strategies, using Kubernetes secret for environment variables, it will not allow pods to automatically read updated environments after a secret change on the external provider. In this case an external tool to detect secrets changes than restart pods it's needed, for example https://github.com/stakater/Reloader.

Mounting instead the secrets as volume, it will automatically be updated by Kubernetes itself, the effect is that pods will automatically see the updated volume content. More details regarding this native Kubernetes behaviour:

• https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod
• https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/main/docs/book/src/topics/secret-auto-rotation.md

In this last case we have to know if the ecosystem (ex. a service mesh configuration, app configurations) is prepared to automatically detects these changes on file system and live reload itself to apply the changes.