Skip to navigation

Secrets and keys

This page applies to the Enterprise plan, where you run the data plane in your own VPC. The Helm chart never creates secret material for a real install. You create one Kubernetes Secret, the app Secret, with whatever tool you already use, and the chart references it. You also supply the control plane’s CA certificate. This page lists what each holds, shows how to create them, and explains how to rotate each value.

Prerequisites

Before you begin, make sure you have the following:

  • The namespace you install the data plane into.
  • Your MySQL password, if you use the MySQL secret store.
  • An enrollment method. Workload attestation needs no token. With a bootstrap token, mint it in the console first. See Connect the data plane to the control plane.

Keys in the app Secret

The gateway pods, the collector sidecar, and the schema migration Job all read from the app Secret. Include the keys that apply to your setup.

KeyWhen you need itSourceNotes
FABRIC_GW_SECRET_KEYAlwaysYou generate itThe 32-byte key that encrypts the credentials the data plane stores. Use the same value on every gateway pod.
FABRIC_GW_SECRET_KEY_OLDOnly while you rotate the keyYouRetired keys, comma separated. The data plane uses them to read values encrypted under an older key.
FABRIC_GW_MYSQL_PASSWith the MySQL secret storeYouThe runtime MySQL user’s password. The migration Job uses it too, unless you give migrations their own user.
FABRIC_GW_SYNC_BOOTSTRAP_TOKENOnly with bootstrap token enrollmentThe consoleThe token the data plane trades for its first certificate. Not needed with workload attestation.
FABRIC_GW_CLICKHOUSE_PASSWORDWith the chart’s default ClickHouse exporterYouIf the key is missing, the collector still starts, but telemetry isn’t exported.
FABRIC_GW_DISTRIBUTED_GOSSIP_KEYWith the distributed secret storeYou generate itMust be identical on every gateway pod. Don’t reuse the encryption key. Any non-empty string works, for example the output of openssl rand -base64 32.
FABRIC_GW_DISTRIBUTED_AUTH_PASSWORDWith the distributed secret storeYou generate itPassword for the store’s traffic between gateway pods. Generate it the same way, for example with openssl rand -base64 32.
FABRIC_GW_ADMIN_KEY_PRIMARY (and FABRIC_GW_ADMIN_KEY_BACKUP for rotation)To use the kill switch in production modeYouAuthenticates kill-switch requests. In production mode, the kill switch fails closed without it.
FABRIC_GW_COORD_REDIS_PASSWORDWith a coordination Redis that needs a passwordYouReferenced by FABRIC_GW_COORD_REDIS_AUTH_REF=env://FABRIC_GW_COORD_REDIS_PASSWORD.
OTEL_EXPORTER_OTLP_HEADERSWhen an OTLP backend needs an API keyYour backendA value in the Secret overrides the same name in the chart’s values.

The encryption key is yours

FABRIC_GW_SECRET_KEY doesn’t come from Postman, and it doesn’t need to match anything on the control plane. The control plane holds only references to credentials and vault settings, never the secret values, so the data plane’s key is independent of the control plane’s. Generate it once, keep it stable, and use the same value on every gateway pod.

The key is 32 bytes, given as raw bytes, hex, or base64. Generate one with:

openssl rand -hex 32 > secret-key.txt

Store the value without a trailing newline. The kubectl example in Create the app Secret strips it for you. If you create the Secret from a file or a secret manager, make sure the stored value is exactly the 64 characters.

Store the encryption key somewhere durable, such as your secret manager, before you install. Credentials your users connect are encrypted with it. If you lose the key, or pods start with different keys, the data plane can’t read those credentials, and users have to connect their accounts again.

The data plane runs in production mode by default. In production mode, a data plane without a usable encryption key refuses to start, and with the distributed store, so does a pod with no gossip key or no data-port password.

Create the app Secret

The default Secret name is fabric-gateway-chart-ext-secret. To use another name, set secrets.app.existingSecret in your values. The chart passes the name to the gateway, the collector sidecar, and the migration Job, so you don’t need to set it anywhere else.

  1. Generate the encryption key, and save it in your secret manager.

  2. Create the Secret in the data plane’s namespace. For example, with kubectl and the MySQL store:

    kubectl -n pm-fabric-dp create secret generic fabric-dp-secrets \
    --from-literal=FABRIC_GW_SECRET_KEY="$(cat secret-key.txt)" \
    --from-literal=FABRIC_GW_MYSQL_PASS='<mysql-password>'

    With a bootstrap token, also add --from-literal=FABRIC_GW_SYNC_BOOTSTRAP_TOKEN='<token-from-console>'. With the default ClickHouse exporter, add FABRIC_GW_CLICKHOUSE_PASSWORD.

  3. In your values file, point the chart at the Secret:

    secrets:
    app:
    existingSecret: fabric-dp-secrets

The Secret must exist before you run helm install, because the migration Job reads the MySQL password from it during the install.

You can produce the same Secret with any tool. The chart only needs a plain Kubernetes Secret with the right name and keys:

  • External Secrets Operator. Create an ExternalSecret whose spec.target.name is the Secret name the chart expects. The chart doesn’t ship an ExternalSecret.
  • Vault. Use the Vault CSI provider or Vault Agent to sync the secret into a Kubernetes Secret with that name.
  • sealed-secrets or SOPS. Commit the encrypted manifest to your GitOps repository. The controller decrypts it into a plain Secret with the same name.

To read the MySQL password from a Secret another chart owns, set mysqlPassword.existingSecret and mysqlPassword.passwordKey. Also set migrations.existingSecret and migrations.passwordKey to the same Secret and key. Otherwise the migration Job still reads FABRIC_GW_MYSQL_PASS from the app Secret.

The chart has a secrets.dev option that creates the Secret from Helm values. It’s for local testing only. Don’t use it in your VPC, because it puts secret values into Helm values and release history.

The control plane CA certificate

The data plane verifies the control plane’s sync server against the control plane’s CA certificate, read from /etc/fabric/sync/ca.crt (FABRIC_GW_SYNC_SERVER_CA). The certificate is public and has no private key, so you can supply it either way:

  • In your values. Set syncCA.pem to the PEM text. The chart renders a fabric-gateway-chart-sync-ca ConfigMap and mounts it. There’s no Secret to create.

    syncCA:
    pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
  • As a Secret. Leave syncCA.pem empty and create a Secret named fabric-dp-sync-ca with a ca.crt key:

    kubectl -n pm-fabric-dp create secret generic fabric-dp-sync-ca \
    --from-file=ca.crt=fabric-cp-ca.pem

To get the certificate, select Download CA bundle when you register an attestation anchor or mint a token in the console. Use the file exactly as it downloads, without editing it. When the data plane enrolls, it compares the bundle with the control plane’s current one. Because the chart mounts it read-only, a bundle that differs in any way makes enrollment fail with enrollment: write trust bundle. The console also shows a SHA-256 fingerprint, which you can use to check the file out of band.

Use a separate database user for migrations

By default, the migration Job connects as the runtime MySQL user, so that user needs DDL grants as well as SELECT, INSERT, UPDATE, and DELETE. To keep the runtime user to data grants only, give migrations their own user and Secret. See Database migrations.

Rotate secrets

The gateway reads the app Secret when a pod starts. After you change a value, restart the gateway so new pods pick it up:

kubectl -n pm-fabric-dp rollout restart deploy/fabric-gateway-chart

With bootstrap token enrollment, put a fresh token in FABRIC_GW_SYNC_BOOTSTRAP_TOKEN before you restart. With the chart’s default store, every new pod enrolls again.

If you use the distributed store, the gateway runs as a StatefulSet, so restart statefulset/fabric-gateway-chart instead.

With the distributed store, you can set container.probes.readiness to { path: /readyz/secret-store, port: 8080 }, so a rollout waits for each pod to re-replicate its entries before it stops the next one. The chart’s default is /healthz.

If a secret operator such as External Secrets Operator manages the Secret, rotate the value in your backend, wait for the Secret to update, and then restart.

Rotate the encryption key

When a pod starts with a new primary key and the old key in FABRIC_GW_SECRET_KEY_OLD, it re-encrypts the stored values under the new key, in the MySQL store and in the distributed store. Keys in FABRIC_GW_SECRET_KEY_OLD are used only to read values, never to write them, and you can list more than one, separated by commas. Pods that are still running with only the old key can’t read values encrypted under the new one, so credential lookups on those pods fail until they restart. To avoid that window, rotate in three restarts:

  1. Add the new key to FABRIC_GW_SECRET_KEY_OLD, keep the current key as FABRIC_GW_SECRET_KEY, and restart the gateway.
  2. Swap them: set the new key as FABRIC_GW_SECRET_KEY, put the old key in FABRIC_GW_SECRET_KEY_OLD, and restart again.
  3. After every pod runs the new key, restart the gateway once more. Check that each pod logs a line that starts with secret store re-encrypted.
  4. Remove the old key from FABRIC_GW_SECRET_KEY_OLD.

Rotate the enrollment token

With workload attestation there’s no token to rotate. The kubelet issues and rotates the pod’s projected token.

With a bootstrap token, a data plane needs the token only to enroll. After that, it renews its certificate with the certificate it already has. New pods, and pods that lost their disk, enroll again, so they need a token that’s still valid. A token lasts 30 minutes at most, so before you add or replace pods, mint a new token in the console, update FABRIC_GW_SYNC_BOOTSTRAP_TOKEN, and restart. For token lifetimes, see Connect the data plane to the control plane.

With token enrollment, leave a token in the Secret after the data plane enrolls. A data plane whose token key is removed from the Secret fails to start, even with a valid certificate. An expired token left in place is fine.

Rotate the control plane CA

When Postman rotates the control plane CA, update syncCA.pem (or the fabric-dp-sync-ca Secret) with the new bundle and run helm upgrade promptly, so the gateway pods restart with it. Until they do, they log config sync: rotation persist failed; continuing with existing material and can’t renew their certificates. With bootstrap token enrollment and the chart’s default store, put a fresh token in the app Secret first, because the new pods enroll again.

If you keep the CA in the Secret and run Stakater Reloader in the cluster, reloader.enabled: true restarts the gateway for you when the Secret changes. It also restarts the gateway when any other Secret or ConfigMap the pods use changes, including the app Secret. With bootstrap token enrollment, update the token in the same change.

Next steps