Skip to navigation

Configure a secret store

This page applies to the Enterprise plan, where you run the data plane in your own VPC. A data plane uses two kinds of secret storage. The secret store holds runtime state, such as the credentials your users connect. To choose one, see Choose a secret store. A vault holds the upstream secrets that you keep in your own system, such as an API key. This page explains how to choose each one, how to connect Azure Key Vault or HashiCorp Vault, and how to fix common errors.

Prerequisites

Before you begin, make sure you have the following:

Choose a secret store and a vault

LayerOptionsUse it for
Secret storedistributed-embedded, mysqlState that the data plane owns: credentials your users connect, OAuth state, and codes it issues. Always set.
VaultAWS Secrets Manager, Azure Key Vault, GCP Secret Manager, HashiCorp VaultSecrets that you store and rotate in your own system. Optional. Add one when a connection or plugin reads a secret by reference.

Choose the secret store first, because the data plane needs it to start:

  • distributed-embedded is the default on a data plane. The gateway pods keep the values in memory and replicate them to each other, so there’s no database to run. It needs the gossip key and data-port password in the app Secret.
  • mysql is optional. It stores the values in your MySQL database, for customers who prefer to run one. The Helm chart selects it unless you set distributed.enabled: true. When you use a vault for your secrets, you don’t need it.

The bbolt store, which is a local file, isn’t supported on a data plane. The gateway fails to start with secrets.type=bbolt is not supported on the dataplane; use distributed-embedded (default) or mysql.

Choose a vault by where your secrets already live. Use Azure Key Vault if you run on Azure, and HashiCorp Vault if you run it already. A vault is independent of the secret store, and you can register more than one.

Settings

The data plane reads these environment variables for the secret store. In the Helm chart, set them under env.vars.

VariableDefaultRequiredWhat it does
FABRIC_GW_SECRETS_TYPEdistributed-embedded on a data plane. The chart sets mysql unless you set distributed.enabled: true.NoSelects the secret store: distributed-embedded or mysql.
FABRIC_GW_SECRETS_DSNBuilt from the FABRIC_GW_MYSQL_* variablesNoThe MySQL connection string, used only with mysql. Leave it unset to use the chart’s MySQL settings.
FABRIC_GW_SECRETS_VAULT_WRITE_FALLBACKfalseNoWhen a write to your vault fails, such as when the identity lacks write permission, store the value in the data plane’s encrypted store instead. The gateway logs the fallback. With the default, a failed write returns an error.
FABRIC_GW_SECRET_KEYNoneYesThe key that encrypts stored values. See Secrets and keys.

Register a vault

You register a vault on the Vaults page. The control plane stores the vault’s settings and sends them to the data plane. The data plane then reads and writes secrets in your vault itself, so secret values don’t pass through the control plane.

  1. In the console, select Vaults under Admin, and then select Add.
  2. Enter a Name. You choose this name later, when you point a credential or plugin at the vault.
  3. For the provider, select Azure Key Vault or HashiCorp Vault.
  4. Enter the authentication fields in one of the following sections, and save the vault.

Both vault types have two optional settings:

SettingDefaultWhat it does
prefixNoneThe path prefix for the secrets that the gateway creates.
cache_ttl60sHow long the gateway caches a value it read. Use a Go duration such as 30s or 5m. An invalid value falls back to 60s.

Azure Key Vault

FieldRequiredWhat it does
vault_urlYesThe vault’s address, such as https://myvault.vault.azure.net. It must be HTTPS, with no port, path, or query. The host must be an Azure Key Vault host.
auth_modeNomanaged_identity, workload_identity, or client_secret. If you leave it empty, the gateway uses client_secret when you set a client secret, then workload_identity when the pod has AZURE_FEDERATED_TOKEN_FILE, and otherwise managed_identity.
client_idFor client_secret. Optional for managed_identity.With managed_identity, selects a user-assigned identity. Leave it empty to use the system-assigned identity.
tenant_idFor client_secretThe Microsoft Entra tenant. Not valid with managed_identity, which uses the identity’s own tenant.
client_secretFor client_secret onlyThe service principal’s secret. Not valid with the other modes. The gateway stores it encrypted.

The gateway doesn’t use the developer credentials on a node, such as the Azure CLI. It uses only the mode you select.

Choose the mode that fits where the data plane runs:

  • managed_identity for a node, VM scale set, or App Service identity. Set only vault_url, and add client_id for a user-assigned identity.
  • workload_identity for AKS. The pod’s ServiceAccount needs the azure.workload.identity/client-id annotation, and the pod needs the azure.workload.identity/use=true label. The AKS webhook then injects AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_FEDERATED_TOKEN_FILE.
  • client_secret when neither is available. You manage the secret’s rotation.

Give the identity one of these Azure roles on the vault:

RoleAllows
Key Vault Secrets UserReading secrets.
Key Vault Secrets OfficerReading, and also storing, rotating, and deleting the credentials that the gateway manages.

Use a Key Vault that’s dedicated to gateway-managed credentials. Key Vault deletes a whole secret, so if another writer adds versions to the same secrets, the gateway detects and rolls back the change, but it can’t block it atomically.

Example for AKS with the Helm chart. The chart sets the ServiceAccount annotation through serviceAccount.annotations:

serviceAccount:
create: true
annotations:
azure.workload.identity/client-id: "<managed-identity-client-id>"

The chart has no value for pod labels, so it can’t add the azure.workload.identity/use=true label for you. Add the label to the data plane’s pod template yourself, for example with a Helm post-renderer or a Kustomize patch. If you can’t, use managed_identity or client_secret instead.

Then register the vault in the console with vault_url set to https://<vault-name>.vault.azure.net and auth_mode set to workload_identity.

HashiCorp Vault

FieldRequiredWhat it does
addressYesThe Vault address, such as https://vault.internal:8200. It must be an http or https URL with a host, and no path, query, or user. Private addresses are allowed.
auth_typeYestoken, approle, or kubernetes.
tokenFor tokenA Vault token. The gateway stores it encrypted. It can’t renew itself, so use a token that doesn’t expire before you rotate it.
role_id and secret_idFor approleThe AppRole credentials. The gateway stores secret_id encrypted. Logs in on the approle auth mount.
roleFor kubernetesThe Vault role that the pod’s ServiceAccount token logs in as. The gateway reads the token from /var/run/secrets/kubernetes.io/serviceaccount/token and logs in on the kubernetes auth mount.
namespaceNoA Vault Enterprise namespace.
mountNoThe KV engine’s mount path. Default: secret.
kv_versionNo1 or 2. Default: 2.

With approle and kubernetes, the gateway logs in again before the token expires, and once more if Vault returns 403.

Give the identity a Vault policy for the paths the gateway uses. For KV v2 with the default mount, the gateway reads secret/data/<path>. To store and delete secrets that it manages, it also writes secret/data/<path> and writes, reads, and deletes secret/metadata/<path>. It stamps each secret it creates with managed-by: fabric-gateway, and deletes only secrets that carry it. The health check calls auth/token/lookup-self.

path "secret/data/*" {
capabilities = ["read", "create", "update"]
}
path "secret/metadata/*" {
capabilities = ["read", "create", "update", "delete"]
}

For read-only use, keep only read on secret/data/*. With KV v1, the gateway can read and write, but it declines to delete, because v1 can’t carry the marker.

Reference a secret

A credential or plugin points at a secret with a vault and a secret reference. In the console, select the vault by name, and then enter the reference. For a plugin, see Secret Injector. Where you enter a secret directly, on a data plane that you run, you enter a reference instead of the value.

VaultReference you can enterNotes
Azure Key VaultThe secret’s name, or its full identifier: https://<vault-name>.vault.azure.net/secrets/<name> or .../secrets/<name>/<version>The identifier must belong to the vault you selected. Enter a Version only if you didn’t put one in the identifier, and the two must match.
HashiCorp VaultA path under the mount, such as partner/api-key, or the full read URL: https://vault.internal:8200/v1/secret/data/partner/api-keyThe URL must match the vault’s address and mount. A secret that the gateway stores is returned as the full URL. Version must be a number.

If a secret has more than one field, enter Property to pick one. A secret with a single value field needs no property.

For secrets that the gateway creates, such as a managed credential, the console stores and returns the full identifier or URL. Keep the full form for those. The gateway refuses a bare Azure name for a managed secret.

Common errors

Each message is in the gateway log. The data plane redacts the secret reference.

MessageCause and fix
azure_key_vault: vault_url must be an https URL such as https://<name>.vault.azure.netThe URL isn’t HTTPS. Correct it.
azure_key_vault: vault_url must not include a port, path, userinfo, query, or fragmentUse only https://<name>.vault.azure.net.
azure_key_vault: vault_url host ... is not an Azure Key Vault hostThe host isn’t a Key Vault domain. Check the name.
azure_key_vault: unknown auth_mode ... (want managed_identity, workload_identity or client_secret)Fix the auth_mode spelling.
azure_key_vault: client_secret auth requires auth.tenant_id and auth.client_id, or auth_mode client_secret requires auth.client_secretAdd the missing fields, or switch to managed_identity.
azure_key_vault: auth.client_secret must not be set with auth_mode ... or auth.tenant_id does not apply to auth_mode managed_identityRemove the field that the mode doesn’t use.
azure credentials: workload_identity (is the pod labelled azure.workload.identity/use=true and its ServiceAccount annotated with azure.workload.identity/client-id?)Add the label and annotation, and restart the pods.
azure kv get ... or azure kv set ... with 403The identity lacks a role. Add Key Vault Secrets User to read, or Key Vault Secrets Officer to write.
azure kv: ref ... is not a secret in https://...The reference points at another vault. Use this vault’s identifier or the secret’s name.
azure kv: secret exists and is not managed by fabric-gatewayThe gateway won’t overwrite or delete a secret it didn’t create. Use a different name or a dedicated vault.
azure kv: encoded secret name exceeds 127 characters; shorten the vault prefixShorten the vault’s prefix.
hashicorp_vault: unsupported auth_type ... (want token, approle, or kubernetes)Fix auth_type.
hashicorp_vault: approle auth requires auth.role_id (or secret_id, role, token)Add the missing field for that auth_type.
hashicorp_vault: kv_version ... must be 1 or 2Set kv_version to 1 or 2, or leave it empty.
hashicorp_vault: address ... must not include a path, query, fragment, or userinfoUse scheme://host[:port] only.
hashicorp_vault: reading kubernetes service account tokenThe pod has no ServiceAccount token at the expected path.
hashicorp_vault ... login: ...Vault rejected the login. Check the role, the auth mount, and the Vault policy.
hashicorp_vault: ref ... is not a secret in ...The URL doesn’t match the vault’s address and mount. Use a path under the mount.
hashicorp_vault: refusing redirect off the configured originVault redirected to another host. Set address to the node that serves requests.
hashicorp_vault delete skipped: secret is not managed by fabric-gateway (warning)The secret wasn’t created by the gateway, so it stays.

The console’s Dependencies view shows whether your vault is reachable. See Monitor the data plane.

Next steps