Skip to navigation

Connect the data plane to the control plane

This page applies to the Enterprise plan, where you run the data plane in your own VPC. Before a data plane serves any traffic, it enrolls with the control plane: it proves which tenant it belongs to, gets a certificate, and opens an mTLS connection that carries your configuration. This page explains how enrollment works, how to set it up with workload attestation or a bootstrap token, and what the console shows afterward.

How enrollment works

  1. When a gateway pod starts, it checks its local disk for a certificate. A certificate with more than 5 minutes left and a matching private key is reused, and the pod skips the next two steps.
  2. Otherwise, the pod generates a private key, which never leaves the pod, and sends a certificate request to the control plane’s sync address. It authenticates with its Kubernetes service account token or with a bootstrap token, over TLS that it verifies against the control plane CA certificate you supplied.
  3. The control plane checks the credential and issues a certificate for exactly one tenant and node.
  4. The pod opens its sync connection over mTLS and receives your configuration and policy.
  5. The pod serves client requests with that configuration. It starts listening once it’s enrolled, so a request that arrives before the first configuration can get a 503 or a 404.

Each pod is a separate node. A node’s ID defaults to its pod hostname, so every replica enrolls under its own name.

About 12 hours before the certificate expires, the pod renews it in the background, using its current certificate. Renewal doesn’t interrupt the sync connection or client traffic. If the control plane is unreachable, the pod keeps retrying, backing off from 30 seconds up to 30 minutes, until the certificate expires.

If enrollment fails, the pod exits before it ever listens, and Kubernetes restarts it. A pod stuck in CrashLoopBackOff right after install usually has an enrollment problem, not a Kubernetes one. See Troubleshoot enrollment.

Prerequisites

Before you begin, make sure you have the following:

  • Your tenant ID and the control plane’s sync address from Postman.
  • Access to Admin > Gateway health in the console.
  • The data plane’s namespace, and its app Secret or a plan to create it. See Secrets and keys.

To mint bootstrap tokens and revoke certificates, you need the View gateway health permission. To register a workload attestation anchor, you also need the Super admin role in your organization.

Choose an enrollment method

MethodHow the pod proves its identityBest for
Workload attestation (recommended)Its own projected Kubernetes service account token. There’s no shared secret to distribute, rotate, or leak.Every Kubernetes install, especially autoscaled data planes and pods that are recreated often
Bootstrap tokenA short-lived token you mint in the console and put in the app SecretClusters where the control plane can’t verify the cluster’s service account tokens

With attestation, the control plane trusts one service account in one namespace of your cluster, and maps it to your tenant. The tenant comes from that trust anchor, never from the token, and the node ID is the pod’s name.

Bootstrap tokens come in two kinds:

  • Single-use token. Bound to one node ID, and redeemed once.
  • Reusable join token. Bound only to the tenant. Every pod can redeem it until it expires, and each pod still gets its own certificate. Use this for a data plane with more than one replica.

You choose a token’s lifetime in minutes when you mint it. It defaults to 10 minutes, and the control plane keeps it between 1 and 30 minutes. Install the chart soon after you mint the token.

With the chart’s defaults, a pod keeps its certificate on a volume that’s deleted with the pod. A pod that’s rescheduled, replaced in a rollout, or added by an autoscaler has no certificate, so it enrolls again. With a bootstrap token, it needs a token that’s still valid, and once the token expires those pods can’t start. Workload attestation avoids this, because the kubelet issues and rotates the pod’s token for you.

Enroll with workload attestation

  1. Find your cluster’s service account token issuer. It must be an https URL. On EKS, run:

    aws eks describe-cluster --name <cluster> --query cluster.identity.oidc.issuer --output text

    On other clusters, read the issuer field from kubectl get --raw /.well-known/openid-configuration.

  2. In the console, open Admin, and then select Gateway health.

  3. Select Enroll data plane, and then select Workload attestation.

  4. Enter the following:

    • Cluster OIDC issuer URL: the issuer from step 1.
    • Namespace: the namespace the data plane runs in, for example pm-fabric-dp.
    • Service account: the gateway’s ServiceAccount. The chart names it <release-name>-sa, for example fabric-dp-sa, unless you set serviceAccount.name.
    • Audience: fabric-cp. This must equal attestation.audience in your values, which defaults to fabric-cp.
    • Tenant ID: keep the prefilled value. The field is required, but the anchor always maps to the tenant you’re signed in to.
  5. Under Cluster keys (JWKS), select one of the following:

    • JWKS URL: leave the field blank to let the control plane discover your cluster’s keys from the issuer. This needs the issuer to be reachable from the internet.
    • Paste JWKS: for a private cluster that doesn’t expose its keys. Paste the output of kubectl get --raw /openid/v1/jwks. The control plane then verifies tokens offline.
  6. Select Register anchor.

  7. Under Control plane CA certificate, note the Sync endpoint and select Download CA bundle.

  8. In your values file, turn on attestation and set the tenant, sync address, and CA certificate:

    attestation:
    enabled: true
    syncCA:
    pem: |
    <contents of the CA bundle>
    env:
    vars:
    FABRIC_GW_TENANT_ID: "<your-tenant-id>"
    FABRIC_GW_CP_ADDRESS: "<sync-endpoint>"
  9. Install or upgrade the chart. See Install with Helm.

The console also shows Data plane env and Data plane pod volume snippets. With the Helm chart you don’t need them: attestation.enabled mounts a projected service account token for the audience at /var/run/secrets/fabric-cp/token and sets the matching environment variables. The token is valid for an hour (attestation.expirationSeconds: 3600), and the kubelet replaces it before it expires. attestation.expirationSeconds must be at least 600, and attestation.mountPath changes where the token mounts.

To move an existing data plane from a bootstrap token to attestation, register the anchor and upgrade with attestation.enabled: true. The upgrade replaces every gateway pod. With the chart’s default store, each new pod enrolls through attestation. With the distributed store, each pod keeps its certificate on its own volume and uses attestation the next time it enrolls. Kubernetes replaces the pods in a rolling update, so the old pods keep serving while the new ones start. If the new pods can’t enroll, the rollout stops. Check that the anchor matches the pod’s namespace, service account, and audience.

The gateway doesn’t fall back to the bootstrap token on its own. To roll back, set attestation.enabled: false, put a fresh token in the app Secret, and upgrade again.

Enroll with a bootstrap token

  1. In the console, open Admin, and then select Gateway health.

  2. Select Enroll data plane, and then select Bootstrap token.

  3. Do one of the following:

    • For a data plane with more than one replica, turn on Reusable join token.
    • For a single node, enter its Node ID. A single-use token enrolls exactly one pod, once. With the Helm chart, set replicaCount: 1, and set env.vars.FABRIC_GW_NODE_ID to the node ID you enter here. If the pod is replaced, it needs a new token. For any other setup, use a reusable join token. If you turn on autoscaling (keda.enabled or autoscaling.enabled), also set scalingConfig.scalingProperties.minReplicas and maxReplicas to 1.
  4. In Token lifetime (minutes), enter 30, the maximum, unless you’re ready to install right away. If you leave it blank, the token lasts 10 minutes.

  5. Select Mint join token or Mint token.

  6. Copy the token. The console shows it only once.

  7. Under Control plane trust anchor, note the Sync endpoint and select Download CA bundle.

  8. Select the checkbox to confirm you saved the token, and then select Done.

  9. Add the token to the app Secret as FABRIC_GW_SYNC_BOOTSTRAP_TOKEN. See Secrets and keys.

  10. In your values file, set the tenant, sync address, and CA certificate:

    syncCA:
    pem: |
    <contents of the CA bundle>
    env:
    vars:
    FABRIC_GW_TENANT_ID: "<your-tenant-id>"
    FABRIC_GW_CP_ADDRESS: "<sync-endpoint>"
  11. Install or upgrade the chart before the token expires.

FABRIC_GW_TENANT_ID must be the tenant the token was minted for. If they differ, the control plane refuses enrollment with an identity mismatch.

The console also shows a config snippet in the gateway’s YAML format. With the Helm chart you don’t need it: the chart already turns enrollment on, and the values above set the same settings.

Anyone with a join token can enroll a node under your tenant until the token expires. Store it like a secret.

The control plane CA certificate

A new data plane trusts nothing yet. It verifies the control plane’s sync server against the control plane’s CA certificate, which the chart mounts at /etc/fabric/sync/ca.crt. The certificate is public and has no private key, so you can put it in your values as syncCA.pem or create it as a Secret.

Both enrollment flows in the console offer the CA bundle for download, with a SHA-256 fingerprint you can use to check the file out of band. Use the bundle exactly as it downloads, without editing it. If it differs from the control plane’s current bundle in any way, enrollment fails with enrollment: write trust bundle. For both ways to supply it and how to rotate it, see Secrets and keys.

Check the data plane in the console

In the console, open Admin, and then select Gateway health. Each gateway pod appears as a node once it opens its sync connection. The page shows the following:

  • Connected, Health, Sync, and Versions across the fleet. Sync reads All in sync when every node has the latest configuration.
  • A table of nodes with Node, Status, Health, Version, Config, Policy, Certificate, Uptime, and Last seen. A node’s status is Healthy, Lagging, or Disconnected.

Select a node to see its details, including its tenant, mTLS subject, the configuration and policy versions it acknowledged, and its certificate’s serial, issuer, expiry, and last renewal.

If a node’s certificate is compromised, or you’re retiring the node, open the node and select Revoke certificate. Choose a reason and confirm. The control plane closes the node’s sync connection and refuses its reconnects.

A revoked or expired node can’t receive configuration updates, but it keeps serving traffic with the last configuration it applied until you stop it. To retire a node, revoke its certificate and also delete the pod.

Troubleshoot enrollment

Check the gateway container’s logs first:

kubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c fabric-gateway-chart

If the pod is restarting, add --previous to see the log of the attempt that failed. With the distributed store, use statefulset/fabric-gateway-chart in place of deploy/fabric-gateway-chart.

Each cause logs its own message. Find it in the log, and then use this table.

Log messageCauseFix
enrollment: bootstrap token expiredThe token’s lifetime ended before the pod enrolled.Mint a new token, update FABRIC_GW_SYNC_BOOTSTRAP_TOKEN in the app Secret, and restart the gateway.
enrollment: tenant_id/node_id do not match the bootstrap token claimsFABRIC_GW_TENANT_ID isn’t the tenant the token was minted for, or a single-use token’s Node ID doesn’t match the pod.Set FABRIC_GW_TENANT_ID to your tenant ID. For more than one replica, use a reusable join token.
enrollment: bootstrap token already consumed (replay denied)A single-use token was already redeemed, by another replica or by a pod that was replaced.Mint a new token. For more than one replica, use a reusable join token.
attestation deniedThe pod’s service account token doesn’t match the anchor’s issuer, namespace, service account, or audience. The message doesn’t say which.Compare each anchor field with the values from Enroll with workload attestation.
enrollment: sync.bootstrap_token is empty under production_modeThe app Secret has no token, and attestation is off.Add FABRIC_GW_SYNC_BOOTSTRAP_TOKEN to the app Secret, or set attestation.enabled: true.
enrollment: server rejected enroll call, followed by a dial errorFABRIC_GW_CP_ADDRESS is wrong, or egress to the sync address is blocked.Set the address to the Sync endpoint the console shows, and allow egress to it. See Network and egress.
enrollment: write trust bundleThe CA bundle you supplied doesn’t match the control plane’s current bundle.Replace it with the file from Download CA bundle, unedited. See The control plane CA certificate.

For problems after enrollment, such as failed renewals or nodes that show Disconnected, see Troubleshoot the data plane.

Next steps