Troubleshoot the data plane
This page applies to the Enterprise plan, where you run the data plane in your own VPC. It lists common data plane problems by symptom, with the likely cause and the fix. For problems with individual requests, such as a 401 or a 429, see Troubleshooting.
The commands use the release name fabric-dp in the namespace pm-fabric-dp. Start with these three:
# Pods and their statekubectl -n pm-fabric-dp get pods# Events and probe failures for one podkubectl -n pm-fabric-dp describe pod <pod-name># The gateway container's logskubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c fabric-gateway-chart
deploy/fabric-gateway-chart reads one pod, chosen for you. To read a specific pod, use its name instead. If the pod is restarting, add --previous to see the logs of the container that exited:
kubectl -n pm-fabric-dp logs <pod-name> -c fabric-gateway-chart --previous
With the distributed store, the gateway is a StatefulSet, so use statefulset/fabric-gateway-chart in place of deploy/fabric-gateway-chart.
Install and migrations
With the MySQL store, every install and upgrade runs a schema migration Job before any gateway pod changes. A failed migration stops the release, so the gateway never runs against a database it doesn’t match. To inspect a failed migration:
kubectl -n pm-fabric-dp get jobs -l app.kubernetes.io/name=fabric-gateway-chartkubectl -n pm-fabric-dp logs job/fabric-gateway-chart-dp-migrate
| Symptom | Likely cause | Fix |
|---|---|---|
Install stops at the migration Job, and its pod shows CreateContainerConfigError | The app Secret doesn’t exist yet, or has no database password key | Create the app Secret before helm install, with the key named by migrations.passwordKey (FABRIC_GW_MYSQL_PASS by default). See Secrets and keys. |
Migration fails with Access denied or CREATE command denied | The database user the migration connects as lacks grants | Grant CREATE, ALTER, INDEX, SELECT, INSERT, UPDATE, and DELETE to that user, or set migrations.dbUser to a user that has them |
| Migration fails with a connection timeout | FABRIC_GW_MYSQL_HOST is wrong or unreachable from the cluster | Fix the host or the network path. The Job retries a slow database 10 times (migrations.connectRetries), but can’t fix a wrong address. |
Migration fails with RSA public key is not available client side | MySQL 8’s default authentication, over a connection without TLS, needs a key exchange the migration Job’s driver doesn’t do | Reach MySQL over TLS, use a database user created with mysql_native_password, or run migrations yourself with migrations.enabled: false. See Database migrations. |
Migration fails with Unknown database | The database doesn’t exist. The chart creates tables, not the database. | Create the database named in FABRIC_GW_MYSQL_DATABASE before you install |
Migration fails with a connection timeout, and MySQL listens on a port other than 3306 | The migration Job ignores FABRIC_GW_MYSQL_PORT and connects on 3306 | Put the port in the host instead: FABRIC_GW_MYSQL_HOST: "<host>:<port>". The gateway accepts this form too. |
Upgrade fails with Migration checksum mismatch | A migration that was already applied was edited | Restore the chart’s original SQL files. Never edit an applied migration. |
On a database that already had the schema, early migrations such as V001 were never applied | migrations.baselineVersion was raised above "0", so Flyway skipped every migration at or below it | Keep migrations.baselineVersion: "0", the default |
| You want to run migrations from your own pipeline | Set migrations.enabled: false, and apply the chart’s SQL files in order before you roll out the gateway. See Database migrations. |
Pods don’t start or aren’t ready
| Symptom | Likely cause | Fix |
|---|---|---|
Pods never become Ready, and probes report 404 | Probes point at a path or port other than /healthz on 8080 | Keep container.probes.liveness, readiness, and startup at /healthz on port 8080. The gateway doesn’t serve /ping or /health. |
| Probes fail with connection refused, and the pod restarts | The pod hasn’t enrolled. A gateway that hasn’t enrolled doesn’t listen. | See Enrollment and the control plane |
Pod exits at startup, and logs show production_mode: secret store requires an encryption key | Production mode is on (the default) and FABRIC_GW_SECRET_KEY is missing | Add the key to the app Secret. See Secrets and keys. |
Gateway pod in ImagePullBackOff, on the fabric-gateway-chart container | The gateway image isn’t in a registry the cluster can pull from, or the pull Secret is missing | Mirror the image, set image.repository, and set image.imagePullSecrets if your registry needs credentials |
Gateway pod in InvalidImageName | image.repository is still the chart’s <your-registry>/fabric/gateway placeholder | Set image.repository to your mirror |
Gateway pod in ImagePullBackOff, on the otel-collector container | The collector image, otel/opentelemetry-collector-contrib:0.149.0, can’t be pulled | Mirror it. To change the image, copy the chart’s whole otel-collector entry into additionalContainers in your values file, and change only image. image.imagePullSecrets covers this container too. |
Gateway pod in ImagePullBackOff, on the opa container | The OPA image, openpolicyagent/opa, can’t be pulled | Mirror it, and set opa.image.repository and opa.image.tag. image.imagePullSecrets covers this container too. |
Embedder pod in ImagePullBackOff | The embedder image, ghcr.io/huggingface/text-embeddings-inference:cpu-1.9, can’t be pulled | Mirror it, set embedder.image.repository, and set embedder.image.imagePullSecrets if your registry needs credentials |
Migration Job pod in ImagePullBackOff | The Flyway image, flyway/flyway:10, can’t be pulled. The Job doesn’t use image.imagePullSecrets. | Mirror it, and set migrations.image. If your registry needs credentials, add the pull Secret to the namespace’s default ServiceAccount, which the Job runs as. |
Pod exits at startup, and logs show secrets: refusing to start without global.tenant_id in production_mode | FABRIC_GW_TENANT_ID is empty, as in the chart’s defaults | Set env.vars.FABRIC_GW_TENANT_ID to the tenant ID Postman gave you |
Pod exits at startup, and logs show opening mysql secret store: pinging mysql | The gateway can’t reach or log in to MySQL | Check FABRIC_GW_MYSQL_HOST, FABRIC_GW_MYSQL_USER, and the password, and allow traffic from the gateway pods to the database |
Pod exits at startup, and logs show key must be exactly 32 raw bytes, 64 hex chars, or 44 base64 chars | FABRIC_GW_SECRET_KEY has the wrong length. got 65 bytes usually means a trailing newline. | Store the key without a trailing newline. See Secrets and keys. |
Gateway pod shows CreateContainerConfigError, with the distributed store | The app Secret doesn’t exist. With the distributed store there’s no migration Job, so the gateway pod is the first to need it. | Create the app Secret before you install. See Secrets and keys. |
Pod stays in ContainerCreating, and its events show that the fabric-dp-sync-ca Secret isn’t found | syncCA.pem is empty, and the fabric-dp-sync-ca Secret doesn’t exist | Set syncCA.pem, or create the Secret. See Secrets and keys. |
Pod stays Pending, and its PersistentVolumeClaim isn’t bound | The cluster has no default StorageClass, and the chart’s volume doesn’t name one | Set distributed.persistence.storageClass for the distributed store, or embedder.persistence.storageClass for the embedder cache |
Pod reads READY 1/2 (or 2/3 with OPA) | One container isn’t ready. During startup or an enrollment problem, it’s usually the gateway container. With opa.enabled: true, 2/3 also appears when the opa container’s readiness check fails, which is a known chart issue. | Run kubectl -n pm-fabric-dp describe pod <pod-name> to see which one. If it’s fabric-gateway-chart, see Enrollment and the control plane. If it’s otel-collector, check its logs with -c otel-collector, and see Telemetry. |
| Values are back to the chart defaults after an upgrade | The upgrade ran without -f or --reuse-values | Run the upgrade again with your values file |
To add a pull Secret to the namespace’s default ServiceAccount, run the following. It replaces any pull Secrets the ServiceAccount already lists.
kubectl -n pm-fabric-dp patch serviceaccount default \-p '{"imagePullSecrets":[{"name":"<pull-secret>"}]}'
Enrollment and the control plane
A gateway pod enrolls with the control plane before it serves any traffic. If enrollment fails, the pod exits and Kubernetes restarts it, so a pod in CrashLoopBackOff right after install usually has an enrollment problem. Look for the dp enrollment line in the gateway logs. A join token, which the console mints with Mint join token, is a reusable bootstrap token.
| Symptom | Likely cause | Fix |
|---|---|---|
Logs show enrollment: bootstrap token expired, or new pods fail after weeks of working | The bootstrap token passed its lifetime, 10 minutes by default and 30 minutes at most. Pods with a valid certificate keep working, but new pods need a valid token. | Mint a new token, update the app Secret, and restart the gateway. Consider workload attestation, which needs no token. See Connect the data plane to the control plane. |
Logs show tenant_id/node_id do not match the bootstrap token claims | FABRIC_GW_TENANT_ID differs from the tenant the token was minted for, or, with a single-use token, the pod’s node ID differs from the Node ID you entered | Set FABRIC_GW_TENANT_ID to the tenant ID Postman gave you. For more than one replica, or any pod that can be rescheduled, mint a reusable join token. |
Pod exits at startup, and logs show enrollment: sync.bootstrap_token is empty under production_mode | The app Secret has no FABRIC_GW_SYNC_BOOTSTRAP_TOKEN, and workload attestation is off | Add the token to the app Secret, or set attestation.enabled: true |
Logs show bootstrap token already consumed (replay denied) | A single-use token was already redeemed, for example by another replica or by a pod that was replaced | Mint a new token. For more than one replica, or any pod that can be rescheduled, use a reusable join token or workload attestation. |
Logs show attestation denied | The pod’s service account token doesn’t match the anchor’s issuer, namespace, service account, or audience. The message doesn’t say which. | Compare the anchor you registered with the cluster’s issuer, the data plane’s namespace, its ServiceAccount (<release-name>-sa by default), and attestation.audience |
Logs show enrollment: write trust bundle | The CA bundle you supplied differs from the control plane’s current bundle. The chart mounts it read-only, so the gateway can’t replace it. | Replace it with the file from Download CA bundle, unedited, and restart the gateway. See Secrets and keys. |
Logs show enrollment: bootstrap token expired for a token you just minted, or a node’s Clock skew component shows clock skew <n>ms exceeds 60000ms tolerance | The node’s clock is more than 60 seconds off | Run NTP on every Kubernetes node |
Pods run and pass /healthz, but serve no routes and never appear in Gateway health | FABRIC_GW_CP_ADDRESS is empty, so the gateway starts without enrolling | Set env.vars.FABRIC_GW_CP_ADDRESS to the Sync endpoint from the console |
| Enrollment times out or the connection is refused | FABRIC_GW_CP_ADDRESS is wrong, or egress to it is blocked | Fix the address, and allow egress to the control plane. See Network and egress. |
| TLS or x509 errors reaching the control plane | The control plane CA certificate is missing or wrong | Supply the CA certificate Postman gave you, with syncCA.pem or the fabric-dp-sync-ca Secret |
Certificate renewal fails, or logs show renewal loop terminated | Something between the data plane and the control plane ends TLS, or the control plane refused the renewal | Let the sync connection pass through with TLS intact. In Istio clusters, set istio.enabled: true. If your sync address doesn’t use port 9090, add its port to istio.excludeOutboundPorts, for example "4317,4318,9090,<port>". Then restart the pod. A restarted pod often enrolls again, for example when its certificate has expired, so with bootstrap token enrollment, put a fresh token in the app Secret first. If renewal still fails, contact Postman. |
| Nodes show Disconnected in Gateway health, but traffic still flows | The data plane lost its connection to the control plane | Running pods keep serving their last configuration. Restore the network path, and avoid rollouts and scale-ups until it’s back. See When the control plane is unreachable. |
For how enrollment works and how to set it up, see Connect the data plane to the control plane.
Traffic
| Symptom | Likely cause | Fix |
|---|---|---|
503 for a few seconds after a pod starts | The pod hasn’t received its configuration yet | This is normal. If it lasts, check that the pod can reach the control plane and its status in Gateway health. With the distributed store, see also Distributed store. |
502 for an upstream on your private network | FABRIC_GW_SSRF_BLOCK_PRIVATE_UPSTREAMS is on, or the upstream is on a link-local or cloud metadata address, which the gateway never calls, or on a loopback address, which it calls only when FABRIC_GW_SSRF_ALLOW_LOOPBACK_UPSTREAMS is "true" | Add the upstream’s host to FABRIC_GW_SSRF_ALLOW_PRIVATE_UPSTREAM_HOSTS. For a loopback upstream, set FABRIC_GW_SSRF_ALLOW_LOOPBACK_UPSTREAMS to "true". FABRIC_GW_SSRF_ALLOW_PRIVATE_HOSTS doesn’t apply to upstreams. See Calls to private hosts. |
| Stored credentials can’t be decrypted after a restart, or only on some replicas | FABRIC_GW_SECRET_KEY changed, or differs between replicas | Use one stable key on every replica. To rotate, see Secrets and keys. |
| Access-control, validator, or transformer plugins fail every request | The OPA sidecar is off, which is the default. Those plugins have no in-process fallback, so they fail according to their on_error setting, fail_closed by default. | Set opa.enabled: true, after you read the readiness warning in Helm values. Or set the plugin’s on_error to fail_open if you accept unchecked requests. |
/readyz answers 503 with opa_bundle_not_loaded | The OPA sidecar is on, but its policy bundle hasn’t loaded yet | The gateway writes the compiled policy bundle to the shared policy volume after it syncs from the control plane, and OPA loads it from there. Wait for the first sync. If it doesn’t load, check the gateway logs for enrollment and sync errors. |
| A rate limit allows more requests than you set | Each pod counts on its own by default | Turn on a coordination Redis, and set FABRIC_GW_RATE_LIMIT_COUNTER_STORE to redis. See Scale and run a highly available data plane. |
Clients time out reaching port 8080, with the distributed store | On a cluster that enforces NetworkPolicy, the distributed store’s policy allows only ports 3320 and 3322 between gateway pods, and blocks client traffic to 8080 | Allow port 8080 in distributed.networkPolicy.extraIngress, as shown after this table |
To allow client traffic with the distributed store, add this to your values. To admit only your ingress controller and in-cluster clients, add a from list to the rule.
distributed:networkPolicy:extraIngress:- ports:- protocol: TCPport: 8080
Telemetry
| Symptom | Likely cause | Fix |
|---|---|---|
| No telemetry arrives, but the sidecar runs | The ClickHouse settings or password are wrong. A missing FABRIC_GW_CLICKHOUSE_PASSWORD doesn’t stop the sidecar, it drops telemetry. | Check otel.clickhouse.* and the password key. Then run kubectl -n pm-fabric-dp port-forward <pod-name> 8888:8888 and curl -s localhost:8888/metrics | grep send_failed to check otelcol_exporter_send_failed_*. |
| Telemetry stopped after you turned off the sidecar | The gateway still sends to localhost:4317 | Set OTEL_EXPORTER_OTLP_ENDPOINT to your backend. |
Nothing on port 8888 | The collector didn’t load its configuration | Check that the collector ConfigMap rolled out, and restart the gateway pods |
| Fewer traces than requests | The gateway samples 10% of new traces, and the sidecar keeps only traces that matched a route or ended in a 5xx | This is expected. To change sampling, set OTEL_TRACES_SAMPLER_ARG. |
Embedder
| Symptom | Likely cause | Fix |
|---|---|---|
| A node drain hangs on the embedder pod | One replica plus the default minAvailable: 1 budget blocks eviction | Set embedder.podDisruptionBudget.maxUnavailable: 1, or run more than one replica |
Embedder is slow to become Ready after every restart | The default emptyDir cache downloads the model each time | Set embedder.persistence.enabled: true |
| Embedder can’t start in an air-gapped cluster | Egress to huggingface.co is blocked and no local model is available | Load the model into the cache or a local embedder.model.id path, confirm it loads, and then set embedder.offline.enabled: true |
Embedder in CrashLoopBackOff after you set a custom image | The image writes outside /data and /tmp, and hits the read-only root filesystem | Mount the extra path as a volume, or relax embedder.securityContext.readOnlyRootFilesystem |
helm install or helm upgrade fails with embedder.persistence with a chart-created ReadWriteOnce PVC supports only one embedder replica | A ReadWriteOnce cache with more than one possible replica | Use ReadWriteMany storage or an existingClaim, or keep one replica |
| Tool search returns an error | The embedder is down, too slow, or unreachable from the gateway | Check the embedder pod and the Tool Search plugin’s embedder URL. If FABRIC_GW_SSRF_BLOCK_PRIVATE_UPSTREAMS is "true", add the embedder’s host to FABRIC_GW_SSRF_ALLOW_PRIVATE_UPSTREAM_HOSTS. |
| Something other than the gateway can’t reach the embedder, such as your own Prometheus | embedder.networkPolicy.enabled is true, and the policy admits only the gateway pods | Add the other clients as peers in embedder.networkPolicy.extraIngress |
Distributed store
| Symptom | Likely cause | Fix |
|---|---|---|
Pods exit at startup after you turn on the distributed store, and logs show production_mode: distributed secret store requires a gossip key | FABRIC_GW_DISTRIBUTED_GOSSIP_KEY or FABRIC_GW_DISTRIBUTED_AUTH_PASSWORD is missing from the app Secret | Add both keys |
New credentials can’t be saved, and requests to routes with stored credentials fail with 503 | Fewer than a majority of pods are running. Reads need a majority too. | Bring pods back. With three replicas, reads and writes need two. |
Routes with stored credentials answer 503 with secret store is restoring, and account-connection pages show “Credential storage is starting. Please try again.” | After a restart, each pod keeps its secret store closed until it restores from the cluster, which needs a majority of pods. The pod reports Ready meanwhile. | Bring a majority of pods up. If it lasts, look for local secret snapshot is unusable in the logs. It means this pod can’t read its own snapshot, for example after an encryption key change, and is waiting for a peer to restore. |
Pods crash-loop on a cold start, and logs show member_count_quorum > 1 gates cold start | distributed.memberCountQuorum is above 1 | Set it back to 1 |
Get help from Postman
If you contact Postman, include the chart version, the gateway image tag, the affected node IDs from Gateway health, and the gateway logs from around the failure. Remove secret values before you share anything.
Next steps
- To check health after a fix, see Monitor the data plane.
- For problems during an upgrade, see Upgrade the data plane.
- For request errors, see Gateway error codes.
- To send telemetry to your own backend, see Export telemetry with OpenTelemetry.
- To run the tool-search embedder, see Run the tool-search embedder.

