Skip to navigation

Troubleshoot the data plane

This page applies to the Enterprise plan, where you run the data plane in your own VPC. It lists common data plane problems by symptom, with the likely cause and the fix. For problems with individual requests, such as a 401 or a 429, see Troubleshooting.

The commands use the release name fabric-dp in the namespace pm-fabric-dp. Start with these three:

# Pods and their state
kubectl -n pm-fabric-dp get pods
# Events and probe failures for one pod
kubectl -n pm-fabric-dp describe pod <pod-name>
# The gateway container's logs
kubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c fabric-gateway-chart

deploy/fabric-gateway-chart reads one pod, chosen for you. To read a specific pod, use its name instead. If the pod is restarting, add --previous to see the logs of the container that exited:

kubectl -n pm-fabric-dp logs <pod-name> -c fabric-gateway-chart --previous

With the distributed store, the gateway is a StatefulSet, so use statefulset/fabric-gateway-chart in place of deploy/fabric-gateway-chart.

Install and migrations

With the MySQL store, every install and upgrade runs a schema migration Job before any gateway pod changes. A failed migration stops the release, so the gateway never runs against a database it doesn’t match. To inspect a failed migration:

kubectl -n pm-fabric-dp get jobs -l app.kubernetes.io/name=fabric-gateway-chart
kubectl -n pm-fabric-dp logs job/fabric-gateway-chart-dp-migrate
SymptomLikely causeFix
Install stops at the migration Job, and its pod shows CreateContainerConfigErrorThe app Secret doesn’t exist yet, or has no database password keyCreate the app Secret before helm install, with the key named by migrations.passwordKey (FABRIC_GW_MYSQL_PASS by default). See Secrets and keys.
Migration fails with Access denied or CREATE command deniedThe database user the migration connects as lacks grantsGrant CREATE, ALTER, INDEX, SELECT, INSERT, UPDATE, and DELETE to that user, or set migrations.dbUser to a user that has them
Migration fails with a connection timeoutFABRIC_GW_MYSQL_HOST is wrong or unreachable from the clusterFix the host or the network path. The Job retries a slow database 10 times (migrations.connectRetries), but can’t fix a wrong address.
Migration fails with RSA public key is not available client sideMySQL 8’s default authentication, over a connection without TLS, needs a key exchange the migration Job’s driver doesn’t doReach MySQL over TLS, use a database user created with mysql_native_password, or run migrations yourself with migrations.enabled: false. See Database migrations.
Migration fails with Unknown databaseThe database doesn’t exist. The chart creates tables, not the database.Create the database named in FABRIC_GW_MYSQL_DATABASE before you install
Migration fails with a connection timeout, and MySQL listens on a port other than 3306The migration Job ignores FABRIC_GW_MYSQL_PORT and connects on 3306Put the port in the host instead: FABRIC_GW_MYSQL_HOST: "<host>:<port>". The gateway accepts this form too.
Upgrade fails with Migration checksum mismatchA migration that was already applied was editedRestore the chart’s original SQL files. Never edit an applied migration.
On a database that already had the schema, early migrations such as V001 were never appliedmigrations.baselineVersion was raised above "0", so Flyway skipped every migration at or below itKeep migrations.baselineVersion: "0", the default
You want to run migrations from your own pipelineSet migrations.enabled: false, and apply the chart’s SQL files in order before you roll out the gateway. See Database migrations.

Pods don’t start or aren’t ready

SymptomLikely causeFix
Pods never become Ready, and probes report 404Probes point at a path or port other than /healthz on 8080Keep container.probes.liveness, readiness, and startup at /healthz on port 8080. The gateway doesn’t serve /ping or /health.
Probes fail with connection refused, and the pod restartsThe pod hasn’t enrolled. A gateway that hasn’t enrolled doesn’t listen.See Enrollment and the control plane
Pod exits at startup, and logs show production_mode: secret store requires an encryption keyProduction mode is on (the default) and FABRIC_GW_SECRET_KEY is missingAdd the key to the app Secret. See Secrets and keys.
Gateway pod in ImagePullBackOff, on the fabric-gateway-chart containerThe gateway image isn’t in a registry the cluster can pull from, or the pull Secret is missingMirror the image, set image.repository, and set image.imagePullSecrets if your registry needs credentials
Gateway pod in InvalidImageNameimage.repository is still the chart’s <your-registry>/fabric/gateway placeholderSet image.repository to your mirror
Gateway pod in ImagePullBackOff, on the otel-collector containerThe collector image, otel/opentelemetry-collector-contrib:0.149.0, can’t be pulledMirror it. To change the image, copy the chart’s whole otel-collector entry into additionalContainers in your values file, and change only image. image.imagePullSecrets covers this container too.
Gateway pod in ImagePullBackOff, on the opa containerThe OPA image, openpolicyagent/opa, can’t be pulledMirror it, and set opa.image.repository and opa.image.tag. image.imagePullSecrets covers this container too.
Embedder pod in ImagePullBackOffThe embedder image, ghcr.io/huggingface/text-embeddings-inference:cpu-1.9, can’t be pulledMirror it, set embedder.image.repository, and set embedder.image.imagePullSecrets if your registry needs credentials
Migration Job pod in ImagePullBackOffThe Flyway image, flyway/flyway:10, can’t be pulled. The Job doesn’t use image.imagePullSecrets.Mirror it, and set migrations.image. If your registry needs credentials, add the pull Secret to the namespace’s default ServiceAccount, which the Job runs as.
Pod exits at startup, and logs show secrets: refusing to start without global.tenant_id in production_modeFABRIC_GW_TENANT_ID is empty, as in the chart’s defaultsSet env.vars.FABRIC_GW_TENANT_ID to the tenant ID Postman gave you
Pod exits at startup, and logs show opening mysql secret store: pinging mysqlThe gateway can’t reach or log in to MySQLCheck FABRIC_GW_MYSQL_HOST, FABRIC_GW_MYSQL_USER, and the password, and allow traffic from the gateway pods to the database
Pod exits at startup, and logs show key must be exactly 32 raw bytes, 64 hex chars, or 44 base64 charsFABRIC_GW_SECRET_KEY has the wrong length. got 65 bytes usually means a trailing newline.Store the key without a trailing newline. See Secrets and keys.
Gateway pod shows CreateContainerConfigError, with the distributed storeThe app Secret doesn’t exist. With the distributed store there’s no migration Job, so the gateway pod is the first to need it.Create the app Secret before you install. See Secrets and keys.
Pod stays in ContainerCreating, and its events show that the fabric-dp-sync-ca Secret isn’t foundsyncCA.pem is empty, and the fabric-dp-sync-ca Secret doesn’t existSet syncCA.pem, or create the Secret. See Secrets and keys.
Pod stays Pending, and its PersistentVolumeClaim isn’t boundThe cluster has no default StorageClass, and the chart’s volume doesn’t name oneSet distributed.persistence.storageClass for the distributed store, or embedder.persistence.storageClass for the embedder cache
Pod reads READY 1/2 (or 2/3 with OPA)One container isn’t ready. During startup or an enrollment problem, it’s usually the gateway container. With opa.enabled: true, 2/3 also appears when the opa container’s readiness check fails, which is a known chart issue.Run kubectl -n pm-fabric-dp describe pod <pod-name> to see which one. If it’s fabric-gateway-chart, see Enrollment and the control plane. If it’s otel-collector, check its logs with -c otel-collector, and see Telemetry.
Values are back to the chart defaults after an upgradeThe upgrade ran without -f or --reuse-valuesRun the upgrade again with your values file

To add a pull Secret to the namespace’s default ServiceAccount, run the following. It replaces any pull Secrets the ServiceAccount already lists.

kubectl -n pm-fabric-dp patch serviceaccount default \
-p '{"imagePullSecrets":[{"name":"<pull-secret>"}]}'

Enrollment and the control plane

A gateway pod enrolls with the control plane before it serves any traffic. If enrollment fails, the pod exits and Kubernetes restarts it, so a pod in CrashLoopBackOff right after install usually has an enrollment problem. Look for the dp enrollment line in the gateway logs. A join token, which the console mints with Mint join token, is a reusable bootstrap token.

SymptomLikely causeFix
Logs show enrollment: bootstrap token expired, or new pods fail after weeks of workingThe bootstrap token passed its lifetime, 10 minutes by default and 30 minutes at most. Pods with a valid certificate keep working, but new pods need a valid token.Mint a new token, update the app Secret, and restart the gateway. Consider workload attestation, which needs no token. See Connect the data plane to the control plane.
Logs show tenant_id/node_id do not match the bootstrap token claimsFABRIC_GW_TENANT_ID differs from the tenant the token was minted for, or, with a single-use token, the pod’s node ID differs from the Node ID you enteredSet FABRIC_GW_TENANT_ID to the tenant ID Postman gave you. For more than one replica, or any pod that can be rescheduled, mint a reusable join token.
Pod exits at startup, and logs show enrollment: sync.bootstrap_token is empty under production_modeThe app Secret has no FABRIC_GW_SYNC_BOOTSTRAP_TOKEN, and workload attestation is offAdd the token to the app Secret, or set attestation.enabled: true
Logs show bootstrap token already consumed (replay denied)A single-use token was already redeemed, for example by another replica or by a pod that was replacedMint a new token. For more than one replica, or any pod that can be rescheduled, use a reusable join token or workload attestation.
Logs show attestation deniedThe pod’s service account token doesn’t match the anchor’s issuer, namespace, service account, or audience. The message doesn’t say which.Compare the anchor you registered with the cluster’s issuer, the data plane’s namespace, its ServiceAccount (<release-name>-sa by default), and attestation.audience
Logs show enrollment: write trust bundleThe CA bundle you supplied differs from the control plane’s current bundle. The chart mounts it read-only, so the gateway can’t replace it.Replace it with the file from Download CA bundle, unedited, and restart the gateway. See Secrets and keys.
Logs show enrollment: bootstrap token expired for a token you just minted, or a node’s Clock skew component shows clock skew <n>ms exceeds 60000ms toleranceThe node’s clock is more than 60 seconds offRun NTP on every Kubernetes node
Pods run and pass /healthz, but serve no routes and never appear in Gateway healthFABRIC_GW_CP_ADDRESS is empty, so the gateway starts without enrollingSet env.vars.FABRIC_GW_CP_ADDRESS to the Sync endpoint from the console
Enrollment times out or the connection is refusedFABRIC_GW_CP_ADDRESS is wrong, or egress to it is blockedFix the address, and allow egress to the control plane. See Network and egress.
TLS or x509 errors reaching the control planeThe control plane CA certificate is missing or wrongSupply the CA certificate Postman gave you, with syncCA.pem or the fabric-dp-sync-ca Secret
Certificate renewal fails, or logs show renewal loop terminatedSomething between the data plane and the control plane ends TLS, or the control plane refused the renewalLet the sync connection pass through with TLS intact. In Istio clusters, set istio.enabled: true. If your sync address doesn’t use port 9090, add its port to istio.excludeOutboundPorts, for example "4317,4318,9090,<port>". Then restart the pod. A restarted pod often enrolls again, for example when its certificate has expired, so with bootstrap token enrollment, put a fresh token in the app Secret first. If renewal still fails, contact Postman.
Nodes show Disconnected in Gateway health, but traffic still flowsThe data plane lost its connection to the control planeRunning pods keep serving their last configuration. Restore the network path, and avoid rollouts and scale-ups until it’s back. See When the control plane is unreachable.

For how enrollment works and how to set it up, see Connect the data plane to the control plane.

Traffic

SymptomLikely causeFix
503 for a few seconds after a pod startsThe pod hasn’t received its configuration yetThis is normal. If it lasts, check that the pod can reach the control plane and its status in Gateway health. With the distributed store, see also Distributed store.
502 for an upstream on your private networkFABRIC_GW_SSRF_BLOCK_PRIVATE_UPSTREAMS is on, or the upstream is on a link-local or cloud metadata address, which the gateway never calls, or on a loopback address, which it calls only when FABRIC_GW_SSRF_ALLOW_LOOPBACK_UPSTREAMS is "true"Add the upstream’s host to FABRIC_GW_SSRF_ALLOW_PRIVATE_UPSTREAM_HOSTS. For a loopback upstream, set FABRIC_GW_SSRF_ALLOW_LOOPBACK_UPSTREAMS to "true". FABRIC_GW_SSRF_ALLOW_PRIVATE_HOSTS doesn’t apply to upstreams. See Calls to private hosts.
Stored credentials can’t be decrypted after a restart, or only on some replicasFABRIC_GW_SECRET_KEY changed, or differs between replicasUse one stable key on every replica. To rotate, see Secrets and keys.
Access-control, validator, or transformer plugins fail every requestThe OPA sidecar is off, which is the default. Those plugins have no in-process fallback, so they fail according to their on_error setting, fail_closed by default.Set opa.enabled: true, after you read the readiness warning in Helm values. Or set the plugin’s on_error to fail_open if you accept unchecked requests.
/readyz answers 503 with opa_bundle_not_loadedThe OPA sidecar is on, but its policy bundle hasn’t loaded yetThe gateway writes the compiled policy bundle to the shared policy volume after it syncs from the control plane, and OPA loads it from there. Wait for the first sync. If it doesn’t load, check the gateway logs for enrollment and sync errors.
A rate limit allows more requests than you setEach pod counts on its own by defaultTurn on a coordination Redis, and set FABRIC_GW_RATE_LIMIT_COUNTER_STORE to redis. See Scale and run a highly available data plane.
Clients time out reaching port 8080, with the distributed storeOn a cluster that enforces NetworkPolicy, the distributed store’s policy allows only ports 3320 and 3322 between gateway pods, and blocks client traffic to 8080Allow port 8080 in distributed.networkPolicy.extraIngress, as shown after this table

To allow client traffic with the distributed store, add this to your values. To admit only your ingress controller and in-cluster clients, add a from list to the rule.

distributed:
networkPolicy:
extraIngress:
- ports:
- protocol: TCP
port: 8080

Telemetry

SymptomLikely causeFix
No telemetry arrives, but the sidecar runsThe ClickHouse settings or password are wrong. A missing FABRIC_GW_CLICKHOUSE_PASSWORD doesn’t stop the sidecar, it drops telemetry.Check otel.clickhouse.* and the password key. Then run kubectl -n pm-fabric-dp port-forward <pod-name> 8888:8888 and curl -s localhost:8888/metrics | grep send_failed to check otelcol_exporter_send_failed_*.
Telemetry stopped after you turned off the sidecarThe gateway still sends to localhost:4317Set OTEL_EXPORTER_OTLP_ENDPOINT to your backend.
Nothing on port 8888The collector didn’t load its configurationCheck that the collector ConfigMap rolled out, and restart the gateway pods
Fewer traces than requestsThe gateway samples 10% of new traces, and the sidecar keeps only traces that matched a route or ended in a 5xxThis is expected. To change sampling, set OTEL_TRACES_SAMPLER_ARG.

Embedder

SymptomLikely causeFix
A node drain hangs on the embedder podOne replica plus the default minAvailable: 1 budget blocks evictionSet embedder.podDisruptionBudget.maxUnavailable: 1, or run more than one replica
Embedder is slow to become Ready after every restartThe default emptyDir cache downloads the model each timeSet embedder.persistence.enabled: true
Embedder can’t start in an air-gapped clusterEgress to huggingface.co is blocked and no local model is availableLoad the model into the cache or a local embedder.model.id path, confirm it loads, and then set embedder.offline.enabled: true
Embedder in CrashLoopBackOff after you set a custom imageThe image writes outside /data and /tmp, and hits the read-only root filesystemMount the extra path as a volume, or relax embedder.securityContext.readOnlyRootFilesystem
helm install or helm upgrade fails with embedder.persistence with a chart-created ReadWriteOnce PVC supports only one embedder replicaA ReadWriteOnce cache with more than one possible replicaUse ReadWriteMany storage or an existingClaim, or keep one replica
Tool search returns an errorThe embedder is down, too slow, or unreachable from the gatewayCheck the embedder pod and the Tool Search plugin’s embedder URL. If FABRIC_GW_SSRF_BLOCK_PRIVATE_UPSTREAMS is "true", add the embedder’s host to FABRIC_GW_SSRF_ALLOW_PRIVATE_UPSTREAM_HOSTS.
Something other than the gateway can’t reach the embedder, such as your own Prometheusembedder.networkPolicy.enabled is true, and the policy admits only the gateway podsAdd the other clients as peers in embedder.networkPolicy.extraIngress

Distributed store

SymptomLikely causeFix
Pods exit at startup after you turn on the distributed store, and logs show production_mode: distributed secret store requires a gossip keyFABRIC_GW_DISTRIBUTED_GOSSIP_KEY or FABRIC_GW_DISTRIBUTED_AUTH_PASSWORD is missing from the app SecretAdd both keys
New credentials can’t be saved, and requests to routes with stored credentials fail with 503Fewer than a majority of pods are running. Reads need a majority too.Bring pods back. With three replicas, reads and writes need two.
Routes with stored credentials answer 503 with secret store is restoring, and account-connection pages show “Credential storage is starting. Please try again.”After a restart, each pod keeps its secret store closed until it restores from the cluster, which needs a majority of pods. The pod reports Ready meanwhile.Bring a majority of pods up. If it lasts, look for local secret snapshot is unusable in the logs. It means this pod can’t read its own snapshot, for example after an encryption key change, and is waiting for a peer to restore.
Pods crash-loop on a cold start, and logs show member_count_quorum > 1 gates cold startdistributed.memberCountQuorum is above 1Set it back to 1

Get help from Postman

If you contact Postman, include the chart version, the gateway image tag, the affected node IDs from Gateway health, and the gateway logs from around the failure. Remove secret values before you share anything.

Next steps