Monitor the data plane
This page applies to the Enterprise plan, where you run the data plane in your own VPC. You can check the data plane’s health in four places: Kubernetes itself, the gateway’s health endpoints, the Gateway health page in the console, and the logs, metrics, and traces the data plane exports to your own backend. This page describes each one and what to alert on.
The commands on this page use the release name fabric-dp in the namespace pm-fabric-dp. Resource names come from the chart, not the release: the gateway Deployment, Service, and container are all named fabric-gateway-chart, and the embedder is always fabric-gateway-embedder.
Check health with kubectl
Run these checks after an install or upgrade, after a node drain, or as a routine check.
-
Check the gateway pods:
A healthy gateway pod is
RunningandREADY 2/2: the gateway container plus the OpenTelemetry Collector sidecar. If you turned the sidecar off, it reads1/1. Withopa.enabled: true, add one container to the count. The list can also include a schema migration pod, which showsErrorif the migration failed. See step 4. -
Check the embedder pod:
A healthy embedder pod is
RunningandREADY 1/1. -
Check that the gateway enrolled with the control plane and started cleanly:
You should see the
dp enrollmentline with its outcome, sync activity, and astartedline, with no repeating restarts. With the distributed store, the gateway runs as a StatefulSet, so usestatefulset/fabric-gateway-chartinstead ofdeploy/fabric-gateway-chart. -
Around an install or upgrade with the MySQL store, check the schema migration Job:
Helm deletes the Job as soon as it succeeds, so on a steady-state cluster it’s normally gone. If it’s still there, the migration failed. See Troubleshoot the data plane. With the distributed store, there’s no migration Job.
Health endpoints
The gateway answers three endpoints on its client port, 8080, on any hostname. None of them needs credentials.
The chart points the liveness, readiness, and startup probes at /healthz on port 8080. Keep them on the gateway’s own endpoints: the gateway doesn’t serve /ping or /health, and requests to those paths go to route matching like any other request, so probes on them fail. Paths are matched exactly, so /healthz/ isn’t a health endpoint either.
With the distributed store, point the readiness probe at /readyz/secret-store instead, so a rolling restart waits for each pod to re-replicate before it stops the next one. Keep liveness and startup on /healthz.
None of these endpoints goes through your routes or plugins, and none depends on your configuration. A pod that hasn’t enrolled yet doesn’t listen at all, so its probes fail with connection refused rather than an error status. Once a pod has enrolled, /readyz answers 200 before its first configuration arrives, so don’t use it to tell whether a pod has your configuration. Check the node’s Config sync in Gateway health for that.
Gateway health in the console
In the console, select Admin > Gateway health to see every node that has connected to the control plane:
- Fleet totals for Connected, Health, Sync, and Versions.
- Per node: Status (Healthy, Lagging, or Disconnected), Health (OK, Degraded, Critical, or Unknown), gateway Version, the Config and Policy versions it acknowledged, Certificate state, Uptime, and Last seen.
Each gateway pod is its own node. Select a node for its certificate details and per-stream status. A node reports its health over its sync connection, so a node that can’t connect shows as Disconnected, not with its own health. For enrollment and certificate details, see Connect the data plane to the control plane.
Below the fleet totals, a component health row rolls up Dependencies, Upstreams, Certificates, and Credentials across the fleet. Select a tile to see the nodes and conditions behind it.
A node’s details also include a Components section:
- Dependencies: the Runtime database (MySQL), your vault, and Control plane sync, with whether each is reachable.
- Subsystems: the listener, Config sync and Policy sync with the version each applied, upstreams, MCP servers, and plugins.
- Reported components: the node’s own checks, such as Cert renewal, Secret store, Config apply, Telemetry export, and Clock skew.
To confirm that a pod has your configuration, open its node and check that Config sync shows Connected and an Applied version.
What the data plane exports
The gateway emits traces, metrics, and logs over OTLP/gRPC. By default, the chart runs an OpenTelemetry Collector sidecar in each gateway pod, and the gateway sends to it on localhost:4317. The sidecar does the following:
- Receives the gateway’s traces, metrics, and logs.
- Scrapes its own metrics and the embedder’s metrics, and ships them with the gateway’s.
- Drops traces of health checks and metrics scrapes.
- Keeps traces of requests that matched a route or ended in a
5xxerror, and drops the rest. - Exports everything to a ClickHouse database that you provide, set with
otel.clickhouse.endpoint,otel.clickhouse.database, andotel.clickhouse.username. The password comes fromFABRIC_GW_CLICKHOUSE_PASSWORDin the app Secret. The collector doesn’t create tables, so createotel_tracesandotel_logs, and the metrics tables the ClickHouse exporter expects, in the database first. Failed exports retry from a queue of 1,000 items, so a long backend outage can drop telemetry.
The ClickHouse password key is optional. If it’s missing, the sidecar still starts, but telemetry doesn’t arrive. Until you point it at your backend, no telemetry leaves your cluster.
To send telemetry to another backend, or to turn export off, see Export telemetry with OpenTelemetry. Usage counts for billing are separate: they travel on the data plane’s own connection to the control plane, so telemetry settings never affect them.
Logs
The gateway writes structured logs to standard output, as JSON by default, and also exports them over OTLP with your other telemetry. Set the level and format with environment variables under env.vars:
Every request leaves exactly one access record, with the message request completed. It’s written at INFO level, or at ERROR when the status is 500 or higher. Its fields include the following:
Log lines written while handling a request also carry trace_id and span_id when the request is traced, so you can jump from a log line to its trace. Credentials such as Authorization header values, bearer tokens, and API keys are redacted before they reach logs.
The chart sets FABRIC_MCP_DEBUG_LOGS to "1", which adds several diagnostic log lines to each MCP request. To turn them off, set it to "0".
To read the gateway container’s logs directly:
Metrics
The gateway exports its own metrics over OTLP. Useful series include the following:
Token counts are recorded only on routes with the LLM observability plugin. Without it, the gateway records no token counts.
The collector sidecar and the embedder also serve Prometheus metrics of their own, and the collector scrapes and ships both by default:
- Collector metrics (
otelcol_receiver_*,otelcol_processor_*,otelcol_exporter_*), on port8888of each gateway pod. They count accepted, refused, dropped, and sent data, queue depth, and export failures.otelcol_exporter_send_failed_*is the earliest sign that telemetry isn’t reaching your backend. - Embedder metrics (
te_*), at/metricson the embedder Service.
To check the raw endpoints yourself:
If you’d rather have your own Prometheus pull the sidecar metrics, turn on one of these. Both are off by default:
metrics.serviceMonitor.enabled: truerenders a PodMonitor for the collector and a ServiceMonitor for the embedder. It needs the Prometheus Operator. Setmetrics.serviceMonitor.labelsto match your operator’s selector, for examplerelease: kube-prometheus-stack.metrics.annotations.enabled: trueaddsprometheus.io/scrapeannotations to the gateway and embedder pods, for an annotation-based scraper.
Traces
The gateway reads W3C traceparent and B3 headers, starts a server span for each request, and passes trace context on to the upstream. Health endpoints aren’t traced. The root span records where the gateway’s own time went, for example fabric.gateway.overhead.duration_ms for time spent in the gateway and fabric.gateway.io_wait.duration_ms for time waiting on the upstream and the client.
The chart samples 10% of new traces (OTEL_TRACES_SAMPLER_ARG: "0.10"). LLM prompts and replies aren’t captured unless you set FABRIC_GW_TRACING_CAPTURE_MESSAGE_CONTENT to "true". For both settings, see Export telemetry with OpenTelemetry.
What to alert on
- Gateway or embedder pods restarting, or not
Ready. - Nodes that show Disconnected, Lagging, Degraded, or Critical in Gateway health.
- A rising rate of 5xx responses in
http.server.request.duration. To tell gateway faults from upstream ones, group failed access records byfabric.error.origin. - A node component that isn’t healthy in Gateway health, especially Cert renewal, Secret store, and Clock skew.
- Any increase in
otelcol_exporter_send_failed_*. - A
renewal loop terminatederror in the gateway logs, which means a pod can no longer renew its certificate. - With the MySQL store, your database’s availability. With the distributed store, fewer running pods than
distributed.replicaCount.
Next steps
- To fix a problem you found, see Troubleshoot the data plane.
- To send telemetry to your backend, see Export telemetry with OpenTelemetry.
- To run more replicas, see Scale and run a highly available data plane.

