Skip to navigation

Monitor the data plane

This page applies to the Enterprise plan, where you run the data plane in your own VPC. You can check the data plane’s health in four places: Kubernetes itself, the gateway’s health endpoints, the Gateway health page in the console, and the logs, metrics, and traces the data plane exports to your own backend. This page describes each one and what to alert on.

The commands on this page use the release name fabric-dp in the namespace pm-fabric-dp. Resource names come from the chart, not the release: the gateway Deployment, Service, and container are all named fabric-gateway-chart, and the embedder is always fabric-gateway-embedder.

Check health with kubectl

Run these checks after an install or upgrade, after a node drain, or as a routine check.

  1. Check the gateway pods:

    kubectl -n pm-fabric-dp get pods -l app.kubernetes.io/name=fabric-gateway-chart

    A healthy gateway pod is Running and READY 2/2: the gateway container plus the OpenTelemetry Collector sidecar. If you turned the sidecar off, it reads 1/1. With opa.enabled: true, add one container to the count. The list can also include a schema migration pod, which shows Error if the migration failed. See step 4.

  2. Check the embedder pod:

    kubectl -n pm-fabric-dp get pods -l app=fabric-gateway-embedder

    A healthy embedder pod is Running and READY 1/1.

  3. Check that the gateway enrolled with the control plane and started cleanly:

    kubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c fabric-gateway-chart \
    | grep -iE "dp enrollment|sync|started"

    You should see the dp enrollment line with its outcome, sync activity, and a started line, with no repeating restarts. With the distributed store, the gateway runs as a StatefulSet, so use statefulset/fabric-gateway-chart instead of deploy/fabric-gateway-chart.

  4. Around an install or upgrade with the MySQL store, check the schema migration Job:

    kubectl -n pm-fabric-dp get jobs -l app.kubernetes.io/name=fabric-gateway-chart

    Helm deletes the Job as soon as it succeeds, so on a steady-state cluster it’s normally gone. If it’s still there, the migration failed. See Troubleshoot the data plane. With the distributed store, there’s no migration Job.

Health endpoints

The gateway answers three endpoints on its client port, 8080, on any hostname. None of them needs credentials.

EndpointAnswer
/healthz200 with {"status":"ok"} whenever the process is serving.
/readyz200 with {"status":"ready"}. Answers 503 with {"status":"not_ready","reason":"opa_bundle_not_loaded"} while the OPA sidecar is enabled and its policy bundle hasn’t loaded yet.
/readyz/secret-storeWith the distributed store. Answers 503 with {"status":"not_ready","reason":"secret_store_rereplication_pending"} until the pod finishes re-replicating credentials after it joins its peers, or until 5 minutes pass.

The chart points the liveness, readiness, and startup probes at /healthz on port 8080. Keep them on the gateway’s own endpoints: the gateway doesn’t serve /ping or /health, and requests to those paths go to route matching like any other request, so probes on them fail. Paths are matched exactly, so /healthz/ isn’t a health endpoint either.

With the distributed store, point the readiness probe at /readyz/secret-store instead, so a rolling restart waits for each pod to re-replicate before it stops the next one. Keep liveness and startup on /healthz.

container:
probes:
readiness: { path: /readyz/secret-store, port: 8080 }

None of these endpoints goes through your routes or plugins, and none depends on your configuration. A pod that hasn’t enrolled yet doesn’t listen at all, so its probes fail with connection refused rather than an error status. Once a pod has enrolled, /readyz answers 200 before its first configuration arrives, so don’t use it to tell whether a pod has your configuration. Check the node’s Config sync in Gateway health for that.

Gateway health in the console

In the console, select Admin > Gateway health to see every node that has connected to the control plane:

  • Fleet totals for Connected, Health, Sync, and Versions.
  • Per node: Status (Healthy, Lagging, or Disconnected), Health (OK, Degraded, Critical, or Unknown), gateway Version, the Config and Policy versions it acknowledged, Certificate state, Uptime, and Last seen.

Each gateway pod is its own node. Select a node for its certificate details and per-stream status. A node reports its health over its sync connection, so a node that can’t connect shows as Disconnected, not with its own health. For enrollment and certificate details, see Connect the data plane to the control plane.

Below the fleet totals, a component health row rolls up Dependencies, Upstreams, Certificates, and Credentials across the fleet. Select a tile to see the nodes and conditions behind it.

A node’s details also include a Components section:

  • Dependencies: the Runtime database (MySQL), your vault, and Control plane sync, with whether each is reachable.
  • Subsystems: the listener, Config sync and Policy sync with the version each applied, upstreams, MCP servers, and plugins.
  • Reported components: the node’s own checks, such as Cert renewal, Secret store, Config apply, Telemetry export, and Clock skew.

To confirm that a pod has your configuration, open its node and check that Config sync shows Connected and an Applied version.

What the data plane exports

The gateway emits traces, metrics, and logs over OTLP/gRPC. By default, the chart runs an OpenTelemetry Collector sidecar in each gateway pod, and the gateway sends to it on localhost:4317. The sidecar does the following:

  • Receives the gateway’s traces, metrics, and logs.
  • Scrapes its own metrics and the embedder’s metrics, and ships them with the gateway’s.
  • Drops traces of health checks and metrics scrapes.
  • Keeps traces of requests that matched a route or ended in a 5xx error, and drops the rest.
  • Exports everything to a ClickHouse database that you provide, set with otel.clickhouse.endpoint, otel.clickhouse.database, and otel.clickhouse.username. The password comes from FABRIC_GW_CLICKHOUSE_PASSWORD in the app Secret. The collector doesn’t create tables, so create otel_traces and otel_logs, and the metrics tables the ClickHouse exporter expects, in the database first. Failed exports retry from a queue of 1,000 items, so a long backend outage can drop telemetry.

The ClickHouse password key is optional. If it’s missing, the sidecar still starts, but telemetry doesn’t arrive. Until you point it at your backend, no telemetry leaves your cluster.

To send telemetry to another backend, or to turn export off, see Export telemetry with OpenTelemetry. Usage counts for billing are separate: they travel on the data plane’s own connection to the control plane, so telemetry settings never affect them.

Logs

The gateway writes structured logs to standard output, as JSON by default, and also exports them over OTLP with your other telemetry. Set the level and format with environment variables under env.vars:

VariableValuesDefault
LOG_LEVELdebug, info, warn, errorinfo
LOG_FORMATjson, textjson

Every request leaves exactly one access record, with the message request completed. It’s written at INFO level, or at ERROR when the status is 500 or higher. Its fields include the following:

FieldDescription
http.request.method, url.path, http.routeThe request and the route it matched.
http.response.status_codeThe status the client received.
http.server.request.durationDuration, in seconds.
fabric.request.idThe request ID, also returned in the X-Request-Id header.
fabric.tenant.id, fabric.namespace.id, fabric.consumer, fabric.service.nameWho made the request and which service handled it.
server.addressThe upstream host.
gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokensLLM provider, model, and token counts, on LLM requests.
error.type, fabric.error.reason, fabric.error.origin, errorOn failures only. fabric.error.origin is client, gateway, upstream, or unknown.

Log lines written while handling a request also carry trace_id and span_id when the request is traced, so you can jump from a log line to its trace. Credentials such as Authorization header values, bearer tokens, and API keys are redacted before they reach logs.

The chart sets FABRIC_MCP_DEBUG_LOGS to "1", which adds several diagnostic log lines to each MCP request. To turn them off, set it to "0".

To read the gateway container’s logs directly:

kubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c fabric-gateway-chart

Metrics

The gateway exports its own metrics over OTLP. Useful series include the following:

MetricWhat it measures
http.server.request.durationRequest latency and count, by route, status, and tenant.
gen_ai.client.token.usageLLM tokens by type (input, output, cache read, cache write).
gen_ai.client.operation.durationLLM call duration.
fabric.server.connectionsOpen client connections.
fabric.upstream.connections.open, fabric.upstream.connection.dial.errorsConnections to upstreams, and failures to open them.
fabric.upstream.requests.in_flightRequests waiting on an upstream.
fabric.process.open_fds, fabric.process.max_fdsFile descriptors in use and the limit.

Token counts are recorded only on routes with the LLM observability plugin. Without it, the gateway records no token counts.

The collector sidecar and the embedder also serve Prometheus metrics of their own, and the collector scrapes and ships both by default:

  • Collector metrics (otelcol_receiver_*, otelcol_processor_*, otelcol_exporter_*), on port 8888 of each gateway pod. They count accepted, refused, dropped, and sent data, queue depth, and export failures. otelcol_exporter_send_failed_* is the earliest sign that telemetry isn’t reaching your backend.
  • Embedder metrics (te_*), at /metrics on the embedder Service.

To check the raw endpoints yourself:

kubectl -n pm-fabric-dp port-forward deploy/fabric-gateway-chart 8888:8888
curl -s localhost:8888/metrics | grep '^otelcol_'
kubectl -n pm-fabric-dp port-forward svc/fabric-gateway-embedder 8080:80
curl -s localhost:8080/metrics | grep '^te_'

If you’d rather have your own Prometheus pull the sidecar metrics, turn on one of these. Both are off by default:

  • metrics.serviceMonitor.enabled: true renders a PodMonitor for the collector and a ServiceMonitor for the embedder. It needs the Prometheus Operator. Set metrics.serviceMonitor.labels to match your operator’s selector, for example release: kube-prometheus-stack.
  • metrics.annotations.enabled: true adds prometheus.io/scrape annotations to the gateway and embedder pods, for an annotation-based scraper.

Traces

The gateway reads W3C traceparent and B3 headers, starts a server span for each request, and passes trace context on to the upstream. Health endpoints aren’t traced. The root span records where the gateway’s own time went, for example fabric.gateway.overhead.duration_ms for time spent in the gateway and fabric.gateway.io_wait.duration_ms for time waiting on the upstream and the client.

The chart samples 10% of new traces (OTEL_TRACES_SAMPLER_ARG: "0.10"). LLM prompts and replies aren’t captured unless you set FABRIC_GW_TRACING_CAPTURE_MESSAGE_CONTENT to "true". For both settings, see Export telemetry with OpenTelemetry.

What to alert on

  • Gateway or embedder pods restarting, or not Ready.
  • Nodes that show Disconnected, Lagging, Degraded, or Critical in Gateway health.
  • A rising rate of 5xx responses in http.server.request.duration. To tell gateway faults from upstream ones, group failed access records by fabric.error.origin.
  • A node component that isn’t healthy in Gateway health, especially Cert renewal, Secret store, and Clock skew.
  • Any increase in otelcol_exporter_send_failed_*.
  • A renewal loop terminated error in the gateway logs, which means a pod can no longer renew its certificate.
  • With the MySQL store, your database’s availability. With the distributed store, fewer running pods than distributed.replicaCount.

Next steps