> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.fabricgateway.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.fabricgateway.ai/_mcp/server.

# Monitor the data plane

> Check data plane health with kubectl, its health endpoints, and the console, and collect its logs, metrics, and traces in your own observability stack.

This page applies to the Enterprise plan, where you run the data plane in your own VPC. You can check the data plane's health in four places: Kubernetes itself, the gateway's health endpoints, the **Gateway health** page in the console, and the logs, metrics, and traces the data plane exports to your own backend. This page describes each one and what to alert on.

The commands on this page use the release name `fabric-dp` in the namespace `pm-fabric-dp`. Resource names come from the chart, not the release: the gateway Deployment, Service, and container are all named `fabric-gateway-chart`, and the embedder is always `fabric-gateway-embedder`.

## Check health with kubectl

Run these checks after an install or upgrade, after a node drain, or as a routine check.

1. Check the gateway pods:

   ```bash
   kubectl -n pm-fabric-dp get pods -l app.kubernetes.io/name=fabric-gateway-chart
   ```

   A healthy gateway pod is `Running` and `READY 2/2`: the gateway container plus the OpenTelemetry Collector sidecar. If you turned the sidecar off, it reads `1/1`. With `opa.enabled: true`, add one container to the count. The list can also include a schema migration pod, which shows `Error` if the migration failed. See step 4.

2. Check the embedder pod:

   ```bash
   kubectl -n pm-fabric-dp get pods -l app=fabric-gateway-embedder
   ```

   A healthy embedder pod is `Running` and `READY 1/1`.

3. Check that the gateway enrolled with the control plane and started cleanly:

   ```bash
   kubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c fabric-gateway-chart \
     | grep -iE "dp enrollment|sync|started"
   ```

   You should see the `dp enrollment` line with its outcome, sync activity, and a `started` line, with no repeating restarts. With the distributed store, the gateway runs as a StatefulSet, so use `statefulset/fabric-gateway-chart` instead of `deploy/fabric-gateway-chart`.

4. Around an install or upgrade with the MySQL store, check the schema migration Job:

   ```bash
   kubectl -n pm-fabric-dp get jobs -l app.kubernetes.io/name=fabric-gateway-chart
   ```

   Helm deletes the Job as soon as it succeeds, so on a steady-state cluster it's normally gone. If it's still there, the migration failed. See [Troubleshoot the data plane](/run/troubleshooting#install-and-migrations). With the distributed store, there's no migration Job.

## Health endpoints

The gateway answers three endpoints on its client port, `8080`, on any hostname. None of them needs credentials.

| Endpoint               | Answer                                                                                                                                                                                                                     |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/healthz`             | `200` with `{"status":"ok"}` whenever the process is serving.                                                                                                                                                              |
| `/readyz`              | `200` with `{"status":"ready"}`. Answers `503` with `{"status":"not_ready","reason":"opa_bundle_not_loaded"}` while the OPA sidecar is enabled and its policy bundle hasn't loaded yet.                                    |
| `/readyz/secret-store` | With the distributed store. Answers `503` with `{"status":"not_ready","reason":"secret_store_rereplication_pending"}` until the pod finishes re-replicating credentials after it joins its peers, or until 5 minutes pass. |

The chart points the liveness, readiness, and startup probes at `/healthz` on port `8080`. Keep them on the gateway's own endpoints: the gateway doesn't serve `/ping` or `/health`, and requests to those paths go to route matching like any other request, so probes on them fail. Paths are matched exactly, so `/healthz/` isn't a health endpoint either.

With the distributed store, point the readiness probe at `/readyz/secret-store` instead, so a rolling restart waits for each pod to re-replicate before it stops the next one. Keep liveness and startup on `/healthz`.

```yaml
container:
  probes:
    readiness: { path: /readyz/secret-store, port: 8080 }
```

None of these endpoints goes through your routes or plugins, and none depends on your configuration. A pod that hasn't enrolled yet doesn't listen at all, so its probes fail with connection refused rather than an error status. Once a pod has enrolled, `/readyz` answers `200` before its first configuration arrives, so don't use it to tell whether a pod has your configuration. Check the node's **Config sync** in **Gateway health** for that.

## Gateway health in the console

In the console, select **Admin** > **Gateway health** to see every node that has connected to the control plane:

* Fleet totals for **Connected**, **Health**, **Sync**, and **Versions**.
* Per node: **Status** (**Healthy**, **Lagging**, or **Disconnected**), **Health** (**OK**, **Degraded**, **Critical**, or **Unknown**), gateway **Version**, the **Config** and **Policy** versions it acknowledged, **Certificate** state, **Uptime**, and **Last seen**.

Each gateway pod is its own node. Select a node for its certificate details and per-stream status. A node reports its health over its sync connection, so a node that can't connect shows as **Disconnected**, not with its own health. For enrollment and certificate details, see [Connect the data plane to the control plane](/run/connect-a-data-plane).

Below the fleet totals, a component health row rolls up **Dependencies**, **Upstreams**, **Certificates**, and **Credentials** across the fleet. Select a tile to see the nodes and conditions behind it.

A node's details also include a **Components** section:

* **Dependencies**: the **Runtime database (MySQL)**, your vault, and **Control plane sync**, with whether each is reachable.
* **Subsystems**: the listener, **Config sync** and **Policy sync** with the version each applied, upstreams, MCP servers, and plugins.
* **Reported components**: the node's own checks, such as **Cert renewal**, **Secret store**, **Config apply**, **Telemetry export**, and **Clock skew**.

To confirm that a pod has your configuration, open its node and check that **Config sync** shows **Connected** and an **Applied** version.

## What the data plane exports

The gateway emits traces, metrics, and logs over OTLP/gRPC. By default, the chart runs an OpenTelemetry Collector sidecar in each gateway pod, and the gateway sends to it on `localhost:4317`. The sidecar does the following:

* Receives the gateway's traces, metrics, and logs.
* Scrapes its own metrics and the embedder's metrics, and ships them with the gateway's.
* Drops traces of health checks and metrics scrapes.
* Keeps traces of requests that matched a route or ended in a `5xx` error, and drops the rest.
* Exports everything to a ClickHouse database that you provide, set with `otel.clickhouse.endpoint`, `otel.clickhouse.database`, and `otel.clickhouse.username`. The password comes from `FABRIC_GW_CLICKHOUSE_PASSWORD` in the app Secret. The collector doesn't create tables, so create `otel_traces` and `otel_logs`, and the metrics tables the ClickHouse exporter expects, in the database first. Failed exports retry from a queue of 1,000 items, so a long backend outage can drop telemetry.

The ClickHouse password key is optional. If it's missing, the sidecar still starts, but telemetry doesn't arrive. Until you point it at your backend, no telemetry leaves your cluster.

To send telemetry to another backend, or to turn export off, see [Export telemetry with OpenTelemetry](/guides/observability/opentelemetry). Usage counts for billing are separate: they travel on the data plane's own connection to the control plane, so telemetry settings never affect them.

## Logs

The gateway writes structured logs to standard output, as JSON by default, and also exports them over OTLP with your other telemetry. Set the level and format with environment variables under `env.vars`:

| Variable     | Values                           | Default |
| ------------ | -------------------------------- | ------- |
| `LOG_LEVEL`  | `debug`, `info`, `warn`, `error` | `info`  |
| `LOG_FORMAT` | `json`, `text`                   | `json`  |

Every request leaves exactly one access record, with the message `request completed`. It's written at `INFO` level, or at `ERROR` when the status is 500 or higher. Its fields include the following:

| Field                                                                                                     | Description                                                                               |
| --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- |
| `http.request.method`, `url.path`, `http.route`                                                           | The request and the route it matched.                                                     |
| `http.response.status_code`                                                                               | The status the client received.                                                           |
| `http.server.request.duration`                                                                            | Duration, in seconds.                                                                     |
| `fabric.request.id`                                                                                       | The request ID, also returned in the `X-Request-Id` header.                               |
| `fabric.tenant.id`, `fabric.namespace.id`, `fabric.consumer`, `fabric.service.name`                       | Who made the request and which service handled it.                                        |
| `server.address`                                                                                          | The upstream host.                                                                        |
| `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | LLM provider, model, and token counts, on LLM requests.                                   |
| `error.type`, `fabric.error.reason`, `fabric.error.origin`, `error`                                       | On failures only. `fabric.error.origin` is `client`, `gateway`, `upstream`, or `unknown`. |

Log lines written while handling a request also carry `trace_id` and `span_id` when the request is traced, so you can jump from a log line to its trace. Credentials such as `Authorization` header values, bearer tokens, and API keys are redacted before they reach logs.

The chart sets `FABRIC_MCP_DEBUG_LOGS` to `"1"`, which adds several diagnostic log lines to each MCP request. To turn them off, set it to `"0"`.

To read the gateway container's logs directly:

```bash
kubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c fabric-gateway-chart
```

## Metrics

The gateway exports its own metrics over OTLP. Useful series include the following:

| Metric                                                                       | What it measures                                             |
| ---------------------------------------------------------------------------- | ------------------------------------------------------------ |
| `http.server.request.duration`                                               | Request latency and count, by route, status, and tenant.     |
| `gen_ai.client.token.usage`                                                  | LLM tokens by type (input, output, cache read, cache write). |
| `gen_ai.client.operation.duration`                                           | LLM call duration.                                           |
| `fabric.server.connections`                                                  | Open client connections.                                     |
| `fabric.upstream.connections.open`, `fabric.upstream.connection.dial.errors` | Connections to upstreams, and failures to open them.         |
| `fabric.upstream.requests.in_flight`                                         | Requests waiting on an upstream.                             |
| `fabric.process.open_fds`, `fabric.process.max_fds`                          | File descriptors in use and the limit.                       |

Token counts are recorded only on routes with the LLM observability plugin. Without it, the gateway records no token counts.

The collector sidecar and the embedder also serve Prometheus metrics of their own, and the collector scrapes and ships both by default:

* **Collector metrics** (`otelcol_receiver_*`, `otelcol_processor_*`, `otelcol_exporter_*`), on port `8888` of each gateway pod. They count accepted, refused, dropped, and sent data, queue depth, and export failures. `otelcol_exporter_send_failed_*` is the earliest sign that telemetry isn't reaching your backend.
* **Embedder metrics** (`te_*`), at `/metrics` on the embedder Service.

To check the raw endpoints yourself:

```bash
kubectl -n pm-fabric-dp port-forward deploy/fabric-gateway-chart 8888:8888
curl -s localhost:8888/metrics | grep '^otelcol_'
```

```bash
kubectl -n pm-fabric-dp port-forward svc/fabric-gateway-embedder 8080:80
curl -s localhost:8080/metrics | grep '^te_'
```

If you'd rather have your own Prometheus pull the sidecar metrics, turn on one of these. Both are off by default:

* `metrics.serviceMonitor.enabled: true` renders a PodMonitor for the collector and a ServiceMonitor for the embedder. It needs the Prometheus Operator. Set `metrics.serviceMonitor.labels` to match your operator's selector, for example `release: kube-prometheus-stack`.
* `metrics.annotations.enabled: true` adds `prometheus.io/scrape` annotations to the gateway and embedder pods, for an annotation-based scraper.

## Traces

The gateway reads W3C `traceparent` and B3 headers, starts a server span for each request, and passes trace context on to the upstream. Health endpoints aren't traced. The root span records where the gateway's own time went, for example `fabric.gateway.overhead.duration_ms` for time spent in the gateway and `fabric.gateway.io_wait.duration_ms` for time waiting on the upstream and the client.

The chart samples 10% of new traces (`OTEL_TRACES_SAMPLER_ARG: "0.10"`). LLM prompts and replies aren't captured unless you set `FABRIC_GW_TRACING_CAPTURE_MESSAGE_CONTENT` to `"true"`. For both settings, see [Export telemetry with OpenTelemetry](/guides/observability/opentelemetry#tune-what-the-data-plane-exports).

## What to alert on

* Gateway or embedder pods restarting, or not `Ready`.
* Nodes that show **Disconnected**, **Lagging**, **Degraded**, or **Critical** in **Gateway health**.
* A rising rate of 5xx responses in `http.server.request.duration`. To tell gateway faults from upstream ones, group failed access records by `fabric.error.origin`.
* A node component that isn't healthy in **Gateway health**, especially **Cert renewal**, **Secret store**, and **Clock skew**.
* Any increase in `otelcol_exporter_send_failed_*`.
* A `renewal loop terminated` error in the gateway logs, which means a pod can no longer renew its certificate.
* With the MySQL store, your database's availability. With the distributed store, fewer running pods than `distributed.replicaCount`.

## Next steps

* To fix a problem you found, see [Troubleshoot the data plane](/run/troubleshooting).
* To send telemetry to your backend, see [Export telemetry with OpenTelemetry](/guides/observability/opentelemetry).
* To run more replicas, see [Scale and run a highly available data plane](/run/scale-and-high-availability).