Skip to navigation

Export telemetry with OpenTelemetry

A data plane emits traces, metrics, and logs over OTLP/gRPC, so you can send them to any OpenTelemetry backend. It exports only to destinations that you configure. This page shows the two ways to route that telemetry, how to point the data plane at an OTLP endpoint, how to tune or turn off export, and how to confirm that telemetry arrives.

This applies to the Enterprise plan, where the data plane runs in your VPC. On the Free plan, Postman runs the data plane, and you review traffic in the console instead. For more information, see Monitor gateway traffic.

Choose how telemetry leaves the pod

The gateway always sends OTLP/gRPC to OTEL_EXPORTER_OTLP_ENDPOINT. What sits at that address is up to you:

ApproachHow it worksUse it when
Collector sidecar (chart default)The chart runs an OpenTelemetry Collector in each gateway pod, and the gateway sends to it on localhost:4317. The sidecar also scrapes its own and the embedder’s metrics, drops health-check traces, keeps traces that matched a route or ended in a 5xx, and exports to ClickHouse.You use ClickHouse, or your backend accepts only OTLP over HTTP, such as Grafana Cloud.
DirectYou turn off the sidecar and point the gateway straight at your collector or backend.Your backend accepts OTLP/gRPC, such as Honeycomb, New Relic, a Datadog Agent, or your own OpenTelemetry Collector.

With the direct approach, the sidecar’s extra processing goes away: you receive every sampled trace, and the collector and embedder metrics aren’t shipped. For the sidecar’s details, see Monitor the data plane.

Usage metering travels to the control plane on the data plane’s own connection, so nothing on this page affects billing or usage data.

Prerequisites

Before you begin, make sure you have the following:

  • A data plane installed with the Helm chart. For the steps, see Install the data plane.
  • An OTLP/gRPC endpoint that the gateway pods can reach, and any credentials that your backend needs.
  • A way to add keys to the data plane’s app Secret. See Manage data plane secrets and keys.

Send telemetry straight to an OTLP endpoint

  1. In your values file, turn off the collector sidecar and set your endpoint:

    otelCollector:
    enabled: false
    env:
    vars:
    OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector.observability.svc.cluster.local:4317"

    Use an https:// endpoint for a backend that needs TLS.

  2. If your backend needs an API key or other headers, add OTEL_EXPORTER_OTLP_HEADERS to the app Secret rather than to the values file. The value is a comma-separated list of name=value pairs. A key in the app Secret overrides the same key in env.vars. For example, with the app Secret fabric-dp-secrets from Install the data plane:

    kubectl -n pm-fabric-dp patch secret fabric-dp-secrets --type merge \
    -p '{"stringData":{"OTEL_EXPORTER_OTLP_HEADERS":"api-key=YOUR_KEY"}}'

    If you manage the Secret through a secret manager, add the key there instead.

  3. Apply the change:

    helm upgrade fabric-dp ./fabric-gateway-chart -n pm-fabric-dp -f my-values.yaml

    If you changed only the Secret, restart the gateway so it reads the new value:

    kubectl -n pm-fabric-dp rollout restart deploy/fabric-gateway-chart

    With the distributed store, the gateway runs as a StatefulSet, so use statefulset/fabric-gateway-chart in place of deploy/fabric-gateway-chart.

If you turn off the sidecar without changing OTEL_EXPORTER_OTLP_ENDPOINT, the gateway still sends to localhost:4317, and the telemetry is dropped.

The chart’s own examples drop the sidecar with additionalContainers: [] instead. That also works, but it removes any other sidecar you added to that list, so prefer otelCollector.enabled: false.

Tune what the data plane exports

The gateway reads the standard OpenTelemetry variables. Set them under env.vars in your values file:

VariableWhat it does
OTEL_EXPORTER_OTLP_ENDPOINTThe OTLP/gRPC endpoint for every signal. The chart sets http://localhost:4317, the sidecar.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_LOGS_ENDPOINTAn endpoint for one signal, which overrides the shared endpoint.
OTEL_EXPORTER_OTLP_HEADERSHeaders sent with every export, such as an API key. Put it in the app Secret.
OTEL_TRACES_EXPORTER, OTEL_METRICS_EXPORTER, OTEL_LOGS_EXPORTERSet to none to stop exporting that signal.
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARGThe trace sampler and its rate. The chart samples 10% of new traces with parentbased_traceidratio.
OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTESThe service name and resource attributes on exported data. The chart uses fabric-gateway-dp, with service.namespace=fabric-gateway,deployment.environment=production. Change deployment.environment to match your own environment names.
FABRIC_GW_TRACING_CAPTURE_MESSAGE_CONTENTSet to "true" to add LLM messages and MCP tool arguments and results to spans. Off by default.
FABRIC_GW_TRACING_MAX_PAYLOAD_BYTESThe maximum size of captured content. The chart sets 16384.

Captured message content can contain prompts and personal data. When the PII Redaction plugin runs on a request, the exported content is the scrubbed version. Otherwise it’s exported as sent. Turn capture on only if your data-handling rules allow it.

Turn off telemetry export

To export nothing, turn off the sidecar and every signal:

otelCollector:
enabled: false
env:
vars:
OTEL_TRACES_EXPORTER: "none"
OTEL_METRICS_EXPORTER: "none"
OTEL_LOGS_EXPORTER: "none"

The gateway still reads and passes on trace context, so traces your own services start stay connected across the gateway. Logs still go to standard output.

Confirm that telemetry arrives

  1. Send a request through a route on the data plane. For the steps, see Authenticate and send a request.
  2. In your backend, search for traces from the service fabric-gateway-dp, or the name that you set in OTEL_SERVICE_NAME. The chart samples 10% of new traces, so send several requests.

If nothing arrives, check the gateway logs:

kubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c fabric-gateway-chart | grep -i otlp

With the distributed store, use statefulset/fabric-gateway-chart in place of deploy/fabric-gateway-chart.

A warning that tracing, metrics, or logs are enabled but no endpoint is configured means the gateway found no OTLP endpoint for that signal. If you kept the sidecar, check its export failures, as shown in Monitor the data plane.

Next steps