Skip to navigation

Send data plane telemetry to Grafana Cloud

Grafana Cloud’s OTLP endpoint accepts OTLP over HTTP only, and the gateway exports over gRPC, so the gateway can’t send to it directly. Instead, you replace the chart’s collector sidecar with one that receives the gateway’s OTLP/gRPC on localhost:4317 and re-exports it to Grafana Cloud over HTTP. This page shows how to set up that sidecar, keep your Grafana Cloud credentials in the app Secret, and confirm that telemetry arrives.

This applies to the Enterprise plan, where the data plane runs in your VPC. For how data plane telemetry export works in general, see Export telemetry with OpenTelemetry.

Prerequisites

Before you begin, make sure you have the following:

  • A data plane installed with the Helm chart. See Install the data plane.
  • Your Grafana Cloud stack’s OTLP endpoint, instance ID, and an access token that can write telemetry.
  • Egress from the gateway pods to your Grafana Cloud OTLP endpoint.
  • A way to add keys to the data plane’s app Secret. See Manage data plane secrets and keys.

Add your credentials to the app Secret

The sidecar authenticates with HTTP Basic auth. Add the base64 encoding of <instance-id>:<token> to the app Secret as GRAFANA_CLOUD_AUTH:

AUTH=$(printf '%s:%s' "<instance-id>" "<token>" | base64 | tr -d '\n')
kubectl -n pm-fabric-dp patch secret fabric-dp-secrets --type merge \
-p "{\"stringData\":{\"GRAFANA_CLOUD_AUTH\":\"$AUTH\"}}"

tr -d '\n' removes the line breaks that base64 adds to long output on Linux, which would make the patch invalid JSON. Replace fabric-dp-secrets with the name of your app Secret, the one in secrets.app.existingSecret. If you manage the Secret through a secret manager, add the key there instead.

Replace the collector sidecar

  1. In your values file, replace the chart’s sidecar with this one. Set the endpoint to your stack’s region, and keep otelCollector.enabled at its default of true. Setting it to false removes any sidecar named otel-collector, including this one:

    additionalContainers:
    - name: otel-collector
    image: otel/opentelemetry-collector-contrib:0.149.0
    imagePullPolicy: IfNotPresent
    args:
    - "--config=yaml:{receivers:{otlp:{protocols:{grpc:{endpoint:0.0.0.0:4317}}}},processors:{batch:{}},exporters:{otlphttp:{endpoint:\"${env:GRAFANA_CLOUD_OTLP_ENDPOINT}\",headers:{Authorization:\"Basic ${env:GRAFANA_CLOUD_AUTH}\"}}},service:{pipelines:{traces:{receivers:[otlp],processors:[batch],exporters:[otlphttp]},metrics:{receivers:[otlp],processors:[batch],exporters:[otlphttp]},logs:{receivers:[otlp],processors:[batch],exporters:[otlphttp]}}}}"
    env:
    - name: GRAFANA_CLOUD_OTLP_ENDPOINT
    value: "https://otlp-gateway-<region>.grafana.net/otlp"
    - name: GRAFANA_CLOUD_AUTH
    valueFrom:
    secretKeyRef:
    name: '{{ include "appSecretName" $ }}'
    key: GRAFANA_CLOUD_AUTH
    ports:
    - { containerPort: 4317, name: otlp-grpc }

    The chart fills in the app Secret’s name for {{ include "appSecretName" $ }}, so you don’t need to change it. Leave OTEL_EXPORTER_OTLP_ENDPOINT at the chart’s default, http://localhost:4317, so the gateway sends to this sidecar. If you mirror images, mirror the collector image too and change image.

  2. Apply the change:

    helm upgrade fabric-dp ./fabric-gateway-chart -n pm-fabric-dp -f my-values.yaml

This sidecar forwards only what the gateway sends. It doesn’t scrape the collector’s or the embedder’s metrics, and it doesn’t drop health-check traces or apply the default sidecar’s trace filtering. Add those to its configuration if you need them.

Confirm that telemetry arrives

  1. Check that the gateway pods are READY 2/2:

    kubectl -n pm-fabric-dp get pods -l app.kubernetes.io/name=fabric-gateway-chart
  2. Send several requests through a route on the data plane. The chart samples 10% of new traces. For the steps, see Authenticate and send a request.

  3. In Grafana Cloud, search your traces for the service fabric-gateway-dp, or the name you set in OTEL_SERVICE_NAME.

If nothing arrives, check the sidecar’s logs for export errors:

kubectl -n pm-fabric-dp logs deploy/fabric-gateway-chart -c otel-collector

With the distributed store, use statefulset/fabric-gateway-chart in place of deploy/fabric-gateway-chart.

An authentication error usually means GRAFANA_CLOUD_AUTH isn’t the base64 encoding of <instance-id>:<token>.

Next steps