Skip to navigation

Data plane production checklist

This page applies to the Enterprise plan, where you run the data plane in your own VPC. Work through this checklist before the data plane carries production traffic. The chart ships with production-oriented defaults, so most items either confirm that you kept a default or ask you to replace a placeholder. Each item links to the page that explains it.

Defaults to keep

The chart turns these on for you. Keep them on.

  • Production mode. FABRIC_GW_PRODUCTION_MODE is "true". The data plane then refuses to start without its encryption key, a tenant ID, or the control plane CA certificate, or without a bootstrap token unless workload attestation is on. It also refuses to start with FABRIC_GW_STORE_AUTO_MIGRATE or FABRIC_GW_DEBUG_SCOPES set to "true", and credential brokering fails closed. Set it to "false" only for local testing.
  • Pod security. The gateway and embedder containers run as non-root with the RuntimeDefault seccomp profile, drop all Linux capabilities, block privilege escalation, and use a read-only root filesystem. The OpenTelemetry Collector sidecar runs as non-root with the RuntimeDefault profile, but has no container security context of its own. If your policy requires one, copy the chart’s whole otel-collector entry into additionalContainers in your values file, and add a securityContext. Writable state lives only in mounted volumes: /data/cache for the gateway, and /data and /tmp for the embedder.
  • Health probes. Liveness, readiness, and startup probes all point at /healthz on port 8080. See Monitor the data plane.
  • Disruption budgets. The gateway and embedder each have a PodDisruptionBudget with minAvailable: 1.
  • Tracing. The gateway samples 10% of new traces (OTEL_TRACES_SAMPLER_ARG: "0.10"), and doesn’t capture LLM messages or MCP tool arguments and results (FABRIC_GW_TRACING_CAPTURE_MESSAGE_CONTENT: "false"). Raise the sample rate or turn on message capture only after you review the cost and your data-handling rules.
  • Private hosts. FABRIC_GW_SSRF_ALLOW_PRIVATE_HOSTS is empty, so the gateway’s own authentication calls reach no private host you haven’t listed. The gateway’s authentication calls and upstream calls never reach link-local or cloud metadata addresses. Upstream calls to loopback are refused unless you set FABRIC_GW_SSRF_ALLOW_LOOPBACK_UPSTREAMS to "true".

Install and configuration

  • Every image the chart pulls is mirrored into your registry and pinned:

    • The gateway: image.repository, with image.digest or an exact image.tag.
    • The embedder, ghcr.io/huggingface/text-embeddings-inference:cpu-1.9: embedder.image.repository and embedder.image.digest.
    • The migration Job, flyway/flyway:10: migrations.image, pinned to an exact tag.
    • The collector sidecar, otel/opentelemetry-collector-contrib:0.149.0: the image of the otel-collector entry in additionalContainers.
    • OPA, if you turn it on: opa.image.repository and opa.image.tag.

    The migration Job doesn’t use image.imagePullSecrets. If your registry needs credentials, add the pull Secret to the namespace’s default ServiceAccount, which the Job runs as. See Install the data plane with Helm.

  • Every placeholder in the chart’s values is replaced: image.repository, FABRIC_GW_TENANT_ID, FABRIC_GW_CP_ADDRESS, FABRIC_GW_MYSQL_HOST, FABRIC_GW_ADMIN_EXTERNAL_URL, FABRIC_GW_BROKER_EXTERNAL_URL, and otel.clickhouse.* if you use the default collector. See Helm values.

  • Your values file is in version control, holds no secret values, and is passed on every helm upgrade.

  • The control plane CA certificate is the one Postman gave you, supplied with syncCA.pem or the fabric-dp-sync-ca Secret.

  • FABRIC_GW_ADMIN_EXTERNAL_URL and FABRIC_GW_BROKER_EXTERNAL_URL match the external URL clients use, and Admin > Data plane > Base URL in the console is set to the same URL. See Expose the gateway to your clients.

  • You decided whether routes may reach upstreams on private addresses. They can by default. To block them, set FABRIC_GW_SSRF_BLOCK_PRIVATE_UPSTREAMS to "true" and list the internal hosts your routes need in FABRIC_GW_SSRF_ALLOW_PRIVATE_UPSTREAM_HOSTS. Include the embedder’s host from your Tool Search plugin’s embedder URL, or tool search stops working. See Calls to private hosts.

  • FABRIC_GW_SSRF_ALLOW_PRIVATE_HOSTS lists only identity provider or OAuth hosts on your private network, if any.

  • If you use the access-control, validator, or transformer policy plugins, you know that they need the OPA sidecar (opa.enabled) and fail closed without it. Before you turn it on, read the readiness warning in Helm values: in chart 0.6.0, the sidecar’s readiness check can’t succeed and pods stay at 2/3.

Secrets and identity

  • The app Secret is created from your secret manager and named in secrets.app.existingSecret. secrets.dev.enabled is false. See Secrets and keys.
  • FABRIC_GW_SECRET_KEY is backed up somewhere durable and is the same on every pod.
  • You have a plan for new pods to enroll: workload attestation (recommended), or a bootstrap token that you mint in the console when you add or replace pods. A token lasts 10 minutes by default and 30 minutes at most. See Connect the data plane to the control plane.
  • Bootstrap tokens are stored like secrets, and the people who can mint them are limited.
  • You know how to revoke a node’s certificate from Admin > Gateway health, and that you also need to delete the pod to stop its traffic.

Network

  • Egress is allowed to the control plane sync address, your secret store, your telemetry backend, your upstreams, and huggingface.co unless the embedder runs offline. See Network and egress.
  • Nothing between the data plane and the control plane ends TLS on the sync connection. In Istio clusters, istio.enabled is true, and the sync address’s port is in istio.excludeOutboundPorts (4317,4318,9090 by default).
  • Clients reach the gateway through your ingress, with TLS ending there. See Expose the gateway to your clients.
  • On a cluster with a NetworkPolicy controller, embedder.networkPolicy.enabled restricts the embedder to the gateway pods.

Availability

  • At least two gateway replicas (replicaCount). With the distributed store, the gateway runs as a StatefulSet sized by distributed.replicaCount, which defaults to 3. See Scale and run a highly available data plane.
  • If an autoscaler is on, scalingConfig.scalingProperties.maxReplicas is above 2. The chart’s autoscalers scale only the Deployment, so they don’t apply with the distributed store.
  • NTP runs on every Kubernetes node. Enrollment allows only 60 seconds of clock skew, and Gateway health flags a node whose clock is further off.
  • The embedder doesn’t stall node drains: it runs two or more replicas, or has embedder.podDisruptionBudget.maxUnavailable: 1. See Run the tool-search embedder.
  • The embedder’s model cache is on a volume (embedder.persistence.enabled), so restarts don’t download the model again. With more than one embedder replica, or an autoscaler that can add one, set embedder.persistence.accessModes to [ReadWriteMany] or use embedder.persistence.existingClaim. With the default ReadWriteOnce, keep one replica and set embedder.podDisruptionBudget.maxUnavailable: 1.
  • If more than one pod must share rate limits, a coordination Redis is configured, and FABRIC_GW_RATE_LIMIT_COUNTER_STORE is "redis". It defaults to in_memory, which counts on each pod separately.

Secret store

With MySQL:

  • The database is dedicated to the data plane, backed up with its encryption key, and reached over TLS, with FABRIC_GW_MYSQL_TLS set to "true" under env.vars. See Maintain the data plane and its data.
  • The migration user has DDL and data grants. If you want least privilege, the runtime user has only SELECT, INSERT, UPDATE, and DELETE. See Use a least-privilege migration user.

With the distributed store:

  • FABRIC_GW_DISTRIBUTED_GOSSIP_KEY and FABRIC_GW_DISTRIBUTED_AUTH_PASSWORD are in the app Secret, under exactly these key names. Setting distributed.gossipKeySecretKey or distributed.authPasswordSecretKey in your values has no effect.

  • distributed.networkPolicy.enabled is true, and your cluster enforces NetworkPolicies. The policy allows only the store’s ports, 3320 and 3322, between gateway pods, so it also blocks client traffic to port 8080. Allow it with distributed.networkPolicy.extraIngress:

    distributed:
    networkPolicy:
    extraIngress:
    - ports:
    - protocol: TCP
    port: 8080
  • Each pod has its own persistent volume.

Observability

  • Telemetry reaches your backend, and you alert on otelcol_exporter_send_failed_* if you use the collector sidecar. See Monitor the data plane.
  • You alert on pod restarts and on nodes that show Disconnected, Lagging, Degraded, or Critical in Gateway health. See What to alert on.
  • You set FABRIC_MCP_DEBUG_LOGS to "0", unless you want its extra diagnostic log lines for each MCP request. The chart turns it on.

Next steps