Scale and run a highly available data plane
This page applies to the Enterprise plan, where you run the data plane in your own VPC. It explains how to run and autoscale more than one gateway replica, how the chart keeps replicas available during drains and rollouts, how the secret store you choose affects availability, and what the data plane keeps doing when it loses its connection to the control plane.
Run several replicas
The chart runs two gateway replicas by default. To change the count, set replicaCount:
Each replica is a separate node: it enrolls under its own hostname, gets its own certificate, and appears as its own row in Admin > Gateway health.
A new replica has to enroll before it serves traffic, so it needs a working enrollment credential when it starts. With the chart’s defaults, a pod keeps its certificate on a volume that’s deleted with the pod, so this applies to every replacement pod too. Use workload attestation, or keep a valid reusable join token in the app Secret. See Connect a data plane to the control plane.
How the chart spreads and drains pods
The chart sets the following for the gateway pods. You don’t need to change them for a typical install.
- Spread. Pods spread across zones and across nodes where possible. The spread is a preference, so pods still schedule when the cluster can’t satisfy it.
- PodDisruptionBudget. On by default with
minAvailable: 1, so a node drain or upgrade always leaves at least one gateway pod serving. - Graceful shutdown. A stopping pod waits 60 seconds before it shuts down, so your load balancer can stop sending it traffic, within a 100-second termination grace period.
To change the gateway’s budget, set podDisruptionBudget.minAvailable. For the gateway, minAvailable takes precedence over maxUnavailable, so to use maxUnavailable you must also remove minAvailable.
Add autoscaling
You can hand the gateway replica count to one autoscaler:
- HorizontalPodAutoscaler. Set
autoscaling.enabled: true. It needs metrics-server, and scales on CPU (target 60%) and memory (target 75%) by default. Change the targets withautoscaling.targetCPUUtilizationPercentageandautoscaling.targetMemoryUtilizationPercentage. - KEDA. Set
keda.enabled: true. It needs the KEDA operator, and scales on CPU (target 60%) and memory (target 75%). These targets are fixed: theautoscaling.*target values don’t change them.
Enable only one. The chart refuses to render if both are on. Both render only when service.enabled is true. KEDA polls every 20 seconds and scales down slowly: it waits 300 seconds, then removes at most 10% of pods every 120 seconds. To add triggers, set scalingConfig.triggers. They authenticate through a ClusterTriggerAuthentication named keda-cluster-trigger-authentication, which you create.
Both autoscalers use scalingConfig.scalingProperties.minReplicas and maxReplicas, which both default to 2. Raise maxReplicas, or the autoscaler can’t add pods:
With an autoscaler on, keep replicaCount equal to scalingConfig.scalingProperties.minReplicas, because each helm upgrade resets the Deployment to replicaCount. Autoscaling applies to the default Deployment. With the distributed store, the gateway runs as a StatefulSet, and you set its size with distributed.replicaCount instead.
Share state across replicas
Some state is held in each pod’s memory:
- Rate-limit counters. Each pod counts on its own by default, so a limit applies per pod. To enforce one limit across the data plane, use a coordination Redis: set
FABRIC_GW_COORD_REDIS_ENABLEDto"true",FABRIC_GW_COORD_REDIS_ADDRESSto your Redis, andFABRIC_GW_RATE_LIMIT_COUNTER_STOREtoredis. AddFABRIC_GW_COORD_REDIS_TLSfor a TLS endpoint, and the password in the app Secret.FABRIC_GW_RATE_LIMIT_ON_ERROR(localorfail_open) sets what happens if Redis fails. See Helm chart values. - MCP notification streams. A client’s notification stream stays on the pod that received it. When a user connects an upstream account on another pod, that pod tells the others through the coordination Redis. Without Redis, a client whose stream is on another pod doesn’t hear that its tool list changed.
Scale the embedder
The embedder runs one replica by default, with its own PodDisruptionBudget. With a single replica, that budget blocks node drains. For scaling, the model cache, and the drain fix, see Run the tool-search embedder.
Choose a secret store
The data plane stores the credentials users connect, client registrations, and short-lived login state in a secret store. All replicas share it.
Run the distributed store
Set distributed.enabled: true. The chart then does the following:
- Runs the gateway as a StatefulSet (three replicas by default, set by
distributed.replicaCount), with its own volume per pod at the cache path (distributed.persistence.size,1Giby default). Because the volume survives restarts, each pod also keeps its certificate and configuration cache. - Adds a headless Service so pods find each other by DNS.
- Opens ports
3320(data) and3322(gossip), and adds a NetworkPolicy that allows only gateway pods to reach them. - Skips the MySQL migration Job. This mode needs no MySQL.
Add FABRIC_GW_DISTRIBUTED_GOSSIP_KEY and FABRIC_GW_DISTRIBUTED_AUTH_PASSWORD to the app Secret. In production mode, a pod without them refuses to start. See Manage data plane secrets and keys.
Traffic between pods on ports 3320 and 3322 isn’t encrypted in transit. Stored values are encrypted, but anyone who can reach the data port can list, overwrite, or delete them. Keep distributed.networkPolicy.enabled on, and use an encrypted overlay or mesh if pods talk across a network you don’t trust.
Plan for the following:
- Writes need a majority. With three replicas, a credential write needs two pods. A single surviving pod can’t save new credentials.
- Snapshots run every 30 seconds (
distributed.snapshotInterval). If every pod stops at once without shutting down cleanly, credential changes made since the last snapshot can be lost. - A full restart restores from the newest snapshot. While a cold data plane restores, credential operations wait, but
/healthzstill answers and requests that need no stored credential are served. - Clocks must agree. The store uses wall-clock time for expiry and conflict resolution. Run NTP on every node.
- Keep
distributed.memberCountQuorumat1. A higher value stops a lone pod from starting.
If these trade-offs don’t suit you, use MySQL.
When the control plane is unreachable
The data plane serves traffic on its own. The control plane isn’t in the request path, so an outage affects configuration and identity, not requests:
- Running pods keep serving the last configuration they applied, for as long as the outage lasts. Configuration and policy changes you make in the console reach them when the connection returns.
- A pod that restarts with its cache and certificate intact starts and serves its cached configuration without contacting the control plane. With the chart’s defaults, this holds for a container restart inside the same pod. With the distributed store, it also holds for a replaced pod, because its volume survives.
- A new pod can’t enroll, so it exits and Kubernetes restarts it until the control plane is back. This includes pods added by an autoscaler and replacement pods without a persistent volume. Avoid rollouts and scale-ups during an outage.
- Certificate renewal retries with back-off. As long as the control plane returns before a pod’s certificate expires, the pod renews without help.
- The console shows the affected nodes as Disconnected.
Next steps
- To set the values named on this page, see Helm chart values.
- To watch replicas and the store, see Monitor the data plane.
- Before you go live, see the Production checklist.

