Skip to navigation

Scale and run a highly available data plane

This page applies to the Enterprise plan, where you run the data plane in your own VPC. It explains how to run and autoscale more than one gateway replica, how the chart keeps replicas available during drains and rollouts, how the secret store you choose affects availability, and what the data plane keeps doing when it loses its connection to the control plane.

Run several replicas

The chart runs two gateway replicas by default. To change the count, set replicaCount:

helm upgrade fabric-dp ./fabric-gateway-chart -n pm-fabric-dp -f my-values.yaml --set replicaCount=4

Each replica is a separate node: it enrolls under its own hostname, gets its own certificate, and appears as its own row in Admin > Gateway health.

A new replica has to enroll before it serves traffic, so it needs a working enrollment credential when it starts. With the chart’s defaults, a pod keeps its certificate on a volume that’s deleted with the pod, so this applies to every replacement pod too. Use workload attestation, or keep a valid reusable join token in the app Secret. See Connect a data plane to the control plane.

How the chart spreads and drains pods

The chart sets the following for the gateway pods. You don’t need to change them for a typical install.

  • Spread. Pods spread across zones and across nodes where possible. The spread is a preference, so pods still schedule when the cluster can’t satisfy it.
  • PodDisruptionBudget. On by default with minAvailable: 1, so a node drain or upgrade always leaves at least one gateway pod serving.
  • Graceful shutdown. A stopping pod waits 60 seconds before it shuts down, so your load balancer can stop sending it traffic, within a 100-second termination grace period.

To change the gateway’s budget, set podDisruptionBudget.minAvailable. For the gateway, minAvailable takes precedence over maxUnavailable, so to use maxUnavailable you must also remove minAvailable.

Add autoscaling

You can hand the gateway replica count to one autoscaler:

  • HorizontalPodAutoscaler. Set autoscaling.enabled: true. It needs metrics-server, and scales on CPU (target 60%) and memory (target 75%) by default. Change the targets with autoscaling.targetCPUUtilizationPercentage and autoscaling.targetMemoryUtilizationPercentage.
  • KEDA. Set keda.enabled: true. It needs the KEDA operator, and scales on CPU (target 60%) and memory (target 75%). These targets are fixed: the autoscaling.* target values don’t change them.

Enable only one. The chart refuses to render if both are on. Both render only when service.enabled is true. KEDA polls every 20 seconds and scales down slowly: it waits 300 seconds, then removes at most 10% of pods every 120 seconds. To add triggers, set scalingConfig.triggers. They authenticate through a ClusterTriggerAuthentication named keda-cluster-trigger-authentication, which you create.

Both autoscalers use scalingConfig.scalingProperties.minReplicas and maxReplicas, which both default to 2. Raise maxReplicas, or the autoscaler can’t add pods:

autoscaling:
enabled: true
scalingConfig:
scalingProperties:
minReplicas: 2
maxReplicas: 6

With an autoscaler on, keep replicaCount equal to scalingConfig.scalingProperties.minReplicas, because each helm upgrade resets the Deployment to replicaCount. Autoscaling applies to the default Deployment. With the distributed store, the gateway runs as a StatefulSet, and you set its size with distributed.replicaCount instead.

Share state across replicas

Some state is held in each pod’s memory:

  • Rate-limit counters. Each pod counts on its own by default, so a limit applies per pod. To enforce one limit across the data plane, use a coordination Redis: set FABRIC_GW_COORD_REDIS_ENABLED to "true", FABRIC_GW_COORD_REDIS_ADDRESS to your Redis, and FABRIC_GW_RATE_LIMIT_COUNTER_STORE to redis. Add FABRIC_GW_COORD_REDIS_TLS for a TLS endpoint, and the password in the app Secret. FABRIC_GW_RATE_LIMIT_ON_ERROR (local or fail_open) sets what happens if Redis fails. See Helm chart values.
  • MCP notification streams. A client’s notification stream stays on the pod that received it. When a user connects an upstream account on another pod, that pod tells the others through the coordination Redis. Without Redis, a client whose stream is on another pod doesn’t hear that its tool list changed.

Scale the embedder

The embedder runs one replica by default, with its own PodDisruptionBudget. With a single replica, that budget blocks node drains. For scaling, the model cache, and the drain fix, see Run the tool-search embedder.

Choose a secret store

The data plane stores the credentials users connect, client registrations, and short-lived login state in a secret store. All replicas share it.

MySQL (chart default)Distributed store
Where data livesYour MySQL databaseIn memory across the gateway pods, with an encrypted snapshot on each pod’s volume
What you runA MySQL instance, with backupsNothing extra, but a StorageClass and a NetworkPolicy controller
Workload typeDeploymentStatefulSet
Schema migrationsA Job on each install and upgradeNone
AutoscalingHorizontalPodAutoscaler or KEDANone. Set distributed.replicaCount.

Run the distributed store

Set distributed.enabled: true. The chart then does the following:

  • Runs the gateway as a StatefulSet (three replicas by default, set by distributed.replicaCount), with its own volume per pod at the cache path (distributed.persistence.size, 1Gi by default). Because the volume survives restarts, each pod also keeps its certificate and configuration cache.
  • Adds a headless Service so pods find each other by DNS.
  • Opens ports 3320 (data) and 3322 (gossip), and adds a NetworkPolicy that allows only gateway pods to reach them.
  • Skips the MySQL migration Job. This mode needs no MySQL.

Add FABRIC_GW_DISTRIBUTED_GOSSIP_KEY and FABRIC_GW_DISTRIBUTED_AUTH_PASSWORD to the app Secret. In production mode, a pod without them refuses to start. See Manage data plane secrets and keys.

Traffic between pods on ports 3320 and 3322 isn’t encrypted in transit. Stored values are encrypted, but anyone who can reach the data port can list, overwrite, or delete them. Keep distributed.networkPolicy.enabled on, and use an encrypted overlay or mesh if pods talk across a network you don’t trust.

Plan for the following:

  • Writes need a majority. With three replicas, a credential write needs two pods. A single surviving pod can’t save new credentials.
  • Snapshots run every 30 seconds (distributed.snapshotInterval). If every pod stops at once without shutting down cleanly, credential changes made since the last snapshot can be lost.
  • A full restart restores from the newest snapshot. While a cold data plane restores, credential operations wait, but /healthz still answers and requests that need no stored credential are served.
  • Clocks must agree. The store uses wall-clock time for expiry and conflict resolution. Run NTP on every node.
  • Keep distributed.memberCountQuorum at 1. A higher value stops a lone pod from starting.

If these trade-offs don’t suit you, use MySQL.

When the control plane is unreachable

The data plane serves traffic on its own. The control plane isn’t in the request path, so an outage affects configuration and identity, not requests:

  • Running pods keep serving the last configuration they applied, for as long as the outage lasts. Configuration and policy changes you make in the console reach them when the connection returns.
  • A pod that restarts with its cache and certificate intact starts and serves its cached configuration without contacting the control plane. With the chart’s defaults, this holds for a container restart inside the same pod. With the distributed store, it also holds for a replaced pod, because its volume survives.
  • A new pod can’t enroll, so it exits and Kubernetes restarts it until the control plane is back. This includes pods added by an autoscaler and replacement pods without a persistent volume. Avoid rollouts and scale-ups during an outage.
  • Certificate renewal retries with back-off. As long as the control plane returns before a pod’s certificate expires, the pod renews without help.
  • The console shows the affected nodes as Disconnected.

Next steps