> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.fabricgateway.ai/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.fabricgateway.ai/_mcp/server.

# Scale and run a highly available data plane

> Run several gateway replicas, add autoscaling, keep node drains working, choose a secret store, and understand what the data plane does when the control plane is unreachable.

This page applies to the Enterprise plan, where you run the data plane in your own VPC. It explains how to run and autoscale more than one gateway replica, how the chart keeps replicas available during drains and rollouts, how the secret store you choose affects availability, and what the data plane keeps doing when it loses its connection to the control plane.

## Run several replicas

The chart runs two gateway replicas by default. To change the count, set `replicaCount`:

```bash
helm upgrade fabric-dp ./fabric-gateway-chart -n pm-fabric-dp -f my-values.yaml --set replicaCount=4
```

Each replica is a separate node: it enrolls under its own hostname, gets its own certificate, and appears as its own row in **Admin** > **Gateway health**.

A new replica has to enroll before it serves traffic, so it needs a working enrollment credential when it starts. With the chart's defaults, a pod keeps its certificate on a volume that's deleted with the pod, so this applies to every replacement pod too. Use workload attestation, or keep a valid reusable join token in the app Secret. See [Connect a data plane to the control plane](/run/connect-a-data-plane).

## How the chart spreads and drains pods

The chart sets the following for the gateway pods. You don't need to change them for a typical install.

* **Spread.** Pods spread across zones and across nodes where possible. The spread is a preference, so pods still schedule when the cluster can't satisfy it.
* **PodDisruptionBudget.** On by default with `minAvailable: 1`, so a node drain or upgrade always leaves at least one gateway pod serving.
* **Graceful shutdown.** A stopping pod waits 60 seconds before it shuts down, so your load balancer can stop sending it traffic, within a 100-second termination grace period.

To change the gateway's budget, set `podDisruptionBudget.minAvailable`. For the gateway, `minAvailable` takes precedence over `maxUnavailable`, so to use `maxUnavailable` you must also remove `minAvailable`.

## Add autoscaling

You can hand the gateway replica count to one autoscaler:

* **HorizontalPodAutoscaler.** Set `autoscaling.enabled: true`. It needs metrics-server, and scales on CPU (target 60%) and memory (target 75%) by default. Change the targets with `autoscaling.targetCPUUtilizationPercentage` and `autoscaling.targetMemoryUtilizationPercentage`.
* **KEDA.** Set `keda.enabled: true`. It needs the KEDA operator, and scales on CPU (target 60%) and memory (target 75%). These targets are fixed: the `autoscaling.*` target values don't change them.

Enable only one. The chart refuses to render if both are on. Both render only when `service.enabled` is `true`. KEDA polls every 20 seconds and scales down slowly: it waits 300 seconds, then removes at most 10% of pods every 120 seconds. To add triggers, set `scalingConfig.triggers`. They authenticate through a `ClusterTriggerAuthentication` named `keda-cluster-trigger-authentication`, which you create.

Both autoscalers use `scalingConfig.scalingProperties.minReplicas` and `maxReplicas`, which both default to `2`. Raise `maxReplicas`, or the autoscaler can't add pods:

```yaml
autoscaling:
  enabled: true
scalingConfig:
  scalingProperties:
    minReplicas: 2
    maxReplicas: 6
```

With an autoscaler on, keep `replicaCount` equal to `scalingConfig.scalingProperties.minReplicas`, because each `helm upgrade` resets the Deployment to `replicaCount`. Autoscaling applies to the default Deployment. With the distributed store, the gateway runs as a StatefulSet, and you set its size with `distributed.replicaCount` instead.

### Share state across replicas

Some state is held in each pod's memory:

* **Rate-limit counters.** Each pod counts on its own by default, so a limit applies per pod. To enforce one limit across the data plane, use a coordination Redis: set `FABRIC_GW_COORD_REDIS_ENABLED` to `"true"`, `FABRIC_GW_COORD_REDIS_ADDRESS` to your Redis, and `FABRIC_GW_RATE_LIMIT_COUNTER_STORE` to `redis`. Add `FABRIC_GW_COORD_REDIS_TLS` for a TLS endpoint, and the password in the app Secret. `FABRIC_GW_RATE_LIMIT_ON_ERROR` (`local` or `fail_open`) sets what happens if Redis fails. See [Helm chart values](/run/helm-values).
* **MCP notification streams.** A client's notification stream stays on the pod that received it. When a user connects an upstream account on another pod, that pod tells the others through the coordination Redis. Without Redis, a client whose stream is on another pod doesn't hear that its tool list changed.

## Scale the embedder

The embedder runs one replica by default, with its own PodDisruptionBudget. With a single replica, that budget blocks node drains. For scaling, the model cache, and the drain fix, see [Run the tool-search embedder](/run/embedder).

## Choose a secret store

The data plane stores the credentials users connect, client registrations, and short-lived login state in a secret store. All replicas share it.

|                   | MySQL (chart default)             | Distributed store                                                                  |
| ----------------- | --------------------------------- | ---------------------------------------------------------------------------------- |
| Where data lives  | Your MySQL database               | In memory across the gateway pods, with an encrypted snapshot on each pod's volume |
| What you run      | A MySQL instance, with backups    | Nothing extra, but a StorageClass and a NetworkPolicy controller                   |
| Workload type     | Deployment                        | StatefulSet                                                                        |
| Schema migrations | A Job on each install and upgrade | None                                                                               |
| Autoscaling       | HorizontalPodAutoscaler or KEDA   | None. Set `distributed.replicaCount`.                                              |

### Run the distributed store

Set `distributed.enabled: true`. The chart then does the following:

* Runs the gateway as a StatefulSet (three replicas by default, set by `distributed.replicaCount`), with its own volume per pod at the cache path (`distributed.persistence.size`, `1Gi` by default). Because the volume survives restarts, each pod also keeps its certificate and configuration cache.
* Adds a headless Service so pods find each other by DNS.
* Opens ports `3320` (data) and `3322` (gossip), and adds a NetworkPolicy that allows only gateway pods to reach them.
* Skips the MySQL migration Job. This mode needs no MySQL.

Add `FABRIC_GW_DISTRIBUTED_GOSSIP_KEY` and `FABRIC_GW_DISTRIBUTED_AUTH_PASSWORD` to the app Secret. In production mode, a pod without them refuses to start. See [Manage data plane secrets and keys](/run/secrets-and-keys).

> **Warning**
>
> Traffic between pods on ports `3320` and `3322` isn't encrypted in transit. Stored values are encrypted, but anyone who can reach the data port can list, overwrite, or delete them. Keep `distributed.networkPolicy.enabled` on, and use an encrypted overlay or mesh if pods talk across a network you don't trust.

Plan for the following:

* **Writes need a majority.** With three replicas, a credential write needs two pods. A single surviving pod can't save new credentials.
* **Snapshots run every 30 seconds** (`distributed.snapshotInterval`). If every pod stops at once without shutting down cleanly, credential changes made since the last snapshot can be lost.
* **A full restart restores from the newest snapshot.** While a cold data plane restores, credential operations wait, but `/healthz` still answers and requests that need no stored credential are served.
* **Clocks must agree.** The store uses wall-clock time for expiry and conflict resolution. Run NTP on every node.
* **Keep `distributed.memberCountQuorum` at `1`.** A higher value stops a lone pod from starting.

If these trade-offs don't suit you, use MySQL.

## When the control plane is unreachable

The data plane serves traffic on its own. The control plane isn't in the request path, so an outage affects configuration and identity, not requests:

* **Running pods keep serving** the last configuration they applied, for as long as the outage lasts. Configuration and policy changes you make in the console reach them when the connection returns.
* **A pod that restarts with its cache and certificate intact** starts and serves its cached configuration without contacting the control plane. With the chart's defaults, this holds for a container restart inside the same pod. With the distributed store, it also holds for a replaced pod, because its volume survives.
* **A new pod can't enroll**, so it exits and Kubernetes restarts it until the control plane is back. This includes pods added by an autoscaler and replacement pods without a persistent volume. Avoid rollouts and scale-ups during an outage.
* **Certificate renewal retries** with back-off. As long as the control plane returns before a pod's certificate expires, the pod renews without help.
* **The console** shows the affected nodes as **Disconnected**.

## Next steps

* To set the values named on this page, see [Helm chart values](/run/helm-values).
* To watch replicas and the store, see [Monitor the data plane](/run/monitor).
* Before you go live, see the [Production checklist](/run/production-checklist).