Skip to navigation

Maintain the data plane and its data

This page applies to the Enterprise plan, where you run the data plane in your own VPC. The gateway and embedder pods keep no durable state of their own. Everything that has to survive a restart, an upgrade, or a reinstall lives outside the Helm release, in systems you run. This page lists that state, what to back up, how long stored data lasts, and the routine tasks to put on your calendar.

What state the data plane keeps

StateWhere it livesSurvives pod replacementWho backs it up
The credentials users connect, client registrations, and short-lived login stateYour MySQL database, or the distributed store’s per-pod snapshotsYesYou
The encryption key, FABRIC_GW_SECRET_KEY, that protects stored credentialsThe app SecretYesYou
Traces, metrics, and logsYour telemetry backend, such as ClickHouseYesYou
Each pod’s certificate, private key, and configuration cache/data/cache in the pod. An emptyDir by default, or a per-pod volume with the distributed store.Only with the distributed storeNobody. A pod without them enrolls again.
The embedder’s modelAn emptyDir, or a PVC with embedder.persistence.enabledOnly with a PVCNobody. The model downloads again.
Routes, services, plugins, and the rest of your configurationThe control plane, run by PostmanYesPostman

Your configuration isn’t stored in the data plane. Each pod receives it from the control plane and caches it locally, so a data plane you rebuild from scratch gets the same configuration back when it enrolls.

Back up the secret store

Backups of the secret store are your responsibility. The chart doesn’t back anything up, and doesn’t rely on Helm’s release history to restore anything.

  • With MySQL, back up the data plane’s database on your usual schedule.
  • With the distributed store, each pod’s snapshot volume holds an encrypted copy of the store. Back up those volumes with your volume snapshot tooling.

Back up FABRIC_GW_SECRET_KEY with the same care, and keep it with your database backups. Stored credentials are encrypted with this key, so a database restored without its key is unreadable. Every replica must use the same key. To change it safely, see Manage data plane secrets and keys.

If you lose the secret store without a backup, users have to connect their accounts again, and MCP clients have to register again.

How long stored data lasts

With the MySQL store, each gateway pod removes expired rows every 60 seconds:

DataRemoved after
Login state and one-time codes from OAuth flows10 minutes
MCP client registrations90 days without use. Each use resets the clock, so registrations in use aren’t removed.
Credentials users connectNever. They stay until the user or an admin removes them.

Telemetry retention is set by your backend. The chart’s collector writes to ClickHouse but doesn’t create its tables, so retention follows the tables you create.

Schema migrations

With MySQL, every install and upgrade runs a schema migration Job before the gateway pods change, and the Job applies only migrations the database hasn’t recorded yet. Day-to-day operation needs nothing else. Two rules keep it working:

  • Never edit a migration that has already been applied. The next upgrade stops with a checksum mismatch.
  • Keep migrations.baselineVersion at "0" if you adopt a database that already has the schema.

To run migrations from your own pipeline instead, set migrations.enabled: false. For least privilege, give the Job its own DDL-capable user with migrations.dbUser and migrations.existingSecret.

Because the release holds no state, you can uninstall and reinstall the chart against the same MySQL database. The migration Job adopts the existing schema and applies only what’s missing. For the full model, see Database migrations.

Routine maintenance

  • Keep a way to enroll new pods, if you enroll with a bootstrap token. You mint the token in the console, and it lasts 10 minutes by default and 30 minutes at most, so pods created after it expires can’t enroll. Before you add or replace pods, mint a fresh token and put it in the app Secret. Workload attestation needs no token. See Connect the data plane to the control plane.

  • Rotate other secrets in the app Secret, or in the secret manager that syncs it. The gateway reads new values when its pods restart:

    kubectl -n pm-fabric-dp rollout restart deploy/fabric-gateway-chart

    With the distributed store, the gateway runs as a StatefulSet, so use statefulset/fabric-gateway-chart instead.

  • Update the image on every planned upgrade, by setting image.tag or pinning image.digest. See Upgrade the data plane.

  • Watch telemetry export. An increase in otelcol_exporter_send_failed_* means telemetry isn’t reaching your backend, before it turns into a gap in your data. See Monitor the data plane.

  • Back up the secret store and its key, and test a restore.

Certificate renewal needs no maintenance. Each pod renews its own certificate before it expires.

Next steps