Skip to navigation

Upgrade the data plane

This page applies to the Enterprise plan, where you run the data plane in your own VPC. You upgrade the data plane with helm upgrade, either to a new chart version or to a new gateway image. Before any pod changes, Helm applies new database schema migrations, and a failed migration stops the upgrade. This page shows how to upgrade safely, what happens during an upgrade, how to check the result, and how to roll back.

Prerequisites

Before you begin, make sure you have the following:

  • The new chart version or image tag from Postman, and the image mirrored into your registry.
  • The values file you installed with, from version control.
  • With the MySQL store, a recent database backup.
  • A working enrollment credential, if your pods don’t keep their certificates: workload attestation, or a bootstrap token that hasn’t expired. See Connect a data plane to the control plane.

Check compatibility

Postman runs the control plane and upgrades it independently of your data plane. Before you upgrade, check the release notes Postman gives you for the supported data plane versions and any required order of upgrades.

Then read the chart release notes for every version between yours and the new one. Each version lists the changes that need action, such as chart 0.6.0 turning on production mode by default.

What happens during an upgrade

  1. With the MySQL store, Helm runs the schema migration Job. It applies only migrations the database hasn’t recorded yet, so an upgrade with no new migrations changes nothing in the schema. If the Job fails, the upgrade stops and no gateway pod changes.
  2. Helm applies the rest of the release. The gateway does a rolling update, and its PodDisruptionBudget keeps at least one pod serving. The embedder rolls too, if its settings changed.
  3. Each stopping gateway pod waits 60 seconds before it shuts down, so in-flight traffic can drain.
  4. New gateway pods enroll with the control plane, receive their configuration, and start serving.

With the chart’s default storage, new pods start without a certificate, so each one enrolls again. With the distributed store, the gateway is a StatefulSet that replaces pods one at a time, and each pod keeps its volume, so it keeps its certificate.

Upgrade

  1. Preview what changes:

    helm template fabric-dp ./fabric-gateway-chart -n pm-fabric-dp -f my-values.yaml > new.yaml
    helm get manifest fabric-dp -n pm-fabric-dp > current.yaml
    diff current.yaml new.yaml
  2. Run the upgrade with your values file:

    helm upgrade fabric-dp ./fabric-gateway-chart -n pm-fabric-dp -f my-values.yaml

    To change only the image, set the new tag in your values file, or pin a digest:

    helm upgrade fabric-dp ./fabric-gateway-chart -n pm-fabric-dp -f my-values.yaml \
    --set image.digest=sha256:<digest>

Always pass your values file (-f my-values.yaml) or --reuse-values. A bare helm upgrade resets every value to the chart’s default, including the image, tenant ID, and control plane address. If you used the chart-managed development Secret, a bare upgrade re-creates it from the chart’s default values, which are empty, so it loses your keys. Use an app Secret you create yourself, so a values mistake can’t affect it.

Verify the upgrade

  1. Check that the migration Job is gone, which means it succeeded:

    kubectl -n pm-fabric-dp get jobs -l app.kubernetes.io/name=fabric-gateway-chart
  2. Check that the rollout finished:

    kubectl -n pm-fabric-dp rollout status deploy/fabric-gateway-chart

    With the distributed store, use statefulset/fabric-gateway-chart.

  3. In the console, select Admin > Gateway health, and check that every node shows the new Version and the status Healthy.

For the full set of health checks, see Monitor the data plane.

Roll back

To go back to the previous release, run:

helm rollback fabric-dp -n pm-fabric-dp

The migration Job runs on installs and upgrades only, so a rollback doesn’t run it, and doesn’t undo migrations that already ran. The previous gateway version runs against the newer schema.

Troubleshoot an upgrade

SymptomLikely causeFix
Upgrade stops at the migration Job with Migration checksum mismatchA migration file that was already applied was editedRestore the chart’s original SQL files. Never edit an applied migration.
Migration Job fails with Access denied or CREATE command deniedThe migration user lacks grantsGrant CREATE, ALTER, INDEX, SELECT, INSERT, UPDATE, and DELETE, or set migrations.dbUser to a user that has them
Migration Job times outMySQL is unreachable from the clusterFix FABRIC_GW_MYSQL_HOST or the network path
New pods restart in a loop after the upgradeThe bootstrap token expired, so new pods can’t enrollMint a new token, update the app Secret, and restart
Pods won’t start after an upgrade to chart 0.6.0Production mode is now on, and FABRIC_GW_SECRET_KEY is missingAdd the key to the app Secret.
Values are back to defaultsThe upgrade ran without -f or --reuse-valuesRun the upgrade again with your values file

For more, see Troubleshoot the data plane.

Next steps