Run the tool-search embedder
This page applies to the Enterprise plan, where you run the data plane in your own VPC. The data plane chart deploys an embedder next to the gateway: a small embedding service that turns MCP tool descriptions and search queries into vectors for the Tool Search plugin. This page explains when you need it, and how to keep its model cache, replicas, node drains, and network access the way you want them.
What the embedder is
The embedder runs HuggingFace text-embeddings-inference (TEI) as its own Deployment and Service, both named fabric-gateway-embedder. By default it serves the BAAI/bge-small-en-v1.5 model, which returns 384-dimension vectors, on a CPU image.
The Service name is fixed. It doesn’t change with your release or chart name, because the gateway and the collector sidecar find the embedder by that name.
When you need it
You need the embedder only if a route uses the Tool Search plugin with the embedder as its embedding service. Tool search needs an OpenAI-compatible embedding service, and the embedder is one that runs inside your cluster, so tool descriptions and queries don’t leave it. If you don’t use tool search, the embedder sits idle.
To use it, set the Tool Search plugin’s embedder to the embedder Service. For example, in the namespace pm-fabric-dp:
Keep /v1 at the end of base_url. The gateway sends embedding requests to <base_url>/embeddings. The embedder has a private address inside your cluster, which the gateway reaches by default. If you turn on FABRIC_GW_SSRF_BLOCK_PRIVATE_UPSTREAMS, add the embedder’s host to FABRIC_GW_SSRF_ALLOW_PRIVATE_UPSTREAM_HOSTS. See Calls to private hosts.
To point every Tool Search instance on the data plane at the same embedder, set FABRIC_GW_TOOL_SEARCH_EMBEDDER_BASE_URL under env.vars instead. It replaces the base_url in each instance’s configuration. The chart doesn’t set it.
If the embedder is down or too slow, a tool search returns an error rather than an empty result. See Troubleshoot the data plane.
Keep the model cache across restarts
By default, the model cache is an emptyDir, so every embedder pod downloads the model from huggingface.co each time it starts. To keep the model on a PersistentVolumeClaim (PVC) instead, so restarts are fast and need no egress once the cache is warm:
The chart creates a PVC named fabric-gateway-embedder-model-cache and mounts it at /data. Set embedder.persistence.storageClass to choose a StorageClass, or embedder.persistence.existingClaim to use a PVC you already own. The volume is 5Gi by default (embedder.persistence.size), and embedder.persistence.annotations adds annotations to the PVC.
The chart-created PVC is ReadWriteOnce by default, which works only with a single embedder replica. The chart refuses to render if you combine it with embedder.replicaCount above 1, or with autoscaling and embedder.autoscaling.maxReplicas above 1. For more than one replica, do one of the following:
-
Use
ReadWriteManystorage, such as EFS on EKS: -
Set
embedder.persistence.existingClaimto a PVC that supports your topology. The chart doesn’t check an existing claim’s access mode.
Run without internet access
If your cluster can’t reach huggingface.co, run the embedder offline. embedder.offline.enabled: true stops the embedder from contacting huggingface.co, but the model must already be available locally, or the pod fails to start. Warm the cache first, then go offline:
-
Turn on the persistent cache, with offline mode still off:
-
Confirm that the model loaded. Look for a
Readyline: -
Turn on offline mode, keeping persistence on:
Instead of warming the cache, you can also bake the model into an image you mirror and set embedder.model.id to its local directory, or pre-load the cache volume with an init container in embedder.initContainers. Mirror the embedder image too, and set embedder.image.imagePullSecrets if your registry needs credentials.
Scale the embedder
Scale the embedder with a static count or with its own HorizontalPodAutoscaler, which needs metrics-server:
The autoscaler targets 60% CPU and 75% memory by default (embedder.autoscaling.targetCPUUtilizationPercentage and targetMemoryUtilizationPercentage), with 1 to 3 replicas. While it’s on, the chart ignores embedder.replicaCount.
If the embedder can ever run more than one replica and you use a persistent cache, the cache needs ReadWriteMany storage or an existing claim. See Keep the model cache across restarts.
To tune throughput, set embedder.args.maxConcurrentRequests (default 64) and embedder.args.maxBatchTokens (default 16384). Any other TEI flag goes in embedder.extraArgs. Leave embedder.args.dtype, pooling, and autoTruncate empty to let the embedder detect them from the model. Set embedder.extraEnv for variables such as HF_TOKEN.
Keep node drains working
The embedder has its own PodDisruptionBudget, on by default with minAvailable: 1. With the default single replica, that budget blocks eviction of the only embedder pod, so kubectl drain on its node waits forever. Do one of the following:
-
Keep one replica and allow it to be evicted:
For the embedder,
maxUnavailabletakes precedence overminAvailable, so you don’t need to remove the default. Tool search is unavailable while the pod moves. -
Run two or more replicas, statically or with autoscaling, so the budget always has a second pod.
Restrict network access
On a cluster with a NetworkPolicy controller, set embedder.networkPolicy.enabled: true to let only the gateway pods reach the embedder. That includes the collector sidecar’s metrics scrape. To allow other clients, add them to embedder.networkPolicy.extraIngress.
Security defaults
The embedder runs with the same posture as the gateway: as a non-root user (UID 1000), with the RuntimeDefault seccomp profile, all Linux capabilities dropped, no privilege escalation, and a read-only root filesystem. Its only writable paths are the model cache at /data and /tmp. If you supply a custom embedder.image that writes anywhere else, mount that path as a volume or relax embedder.securityContext.readOnlyRootFilesystem.
Next steps
- To configure tool search on a route, see Tool Search.
- To watch the embedder’s metrics, see Monitor the data plane.
- To fix embedder problems, see Troubleshoot the data plane.

