MOKS, my SRE investigation portal, searches its own knowledge base (runbooks, design docs, skills) for almost every question it answers. On mk4, my three-node k3s cluster on Hetzner, that search ran on a bundled OpenSearch pod. On 19 November 2026 I replaced it with PostgreSQL and pgvector, embedded by BAAI bge-m3 running on Ollama inside the cluster.
The same bge-m3 pod also serves seo-agent, the service behind this site's Ask the Blog and news desk. Changing the model there turned out to be its own project, so it has its own article (opens in a new tab). This one covers the shared model service and the MOKS side.
The problem: a 2.5 GiB search engine doing keyword search
mk4 runs on three Hetzner cax21 servers: 4 arm64 vCPUs and 8 GiB of memory each. Before this change, memory requests on the three nodes stood at 82%, 89% and 65%. MOKS's search stack accounted for a good share of that:
- the
moks-opensearchStatefulSet, requesting 2 GiB, with a 20 Gi local-path volume (reclaim policyRetain) on worker2; - OpenSearch Dashboards, requesting another 512 Mi;
- two public routes,
/opensearchand/opensearch-dashboards, behind basic auth, plus alerts and a Grafana row for them.
When I looked at what that pod was actually indexing, it was worse than I expected. MOKS keeps a history of its index generations in PostgreSQL, and mk4 had exactly one, created on 10 November:
backend: opensearch model_key: keyword-only dimensions: None status: activeNo embedding model had ever been active on mk4: the Helm values seeded no LLM providers. So 2.5 GiB of memory requests were serving BM25 over 730 chunks, which PostgreSQL already does with tsvector. And the half of hybrid search that copes with paraphrase (the question says "pod keeps restarting", the runbook says "CrashLoopBackOff") wasn't there at all.
Choosing bge-m3, and why Ollama
I wanted one embedding model that both MOKS and seo-agent could share, so one copy of the weights in memory serves the whole cluster. bge-m3 fit:
- 1024-dimension dense vectors, L2-normalised, with no query or passage prefix to get wrong.
- Multilingual and trained on long inputs. The model card describes inputs of up to 8,192 tokens. I cap input at 2,048 tokens because both callers chunk to about 350 to 480 tokens.
- Much larger than what seo-agent used before: bge-small-en-v1.5, English only, 384 dimensions. Whether that buys better retrieval on my own content is something to measure, not assume; the companion article covers that.
The obvious server for it is Hugging Face's text-embeddings-inference (TEI). But TEI publishes CPU images for amd64 only, and every mk4 node is arm64. Ollama has arm64 images and bge-m3 in its library (bge-m3:567m, the F16 build, 1.16 GB), so that decided it.
The trade-off: Ollama serves only bge-m3's dense vectors, not its sparse or multi-vector outputs. That's acceptable here, because both callers already do keyword search in PostgreSQL. The model only has to supply the semantic half.
Deploying the model with Terraform
The service lives in the kubernetes repository: a Terraform file and a small local Helm chart. The Terraform is short:
# minikube-infrastructure/embeddings.tf (excerpt)
resource "kubernetes_namespace_v1" "embeddings" {
metadata {
name = "embeddings"
labels = {
"app.kubernetes.io/part-of" = "embeddings"
"istio-injection" = "enabled"
}
}
depends_on = [helm_release.istiod]
}
resource "helm_release" "bge_m3" {
name = "bge-m3"
namespace = kubernetes_namespace_v1.embeddings.metadata[0].name
chart = "${local.minikube}/embeddings/bge-m3"
values = [local.chart_checksum["embeddings/bge-m3"]]
wait = true
# The first start pulls a 2.9 GB image, attaches a new Hetzner volume and downloads 1.16 GB of weights.
timeout = max(var.helm_timeout, 1200)
}The interesting parts are in the chart.
Pin the weights, not just the image
The Ollama image is pinned by its multi-arch index digest, as usual. But a model tag such as bge-m3:567m can be re-pushed. If the weights change, every stored vector quietly becomes incomparable with new ones. So the chart pins the weights' digest too, and an init container refuses anything else:
# deploy/minikube/embeddings/bge-m3/templates/deployment.yaml, init container fetch-model (excerpt)
if [ -f "$manifest" ] && [ -f "$blob" ] && grep -q "$MODEL_DIGEST" "$manifest"; then
echo "$MODEL ($MODEL_DIGEST) is already on the volume"
exit 0
fi
# ...start a temporary ollama serve, then:
ollama pull "$MODEL"
if ! grep -q "$MODEL_DIGEST" "$manifest"; then
echo "$MODEL: the downloaded weights are not $MODEL_DIGEST (was the tag re-pushed?); refusing to serve them" >&2
exit 1
fiThe weights go onto a 10 Gi Hetzner volume (about €0.50 a month) rather than node-local storage, so the pod can start on any node without downloading again. The claim carries helm.sh/resource-policy: keep, and the Deployment uses Recreate because the volume is ReadWriteOnce. The first start pulled about 2.9 GB of image and 1.16 GB of weights. A restart pulls nothing.
Keep it loaded, and don't set a CPU limit
# values.yaml (excerpt)
ollama:
contextLength: 2048
numParallel: 2
resources:
requests:
cpu: 500m
memory: 1536Mi
limits:
memory: 2560MiThe container also sets OLLAMA_KEEP_ALIVE=-1 and OLLAMA_MAX_LOADED_MODELS=1. Without keep-alive, the first request after an idle period pays for loading 1.16 GB.
There is no CPU limit, on purpose. Ollama's runner sizes its thread pool to the node's cores. A CFS quota smaller than that thread count stalls it on every batch. The 500m request decides its share when the node is busy.
Only two callers
Embeddings of private runbooks are worth protecting, and an open model endpoint is free compute for anything in the cluster. So the chart lists its callers, and two independent policies enforce the list:
# values.yaml
callers:
- namespace: jithendar # seo-agent
serviceAccount: default
podLabels:
app: seo-agent
- namespace: moks # moks-api, moks-worker, moks-crewai
serviceAccount: moks
podLabels:
app: moksThe NetworkPolicy admits those namespaces and labels. The Istio AuthorizationPolicy, enforced rather than dry-run, admits only their mTLS identities (cluster.local/ns/moks/sa/moks and so on) on port 11434. That's why MOKS's command executor and web terminal can't reach the model, even though they run in the moks namespace: they use other service accounts. The pod itself runs as non-root, with all capabilities dropped and no service account token mounted.
A new caller needs three things: an entry in callers, an egress rule to embeddings:11434 in its own NetworkPolicy, and a sidecar.
What it measured
On mk4, with the real model:
| Measured | |
|---|---|
| Output | 1024 dimensions, L2-normalised |
| Cold load | about 5 s |
| One warm query | about 650 ms |
| Throughput | about 0.7 chunks/s for chunks of about 325 tokens |
| Batch of 32 chunks | about 45 s |
| Memory | about 1.3 GiB resident |
| CPU while embedding | about 3.5 of 4 cores |
One surprise: I set OLLAMA_NUM_PARALLEL=2, and the server's log shows it, yet the runner starts with -np 1 (n_seq_max = 1). Ollama embeds one request at a time, whatever that setting says. Everything that calls the model has to plan for a single queue.
That number shapes the rest of this post. 0.7 chunks a second is fine for answering a question (one short embedding) and slow for indexing.
Pointing MOKS at the model
MOKS keeps its LLM and embedding providers in its own database, and the Helm chart can seed them. The mk4 values now seed one embedding config:
# deploy/helm/moks/values-hetzner.yaml (excerpt)
seed:
llm:
- name: bge-m3
purpose: embedding
provider: ollama
model: bge-m3:567m # the chart's model.name; a new tag or digest means a reindex
endpoint: http://bge-m3.embeddings:11434
embedding_dimensions: 1024
timeout_seconds: 300
max_concurrency: 1
is_active: trueThe seed is create-only. It activates the entry only when no embedding model is active yet, so a choice made later in the portal is never overwritten by a pod restart. MOKS's existing ollama provider already had an embed() that calls /api/embed, so no new code was needed.
The two limits come straight from the measurements:
timeout_seconds: 300. MOKS indexes in batches of 32, which take about 45 s. A timeout tuned for chat calls would cut them off.max_concurrency: 1. MOKS enforces this with reservation rows in PostgreSQL, taken under an advisory lock, so the limit holds across both API replicas, the worker and the CrewAI pod. A search asks for the slot with a short wait:
# backend/moks/rag/service.py, _query_vector (excerpt)
try:
return (await budget.embed(provider_for(cfg), cfg, [query], user_id=uuid.UUID(principal.user_id),
purpose="search", wait_seconds=2))[0], None
except (LLMError, ConnectorError) as exc:
log.warning("vector search unavailable", extra={"event": "vector_search_failed", "errorType": type(exc).__name__})
return None, f"embedding failed ({getattr(exc, 'kind', 'error')}), so this is keyword only"If an index batch holds the slot, a question waits two seconds and then answers keyword-only, with the reason attached, instead of queueing for up to 45 s. The same function returns a reason for every other way vectors can be missing: no index yet, no active model, an index built by another model, no pgvector. Every search response reports which mode actually ran, and why.
The last piece is an egress rule in MOKS's NetworkPolicy: namespace embeddings, port 11434.
From OpenSearch to PostgreSQL and pgvector
My first plan was a new LanceDB backend for MOKS, storing vectors in Cloudflare R2. I dropped it in favour of something MOKS already had: its PostgresBackend.
- PostgreSQL is already the source of truth for MOKS's documents, chunk text and access rules. It already runs two of the three rankers (keyword and exact identifiers).
- The shared CloudNativePG image already contains pgvector 0.8.6.
- At 730 chunks, an exact k-NN scan is cheap. There's no approximate index to tune.
- One store instead of two means no bucket, no credentials and no second copy to keep in sync.
Installing an untrusted extension for a tenant
There was one obstacle. pgvector is not a trusted extension, so the moks database role can't CREATE EXTENSION vector itself. MOKS's migration 0002 tries it, and on the shared cluster it had quietly skipped the vector column. The fix belongs to the cluster, not the application. The kubernetes repository's Terraform now lets each tenant database list extensions, and the CloudNativePG operator creates them as the superuser:
# minikube-infrastructure/vars/mk4.tfvars (excerpt)
postgres_databases = {
jithendar = { namespace = "jithendar" }
moks = { namespace = "moks", extensions = ["vector"] } # pgvector for its knowledge search (rag.backend postgres)
}Then MOKS adds the column itself, the next time it builds an index:
# backend/moks/rag/backends.py, PostgresBackend
async def enable_vector(self) -> bool:
if await self.has_vector():
return True
if (await self.db.execute(text("SELECT 1 FROM pg_extension WHERE extname = 'vector'"))).first() is None:
return False
await self.db.execute(text("ALTER TABLE chunk_embeddings ADD COLUMN IF NOT EXISTS embedding vector"))
return TrueWithout the extension, index builds stay keyword-only as before, and moks reindex explains how to install it.
Three rankings, one fusion
For each question the PostgresBackend produces three ranked lists of chunk ids:
- Keyword:
websearch_to_tsqueryranked byts_rank_cd. If nothing matches, it retries with OR semantics so partial matches still rank. - Identifiers: error codes, resource names, paths and config keys pulled out of the question with a regular expression and matched exactly with
ILIKE. Embeddings are poor atERR_PAY_4071. An engineer pasting it is giving the best clue they have. - Vector:
ORDER BY e.embedding <=> CAST(:v AS vector), cosine distance against the active index generation.
MOKS fuses the three with Reciprocal Rank Fusion, which needs no score calibration between rankers:
# backend/moks/rag/backends.py
RRF_K = 60
def rrf(*rankings: list[str]) -> list[tuple[str, float]]:
scores: dict[str, float] = {}
for ranking in rankings:
for rank, cid in enumerate(ranking):
scores[cid] = scores.get(cid, 0.0) + 1.0 / (RRF_K + rank + 1)
return sorted(scores.items(), key=lambda kv: kv[1], reverse=True)Only after fusion does MOKS load the chunks and check access on each one. Whatever a ranker returned is a suggestion, and PostgreSQL's access check decides. Reranking, when it's on, comes after that (the reranker article (opens in a new tab) covers it).
The column is a plain vector with no fixed size, and there's no HNSW index. An HNSW index needs a fixed dimension, and at 730 chunks a sequential scan is quick. If the knowledge base grows by an order of magnitude, that's the first thing to revisit.
Rolling it out without a gap
Four pull requests across two repositories, merged on the same day:
- kubernetes#10 deployed bge-m3. Nothing called it yet.
- moks-sre#55 seeded the bge-m3 config and the egress rule. MOKS was still on OpenSearch.
- kubernetes#12 installed pgvector for
moksand removed the public/opensearchroutes, their logins, the three OpenSearch alerts and the Harbor mirror of the image. - moks-sre#67 set
rag.backend: postgres, removed the OpenSearch settings from the mk4 values, and moved themoks-opensearchArgo CD application tooptional/. The OpenSearch backend stays in the product for other environments.
The last two were designed to merge in either order. If the cluster change landed first, MOKS kept using OpenSearch over the in-cluster Service, and only the public URLs stopped answering. If the MOKS change landed first, search ran keyword-only in PostgreSQL until the extension appeared, and the next index build added the vector column. Neither order breaks search.
What neither PR does is delete OpenSearch. Argo CD never prunes on mk4, so moving the application out of the sync left the StatefulSet, Dashboards, Services, NetworkPolicies, VirtualServices and the volume running. That's deliberate: deleting a data volume should be a decision someone makes, not a side effect of a merge. MOKS's operator guide has the runbook:
kubectl -n argocd delete application moks-opensearch
kubectl -n moks delete statefulset/moks-opensearch deployment/moks-opensearch-dashboards \
service/moks-opensearch service/moks-opensearch-dashboards \
networkpolicy/moks-opensearch networkpolicy/moks-opensearch-dashboards \
virtualservice/moks-opensearch virtualservice/moks-opensearch-dashboards
kubectl -n moks delete pvc data-moks-opensearch-0
kubectl get pv | grep data-moks-opensearch-0 # reclaim policy Retain: delete the PV by nameA Retain local-path volume also leaves its directory on the node, so the last step is removing it from worker2 to get the disk back.
Before that, Rebuild index on the Knowledge page (or moks reindex) builds the first generation with vectors. At about 0.7 chunks a second, 730 chunks should take about 17 minutes. Search stays on keyword until the new generation is complete, then switches.
What it costs, and what I gave up
Memory requests go down by about 1 GiB. OpenSearch's 2 GiB and Dashboards' 512 Mi go, and bge-m3's 1.5 GiB comes in, and that 1.5 GiB also serves seo-agent. Disk swaps a 20 Gi local-path volume on one node for a 10 Gi Hetzner volume that can follow the pod.
Indexing is slow. 0.7 chunks a second is the price of a 567M-parameter model on four ARM cores. MOKS's 730 chunks take about a quarter of an hour, and seo-agent's 2,117 take about 50 minutes. Both are rare, deliberate operations, and search keeps working on keyword while they run.
Both callers share one queue. MOKS's two-second slot wait only covers MOKS's own calls. While seo-agent is reindexing, a MOKS question's embedding waits in Ollama's queue behind one of seo-agent's batches of 8. By the throughput above, that's up to roughly 11 seconds. That's an estimate, not a measurement, and the first thing I'll watch once both callers are live.
One replica. If the bge-m3 pod is down or restarting, MOKS answers keyword-only with a reason, and seo-agent's Ask the Blog falls back. Nothing fails hard, but nothing is semantic either. A second replica would double the memory, which isn't worth it for this cluster.
Dense only. bge-m3 can produce sparse lexical weights too, but Ollama doesn't serve them. PostgreSQL's full-text search fills that role.
What's next
The rebuild and the clean-up come first. After that, the same model gets its second caller: seo-agent, where swapping bge-small for bge-m3 meant recalibrating five similarity cutoffs and migrating a news desk's stored vectors in place. That's the companion article (opens in a new tab).



Discussion comments
No comments yet. Start the discussion.