MOKS is the SRE investigation portal I build and run on mk4, my three-node k3s cluster on Hetzner. Its code lives in one public repository, mrjithendar/moks-sre (opens in a new tab), and one GitHub Actions workflow, .github/workflows/ci.yml (opens in a new tab), tests every change and ships five container images.
I first wrote about this pipeline from a screenshot of the Actions page. That version guessed at what each job did and how they depended on each other. This one is written from the workflow file and from the GitHub Actions API: every job, every needs, and the timings and failures of real runs. Some of what I found surprised me, including two gaps that let untested code reach the registry.
What the pipeline ships
MOKS is not one image. The workflow builds five, all for arm64 because every mk4 node is a Hetzner cax21 (4 arm64 vCPUs, 8 GiB):
| Image | Built by | What it is |
|---|---|---|
moks | release | The product: FastAPI backend and React UI in one image. The API, worker, executor and terminal-service Deployments all run it. |
moks-crewai | release-crewai | The same Dockerfile with MOKS_CREWAI=true, which adds a pinned CrewAI. Only the CrewAI worker runs it. |
moks-reranker | reranker-image | A small model service around cross-encoder/ms-marco-MiniLM-L6-v2, revision pinned in reranker/model.lock.json, weights baked in. |
moks-embedder | embedder-image | The same service code built around BAAI/bge-base-en-v1.5 (reranker/embedder.lock.json). Added in #85 (opens in a new tab). |
moks-terminal | terminal-image | The diagnostic shell for terminal sessions (aws, kubectl, curl, jq, dig, tmux), for arm64 and amd64. Each new session pulls it. |
Two names in the first draft of this article are gone. There is no OpenSearch image any more: MOKS has searched its knowledge base in PostgreSQL with pgvector since #67 (opens in a new tab) (19 November), and #91 (opens in a new tab) deleted the OpenSearch backend and the opensearch-image job the next day. The bge-m3 article (opens in a new tab) covers that move. And there's no separate "backend" image: the backend and the UI ship together as moks.
Every image gets exactly one tag, latest. The commit is baked inside it instead (--build-arg GIT_COMMIT), and /moks/api/health reports it. More on that choice in the release section.
Triggers, permissions and concurrency
The top of the file sets three things that shape everything below it (lines 23 to 34 (opens in a new tab)):
on:
workflow_dispatch:
push:
branches: [master]
pull_request:
permissions:
contents: read
concurrency:
group: moks-${{ github.ref }}
cancel-in-progress: false- Three triggers, three different runs. A pull request runs the six test jobs. A push to
masterruns those plus the six that build and deploy. A manual run looks like a push but isn't one: every build job checksgithub.event_name == 'push', and the terminal job checks forpushorpull_request, so a manual run only tests. - Read-only token. No job asks for more than
contents: read, and none asks forid-token: write, so this workflow holds no cloud credentials at all. Its only secrets are a Harbor robot account and an Argo CD token, and onlymasterjobs use them. - One run per branch at a time. With
cancel-in-progress: falsea new push doesn't stop the running build: it waits. GitHub keeps only one pending run per concurrency group, though, so in a burst the waiting runs replace each other. On 19 November I merged eight Dependabot pull requests within 82 seconds. Runs #214 to #220 were each cancelled about 12 seconds after they were queued, without starting a job, and run #221 tested the combined result. That is the behaviour I want for a release branch: the last commit gets a full run, and no release is cut halfway.
The 12 jobs and how they connect
| Job | Runs on | Runner | needs | Median |
|---|---|---|---|---|
check (opens in a new tab) | PR, push | ubuntu-latest | 7 | |
crewai (opens in a new tab) | PR, push | ubuntu-latest | 7 | |
reranker (opens in a new tab) | PR, push | ubuntu-24.04 | 1 | |
image (opens in a new tab) | PR, push | ubuntu-24.04-arm | 1 | |
e2e (opens in a new tab) | PR, push | ubuntu-24.04 | 2 | |
terminal (opens in a new tab) (×2) | PR, push | ubuntu-24.04, ubuntu-24.04-arm | 3 | |
release (opens in a new tab) | push to master | ubuntu-24.04-arm | check | 1 |
release-crewai (opens in a new tab) | push to master | ubuntu-24.04-arm | check, crewai | 1 |
reranker-image (opens in a new tab) | push to master | ubuntu-24.04-arm | reranker | 2 |
embedder-image (opens in a new tab) | push to master | ubuntu-24.04-arm | reranker | 2 |
terminal-image (opens in a new tab) | push to master | ubuntu-24.04-arm | 2 | |
deploy (opens in a new tab) | push to master, if MK4_DEPLOY | ubuntu-latest | release, release-crewai | skipped |
The rule behind the needs lines is written into the release job itself: nothing untested is pushed as latest.
release:
needs: check # must have passed on this commit: nothing untested is pushed as latest
if: ${{ github.event_name == 'push' && github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
runs-on: ubuntu-24.04-armcheck runs again after the merge, on purpose. A merge commit (or a direct push) is not the commit the pull request tested, and master has no branch protection: the API answered "Branch not protected" when I checked on 20 November, and there are no rulesets. So the post-merge check is the only gate that always runs on the code that ships.
But read the graph again for what isn't there. Three test jobs, image, e2e and terminal, have no job depending on them. On master they report and stop nothing. I come back to what that cost in the failures section.
The test jobs
check: the release gate
check is the longest job and the only one release waits for. It starts a PostgreSQL service from pgvector/pgvector:0.8.6-pg17, the same pgvector version as the CloudNativePG image on mk4, so vector search is tested on the real extension and not only on the keyword fallback. Then it runs, in order:
- a check that no symlink points outside the repository. #90 (opens in a new tab) added it after a committed
frontend/node_moduleslink to a path on my laptop put every MOKS Argo CD Application intoComparisonError, because Argo CD refuses out-of-bounds symlinks; - the backend tests, about 1,500 of them (run #228 reported 1,471 passed, 49 skipped and 1 failed);
rufffor undefined names and syntax errors only (E9,F63,F7,F82);- the frontend:
npm ci, build, and its test suite; helm lintandhelm templateagainst the Hetzner, EKS and kind values;scripts/validate-argocd-split.py, which checks that the per-component Argo CD Applications add up to the chart (123 applications, two environments, about 30 scenarios each);terraform fmt -checkandterraform validate.
In run #279 the backend tests took 7 of check's 9. Everything else took about 1 together. One detail keeps that number down on master: line coverage is measured only on pull requests. The comment in the workflow records why. Two runs on 16 November took 6 and 9 with coverage, and 4 and 4 without it, so coverage is measured where someone reads it and a release doesn't wait for it.
crewai: the same suite on a second dependency set
The CrewAI image needs a different pydantic and the crewai package, which pull in their own versions of shared libraries. So crewai runs the whole backend suite a second time against requirements.txt minus its pydantic pin plus requirements-crewai.txt.
Running 1,500 tests twice looks wasteful until it catches something. In run #234 check passed and crewai failed: on the CrewAI dependency set, a SQLAlchemy text() query raised "This text() construct doesn't define a bound parameter named 're0'". And in runs #201 and #221 a Dependabot bump installed crewai 1.15.41 while MOKS pins and checks 1.15.36, so the suite reported "CrewAI investigation unavailable" until #66 (opens in a new tab) moved the pin. Neither failure could show up on the default dependency set.
reranker: one test job, two images
The reranker and embedder are the same service code with different model lock files, so one test job (reranker, about a minute) gates both reranker-image and embedder-image. Their smoke tests run the built image the way the pods do: --network none, --read-only, user 10001. A model that tried to download at start would fail here rather than on the cluster. The embedder's test goes further and calls /embed, then asserts the vector has unit length:
v = json.loads(urllib.request.urlopen(req).read())[0]
assert abs(math.sqrt(sum(x * x for x in v)) - 1) < 1e-3, 'not unit length'image, e2e and terminal: real-world checks that only report
imagebuilds themoksimage on an arm64 runner, the wayreleasewill, starts it, checks/healthz, the UI and that/moks/api/healthreports this commit, and scans it with Trivy (CRITICAL,HIGH, unfixed ignored, exit code 1). About a minute.e2einstalls the Helm chart on a kind cluster with its NetworkPolicy, RBAC and bundled PostgreSQL, has MOKS investigate a crash-looping lab workload without a language model, and has the executor restart a Deployment through an approved action plan (scripts/e2e-kind.sh). In run #279: build 33 s, kind 36 s, the scenario 48 s.terminalis the only matrix in the file:
terminal:
if: github.event_name == 'pull_request' || github.event_name == 'push'
strategy:
fail-fast: false
matrix:
runner: [ubuntu-24.04, ubuntu-24.04-arm]
runs-on: ${{ matrix.runner }}It runs on both architectures because terminal-image publishes both. Each leg checks that the image runs read-only, without capabilities, as uid 10002, with every tool present; that a tmux session survives a detach (a browser reconnect depends on that); runs a Trivy scan; and then runs a real isolation test on kind. From inside a session pod, the instance metadata address 169.254.169.254, the ECS and EKS credential endpoints, PostgreSQL, the kubelet and the internet must all be blocked, and only the Kubernetes API reachable.
That isolation test has already paid for itself once. Dependabot bumped helm/kind-action (#43 (opens in a new tab)), which changed the default kind from v0.31.0 to v0.33.0, and in run #201 the test printed:
FAIL instance metadata is reachable (169.254.169.254:80)
ok ECS credentials endpoint is blocked (169.254.170.2:80)
ok EKS Pod Identity agent is blocked (169.254.170.23:80)#56 (opens in a new tab) pinned kind back to v0.31.0 (kubectl v1.35.0). The cause isn't understood yet, so the comment above the pin says to move it only once the isolation script passes on the newer kind. Unless you test the network policy on a real cluster, a dependency bump like that changes your security boundary without anyone noticing.
One real run, second by second
Run #279 (opens in a new tab), the merge of #91, is a typical green run. Every job without needs started within five seconds of the push, on GitHub-hosted runners that were never short. Jobs queued for 2 to 7 seconds in all runs except #221, where a burst of Dependabot runs made them wait 25 to 45 seconds.
Across the 14 green runs:
| Median | Fastest | Slowest | |
|---|---|---|---|
| Push to last job (wall clock) | 10 | 8 | 12 |
| Runner time, all jobs added up | 37 | 34 | 42 |
| check | 7 | 6 | 10 |
| crewai | 7 | 5 | 9 |
| release-crewai | 1 | 1 | 2 |
So the parallel layout runs about 3.6 times faster than the same jobs one after another. And the critical path is always the same shape: the slower of check and crewai, then release-crewai. check usually finishes last, but in 5 of the 14 runs crewai did. Everything else fits inside those two jobs with minutes to spare.
That also explains the in-progress screenshot I started from: Artifacts: 2, check and crewai still running, both release jobs waiting. The two artifacts are the reranker and embedder release records, uploaded about three minutes in. A finished green run has four: release-moks-, release-moks-crewai-, release-moks-reranker- and release-moks-embedder-<sha>.
Release: one tag, a smoke test and a record
All five build jobs follow the same pattern: log in to Harbor (registry.jithendar.com, project jithendar) with a robot account that can push and pull in that one project, build, run the image, push latest, and write down what was pushed. The core of release:
IMAGE="$REGISTRY/$REPOSITORY:latest"
docker build --build-arg GIT_COMMIT=${{ github.sha }} -t "$IMAGE" .
# The image must start and answer its probes without a database (readyz reports 503).
docker run -d --name smoke -e MOKS_ENV=development -e MOKS_AUTH_MODE=local -p 8080:8080 "$IMAGE"
for i in $(seq 1 30); do curl -fsS localhost:8080/healthz && break; sleep 1; done
curl -fsS localhost:8080/moks/ | grep -q '<div id="root">'
curl -fsS localhost:8080/moks/api/health | grep -q '${{ github.sha }}'
docker rm -f smoke
docker push "$IMAGE"
DIGEST="$(docker inspect --format '{{index .RepoDigests 0}}' "$IMAGE" | cut -d@ -f2)"Why only latest? mk4 runs every MOKS Deployment with image: …:latest and pullPolicy: Always, so a release never needs a values change or a Git commit to bump a tag. The price is that Git no longer says which build is running. Two things make up for it:
- The commit is inside the image.
/moks/api/healthreports it, and the release smoke test refuses to push an image that reports any other commit. - A release record per image. Each build job uploads a small JSON artifact, kept 400 days, mapping the commit to the digest it became, and for the model services, the model and revision:
{"component":"moks","commit":"7e09449d…","repository":"moks","tag":"latest","digest":"sha256:…","run":"44719059177"}That record is also the rollback. docs/RELEASES.md (opens in a new tab) points latest back at an earlier digest with docker buildx imagetools create --tag …:latest …@sha256:… and then restarts. Harbor keeps the last ten builds of each repository, so in practice the rollback window is ten builds.
release-crewai adds one more check before it pushes: inside the image, read-only and with no network, CrewAI must import at exactly the pinned version with its telemetry switched off. Otherwise MOKS would report the CrewAI capability as unavailable at run time.
Deploy: designed, but switched off
The deploy job is the most careful part of the file, and it hasn't run once in these 22 runs. It runs only when the repository variable MK4_DEPLOY is true, and the only repository variable set on moks-sre is an unrelated AWS_ROLE_ARN.
When enabled, it would:
- install the argocd CLI v3.5.6 and check its SHA-256 before running it, because the CLI holds the token;
- authenticate as the Argo CD project role
moks/ci, which may get, sync and restart Applications in projectmoksand nothing else. No delete, no exec, no logs, and no Kubernetes credentials in CI at all (deploy/argocd/hetzner/project.yaml(opens in a new tab)). A Cloudflare Access service token is sent as two headers, because/argocdsits behind Access; - sync the nine MOKS Applications from Git in a fixed order (foundation, database, api, worker, crewai, executor, terminal, routing, reranker), never with
--prune; - restart the Deployments that run
moks:latest, worker first so running investigations finish within its grace period, then the API, whose first pod runs pending migrations; thenmoks-crewai, but only ifrelease-crewaisucceeded for this commit; - poll
https://jithendar.com/moks/api/healthuntil it reports the new commit, for up to five minutes.
Note the if on deploy: !cancelled() && needs.release.result == 'success'. It waits for release-crewai but only requires release. A broken CrewAI build leaves the CrewAI worker on its previous image with a warning, instead of blocking every other component.
What happens without it
Two things move MOKS on mk4 today. The Argo CD Applications auto-sync from Git (selfHeal: true, prune: false), so chart changes arrive on their own a few minutes after a merge. New images arrive only when someone runs kubectl -n moks-sre rollout restart. These two paths aren't coordinated, and on 19 November that bit me:
#67 (opens in a new tab) changed both the chart (switch search to PostgreSQL) and the code (add the pgvector column on the next index build). Argo CD applied the chart about two minutes after the merge. The pod-template change restarted the pods, and they pulled latest, which still pointed at the previous build, 5b1e709. Run #225 (opens in a new tab) pushed the new image seven minutes later, and nothing restarted the pods again. The result was a new chart running old code: Rebuild index failed with "pgvector is not installed" because the old build had no enable_vector(), and I only found it the next day.
The deploy job exists for exactly this: it syncs and restarts after the image is pushed, then checks the running commit. Switching it on is the single change that would have prevented this.
What broke on master, and which job caught it
Eight of the 22 runs on master were red. Here they are, with the job and step that failed and which images reached Harbor anyway (opensearch-image still existed then):
| Run | Commit | Failed (step) | Cause | Pushed anyway |
|---|---|---|---|---|
| #193 (opens in a new tab) | 37b8238 (#40 (opens in a new tab)) | check (Argo CD split) | "moks-root trusts another repository": the old repository URL left in the AppProjects | opensearch, reranker, terminal |
| #201 (opens in a new tab) | 92a34b3 (#55 (opens in a new tab)) | check, crewai, terminal ×2 | the same Argo CD check; crewai 1.15.41 against the 1.15.36 pin; metadata reachable on kind v0.33.0 | opensearch, reranker, terminal |
| #221 (opens in a new tab) | 5b1e709 (#65 (opens in a new tab), Dependabot) | crewai (tests) | the crewai pin again | moks, opensearch, reranker, terminal |
| #228 (opens in a new tab) | 73f8dff (direct push) | check (backend tests) | test_resize_reaches_the_pseudo_terminal timed out: 1 of 1,521 tests | opensearch, reranker, terminal |
| #234 (opens in a new tab) | 7954ce1 (#72 (opens in a new tab)) | crewai (tests) | SQLAlchemy bound parameter re0, on the CrewAI dependency set only | moks, opensearch, reranker, terminal |
| #245 (opens in a new tab) | ef90ed9 (#76 (opens in a new tab)) | terminal ×2 (isolation) | the test still looked for namespace moks after #75 (opens in a new tab) moved MOKS to moks-sre | everything |
| #257 (opens in a new tab) | 4b90d13 (#82 (opens in a new tab)) | check (frontend), terminal ×2 (Trivy) | a failing Terminal.test.tsx; a HIGH Go standard library CVE in the terminal image, fixed in Go 1.26.11 | opensearch, reranker, terminal |
| #270 (opens in a new tab) | 9b3176a (#86 (opens in a new tab)) | check (Argo CD split) | "hetzner/reranker-off: group reranker should be empty", fixed in #88 (opens in a new tab) | opensearch, reranker, embedder, terminal |
Three things stand out.
check held where it was meant to. In all five runs where check failed, moks:latest didn't move. Three of those five failures came from the Argo CD split validator, a configuration check, not from a test.
The second dependency set earns its seven minutes. crewai failed three times, each for a reason check couldn't see. In #221 and #234, where check passed, release pushed moks and release-crewai was skipped, so the two images briefly came from different commits. That's exactly the case deploy handles by restarting the CrewAI worker only when its own image was built.
The jobs that only report let two things through. In #245, everything was released with both terminal legs red. That time it was harmless, since a test script was out of date, not the image. But in #257 the terminal scan failed on a HIGH CVE in the image that terminal-image pushed in the same run, because terminal-image needs nothing. The next run, 47 minutes later, passed the scan with no change to the terminal's files, so the finding most likely changed in Trivy's database, not in the image. Either way, for those 47 minutes new terminal sessions started from an image the pipeline's own scan had rejected.
Security: what the workflow does, and what it doesn't yet
What it does:
contents: readfor every job, no OIDC, no cloud keys. The secrets are a Harbor robot account scoped to one project and an Argo CD token scoped to get, sync and restart, both used only by jobs that run on pushes tomaster. No job that runs on a pull request refers to them.- Every action that isn't GitHub's own is pinned to a full commit SHA with the release in a comment (
azure/setup-helm,docker/setup-buildx-action,helm/kind-action,hashicorp/setup-terraform,aquasecurity/trivy-action), and Dependabot moves both.check,crewai,image,e2eandterminalcheck out withpersist-credentials: false. - Trivy fails the job on fixable HIGH or CRITICAL findings in the
moksandmoks-terminalimages. - Downloaded binaries are verified (the argocd CLI's SHA-256) and model weights are pinned by revision and file hashes in the lock files.
- Smoke tests run images as they run in the cluster: read-only root, a non-root user, no capabilities, and for the model services, no network.
What it doesn't do yet:
- The scan runs in
image, which no release job needs, and themoks-crewai,moks-rerankerandmoks-embedderimages aren't scanned at all. - No SBOM, signature or build provenance;
terminal-imagebuilds with--provenance=false. The release record (commit to digest) is the only link between a running image and its source. - The checkouts in
rerankerand the six build jobs keep the default persisted token. It's read-only, butpersist-credentials: falsecosts nothing there too. masterhas no branch protection, so the pull request checks gate a merge only by habit, and direct pushes (run #228) skip them.
What I'd change next
Each item here comes from the runs above, and none needs a new tool:
- Make the push jobs wait for the tests that cover them:
release: needs: [check, image, e2e]andterminal-image: needs: terminal. In all 14 green runs,image,e2eandterminalhad finished by 3, whilechecknever took less than 6, andterminal-imagestarted afterterminalwould still have finished beforecheckin every one of them. So the change would have stopped #257's push without lengthening a single run. The trade-off is that a flakye2ewould block a release, which is the right default forlatest. - Switch on
deploy(MK4_DEPLOY=true), so the image and the chart arrive together and the running commit is checked. The 19 November incident is the argument. - Shorten the critical path where it actually is. The backend suite is 7 of
check, and it runs twice (checkandcrewai). Splitting it across parallel jobs or workers would cut the wall clock directly. Faster Docker builds wouldn't: the build jobs already fit insidecheck.setup-pythonalso runs without a pip cache, while npm has one; the pip install time isn't measured separately yet, so measuring it comes before caching it. - Scan every image that's pushed, and add an SBOM and provenance, so the release record isn't the only evidence of what was built.
- Protect
master, withcheck,crewai,image,e2eandterminalas required checks on pull requests.
Investigating a run with the GitHub CLI
Every number in this article came from these commands, so they work for any run of the workflow:
# Recent runs on master
gh run list --repo mrjithendar/moks-sre --workflow ci.yml --branch master --limit 10
# Per-job timings and runners for one run
gh api "repos/mrjithendar/moks-sre/actions/runs/44719059177/jobs?per_page=100" \
--jq '.jobs[] | [.name, .conclusion, .started_at, .completed_at, (.labels | join(","))] | @tsv'
# Only the failed steps' logs of one job
gh run view --repo mrjithendar/moks-sre --job <job-id> --log-failed
# The release record: which digest a commit became
gh run download 44719059177 --repo mrjithendar/moks-sre -n release-moks-7e09449d11a621eae2f01fc84ba42bb3d7f1d41d
# Re-run only what failed
gh run rerun <run-id> --repo mrjithendar/moks-sre --failedRead a red run from the earliest failure, and read skipped as a consequence, not a cause. In #201 four jobs were red, from three unrelated causes, and the three skipped jobs after them were only the needs graph doing its job.
What I took away
- Draw the
needsgraph before trusting it. The question isn't "which jobs run" but "which failures stop a push". Here, three test jobs stopped nothing. - Gate where the cost is zero. Jobs that finish long before the critical path can become dependencies for free. The timings tell you which ones.
- Test the merged commit, not just the pull request. Without branch protection, the post-merge
checkis the only guarantee, and it held in every failure. - Run the same tests on every dependency set you ship. The second run of the suite found what the first couldn't.
- One mutable tag needs a deploy step that's ordered after the push. Otherwise the chart and the image arrive separately, and a restart can pick up the wrong one.
Building the AI side of MOKS was one project. Getting it reliably from a commit to a running pod turned out to be another, and the second one is the one that broke at 11 on a Thursday.



Discussion comments
No comments yet. Start the discussion.