Every GitHub Actions job in my organization, mrjithendar, now runs on mk4, my k3s cluster on Hetzner Cloud: 52 jobs in 10 repositories, from Terraform plans to Docker image builds. Each job gets a fresh pod that's deleted when the job ends. This article shows how that's built and how it's deployed. It also covers everything that broke on the way there, on a single day, 21 November 2026.
The building blocks are GitHub's Actions Runner Controller (ARC) in its newer "runner scale set" mode, two Helm charts, one Terraform file, a GitHub App, our own runner image, and a Docker daemon in every runner pod. Every code block below is copied from Git and names its file. The error messages are copied from real workflow runs, and the references at the end link every commit.
Why runners on the cluster, and why not the way we tried first
GitHub-hosted runners are good, but they live outside the cluster. Every image build starts with an empty cache, and the arm64 ones are a separate pool. We already run a 3-worker arm64 cluster that sits mostly idle between deploys, so it was the obvious place for CI.
We had tried self-hosted runners once before, and it went badly. On 29 October the runners were systemd services on the old Minikube host, the same machine that ran the cluster. The runner's README in the aws-infra repository still carries the warning:
<!-- aws-infra: runner/README.md (top) -->
> **Not used by CI since 2026-10-29.** `.github/workflows/ci-jithendar-com.yml` runs on GitHub-hosted runners
> (`ubuntu-latest`, `ubuntu-24.04-arm`) ... Builds on the mk4 server competed with the cluster for memory and disk
> and froze the host twice. Do not install these runners on the server that runs the cluster; ...Long-lived runners on the node itself meant a heavy build could starve the kubelet, and leftovers from one job stayed around for the next. This time the runners are pods, scheduled like any other workload on the three workers. Each pod exists for one job only, and its network is fenced off from the rest of the cluster. Kubernetes can cap a pod's resources; we haven't chosen the numbers yet (see Still open).
The architecture
ARC in scale set mode has three moving parts:
| Part | Where | What it does |
|---|---|---|
Controller arc-gha-rs-controller | runners-system, release arc (chart gha-runner-scale-set-controller) | Watches AutoscalingRunnerSet objects and creates the listener and the runner pods |
Listener mk4-…-listener | runners-system | Signs in as the GitHub App and long-polls GitHub for jobs queued for scale set mk4. Tells the controller how many runners it needs |
Scale set mk4 | runners, release arc-runners (chart gha-runner-scale-set) | The AutoscalingRunnerSet, its Secret and the runner pod template. One EphemeralRunner pod per job |
The controller has its own namespace, as GitHub recommends. Its ServiceAccount can manage pods, Secrets and RBAC wherever a scale set lives, so it never shares a namespace with code from a workflow. On the cluster today:
$ kubectl -n runners-system get pods -o wide # columns trimmed
NAME READY STATUS NODE
arc-gha-rs-controller-788ffdb87-5jh2n 1/1 Running w3
mk4-cd7f7986-listener 1/1 Running w3
$ kubectl -n runners get autoscalingrunnerset
NAME MINIMUM RUNNERS MAXIMUM RUNNERS PHASE
mk4 RunningUsing it from a workflow is one line. The scale set's name is the whole label, with no self-hosted and no linux:
# deploy/minikube/runners/README.md, "Using it"
jobs:
test:
runs-on: mk4 # the scale set's name; no "self-hosted" labelHow one job runs
Two settings shape this:
minRunnersandmaxRunnersare unset. No runner sits idle, and nothing caps how many pods CI can start. That's deliberate for now: the values file leaves both for the owner to set once we know what CI costs the workers.- Runners are ephemeral. A runner registers with a one-time (just-in-time) configuration, takes exactly one job and exits. Nothing a job leaves behind, files or Docker images, survives into the next job. The cost is that every job starts with an empty image cache.
The Terraform: runners.tf
mk4's add-ons (Istio, cert-manager, Gatekeeper, monitoring, Argo CD and now the runners) are all helm_releases in one Terraform root, minikube-infrastructure/. A workflow applies it on every merge to master. The runners are one more file in that root.
Versions and names
# minikube-infrastructure/runners.tf
locals {
# Controller and scale set charts must be on the same version. Helm never upgrades the controller's CRDs: a new
# version is an uninstall and reinstall (deploy/minikube/runners/README.md, "Upgrading ARC").
arc_chart_version = "0.15.1"
arc_chart_repository = "oci://ghcr.io/actions/actions-runner-controller-charts"
runners_scale_set = "mk4" # the runs-on label
runners_controller_sa = "arc-gha-rs-controller" # chart gha-runner-scale-set-controller: <release>-gha-rs-controller
}Both charts come from GitHub's OCI registry and must stay on the same version. The CRDs are the trap: Helm installs a chart's CRDs on first install and never upgrades them. So a new ARC version is a planned uninstall and reinstall, not a one-line bump.
Two namespaces, two Pod Security levels
# minikube-infrastructure/runners.tf
resource "kubernetes_namespace_v1" "runners_system" {
metadata {
name = "runners-system"
labels = {
"app.kubernetes.io/part-of" = "github-runners"
"pod-security.kubernetes.io/enforce" = "baseline"
}
}
}
resource "kubernetes_namespace_v1" "runners" {
metadata {
name = "runners"
labels = {
"app.kubernetes.io/part-of" = "github-runners"
"pod-security.kubernetes.io/enforce" = "privileged"
}
}
}runners started at baseline too. It became privileged for the Docker daemon (issue 6). Neither namespace is in the Istio mesh: nothing on mk4 calls a runner, and a runner calls nothing on mk4.
The GitHub App's Secret, and a precondition
# minikube-infrastructure/runners.tf
# The scale set's githubConfigSecret. Runner pods run as ServiceAccount mk4-gha-rs-no-permission, which
# cannot read it.
resource "kubernetes_secret_v1" "runners_github_app" {
metadata {
name = "github-app"
namespace = kubernetes_namespace_v1.runners.metadata[0].name
}
data = {
github_app_id = var.github_runners_app_id
github_app_installation_id = var.github_runners_app_installation_id
github_app_private_key = var.github_runners_app_private_key
}
lifecycle {
precondition {
condition = var.github_runners_app_id != "" && var.github_runners_app_installation_id != "" && nonsensitive(var.github_runners_app_private_key != "")
error_message = "The runners' GitHub App is not configured: set secrets RUNNERS_GITHUB_APP_ID, RUNNERS_GITHUB_APP_INSTALLATION_ID and RUNNERS_GITHUB_APP_PRIVATE_KEY of environment mk4 (locally: export TF_VAR_github_runners_app_*). ..."
}
}
}The three variables default to "", so terraform validate and the tests run without real credentials. A plan against mk4, though, stops here with a message that says exactly which secrets to set. The private key variable is sensitive = true, and its description says it must never go in a tfvars file. Only one value is committed:
# minikube-infrastructure/vars/mk4.tfvars
# GitHub Actions runners (runners.tf): one scale set for every repository of the organization mrjithendar
github_runners_config_url = "https://github.com/mrjithendar"A validation on that variable accepts https://github.com/<organization> or https://github.com/<owner>/<repository>, so the same file can register a scale set for a single repository.
Internet only: the NetworkPolicy
A job runs whatever its workflow says, plus every action, package and image it pulls. So runner pods get the internet and nothing else:
# minikube-infrastructure/runners.tf
# Jobs run whatever their workflow (and every action and package it pulls) says, so runner pods get the internet and
# nothing else: DNS through CoreDNS, then public addresses only, any port (git, apt over http, registries...). No pod,
# Service or node on mk4, and not the Hetzner metadata service (169.254.169.254). Nothing may connect to them.
resource "kubernetes_network_policy_v1" "runners" {
metadata {
name = "runners"
namespace = kubernetes_namespace_v1.runners.metadata[0].name
}
spec {
pod_selector {}
policy_types = ["Ingress", "Egress"]
egress {
to {
namespace_selector {
match_labels = { "kubernetes.io/metadata.name" = "kube-system" }
}
pod_selector {
match_labels = { "k8s-app" = "kube-dns" }
}
}
ports {
port = "53"
protocol = "UDP"
}
ports {
port = "53"
protocol = "TCP"
}
}
egress {
to {
ip_block {
cidr = "0.0.0.0/0"
except = ["10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "100.64.0.0/10", "169.254.0.0/16"]
}
}
}
}
}No ingress block means no ingress at all. The except list covers mk4's private network (10.0.0.0/16), the pod and Service ranges, and the cloud metadata service, which on many clouds hands out credentials to anyone who asks.
The two Helm releases
# minikube-infrastructure/runners.tf
resource "helm_release" "arc_controller" {
name = "arc"
namespace = kubernetes_namespace_v1.runners_system.metadata[0].name
repository = local.arc_chart_repository
chart = "gha-runner-scale-set-controller"
version = local.arc_chart_version
values = [file("${local.minikube}/runners/controller.yaml")]
wait = true
timeout = var.helm_timeout
}
# Helm waits for nothing here (the chart renders an AutoscalingRunnerSet, not pods): the deploy workflow checks the
# listener pod afterwards, which only starts once GitHub has accepted the app's credentials.
resource "helm_release" "arc_runners" {
name = "arc-runners"
namespace = kubernetes_namespace_v1.runners.metadata[0].name
repository = local.arc_chart_repository
chart = "gha-runner-scale-set"
version = local.arc_chart_version
values = [
file("${local.minikube}/runners/values.yaml"),
yamlencode({
githubConfigUrl = var.github_runners_config_url
githubConfigSecret = kubernetes_secret_v1.runners_github_app.metadata[0].name
runnerScaleSetName = local.runners_scale_set
# Named, so the chart does not search the cluster for the controller while rendering.
controllerServiceAccount = {
namespace = helm_release.arc_controller.namespace
name = local.runners_controller_sa
}
}),
]
wait = true
timeout = var.helm_timeout
# The AutoscalingRunnerSet CRD comes with the controller; no runner pod starts before its NetworkPolicy, or before
# Gatekeeper admits its privileged dind sidecar in this namespace.
depends_on = [helm_release.arc_controller, kubernetes_network_policy_v1.runners, helm_release.gatekeeper_constraints]
}Two values files feed arc_runners. Everything about the pod lives in values.yaml, which you can read and review like any manifest. Everything that wires the scale set to the rest of Terraform goes in a yamlencode block, so names are never typed twice. Naming controllerServiceAccount explicitly matters: without it, the chart looks up the controller in the live cluster while rendering, which makes terraform plan depend on cluster state.
The controller's own values file is three lines:
# deploy/minikube/runners/controller.yaml
flags:
# The chart's default is debug: every reconcile of every runner would land in Loki.
logLevel: infoTests
The root has terraform test files with mocked providers. Three runs cover the runners:
# minikube-infrastructure/tests/releases.tftest.hcl (excerpt)
run "github_runners" {
# ...
assert {
condition = (
can(regex("^docker:[0-9.]+-dind$", yamldecode(helm_release.arc_runners.values[0]).template.spec.initContainers[1].image)) &&
yamldecode(helm_release.arc_runners.values[0]).template.spec.initContainers[1].securityContext.privileged &&
yamldecode(helm_release.arc_runners.values[0]).template.spec.initContainers[1].restartPolicy == "Always" &&
kubernetes_namespace_v1.runners.metadata[0].labels["pod-security.kubernetes.io/enforce"] == "privileged" &&
kubernetes_namespace_v1.runners_system.metadata[0].labels["pod-security.kubernetes.io/enforce"] == "baseline" &&
contains(yamldecode(file("${local.minikube}/gatekeeper/constraints/values.yaml")).privilegedContainers.excludedNamespaces, "runners")
)
error_message = "Each runner pod has a pinned, privileged dind sidecar; only namespace runners admits it (Pod Security privileged, Gatekeeper exclusion), the controller's namespace stays baseline."
}
}
run "github_runners_without_app_fails" {
# ...
variables {
github_runners_app_private_key = ""
}
expect_failures = [kubernetes_secret_v1.runners_github_app]
}
run "github_runners_bad_config_url_fails" {
# ...
variables {
github_runners_config_url = "github.com/mrjithendar"
}
expect_failures = [var.github_runners_config_url]
}The first run checks the shape that matters for security: the image tags are pinned, only the dind sidecar is privileged, and only one namespace admits privileged pods. The other two prove that the guard rails actually fail.
The deploy workflow
create-mk-infrastructure.yml applies the root on every push to master that touches minikube-infrastructure/** or deploy/minikube/runners/**. The App's credentials are secrets of the GitHub environment mk4, so only the master deploy job can read them:
# .github/workflows/create-mk-infrastructure.yml (excerpt)
jobs:
terraform:
if: github.ref == 'refs/heads/master' && vars.MINIKUBE_TERRAFORM_APPLY == 'true'
runs-on: mk4
environment: mk4
env:
# The GitHub App the runner controller signs in as (minikube-infrastructure/runners.tf): secrets of environment
# mk4, so only master's deploy job sees the private key (deploy/minikube/runners/README.md, "Credentials").
TF_VAR_github_runners_app_id: ${{ secrets.RUNNERS_GITHUB_APP_ID }}
TF_VAR_github_runners_app_installation_id: ${{ secrets.RUNNERS_GITHUB_APP_INSTALLATION_ID }}
TF_VAR_github_runners_app_private_key: ${{ secrets.RUNNERS_GITHUB_APP_PRIVATE_KEY }}Helm reports success as soon as the AutoscalingRunnerSet object exists, which tells you nothing about whether GitHub accepted the App. So after the apply, the job waits for the listener pod:
# .github/workflows/create-mk-infrastructure.yml, step "Verify releases" (end)
# Helm only waits for the controller. The scale set's listener starts once GitHub has accepted the app's
# credentials and registered scale set mk4 (deploy/minikube/runners/README.md, "Checks").
listener='app.kubernetes.io/component=runner-scale-set-listener,actions.github.com/scale-set-name=mk4'
for attempt in $(seq 1 18); do
phase=$(kubectl --request-timeout=20s -n runners-system get pods -l "$listener" -o jsonpath='{.items[*].status.phase}' 2>/dev/null || true)
[[ "$phase" == Running ]] && break
echo "runner scale set listener: '${phase:-no pod}' (attempt $attempt of 18), retrying in 10 s"
sleep 10
done
if [[ "$phase" != Running ]]; then
echo "::error::The listener of runner scale set mk4 is not running after 3 minutes: GitHub may have refused the app's credentials"
kubectl --request-timeout=20s -n runners-system logs deploy/arc-gha-rs-controller --tail=40 | grep -iE 'error|fail' || true
exit 1
fiNote the first line of that job: runs-on: mk4. The workflow that deploys the runners now runs on them. That's fine for day-to-day changes. But a broken runner image could block its own fix, which is why the image is built elsewhere (The runner image).
Credentials: a GitHub App, not a token
ARC can sign in with a personal access token or a GitHub App. We use an App: it belongs to the organization, not to a person, and its permission is narrow. It needs exactly one organization permission, Self-hosted runners: Read and write. The steps from the runners README:
- Create the App under the organization's settings: Homepage URL
https://github.com/actions/actions-runner-controller, webhook off, permission Organization → Self-hosted runners: Read and write, installable only on this account. - Note the App ID and generate a private key (a
.pemfile). - Install it on the organization. The page you land on ends in
/installations/<installation ID>. - Store all three as secrets of the environment
mk4.
Step 4 plus a first apply from a laptop, as we actually ran it (IDs replaced):
# gh-app.sh (k8s-deployments, IDs replaced)
R=mrjithendar/k8s-deployments
KEY=$(ls -t ~/Downloads/<app>.*.private-key.pem | head -1)
gh secret set RUNNERS_GITHUB_APP_ID --repo $R --env mk4 --body <App ID>
gh secret set RUNNERS_GITHUB_APP_INSTALLATION_ID --repo $R --env mk4 --body <installation ID>
gh secret set RUNNERS_GITHUB_APP_PRIVATE_KEY --repo $R --env mk4 < "$KEY"
cd ~/Desktop/Work/k8s-deployments
git pull
export TF_VAR_github_runners_app_id=<App ID>
export TF_VAR_github_runners_app_installation_id=<installation ID>
export TF_VAR_github_runners_app_private_key="$(cat "$KEY")"
terraform -chdir=minikube-infrastructure apply -var-file=vars/mk4.tfvars -target=helm_release.arc_runners-target=helm_release.arc_runners applies only the runners and what they depend on (the controller, both namespaces, the Secret, the NetworkPolicy and the Gatekeeper constraints), leaving the rest of the root alone. Why we needed a laptop apply at all is issue 3.
The Helm values: a Docker daemon in every pod
ARC's scale set chart has a containerMode: dind switch that renders a Docker-in-Docker layout for you. We don't use it, because it pulls an unpinned docker:dind. Instead the same layout is written out in our values, with every image pinned:
# deploy/minikube/runners/values.yaml (comments trimmed)
# minRunners:
# maxRunners:
template:
spec:
initContainers:
# The runner's externals (its Node.js for JavaScript actions) copied where the Docker daemon sees them, for
# container actions and container: jobs.
- name: init-dind-externals
image: ghcr.io/mrjithendar/mk4-runner:2.338.2-48743dc57f6a
command: ["cp"]
args: ["-r", "/home/runner/externals/.", "/home/runner/tmpDir/"]
volumeMounts:
- name: dind-externals
mountPath: /home/runner/tmpDir
# A native sidecar (restartPolicy: Always): starts, and answers docker info, before the runner; stops after it.
- name: dind
image: docker:29.9.2-dind # the runner image's Docker CLI version
args: ["dockerd", "--host=unix:///var/run/docker.sock", "--group=$(DOCKER_GROUP_GID)"]
env:
- name: DOCKER_GROUP_GID
value: "123" # group docker in the runner image; the runner user is in it
securityContext:
privileged: true
restartPolicy: Always
startupProbe:
exec:
command: ["docker", "info"]
initialDelaySeconds: 0
failureThreshold: 24
periodSeconds: 5
volumeMounts:
# Same paths as in the runner, so docker run -v <path> finds the job's files: the workspace, and /tmp
# (scripts bind-mount mktemp -d folders, e.g. jithendar.com check-content.sh).
- name: work
mountPath: /home/runner/_work
- name: tmp
mountPath: /tmp
- name: dind-sock
mountPath: /var/run
- name: dind-externals
mountPath: /home/runner/externals
# Images and containers on the node's disk (an emptyDir: overlay2 cannot run on the container's overlay).
# Gone with the pod: every job starts with an empty image cache.
- name: dind-storage
mountPath: /var/lib/docker
containers:
- name: runner
image: ghcr.io/mrjithendar/mk4-runner:2.338.2-48743dc57f6a
command: ["/home/runner/run.sh"]
env:
- name: DOCKER_HOST
value: unix:///var/run/docker.sock
- name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
value: "120"
volumeMounts:
- name: work
mountPath: /home/runner/_work
- name: tmp
mountPath: /tmp
- name: dind-sock
mountPath: /var/run
volumes:
- name: work
emptyDir: {}
- name: tmp
emptyDir: {}
- name: dind-sock
emptyDir: {}
- name: dind-externals
emptyDir: {}
- name: dind-storage
emptyDir: {}Each part of that file exists because something went wrong without it. Issue 7 goes through them one by one.
The runner image
GitHub's ghcr.io/actions/actions-runner image is deliberately small: Ubuntu 24.04 with git, curl, jq, unzip, sudo and the Docker CLI. Jobs written for ubuntu-latest expect far more. We surveyed all 52 jobs for tools they call without installing them first, and added exactly those:
# deploy/minikube/runners/image/Dockerfile (excerpt)
FROM ghcr.io/actions/actions-runner:2.338.2
# mk4's nodes are arm64 only; the checksums below are the linux/arm64 builds.
ARG GH_VERSION=2.105.0
ARG GH_SHA256=b1278c75107423d818f1937fd92cafc9e5e9b606631568f68d7fb816f5885a02
ARG KUBECTL_VERSION=v1.37.2
ARG KUBECTL_SHA256=fcc16f63186659d9722014b7e02d5fdd2d4e669cb8c0ab66365d7d3b8ba03509
ARG HELM_VERSION=v4.3.2
ARG HELM_SHA256=df31cb61680ef728f82c770e5280dd565617bee6b039d37e85c677288be4bbb0
USER root
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
git-lfs \
python3 python3-dev python3-pip python3-venv \
build-essential \
libgomp1 \
shellcheck \
openssh-client \
zstd \
&& rm -rf /var/lib/apt/lists/* \
&& git lfs install --system \
&& printf '[global]\nbreak-system-packages = true\n' > /etc/pip.conf
RUN cd /tmp \
&& curl -fsSLo gh.tar.gz "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_arm64.tar.gz" \
&& echo "${GH_SHA256} gh.tar.gz" | sha256sum -c - \
&& tar -xzf gh.tar.gz \
&& install -m 0755 "gh_${GH_VERSION}_linux_arm64/bin/gh" /usr/local/bin/gh \
# ... kubectl and helm the same way: download, check SHA-256, install
&& rm -rf /tmp/*
ENV LANG=C.UTF-8 \
PATH=/home/runner/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
USER runner| Added | Who needs it |
|---|---|
git-lfs | actions/checkout with lfs: true (jithendar.com's content workflow) |
python3 with -dev, -pip, -venv, and pip's break-system-packages | jobs that call python3 or pip install without actions/setup-python, as GitHub's Ubuntu images allow |
build-essential | native npm and pip modules |
libgomp1 | LightGBM's OpenMP runtime (moks-sre-agent) |
shellcheck | five repositories lint their scripts with it |
openssh-client | k8s-deployments' disk clean-up workflow |
zstd | actions/cache compresses with it when present |
gh, kubectl, helm | release binaries, each checked against a pinned SHA-256 |
Languages still come from the setup-* actions (setup-node, setup-python, setup-terraform…), exactly as on GitHub's runners. The image only has to cover what a workflow uses without asking.
Built off-cluster, tagged by its own hash
build-runner-image.yml builds the image on a GitHub-hosted arm64 runner, never on mk4. If the runner image ever broke, a build that needed that same image would be stuck. The tag is the runner version plus the first 12 hex characters of the Dockerfile's SHA-256, and the workflow refuses a change where values.yaml doesn't name that exact tag:
# .github/workflows/build-runner-image.yml, step "Image tag"
dockerfile=deploy/minikube/runners/image/Dockerfile
runner=$(sed -nE 's#^FROM ghcr\.io/actions/actions-runner:([0-9.]+)$#\1#p' "$dockerfile")
[[ -n "$runner" ]] || { echo "::error file=$dockerfile::FROM must be ghcr.io/actions/actions-runner:<version>"; exit 1; }
want="ghcr.io/mrjithendar/mk4-runner:${runner}-$(sha256sum "$dockerfile" | cut -c1-12)"
have=$(sed -nE 's#^ *image: (ghcr\.io/mrjithendar/mk4-runner:[^ ]+)$#\1#p' deploy/minikube/runners/values.yaml)
[[ "$have" == "$want" ]] || { echo "::error file=deploy/minikube/runners/values.yaml::the runner image must be $want (the Dockerfile changed), not ${have:-missing}"; exit 1; }What gets pushed is exactly what Terraform deploys, and a changed Dockerfile can never reuse a tag that nodes already have cached. The push uses the workflow's own GITHUB_TOKEN (packages: write), so no registry password is stored anywhere. The package is public on ghcr.io, which means mk4's nodes pull it without an imagePullSecret. That's fine because the image holds tools, not secrets. Before pushing, the workflow runs every added tool as the runner user (git lfs version, gh --version, kubectl version --client, pip install pyyaml…). Dependabot watches the Dockerfile's FROM, because GitHub stops sending jobs to runner versions that are too old. Look closely at that have= line: it has a bug, issue 9.
Moving every workflow to mk4
With the scale set running, moving a repository is a one-word change per job. We did all ten repositories at once: one commit per repository, pushed straight to master. In each clone:
# every job's runs-on, whatever it was (ubuntu-latest, ubuntu-24.04, ubuntu-24.04-arm), becomes mk4
perl -pi -e 's/^(\s*runs-on:\s*)\S.*$/${1}mk4/' .github/workflows/*.yml52 jobs moved, plus the jobs that call reusable workflows, since those inherit the called file's runs-on. One job needed more than a word. moks-sre-agent's terminal job ran a matrix over both CPU types. On an arm64-only cluster, both entries would have been the same job twice:
# moks-sre-agent: .github/workflows/ci.yml (8e54ac4)
- # The diagnostic image on both architectures, its vulnerability scan, and real isolation checks on
+ # The diagnostic image (arm64, the mk4 runners), its vulnerability scan, and real isolation checks on
terminal:
if: github.event_name == 'pull_request' || github.event_name == 'push'
- strategy:
- fail-fast: false
- matrix:
- runner: [ubuntu-24.04, ubuntu-24.04-arm]
- runs-on: ${{ matrix.runner }}
+ runs-on: mk4Comments that had become false went in the same commits, such as "Runs on GitHub-hosted runners, never on the mk4 server". The organization's runner group needed one setting too (issue 4).
Issues we hit, and how we fixed them
All of this happened on 21 November 2026, in about two hours. The timeline:
One workflow run tells most of the story by itself. jithendar.com's content workflow, run 44874905707, ran on the same commit three times:
| Attempt | Time (UTC) | Pod | Runner image | Result |
|---|---|---|---|---|
| 1 | 06 | mk4-runner-wb4p4 | GitHub's actions-runner:2.338.2 | Unable to locate executable file: git-lfs |
| 2 | 06 | mk4-runner-pmp4w | mk4-runner:2.338.2-48743dc57f6a, no Docker daemon | failed to connect to the docker API at unix:///var/run/docker.sock |
| 3 | 06 | mk4-runner-z2k2l | same image, plus the dind sidecar | passed in 44 s |
1. Runners on the host froze the server
Symptom (29 October): the old Minikube host froze, twice, during CI. Cause: the runners were systemd services on the node that ran the cluster, so builds competed with the kubelet and every pod for memory and disk. Fix: runners are now pods, scheduled across the workers and deleted after each job. Resource limits are the next step (Still open).
2. The deploy stopped at "Check CI settings"
Symptom: the first deploy after PR #24 merged failed within its first minute:
##[error]Not set: TF_VAR_github_runners_app_id TF_VAR_github_runners_app_installation_id TF_VAR_github_runners_app_private_key. Run scripts/setup-minikube-terraform-ci.sh (... TF_VAR_github_runners_app_*: secrets RUNNERS_GITHUB_APP_ID, RUNNERS_GITHUB_APP_INSTALLATION_ID, RUNNERS_GITHUB_APP_PRIVATE_KEY of environment mk4, deploy/minikube/runners/README.md)Cause: expected. The PR shipped the code, and the GitHub App didn't exist yet. Fix: we created the App and set the three secrets (Credentials). The check is there to fail before Terraform touches anything, and it named every missing value.
3. CI could not apply: the OIDC trust after the organization rename
Symptom: with the secrets set, every deploy run got one step further, then failed on mk4's own runners:
##[error]Could not assume role with OIDC: Not authorized to perform sts:AssumeRoleWithWebIdentityCause: Terraform state lives in S3, and the workflow reaches it through an AWS role trusted via GitHub's OIDC. The organization had just been renamed. GitHub's OIDC token now carries a subject like repo:mrjithendar@257155124/k8s-deployments@583559700:…, and the role's trust policy only matched repo:mrjithendar/* and the repository's old name. Moving to self-hosted runners doesn't change OIDC at all: the token comes from GitHub, not from the runner.
Fix (for now): apply from a laptop, terraform apply -var-file=vars/mk4.tfvars -target=helm_release.arc_runners. All three Helm revisions of arc-runners (06, 06 and 06 UTC) came from there: every CI deploy that day stopped at the AWS step. The real fix is a trust-policy change in AWS, and it's still open.
4. Public repositories would have queued forever
Symptom: caught before it happened. The organization's Default runner group showed "allows_public_repositories": false, and five of the ten repositories are public. Their jobs would have waited for a runner that's never offered to them.
Fix: we turned the setting on through the API:
gh api -X PATCH orgs/mrjithendar/actions/runner-groups/1 -F allows_public_repositories=trueThat has a security cost, covered in Security.
5. "Unable to locate executable file: git-lfs"
Symptom: the first wave on mk4. 29 jobs ran there in the following hour: 7 passed and 22 failed. The first error was in checkout itself:
##[error]Unable to locate executable file: git-lfs. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. ...Others failed on shellcheck, gh and friends.
Cause: runs-on: ubuntu-latest quietly gives you a large, preinstalled toolbox. ARC's runner image doesn't.
Fix: our own image (The runner image), from a survey of what all 52 jobs actually call (18b905c). From attempt 2 on, the same run got past checkout, which now logs git-lfs/3.4.1 (GitHub; linux arm64; go 1.22.2).
6. No Docker daemon
Symptom: attempt 2, and every job that touches Docker:
failed to connect to the docker API at unix:///var/run/docker.sock; check if the path is correct and if the daemon is running: dial unix /var/run/docker.sock: connect: no such file or directoryJobs with services: failed before their first step, while the runner was asking Docker for its version:
##[error]Exit code 1 returned from process: file name '/usr/bin/docker', arguments 'version --format '{{.Server.APIVersion}}''.and container actions such as checkov failed with Docker pull failed with exit code 1.
Cause: 24 of the 52 jobs need a Docker daemon: docker build and docker run, the docker/* actions, helm/kind-action, container actions, and services:. The first version of the scale set had none on purpose. Docker-in-Docker needs a privileged container, and mk4's Gatekeeper denies privileged containers in every namespace.
Fix (c06ac72): a privileged Docker daemon beside the runner, with the exception made as small as we could. Gatekeeper's privilegedContainers constraint excludes the one namespace runners:
# deploy/minikube/gatekeeper/constraints/values.yaml (excerpt)
privilegedContainers:
enforcementAction: deny
# runners: each GitHub Actions runner pod's Docker daemon (dind sidecar, runners/values.yaml) must be privileged.
# A job there can take over its node; deploy/minikube/runners/README.md, "Docker".
excludedNamespaces: [runners]The other constraints still apply there: no host network or ports, no host PID or IPC, and no hostPath volumes. The namespace's Pod Security label moved from baseline to privileged, while runners-system stays baseline. In Terraform, arc_runners now depends_on helm_release.gatekeeper_constraints, so the scale set can't come up before the exception exists. Attempt 3 passed: latest: Pulling from jithendar/moks-cms, the content check ran in a container, and the job finished in 44 seconds.
Two more Docker workflows, started by hand afterwards, confirmed it. moks-cms's repository checks (docker run with the Prometheus image, then shellcheck) passed in 15 seconds. roboshop-ai-app's showcase checks started a services: container mongo:8.0 ("Initialize containers": success) and ran its suites against it. That run then failed in the test step itself (1 failure and 2 errors in 140 tests). It was the workflow's first run ever, and we haven't looked into it yet.
7. Getting Docker-in-Docker right
The sidecar took more than privileged: true. Each of these is a line in values.yaml above:
/var/lib/dockeron anemptyDir. Docker's overlay2 storage driver can't run on top of the container's own overlay filesystem. AnemptyDirsits on the node's real disk. The price is that images disappear with the pod.- Same paths in both containers.
docker run -v /some/path:...is resolved by the daemon, in the daemon's filesystem. So the workspace/home/runner/_workand/tmpare shared volumes, mounted at the same paths in the runner and in dind./tmpis easy to forget: jithendar.com'scheck-content.shbind-mounts amktemp -dfolder. - The socket's group. dockerd creates the socket with group
123, which isdockerin the runner image, so the non-rootrunneruser can use it. - Start order. dind is a native sidecar: an init container with
restartPolicy: Always. Kubernetes starts it before the runner and only moves on once itsstartupProbe(docker info) passes. On top of that, the runner waits up toRUNNER_WAIT_FOR_DOCKER_IN_SECONDS=120. When the job ends, the sidecar is stopped after the runner, and the pod can finish. - The runner's externals. Container actions and
container:jobs run the runner's bundled Node.js inside a container started by dind, so the daemon must see those files.init-dind-externalscopies them into a shared volume before anything else starts. - Versions pinned.
docker:29.9.2-dindmatches the Docker CLI in the runner image. The runner image is pinned by tag, and the init container uses the same image, which the tests check.
8. A two-architecture matrix on a one-architecture cluster
Symptom: none yet, which is the point. moks-sre-agent's terminal job tested the image on ubuntu-24.04 (x64) and ubuntu-24.04-arm. With runs-on: ${{ matrix.runner }} replaced by mk4, both entries would run the same arm64 job twice.
Fix: drop the matrix (diff above). The trade-off is real: nothing tests that image on x64 any more. The multi-arch terminal-image build (--platform linux/arm64,linux/amd64) still produces both.
9. The tag check that compares a tag with itself
Symptom: since c06ac72, Build runner image fails with an error that looks impossible:
##[error]the runner image must be ghcr.io/mrjithendar/mk4-runner:2.338.2-48743dc57f6a (the Dockerfile changed), not ghcr.io/mrjithendar/mk4-runner:2.338.2-48743dc57f6aCause: the have= line in the tag check greps values.yaml for every image: ghcr.io/mrjithendar/mk4-runner:… line. Before the dind change there was one. Now the init container uses the same image, so have holds the tag twice, on two lines, and never equals want. The error message prints both, and the log viewer shows only the first:
$ sed -nE 's#^ *image: (ghcr\.io/mrjithendar/mk4-runner:[^ ]+)$#\1#p' deploy/minikube/runners/values.yaml | cat -n
1 ghcr.io/mrjithendar/mk4-runner:2.338.2-48743dc57f6a
2 ghcr.io/mrjithendar/mk4-runner:2.338.2-48743dc57f6aFix: not merged yet. Comparing unique values (sort -u), or failing when the lines differ from each other, would both do. Nothing is broken on the cluster: the image was already built and pushed at 18b905c, and Terraform deploys what values.yaml names. But until it's fixed, every change to values.yaml shows a red build.
Security: what we gave up
Be honest about this part. The runners README says it plainly:
<!-- deploy/minikube/runners/README.md, "Docker" -->
**What that means:** a privileged container is root on its node. Any job (its workflow, or any action, package or
image it pulls) can read the node's disk, including other pods' volumes and Secrets, and the k3s credentials, and
from there take over the cluster. The NetworkPolicy does not stop that. It is only as safe as everything those jobs
run.So what still protects mk4 comes down to who can start a job:
- Private repositories: only people with write access to the organization's repositories. Fork pull requests from outside can't run workflows on private repositories here:
run_workflows_from_fork_pull_requestsisfalse. - Public repositories: anyone can open a pull request from a fork, and a fork's pull request runs its own version of the workflow file. So the guard is GitHub's approval setting for fork pull requests. The README's advice is to require approval for every outside contributor before such a workflow runs, or to keep public repositories off the runner group.
- The NetworkPolicy still matters, just less than it seems. It stops a job from calling Postgres, Valkey or the Kubernetes API over the network. It can't stop a privileged container from reaching the same things through the node itself.
If any of that is too much risk for your cluster, the alternatives are known. You can keep Docker work on GitHub-hosted runners, as our first version did. You can use rootless BuildKit or Kaniko for image builds and ARC's kubernetes container mode for container: jobs. Or you can put the runners on separate, disposable nodes that run nothing else.
Still open
| Item | State on 21 November 2026 | Where |
|---|---|---|
| Tag check reads two image lines | Build runner image red since c06ac72 | issue 9 |
| OIDC trust after the org rename | CI deploys of k8s-deployments and aws-infra fail at "AWS credentials"; runners applied from a laptop | issue 3 |
| practice-programming executor tests | failed on mk4 before the image and dind existed (runner produced no result line); not re-run since | its sandbox runner printed no result inside the pod; needs a look |
| Runner limits | no minRunners, maxRunners, requests or limits; CI can take as much of the workers as it asks for | deploy/minikube/runners/values.yaml |
| Upgrading ARC | a remove-and-reinstall of both releases and the CRDs, not a version bump | runners README, "Upgrading ARC" |
Checks
The commands we use to see the runners, all read-only:
kubectl -n runners-system get pods # arc-gha-rs-controller-…, and mk4-…-listener Running
kubectl -n runners get autoscalingrunnersets,ephemeralrunners,pods
kubectl -n runners-system logs deploy/arc-gha-rs-controller | grep -i error
helm -n runners history arc-runners # one revision per apply that changed the scale setNo listener pod almost always means GitHub refused the App's credentials, and the controller's log says why. On GitHub, the scale set appears under the organization's Settings → Actions → Runners as mk4. Every job's "Set up job" log names the pod it ran on (mk4-runner-xxxxx).
Git references
| Commit | Repository | What |
|---|---|---|
| a9cd274 (opens in a new tab) (PR #24 (opens in a new tab)) | k8s-deployments | ARC 0.15.1 from Terraform: namespaces, App Secret, NetworkPolicy, both releases, tests, listener check |
| c64ca5d (opens in a new tab) | k8s-deployments | its four workflows to runs-on: mk4 (one of ten such commits) |
| 8e54ac4 (opens in a new tab) | moks-sre-agent | runs-on: mk4, the terminal matrix dropped |
| 3d2c3d2 (opens in a new tab), 9a898bd (opens in a new tab), c6409d1 (opens in a new tab) | aws-infra, aws-vpc, roboshop-ai-app | the same move (the private repositories got the same commit) |
| 18b905c (opens in a new tab) | k8s-deployments | the mk4-runner image and build-runner-image.yml |
| c06ac72 (opens in a new tab) | k8s-deployments | the privileged dind sidecar, Gatekeeper exclusion, Pod Security privileged |
| 381b585 (opens in a new tab) | k8s-deployments | gh-app.sh, the secrets and the laptop apply |
The files themselves: minikube-infrastructure/runners.tf (opens in a new tab), deploy/minikube/runners/ (opens in a new tab) (values, controller values, image, README) and .github/workflows/build-runner-image.yml (opens in a new tab).



Discussion comments
No comments yet. Start the discussion.