Self-Hosted GitHub Actions Runners on AKS with Actions Runner Controller (ARC)
Introduction
GitHub-hosted runners are the easiest way to run GitHub Actions: you write runs-on: ubuntu-latest and GitHub gives you a fresh VM. The problem shows up as soon as a workflow has to touch something on your private network: Terraform state in a storage account with a private endpoint, a Key Vault with public access disabled, a private AKS API server or a SQL server that only answers on a private IP. A GitHub-hosted runner simply can't reach any of those.
In an earlier article, Running Azure DevOps Private Agents in AKS, I ran Azure DevOps agents inside AKS to solve the same problem for Azure Pipelines. Static agent setups like that work, but they tend to collect a few habits worth leaving behind: long-lived agent pods, a personal access token as the agent's credential, one agent pool shared by every environment, and privileged Docker-in-Docker.
As our platform workloads started moving from Azure DevOps to GitHub Enterprise, I built the GitHub equivalent using Actions Runner Controller (ARC), GitHub's supported way of running self-hosted runners on Kubernetes. This article walks through that setup end to end:
- Disposable runners, one fresh pod per job, scaling from zero
- A GitHub App instead of a personal access token
- The App's private key flowing from Key Vault to the cluster with Terraform, never stored in Terraform state
- Runner groups that decide which repositories (and which workflows) may use each runner label
- A pinned, scanned runner image built without Docker
- ARC delivered through Argo CD (GitOps)
- Network policies, quotas, metrics and alerts
- A pilot workflow that proves OIDC sign-in and private endpoint access end to end
Work in progress
This is based on a real rollout that is still in progress. The design, Terraform, image and GitOps configuration below are built and validated; results from running real workloads will follow in a later article. See What's Next.
Key Concepts
Before diving into the implementation, here are the building blocks this setup relies on.
What is a GitHub Actions runner?
A runner is the machine that executes a GitHub Actions job. A workflow says which kind of runner it needs with runs-on, GitHub hands the job to a runner with that label, and the runner checks out the code, runs the steps and reports back.
For Example: runs-on: ubuntu-latest gets a GitHub-hosted VM; runs-on: infra-dev gets one of our self-hosted runners inside AKS.
What is a self-hosted runner?
A runner you operate yourself, on your own network. The important detail for security reviews: a self-hosted runner starts the connection to GitHub and polls for work over HTTPS (port 443). GitHub never connects into your network, so no inbound firewall rule is needed. For Example: A runner pod in AKS can reach a Key Vault private endpoint, and it only needs outbound HTTPS to GitHub.
What is Actions Runner Controller (ARC)?
ARC is a Kubernetes operator, maintained by GitHub, that manages self-hosted runners as pods. It is installed as two Helm charts: gha-runner-scale-set-controller (the operator) and gha-runner-scale-set (one release per runner label).
For Example: One controller per cluster, and one scale set each for infra-dev, infra-test, infra-staging and infra-prod.
What is a runner scale set?
A named group of identical, disposable runners that share one label (GitHub's docs call them ephemeral runners: each one runs a single job and is then deleted). For each scale set ARC runs a small listener pod that holds a long-poll connection to GitHub. When a job is queued for that label, the listener asks the controller for a runner pod; when the job finishes, the pod is deleted.
For Example: With minRunners: 0, no runner pods exist at all when nothing is queued.
What is a runner group?
An organization-level setting in GitHub that decides which repositories, and optionally which workflow files, may use a set of runners.
For Example: The infra-prod runner group accepts only one workflow file on main in one repository, so nobody can point an arbitrary workflow at the prod network.
What is a GitHub App (for ARC)?
An organization-owned identity with fine-grained permissions and a private key you can rotate. ARC uses it to register runners and poll for jobs. For Example: Unlike a personal access token, it doesn't expire when someone leaves the team, and it only has the Self-hosted runners permission.
What is OIDC federation for GitHub Actions?
GitHub issues a short-lived OIDC token to each job. Azure trusts that token through a federated credential on an app registration (or managed identity) and exchanges it for an Azure access token. No client secret is stored anywhere.
For Example: azure/login with only a client ID, tenant ID and subscription ID, and permissions: id-token: write in the workflow.
Prerequisites
- An AKS cluster that can already reach your private endpoints (I used our existing infra cluster, which also hosts Argo CD)
- GitHub Enterprise Cloud (or github.com) organization, and the Owner or GitHub App manager role in it
- Azure Key Vault reachable from where Terraform runs
- Azure Container Registry for mirrored and custom images
- Terraform 1.11 or later (ephemeral resources and write-only arguments), with the
azurerm,kubernetesandintegrations/githubproviders - Argo CD on the cluster (optional; plain
helm installworks too) - Azure Managed Prometheus on the cluster, if you want the metrics and alerts
- Outbound HTTPS from the cluster to GitHub (
github.com/*.ghe.com, the API,*.actions.githubusercontent.comand the Actions results storage)
Architecture Overview
The diagram below shows the pieces and who owns them. Terraform owns everything with a secret or an Azure resource in it; Argo CD owns everything that runs in the cluster.
flowchart LR
subgraph GH["GitHub organization"]
WF["Workflow<br/>runs-on: infra-dev"]
RG["Runner group<br/>infra-dev"]
APP["GitHub App<br/>arc-runners"]
end
subgraph AZ["Azure (private network)"]
KV["Key Vault<br/>App private key"]
ACR["Container Registry<br/>runner image, ARC images"]
PE["Private endpoints<br/>Key Vault, Storage, SQL"]
ENTRA["Entra ID<br/>federated credential"]
subgraph AKS["AKS infra cluster"]
subgraph SYS["arc-systems"]
CTRL["ARC controller"]
LIS["Listener<br/>(one per scale set)"]
end
subgraph RUN["arc-runners · node pool runners (tainted)"]
SEC["Secret arc-github-app"]
POD["Disposable runner pod<br/>(one per job)"]
end
ARGO["Argo CD"]
end
end
TF["Terraform"] -->|creates| RG
TF -->|ephemeral read| KV
TF -->|write-only| SEC
ARGO -->|syncs Helm charts| CTRL
LIS -->|long-poll HTTPS| GH
LIS --> CTRL
CTRL -->|creates| POD
POD -->|pulls image| ACR
POD -->|OIDC token exchange| ENTRA
POD -->|private DNS| PE
Key Components
| Component | What it does | Owned by |
|---|---|---|
Runner node pool runners |
Dedicated, tainted nodes so runner load stays off Argo CD and the system pool | Terraform (cluster stack) |
Namespaces arc-systems, arc-runners |
Controller and listeners; runner pods | Terraform |
Secret arc-github-app |
GitHub App ID, installation ID and private key | Terraform, from Key Vault |
Runner groups infra-<env> |
Which repositories and workflows may use each label | Terraform (integrations/github) |
| ARC controller | Watches scale sets and creates runner pods | Argo CD |
| Runner scale sets | One per environment label, minRunners: 0 |
Argo CD |
| Hardening chart | NetworkPolicies, ResourceQuota, LimitRange, PodMonitors | Argo CD |
| Runner image | Official runner base + pinned tools + corporate CA trust, no Docker | ACR Tasks pipeline |
| Alerts | Queued jobs, saturated scale set, controller or listener down | Terraform (monitoring stack) |
Design Decisions
These are the decisions that shaped everything else. I recommend writing your own version of this table before building anything; it saves a lot of debate later.
| Topic | Decision | Why |
|---|---|---|
| Runner technology | ARC runner scale sets | Supported by GitHub, disposable, scale to zero |
| Where they run | Existing infra cluster, dedicated tainted node pool | It already reaches the private endpoints and hosts Argo CD; a new cluster adds cost and network work |
| Container builds | No Docker inside jobs | Terraform and Helm jobs don't need it, and privileged pods are a security risk. Image builds use ACR Tasks |
| Authentication to GitHub | GitHub App | Org-owned, scoped and rotatable. A PAT expires and belongs to one person |
| Who may use the runners | Runner groups limited to named repos; never public repos | The runners sit on the private network |
| Environments | Separate scale sets for dev, test, staging (non-prod cluster) and prod (prod cluster) | Prod jobs never run on a dev-subscription cluster and get their own credentials |
| Delivery | Argo CD from a GitOps repo, charts vendored and pinned | Same path as every other cluster workload |
| Secrets in Terraform | Ephemeral read + write-only argument | The private key never lands in state or plan files |
Comparison with static build agents
| Area | Typical static agent setup | GitHub runners with ARC |
|---|---|---|
| Lifetime | Long-lived agent pods | One fresh pod per job, autoscaled from zero |
| Credential | Personal access token | Org-owned GitHub App |
| Cloud sign-in | Stored service connection credentials | OIDC federation, no secrets |
| Environments | One pool shared by every environment | Separate scale sets for non-prod and prod |
| Docker | Privileged Docker-in-Docker | No Docker in jobs |
| Nodes | Shared system pool | Dedicated tainted pool |
Job Lifecycle Sequence Diagram
What happens between git push and a green check mark:
sequenceDiagram
autonumber
participant Dev as Developer
participant GH as GitHub Actions
participant L as ARC listener
participant C as ARC controller
participant N as runners node pool
participant R as Runner pod
participant E as Entra ID
participant KV as Key Vault (private endpoint)
Dev->>GH: push / pull request
GH->>GH: queue job for label infra-dev<br/>(runner group allows repo?)
L->>GH: long-poll: jobs available?
GH-->>L: 1 job assigned
L->>C: patch EphemeralRunnerSet (desired = 1)
C->>N: create runner pod (node pool scales out if needed)
N-->>R: pod starts from pinned image
R->>GH: register with JIT config, take the job
R->>GH: request OIDC token (id-token: write)
R->>E: exchange OIDC token (federated credential)
E-->>R: Azure access token
R->>KV: terraform plan via private DNS + private endpoint
KV-->>R: data plane response
R->>GH: report result and logs
C->>R: delete pod after job completes
Sequence Flow Explanation:
- Queue: GitHub queues the job for the
infra-devlabel, but only if the repository (and, for restricted groups, the workflow file) is allowed by the runner group. If it isn't, the job just waits; GitHub shows no error. - Listener: The listener for that scale set is already holding a long-poll connection. It gets the job assignment and tells the controller to scale the
EphemeralRunnerSetup. - Runner pod: The controller creates a pod on the dedicated node pool. The pod registers with GitHub using a just-in-time configuration, so it never sees the GitHub App key.
- Cloud sign-in: The job requests an OIDC token from GitHub and exchanges it with Entra ID for an Azure token. Nothing secret is stored in the repo or on the runner.
- Private access: Because the pod is inside the VNet, private DNS resolves the Key Vault to its private endpoint IP.
- Cleanup: When the job ends, the pod is deleted. With
minRunners: 0, the scale set is back to zero pods.
Step 1: Create the GitHub App
ARC needs an identity to register runners. A GitHub App is the right choice: it's owned by the organization, has only the permissions you grant, and its private key can be rotated without anyone's personal account being involved.
Creating an App and generating its private key are the only manual steps in this whole setup; GitHub doesn't offer an API for either.
- Get the rights. You need the organization Owner or GitHub App manager role. Without it, the org's New GitHub App page returns a 404.
- Create the App from the organization's settings, not your personal settings: Organization → Settings → Developer settings → GitHub Apps → New GitHub App. An App created from a personal account is owned by that account.
- Name:
my-org-arc-runners - Webhook: off (ARC polls; it needs no webhook)
- Permissions: Organization → Self-hosted runners: Read and write; Repository → Metadata: Read-only. Nothing else.
- Where can this App be installed: Only on this account
- Name:
- Install the App on the organization (App page → Install App). Choose Only select repositories: the App only reads repository metadata, and the runner group decides who may use the runners.
- Generate a private key on the App page and store it in Key Vault, then delete the downloaded file:
az keyvault secret set \
--vault-name kv-infra-dev \
--name arc-github-app-private-key \
--file ./my-org-arc-runners.private-key.pem
# Remove the local copy
rm ./my-org-arc-runners.private-key.pem
- Note the App ID (App settings page) and the installation ID (the number at the end of the installation's URL). Neither is secret.
Keep the runner App narrow
I created a second GitHub App for Terraform (to manage repositories, runner groups, secrets and environments). The runner App's key ends up as a Kubernetes secret on the cluster, so it must never have repository administration rights. If that key leaked, the worst case is someone registering a runner, not someone rewriting your repositories.
Verification Commands
# The key is in Key Vault (shows metadata only, not the value)
az keyvault secret show --vault-name kv-infra-dev --name arc-github-app-private-key \
--query "{name:name, updated:attributes.updated, enabled:attributes.enabled}" -o table
Step 2: Create the Namespaces and the App Secret with Terraform
ARC's scale set chart reads the GitHub App credentials from a Kubernetes secret with three keys: github_app_id, github_app_installation_id and github_app_private_key.
The obvious Terraform approach, a data "azurerm_key_vault_secret" plus a kubernetes_secret_v1 with data = {...}, stores the private key in plain text in Terraform state. Terraform 1.10 and 1.11 fixed that with two features:
- Ephemeral resources read a value at plan/apply time without ever writing it to state or plan files.
- Write-only arguments (
data_wo) send a value to the provider without storing it. Because Terraform can't diff a value it never stored, you bump a*_revisionnumber when you want it re-sent.
# modules/github_runners/main.tf
data "azurerm_key_vault" "this" {
name = var.key_vault_name
resource_group_name = var.key_vault_resource_group_name
}
# Read the GitHub App private key as an ephemeral value:
# it is never stored in Terraform state or plan files.
ephemeral "azurerm_key_vault_secret" "private_key" {
name = var.github_app.private_key_secret_name
key_vault_id = data.azurerm_key_vault.this.id
}
# Tracks the two non-secret IDs. data_wo is only re-sent when data_wo_revision changes,
# so a changed ID replaces the secret instead (see replace_triggered_by below).
resource "terraform_data" "github_app_ids" {
input = {
app_id = var.github_app.app_id
installation_id = var.github_app.installation_id
}
}
# Secret referenced by the ARC runner scale sets (githubConfigSecret).
resource "kubernetes_secret_v1" "github_app" {
metadata {
name = "arc-github-app"
namespace = "arc-runners"
labels = {
"app.kubernetes.io/managed-by" = "terraform"
"app.kubernetes.io/part-of" = "actions-runner-controller"
}
}
type = "Opaque"
# Everything goes through data_wo (the provider doesn't allow data alongside it).
data_wo = {
github_app_id = var.github_app.app_id
github_app_installation_id = var.github_app.installation_id
github_app_private_key = ephemeral.azurerm_key_vault_secret.private_key.value
}
data_wo_revision = var.github_app.private_key_revision
lifecycle {
replace_triggered_by = [terraform_data.github_app_ids]
}
}
The namespaces are created by Terraform too, in the same stack, so that Argo CD never creates them and the secret always exists before ARC starts. If Argo CD syncs first, its apps simply retry until the namespace and secret are there.
resource "kubernetes_namespace_v1" "arc" {
for_each = toset(["arc-systems", "arc-runners"])
metadata {
name = each.key
}
}
And the stack settings, per environment:
# environments/dev/infra.tfvars
github_runners = {
enabled = true
app_id = "<APP_ID>"
installation_id = "<INSTALLATION_ID>"
private_key_revision = 1 # bump to push a rotated key to the cluster
}
Verification Commands
# Terraform state contains no private key
terraform state show 'module.github_runners[0].kubernetes_secret_v1.github_app' | grep -i private
# Expected output: nothing (data_wo is never stored)
# The secret exists with the three keys ARC expects
kubectl -n arc-runners get secret arc-github-app -o jsonpath='{.data}' | jq 'keys'
# Expected output: ["github_app_id", "github_app_installation_id", "github_app_private_key"]
Step 3: Add a Dedicated Runner Node Pool
Runner jobs are bursty and unpredictable: a terraform init downloads providers, a Helm job renders charts, a test job might eat all the memory it can get. I didn't want any of that sharing nodes with Argo CD or the system pods, so runners get their own node pool with a taint that only ARC's pods tolerate.
resource "azurerm_kubernetes_cluster_node_pool" "runners" {
name = "runners"
kubernetes_cluster_id = azurerm_kubernetes_cluster.infra.id
vm_size = "Standard_D8s_v5"
os_type = "Linux"
vnet_subnet_id = var.aks_subnet_id # same subnet: same reach to private endpoints
auto_scaling_enabled = true
min_count = 1
max_count = 3
host_encryption_enabled = true
os_disk_size_gb = 256
max_pods = 110
node_labels = {
workload = "gha-runner"
}
node_taints = ["workload=gha-runner:NoSchedule"]
}
A few things I learned:
- Name length: AKS Linux node pool names are limited to 12 characters. If your module uses
temporary_name_for_rotation(for example by appendingtmp), keep the name at 9 characters or less. min_count = 1: The pool keeps one node running, which bills even with no runners. The ARC controller and listeners also live on this pool, so one node is always useful. Scaling the pool to zero is possible but means the controller needs a home elsewhere.- Sizing: Runner pods request 1 vCPU / 2 GiB with limits of 2 vCPU / 4 GiB, so one
Standard_D8s_v5holds about 6 runners. Check your regional vCPU quota before raisingmax_count.
Verification Commands
# Pool exists with the label and taint
kubectl get nodes -l workload=gha-runner \
-o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'
# Only system DaemonSet pods should be on it before ARC is installed
kubectl get pods -A --field-selector spec.nodeName=<node-name>
Step 4: Create Runner Groups with Terraform
A runner group is what stops any repository in your organization from sending jobs to runners that sit on your private network. I manage them with the integrations/github Terraform provider, signed in as the separate Terraform GitHub App.
resource "github_actions_runner_group" "this" {
for_each = var.github_runner_groups
name = each.key
visibility = "selected"
allows_public_repositories = false # never: these runners sit on the private network
restricted_to_workflows = each.value.restricted_to_workflows
selected_workflows = each.value.restricted_to_workflows ? each.value.selected_workflows : null
selected_repository_ids = [
for repo in each.value.repositories : data.github_repository.selected[repo].repo_id
]
}
github_runner_groups = {
"infra-dev" = {
repositories = ["platform-sandbox"]
}
# test, staging and prod: one repository, one workflow, only from main
"infra-test" = {
repositories = ["platform-sandbox"]
restricted_to_workflows = true
selected_workflows = ["my-org/platform-sandbox/.github/workflows/arc-smoke.yml@refs/heads/main"]
}
"infra-staging" = {
repositories = ["platform-sandbox"]
restricted_to_workflows = true
selected_workflows = ["my-org/platform-sandbox/.github/workflows/arc-smoke.yml@refs/heads/main"]
}
"infra-prod" = {
repositories = ["platform-sandbox"]
restricted_to_workflows = true
selected_workflows = ["my-org/platform-sandbox/.github/workflows/arc-smoke.yml@refs/heads/main"]
}
}
The runner group name matches the scale set name and the runs-on label. One name for all three makes troubleshooting much easier.
Merge order matters
GitHub rejects a runner group restricted to a workflow file that doesn't exist yet on that branch. The workflow (arc-smoke.yml) has to be on main first, then the Terraform that creates the restricted runner groups.
Verification Commands
# List runner groups and what they allow
gh api /orgs/my-org/actions/runner-groups \
--jq '.runner_groups[] | {name, visibility, restricted_to_workflows, selected_workflows}'
Step 5: Build a Custom Runner Image
The official ghcr.io/actions/actions-runner image is intentionally minimal. Our jobs need Terraform, kubectl, kubelogin, Helm, yq, the Azure CLI and PowerShell, plus the corporate root and sub CA certificates, because internal endpoints use an internal CA.
The rule I followed: a tool a job needs belongs in the image, not in an install step. Jobs can't sudo (Step 8), so they can't install packages anyway, and pinned versions in the image mean every job runs the same bits.
# Base: the official runner image, mirrored to ACR and pinned by digest.
ARG RUNNER_BASE=myregistry.azurecr.io/baseimages/ghcr.io/actions/actions-runner@sha256:<digest>
ARG YQ_IMAGE=myregistry.azurecr.io/baseimages/mikefarah/yq@sha256:<digest>
FROM ${YQ_IMAGE} AS yq
FROM ${RUNNER_BASE}
ARG TERRAFORM_VERSION=1.17.2
ARG TERRAFORM_SHA256=<sha256>
ARG KUBECTL_VERSION=v1.36.6
ARG KUBECTL_SHA256=<sha256>
ARG HELM_VERSION=v4.3.2
ARG HELM_SHA256=<sha256>
ARG AZURE_CLI_VERSION=2.93.0
ARG POWERSHELL_VERSION=7.7.0-1.deb
ARG KUBELOGIN_VERSION=0.2.22
USER root
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
# Corporate root and sub CAs, so jobs trust internal TLS.
COPY certs/*.crt /usr/local/share/ca-certificates/corp/
ENV NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt \
REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl gnupg jq unzip lsb-release \
&& update-ca-certificates \
# Microsoft repositories: Azure CLI, PowerShell and kubelogin, at exact versions
&& . /etc/os-release \
&& curl -fsSL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor -o /usr/share/keyrings/microsoft.gpg \
&& echo "deb [arch=amd64 signed-by=/usr/share/keyrings/microsoft.gpg] https://packages.microsoft.com/repos/azure-cli/ ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/azure-cli.list \
&& echo "deb [arch=amd64 signed-by=/usr/share/keyrings/microsoft.gpg] https://packages.microsoft.com/ubuntu/${VERSION_ID}/prod ${VERSION_CODENAME} main" > /etc/apt/sources.list.d/microsoft-prod.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends \
"azure-cli=${AZURE_CLI_VERSION}-1~${VERSION_CODENAME}" \
"powershell=${POWERSHELL_VERSION}" \
"kubelogin=${KUBELOGIN_VERSION}-ubuntu${VERSION_ID}u1" \
# Terraform, kubectl and Helm: version AND checksum pinned
&& curl -fsSLo /tmp/terraform.zip "https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip" \
&& echo "${TERRAFORM_SHA256} /tmp/terraform.zip" | sha256sum -c - \
&& unzip -q /tmp/terraform.zip terraform -d /usr/local/bin \
&& curl -fsSLo /usr/local/bin/kubectl "https://dl.k8s.io/release/${KUBECTL_VERSION}/bin/linux/amd64/kubectl" \
&& echo "${KUBECTL_SHA256} /usr/local/bin/kubectl" | sha256sum -c - \
&& chmod 0755 /usr/local/bin/kubectl \
&& curl -fsSLo /tmp/helm.tgz "https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz" \
&& echo "${HELM_SHA256} /tmp/helm.tgz" | sha256sum -c - \
&& tar -xzf /tmp/helm.tgz -C /usr/local/bin --strip-components=1 linux-amd64/helm \
&& apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/*
COPY --from=yq /usr/bin/yq /usr/local/bin/yq
# Fail the build if a tool or the CA trust is missing
RUN terraform version && kubectl version --client && helm version --short && kubelogin --version \
&& yq --version && az version --output none && pwsh -NoLogo -Command '$PSVersionTable.PSVersion.ToString()' \
&& openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt /usr/local/share/ca-certificates/corp/corp-subca.crt
USER runner
Building without Docker
Since there's no Docker on these runners (or on our build agents), I build the image with ACR Tasks, which runs the build inside Azure Container Registry:
# Build in the dev registry
az acr build \
--registry myregistrydev \
--image github/actions-runner:2.338.2-1 \
--file Dockerfile .
# Copy the exact same digest to the prod registry
az acr import \
--name myregistryprod \
--source myregistrydev.azurecr.io/github/actions-runner:2.338.2-1 \
--image github/actions-runner:2.338.2-1
I wrapped these two commands in a small Python script that refuses to rebuild an existing tag and prints the resulting digest, and an Azure DevOps pipeline runs it on every change to the Dockerfile. Tags follow <runner version>-<build number>, and old tags are never deleted, so a rollback is just a pull request that puts the previous tag back.
Update the image at least monthly
GitHub stops sending jobs to runners whose version falls too far behind the latest runner release. Treat the runner base like any other base image with a regular update cadence.
Step 6: Install the ARC Controller with Argo CD
Everything the cluster pulls is mirrored to ACR first and pinned by digest: the controller image, the runner base image and both Helm charts. The charts are vendored, unmodified, into the GitOps repository so Argo CD (whose project only allows our GitOps repos) can sync them.
# Mirror the controller image
az acr import --name myregistrydev \
--source ghcr.io/actions/gha-runner-scale-set-controller:0.15.1 \
--image baseimages/ghcr.io/actions/gha-runner-scale-set-controller:0.15.1
# Vendor both charts into the GitOps repo
helm pull oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
--version 0.15.1 --untar --untardir platform/github-runners/vendor
helm pull oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set \
--version 0.15.1 --untar --untardir platform/github-runners/vendor
The controller values:
# platform/github-runners/values/controller.yaml
fullnameOverride: arc-controller
serviceAccount:
create: true
# Fixed name: every scale set refers to it, so the chart never has to look it up.
name: arc-controller
image:
repository: myregistrydev.azurecr.io/baseimages/ghcr.io/actions/gha-runner-scale-set-controller
tag: "0.15.1@sha256:<digest>"
resources:
requests: { cpu: 50m, memory: 128Mi }
limits: { cpu: 500m, memory: 512Mi }
podSecurityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
# On the runner node pool, off the Argo CD and system nodes
nodeSelector:
workload: gha-runner
tolerations:
- key: workload
operator: Equal
value: gha-runner
effect: NoSchedule
# Scraped by Azure Managed Prometheus (Step 9)
metrics:
controllerManagerAddr: ":8080"
listenerAddr: ":8080"
listenerEndpoint: "/metrics"
flags:
logLevel: info
logFormat: json
# Argo CD's tracking label must not be copied onto the runner objects ARC creates,
# or Argo CD sees them as out-of-sync resources and prunes them.
excludeLabelPropagationPrefixes:
- "argocd.argoproj.io/instance"
Two Argo CD gotchas
excludeLabelPropagationPrefixes: ARC copies labels from the scale set onto the resources it creates. Without this flag, the Argo CD instance label gets copied too, and Argo CD prunes ARC's runners.- Server-side apply: ARC's CRDs are too large for client-side apply's annotation. Enable
ServerSideApply=trueon the controller application.
The Argo CD application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: github-runners-controller
namespace: argocd
spec:
project: platform
source:
repoURL: https://github.com/my-org/platform-gitops.git
targetRevision: main
path: platform/github-runners/vendor/gha-runner-scale-set-controller
helm:
releaseName: arc-controller
valueFiles:
- ../../values/controller.yaml
destination:
server: https://kubernetes.default.svc
namespace: arc-systems
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- ServerSideApply=true
No Argo CD? The equivalent with Helm:
helm install arc-controller \
oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller \
--version 0.15.1 \
--namespace arc-systems \
-f values/controller.yaml
Verification Commands
# Controller running on the runner pool
kubectl -n arc-systems get pods -o wide
# ARC CRDs installed
kubectl get crds | grep actions.github.com
# Expected output: autoscalinglisteners, autoscalingrunnersets, ephemeralrunners, ephemeralrunnersets
# Controller logs
kubectl -n arc-systems logs deploy/arc-controller --tail=50
Step 7: Deploy the Runner Scale Sets
Each scale set is a separate Helm release of gha-runner-scale-set. I layer three values files so nothing is repeated:
runners.yaml: shared by every scale set (GitHub URL, secret, listener placement)runners-<cluster>.yaml: the runner pod template for one cluster (image, resources, security)<scale set>.yaml: name, runner group and maximum for one label
# values/runners.yaml (every scale set)
githubConfigUrl: https://github.com/my-org
# Created by Terraform in arc-runners (Step 2). Never put the key in Git.
githubConfigSecret: arc-github-app
# Disposable: a runner pod exists only while a job runs
minRunners: 0
controllerServiceAccount:
namespace: arc-systems
name: arc-controller
# Listener pods run in arc-systems, one per scale set
listenerTemplate:
spec:
nodeSelector:
workload: gha-runner
tolerations:
- key: workload
operator: Equal
value: gha-runner
effect: NoSchedule
containers:
- name: listener
resources:
requests: { cpu: 50m, memory: 64Mi }
limits: { cpu: 250m, memory: 256Mi }
# values/runners-dev.yaml (runner pods on the non-prod cluster)
template:
spec:
# Runner jobs don't call this cluster's Kubernetes API
automountServiceAccountToken: false
nodeSelector:
workload: gha-runner
tolerations:
- key: workload
operator: Equal
value: gha-runner
effect: NoSchedule
securityContext:
runAsNonRoot: true
runAsUser: 1001 # the image's runner user
fsGroup: 123
seccompProfile:
type: RuntimeDefault
containers:
- name: runner
image: myregistrydev.azurecr.io/github/actions-runner:2.338.2-1@sha256:<digest>
command: ["/home/runner/run.sh"]
resources:
requests: { cpu: "1", memory: 2Gi }
limits: { cpu: "2", memory: 4Gi }
securityContext:
# No sudo: a job that needs a package belongs in the image
allowPrivilegeEscalation: false
capabilities:
drop: ["ALL"]
# values/infra-dev.yaml
runnerScaleSetName: infra-dev # the runs-on label
runnerGroup: infra-dev # the runner group from Step 4
maxRunners: 4
On the non-prod cluster I run three scale sets (infra-dev max 4, infra-test max 3, infra-staging max 3). The prod cluster runs only infra-prod (max 3), with its own node pool, namespaces, secret and Argo CD instance.
Each scale set gets an Argo CD application like the controller's, pointing at the gha-runner-scale-set chart with the three values files, in namespace arc-runners, and a sync wave of 1 so it lands after the controller and its CRDs.
Verification Commands
# One listener per scale set, next to the controller
kubectl -n arc-systems get pods
# Expected: arc-controller-... and one infra-<env>-...-listener per scale set, all Running
# The scale sets ARC knows about
kubectl -n arc-runners get autoscalingrunnersets
# No runner pods while nothing is queued
kubectl -n arc-runners get ephemeralrunners,pods
# Listener logs: authentication or runner group errors show up here
kubectl -n arc-systems logs -l app.kubernetes.io/component=runner-scale-set-listener --tail=50
In GitHub, Organization → Settings → Actions → Runner groups → infra-dev shows the scale set registered, with zero runners when idle.
Step 8: Harden the Runner Namespace
Runners execute code from pull requests, on a cluster inside your network. I treated every job as untrusted and put four controls around it in a small Helm chart of its own (github-runners-hardening), also delivered by Argo CD.
1. Network policy for runner pods: no inbound traffic, DNS only through CoreDNS, and outbound to anything outside the cluster except the pod network and the instance metadata endpoint.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: arc-runners
namespace: arc-runners
spec:
podSelector: {}
policyTypes: ["Ingress", "Egress"]
ingress: []
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- { protocol: UDP, port: 53 }
- { protocol: TCP, port: 53 }
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 10.244.0.0/16 # your cluster's pod CIDR (AKS overlay default): no talking to other pods
- 169.254.169.254/32 # instance metadata: no borrowing the node's identity
Blocking 169.254.169.254 is the most important line in this whole article. Without it, any job could ask the instance metadata service for a token as the node's managed identity (the kubelet identity), which usually has far more rights than a CI job should. With it blocked, the only way for a job to sign in to Azure is OIDC with its repository's own identity.
2. Network policy for the controller and listeners: inbound only from kube-system on the metrics port, for Prometheus scraping.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: arc-systems
namespace: arc-systems
spec:
podSelector: {}
policyTypes: ["Ingress"]
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- { protocol: TCP, port: 8080 }
3. ResourceQuota and LimitRange: caps all runner pods together so one busy scale set can't starve the others, and gives any container without resources a default. On the non-prod cluster the quota is sized to the sum of all maxRunners (4 + 3 + 3 = 10) plus one spare.
apiVersion: v1
kind: ResourceQuota
metadata:
name: arc-runners
namespace: arc-runners
spec:
hard:
pods: "11"
requests.cpu: "11"
requests.memory: 22Gi
limits.cpu: "22"
limits.memory: 44Gi
---
apiVersion: v1
kind: LimitRange
metadata:
name: arc-runners
namespace: arc-runners
spec:
limits:
- type: Container
defaultRequest: { cpu: 250m, memory: 256Mi }
default: { cpu: "1", memory: 1Gi }
4. Pod security (already in the runner template, Step 7): non-root user, no privilege escalation, all capabilities dropped, RuntimeDefault seccomp, and no service account token mounted, so jobs can't call the Kubernetes API.
What a job can do: reach GitHub, Azure, the registries and private endpoints, and trust the corporate CAs.
What it can't do: reach other pods, use sudo, run Docker, call the Kubernetes API, or use the node's managed identity.
Step 9: Monitoring and Alerts
The ARC controller and listeners expose Prometheus metrics. With Azure Managed Prometheus, two PodMonitor resources (the Azure Monitor flavor, azmonitoring.coreos.com/v1) scrape them into the cluster's Azure Monitor workspace:
apiVersion: azmonitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: arc-controller
namespace: arc-systems
spec:
selector:
matchLabels:
app.kubernetes.io/part-of: gha-rs-controller
podMetricsEndpoints:
- port: metrics
path: /metrics
interval: 30s
---
apiVersion: azmonitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: arc-listeners
namespace: arc-systems
spec:
selector:
matchLabels:
app.kubernetes.io/component: runner-scale-set-listener
podMetricsEndpoints:
- port: metrics
path: /metrics
interval: 30s
On top of those metrics, four Prometheus alert rules in Terraform cover the failure modes that actually hurt:
| Alert | PromQL | For | Meaning |
|---|---|---|---|
GitHubRunnerJobsQueued |
sum by (name, namespace) (gha_assigned_jobs) - sum by (name, namespace) (gha_running_jobs) > 0 |
15 min | Jobs are assigned but no runner started: pool can't scale, image pull fails, quota full |
GitHubRunnersAtMaximum |
max by (name, namespace) (gha_busy_runners) >= max by (name, namespace) (gha_max_runners) |
30 min | Every runner busy; jobs queue behind each other |
GitHubRunnerControllerDown |
absent(up{job=~"podMonitor/arc-systems/arc-controller.*"} == 1) |
10 min | No controller; also fires when scraping itself is broken |
GitHubRunnerListenerDown |
(count(up{job=~"podMonitor/arc-systems/arc-listeners.*"} == 1) or vector(0)) < <scale sets> |
10 min | A label's jobs will never start |
resource "azurerm_monitor_alert_prometheus_rule_group" "github_runners" {
name = "GitHubRunners-${var.cluster_name}"
resource_group_name = var.resource_group_name
location = data.azurerm_monitor_workspace.this.location
cluster_name = var.cluster_name
rule_group_enabled = true
interval = "PT1M"
scopes = [
data.azurerm_monitor_workspace.this.id,
data.azurerm_kubernetes_cluster.this.id,
]
rule {
alert = "GitHubRunnerJobsQueued"
enabled = true
expression = "sum by (name, namespace) (gha_assigned_jobs) - sum by (name, namespace) (gha_running_jobs) > 0"
for = "PT15M"
severity = 2
annotations = {
description = "Scale set {{ $labels.name }} has had jobs waiting for a runner for 15 minutes."
}
action {
action_group_id = azurerm_monitor_action_group.github_runners.id
}
}
# ... GitHubRunnersAtMaximum, GitHubRunnerControllerDown, GitHubRunnerListenerDown
}
The alerts go to a dedicated action group for the team that owns the clusters, not to the general SRE on-call.
Step 10: Pilot Workflow - OIDC and a Private Endpoint
A pilot that runs echo hello proves nothing. I wanted a pilot that can only succeed from inside the private network, so I picked a read-only Terraform plan against a Key Vault that has public network access disabled.
The workflow's identity (an app registration with a federated credential for this repository) gets exactly two roles, on that one vault: Reader (the provider looks the vault up through ARM) and Key Vault Reader (list secret names, never values).
# fixtures/arc-pilot/main.tf
terraform {
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 4.52"
}
}
}
provider "azurerm" {
features {}
subscription_id = "00000000-0000-0000-0000-000000000000"
resource_provider_registrations = "none" # the identity has no subscription-level rights
}
data "azurerm_key_vault" "pilot" {
name = "kv-private-pilot"
resource_group_name = "rg-pilot"
}
# Data plane call: fails with 403 "public network access is disabled"
# unless the runner reaches the vault through its private endpoint.
data "azurerm_key_vault_secrets" "pilot" {
key_vault_id = data.azurerm_key_vault.pilot.id
}
check "vault_is_private" {
assert {
condition = !data.azurerm_key_vault.pilot.public_network_access_enabled
error_message = "The pilot vault must keep public network access disabled, or this plan no longer proves private reach."
}
}
output "secret_name_count" {
value = length(data.azurerm_key_vault_secrets.pilot.names)
}
# .github/workflows/arc-pilot.yml
name: arc-pilot
on:
pull_request:
branches: [main]
paths: ['fixtures/arc-pilot/**', '.github/workflows/arc-pilot.yml']
workflow_dispatch:
permissions:
id-token: write # OIDC sign-in to Azure
contents: read
env:
ARM_CLIENT_ID: 00000000-0000-0000-0000-000000000000
ARM_TENANT_ID: 00000000-0000-0000-0000-000000000000
ARM_SUBSCRIPTION_ID: 00000000-0000-0000-0000-000000000000
jobs:
plan:
runs-on: infra-dev
timeout-minutes: 15
defaults:
run:
working-directory: fixtures/arc-pilot
steps:
- uses: actions/checkout@v7
- name: Runner
run: |
echo "runner: $RUNNER_NAME"
terraform version
- name: Private DNS
# The vault has a private endpoint only: its name must resolve to a private IP from here
run: getent hosts kv-private-pilot.vault.azure.net
- name: Azure sign-in (OIDC)
uses: azure/login@v2
with:
client-id: ${{ env.ARM_CLIENT_ID }}
tenant-id: ${{ env.ARM_TENANT_ID }}
subscription-id: ${{ env.ARM_SUBSCRIPTION_ID }}
- name: terraform init
run: terraform init -input=false
- name: terraform plan
env:
ARM_USE_OIDC: 'true'
run: terraform plan -input=false -lock=false
If this plan is green, four things are proven at once: the runner picked up the job, OIDC federation works, private DNS resolves the vault to its private endpoint, and the data plane answered through that endpoint.
Step 11: Verification with a Smoke Test
For test, staging and prod, the runner groups only accept one workflow: a manual smoke test that signs in to nothing and checks the runner itself. It runs as a matrix over all four labels.
# .github/workflows/arc-smoke.yml
name: arc-smoke
on:
workflow_dispatch:
inputs:
target:
description: Scale set to test
type: choice
default: all
options: [all, dev, test, staging, prod]
permissions:
contents: read
jobs:
smoke:
strategy:
fail-fast: false
matrix:
env: ${{ fromJSON(inputs.target == 'all' && '["dev","test","staging","prod"]' || format('["{0}"]', inputs.target)) }}
runs-on: infra-${{ matrix.env }}
timeout-minutes: 10
steps:
- name: Runner and tools
run: |
echo "runner: $RUNNER_NAME, scale set infra-${{ matrix.env }}"
terraform version
kubectl version --client
kubelogin --version
helm version --short
yq --version
az version --output table
pwsh -NoLogo -Command '$PSVersionTable.PSVersion.ToString()'
- name: Corporate CA trust
run: |
awk -v cmd='openssl x509 -noout -subject' '/BEGIN CERT/{c=""} {c=c $0 "\n"} /END CERT/{print c | cmd; close(cmd)}' \
/etc/ssl/certs/ca-certificates.crt | grep -i 'corp-root-ca'
- name: Instance metadata blocked
# The network policy blocks 169.254.169.254, so jobs can't borrow the node's managed identity
run: |
if curl -sS -m 5 -H Metadata:true "http://169.254.169.254/metadata/instance?api-version=2021-02-01" -o /dev/null; then
echo "::error::the instance metadata endpoint is reachable from the runner"
exit 1
fi
echo "instance metadata endpoint blocked, as expected"
What to watch while a job runs
# A runner pod appears for the job and disappears afterwards
kubectl -n arc-runners get ephemeralrunners,pods -w
# The scale set's current and desired runner count
kubectl -n arc-runners get autoscalingrunnersets
# Is the node pool scaling out?
kubectl get nodes -l workload=gha-runner -w
Troubleshooting Stuck Jobs
The single most common symptom is a job that says "Waiting for a runner to pick up this job" forever. Work through it in this order:
- Label and runner group.
runs-onmust match the scale set name exactly, and the repository must be in that runner group (and, for restricted groups, the workflow file and branch must match). GitHub shows no error for a repository that isn't allowed; the job just waits. - Controller and listeners.
kubectl -n arc-systems get podsmust show the controller and one listener per scale set. Listener logs show authentication errors (bad or rotated App key) and "runner group not found" errors. - Argo CD sync. An application that keeps retrying is usually waiting for the namespace or the
arc-github-appsecret that Terraform creates. - Runner pods.
Pending: no room on the node pool (kubectl describe pod) or the quota is full (kubectl -n arc-runners describe resourcequota arc-runners).ImagePullBackOff: the tag or digest doesn't exist in that cluster's registry.
- A runner stuck after its job:
kubectl -n arc-runners delete ephemeralrunner <name>. ARC creates a fresh one if a job still needs it. - Everything healthy but jobs still queue: delete the listener pod of that scale set; it re-registers with GitHub.
Day-2 Operations
A short summary of the runbook I wrote alongside the build:
| Task | How |
|---|---|
| Scale up | Raise maxRunners in the scale set's values, raise the namespace quota to fit, raise the node pool max_count if needed (check vCPU quota first). ARC only removes idle runners, so scaling down never cuts a job short |
| Rotate the App key | Generate a new key on the App page → store as a new Key Vault secret version → bump private_key_revision and apply → restart the listeners → delete the old key on the App page |
| Roll out a runner image | Mirror the new runner base → update the Dockerfile digest and tool checksums → pipeline builds <version>-<build> → update the tag in dev values, run arc-smoke → then prod |
| Upgrade ARC | Mirror the new controller image, vendor both charts at the same version, read the release notes for CRD changes, merge in a quiet hour, run arc-smoke |
| Roll back ARC | Revert the GitOps PR. Never delete ARC's CRDs by hand: deleting a CRD deletes every scale set and runner that uses it |
| Onboard a repository | Add it to the runner group, give its app registration least-privilege Azure roles per environment, set runs-on: infra-<env>, run once against dev first |
Lessons Learned
- Write the decisions down first. A one-page decision table (runner technology, where they run, Docker or not, environments, identity) made every review faster and stopped the same debates from coming back.
- Block instance metadata. Self-hosted runners on AKS inherit the node's network position. Without a network policy, any job can use the kubelet's managed identity.
- Ephemeral + write-only is the right Terraform pattern for secrets. The GitHub App key goes from Key Vault to the cluster without ever touching state or plan files.
- Pick a pilot that can only pass from the inside. A Terraform plan against a private-only Key Vault proves OIDC, DNS and the private endpoint in one run. A plan against a public storage account proves nothing.
- Restrict prod runner groups to specific workflows. A runner group needs at least one repository; scoping it to one workflow file on
mainkeeps arbitrary code off the prod network until a real workload moves. - Order your merges. Restricted runner groups require the workflow to exist on
mainfirst. - Not every workflow should move. Workflows that build images with Docker, or that work fine on
ubuntu-latest, should stay on GitHub-hosted runners. Self-hosted runners are for jobs that need the private network.
What's Next
The natural next steps after a setup like this:
- Run the pilot and smoke workflows on every scale set, and confirm the Prometheus job names match the alert rules
- Roll the same setup out to the prod cluster with its own label and credentials
- Automate the runner image build so base-image updates become a routine pull request
- Move workflows one repository at a time, only those that genuinely need the private network
I'll write a follow-up with the results from running real workloads on these runners.
Conclusion
Actions Runner Controller turns self-hosted GitHub runners from a fleet of pets into disposable, autoscaled pods: one clean pod per job and zero pods when idle. Combined with a GitHub App, OIDC federation and Terraform's ephemeral and write-only features, the setup stores no long-lived secret anywhere a job can reach it, and no secret in Terraform state.
The hardening matters as much as the installation. A dedicated tainted node pool, a network policy that blocks instance metadata and pod-to-pod traffic, quotas, and runner groups scoped to specific repositories and workflows are what make it reasonable to run CI jobs inside a private network. With metrics, alerts and a runbook in place from day one, the runners are ready to take on real workloads as they move from Azure DevOps to GitHub.
References
- Running Azure DevOps Private Agents in AKS - the earlier setup these runners replace for GitHub workloads
- AKS Pod Connecting to PostgreSQL using Workload Identity - OIDC federation from the Kubernetes side
- About Actions Runner Controller
- Deploying runner scale sets with ARC
- Authenticating ARC to the GitHub API
- Managing access to self-hosted runners using groups
- Security hardening for self-hosted runners
- actions/actions-runner-controller on GitHub
- Configuring OpenID Connect in Azure
- Terraform ephemeral resources and write-only arguments
- Run ACR Tasks to build images
- Azure Managed Prometheus: custom scrape with PodMonitor
- Use network policies in AKS