Running Gitea and CI inside the cluster they deploy

My Git server lives inside the same Kubernetes cluster it deploys. That sounds circular, and in a home lab it is mostly fine. But over a few months it produced a string of very specific problems: git push redirected to a login page, an account password sitting in plain text in 7 repositories, a Git server that was OOM-killed three times in one evening, and a first CI pipeline that failed twice before it ran 365 tests in 82 seconds.

This post collects those lessons: how to push when the public Git URL is behind single sign-on, how to keep credentials out of remote URLs, how to size Gitea for a large repository, and what to expect from Gitea Actions with act_runner in a cluster.

Contents · 9 sections

TL;DR #

• If your Git server’s public URL is behind forward-auth SSO, Git’s basic auth gets a 302 to the login page. Push to the in-cluster Service DNS name instead.
• Don’t push to a ClusterIP: it changes when the Service is recreated. Use <service>.<namespace>.svc.<cluster-domain>, which the host can resolve if you point its resolver at CoreDNS.
• 7 repositories stored the password in URLs; a scoped token plus a credential helper fixed that. General rule: rotate any credential that ever appeared in a URL or in git history.
• 3 OOM kills in one evening; Gitea’s memory limit was sized for the binary, not for git pack-objects on a half-gigabyte repository. It went from 512 MiB to 2 GiB to 6 GiB.
• 365 tests in 82 seconds on the third CI run; before that I learned that in act_runner, JavaScript actions such as actions/checkout need Node inside the job image, steps run with bash -e -o pipefail, and uses: fetches from GitHub by default.

Why the in-cluster Git server #

All my Kubernetes manifests, about fifty namespaces’ worth, live in one repository on a Gitea instance in the cluster. It has its own persistent storage, so it survived the day my home folder was deleted, and the whole cluster could be recreated from it. That is the main reason I keep it.

Gitea has a public hostname, protected by my SSO provider through a Traefik forward-auth middleware. Humans log in through the browser with two factors. That is exactly right for the web UI and exactly wrong for git push.

Problem 1: git push gets a login page #

With forward auth, every request to the public hostname is first sent to the SSO service. If there is no valid session cookie, the answer is a redirect to the login page. The Git client does not have a cookie; it sends HTTP basic auth or a token, which forward auth does not understand. The result is a 302 and a confusing failure.

The API behaves the same: on the in-cluster address, /api/v1 with basic auth answers 200; on the public hostname it answers 302.

Options:

1. Exempt /<owner>/<repo>.git/ paths and /api/ from forward auth and rely on Gitea’s own auth there.
2. Use SSH.
3. Push from inside the trusted network to the in-cluster Service, which is not behind forward auth.

I chose option 3, because all my pushes come from the same machine or from pods. The first version used the Service’s ClusterIP, which broke whenever the Service was recreated and got a new IP. I wrote a small reconciler, a systemd user timer that ran every two minutes, looked up the current ClusterIP and rewrote the remote URLs. It worked; I broke the remote on purpose and watched it heal.

It was also the wrong fix. The real problem was my assumption that the host could not resolve cluster DNS. It can: point the host’s resolver at CoreDNS for the cluster domain.

# /etc/systemd/resolved.conf.d/<cluster>.conf
[Resolve]
DNS=<coredns-service-ip>
Domains=~<cluster-domain>

Then the remote is stable forever:

git remote set-url origin \
  http://<gitea-service>.<namespace>.svc.<cluster-domain>:<port>/<owner>/<repo>.git

In October 2026 I swept the whole estate for hard-coded cluster IPs: 266 hits in 86 files, of which 92 were live configuration. They were rewritten to DNS names; the reconciler is no longer needed.

Problem 2: a password in every remote URL #

The repositories had originally been cloned with credentials in the URL, http://<user>:<password>@<host>/.... Git stores that in .git/config, and any command that prints the remote prints the password. Seven local repositories carried the full account password this way.

The fix, on 1 August 2026:

• mint a Gitea access token with the minimum scope that allows push and pull (write:repository);
• store it in a credential file readable only by me and set a global credential helper;
• strip credentials from every remote URL.

git config --global credential.helper 'store --file ~/.git-credentials-<host>'
chmod 600 ~/.git-credentials-<host>
git remote set-url origin http://<git-host>/<owner>/<repo>.git

I verified read access on all seven, and write access by pushing and deleting a throwaway ref.

A general point worth making here: removing a password from remote URLs does not make it secret again. A credential that sat in .git/config has very likely also been in shell history, in the output of git remote -v, and in filesystem backups. So the rule is simple: rotate any credential that ever appeared in a URL or in git history. Removing a secret from where you can see it is not the same as revoking it.

Problem 3: Gitea killed three times in one evening #

On 2 September a pipeline scale-out was reported to me as “an OOM on the NLP service”. It was not. dmesg showed memory-cgroup OOM kills of git processes, and mapping the cgroup ID to a container with crictl put them inside the Gitea container:

dmesg -T | grep -i "killed process"
crictl ps -a | grep <container-id-prefix>

Gitea had a 512 MiB limit and had restarted nine times. The kills lined up with pushes and CI fetches of the manifests repository, which had grown to about half a gigabyte. Gitea forks real git processes for receive-pack and pack-objects, and those can use far more memory than the Gitea binary itself. One git process was at about 400 MiB on its own.

I raised the limit to 2 GiB. At the next half hour a git process hit 1.99 GiB and took the container down with it. The kills came exactly on the half hour, 18:30, 19:00 and 19:30, while my own pushes all succeeded. I raised the limit to 6 GiB, and started a process sampler across the next half-hour tick to capture the arguments of whatever runs pack-objects at that cadence. Gitea has been stable since; naming the caller was still open at the time of writing.

Lessons: size a Git server for Git, not for the web app, and before blaming a workload for an OOM, map the killed process’s cgroup to its container.

Problem 4: the first CI pipeline #

Gitea 1.25 had Actions enabled and zero runners. I deployed act_runner 0.6.1 in its own namespace with a Docker-in-Docker sidecar, built a Python job image, and added a workflow for my library’s test suite. Runs 1 and 2 failed; run 3 was green.

The first run failed on actions/checkout@v4. Checkout is a JavaScript action, and act_runner runs it with node from the job container. My Python image had no Node. GitHub’s hosted runners hide this because Node is always there.

Rather than add Node, I replaced checkout with a plain Git fetch of the exact commit from the in-cluster Gitea, sparse and blob-filtered:

- name: checkout
  run: |
    git init -q src && cd src
    git remote add origin "http://<gitea-service>.<namespace>.svc.<cluster-domain>:<port>/${GITHUB_REPOSITORY}.git"
    git sparse-checkout set <path-in-repo>
    git fetch -q --depth 1 --filter=blob:none origin "$GITHUB_SHA"
    git checkout -q FETCH_HEAD

That took 9 seconds instead of 2 minutes 15 seconds. It also avoids another default: uses: actions are downloaded from github.com unless you change the runner’s default actions URL, so every run depended on the internet.

The second run failed in a cleanup step. Steps run with bash -e -o pipefail, so an ls of a directory that did not exist, inside $( ), killed the step with exit code 2 before any output. Optional commands need || true.

The third run: 365 tests passed in 82 seconds, 1 minute 37 seconds end to end.

Then I added notifications. A failure step posts the repository, commit, the pytest summary and up to eight failing test IDs to a Telegram chat, and always exits 0 so a notification problem never hides the real failure. A recovery step uses the job’s built-in token to read the workflow’s previous run and sends “recovered” only when the previous run failed, so green runs stay silent. A workflow_dispatch input, simulate_failure, lets me test the alert path on demand.

Two operational notes: the runner pulls job images through its Docker-in-Docker cache, not the node’s, so the first run after an image bump pulls the whole image again; and the runner was registered at repository level, so other repositories need their own registration or an instance-level runner.

Checklist #

• Push to the Git server’s in-cluster DNS name when the public URL is behind SSO.
• Never hard-code ClusterIPs; make the host resolve cluster DNS.
• Keep credentials out of remote URLs: scoped token + credential helper, and rotate anything that was ever in a URL.
• Size Gitea’s memory for git pack-objects on your largest repository.
• Map OOM-killed processes to containers via their cgroup before blaming a workload.
• For act_runner, put Node in the job image or avoid JavaScript actions, and set the default actions URL if you want offline CI.
• Remember steps run with -e -o pipefail.
• Make failure alerts always succeed and test them with a deliberate failure.

Frequently asked questions #

Why does git push redirect to my SSO login page? #

Forward-auth middleware sends every request without a session cookie to the login page, and Git clients authenticate with basic auth or tokens, not cookies. Push over SSH, exempt the Git and API paths from forward auth, or push to an internal address that is not behind the middleware.

Why does actions/checkout fail on a Gitea Actions runner? #

actions/checkout is a JavaScript action. act_runner executes it with the node binary inside the job container, so a job image without Node cannot run it. Add Node to the image or replace checkout with plain git commands.

How much memory does Gitea need? #

It depends on your largest repository more than on Gitea itself. Gitea starts real git processes for pushes and fetches, and pack-objects on a half-gigabyte repository used close to 2 GiB here. Watch for OOM kills of git processes inside the Gitea container and size the limit for them.

Is removing a password from .git/config enough? #

No. Replacing it with a token stops new exposure, but a credential that was in a URL may already be in shell history, logs, screenshots and backups. Rotate any credential that ever appeared in a URL or in git history.

The lesson #

Every problem here came from a default that suits someone else’s setup: forward auth that suits browsers, credentials in URLs that suit a quick clone, a memory limit that suits the web app, a checkout action that suits GitHub’s runners. None was hard to fix once I saw which assumption was wrong. Running your own Git and CI is mostly the work of finding those defaults and replacing them with ones that fit.