Kargo

Kargo watches the GitLab container registry and writes new image tags back into this repo. ArgoCD still does the actual deploy — Kargo's job stops at the git commit.

Web UI

https://kargo.k8s.instance.onyourmarks.tech — Kargo dashboard.

Install

The Kargo control plane is installed via k8s/app/system/kargo/kargo.app.yaml (Helm chart oci://ghcr.io/akuity/kargo-charts/kargo, pinned in the Application). The admin account secret is committed at k8s/app/system/kargo/config/secret/kargo-api.yaml — only the bcrypt password hash and HS256 signing key are in the repo, never the cleartext password.

To rotate the admin password, regenerate both values and commit the new Secret:

pass=$(openssl rand -base64 48 | tr -d "=+/" | head -c 32)
echo "Password: $pass"
hashed_pass=$(htpasswd -bnBC 10 "" $pass | tr -d ':\n')
echo "Encoded Password Hash: $(echo -n "$hashed_pass" | base64)"
echo "Encoded Signing Key: $(openssl rand -base64 48 | tr -d "=+/" | head -c 32 | base64)"

Stash the cleartext password in a password manager — it's the only thing not committed.

Per-project setup

Nothing is written by hand. The Crossplane composition for XPlatformApp (k8s/app/system/crossplane-system/config/composition/composition-xplatformapps.yaml) generates, per project:

• a Kargo Project named <project>-kargo (which is also the namespace all Kargo objects for that project live in)
• a ProjectConfig with autoPromotionEnabled: true for every stage
• per environment: a Warehouse named <project>-<env> and a Stage named <env>

Adding a project is therefore just k8s/app/platform/<project>/definition/xplatformapp.yaml plus values files, see Publishing a project.

Warehouse names are globally unique across the cluster — <project>-<env> (e.g. ziggo-espn-quiz-be-staging). The cluster-wide webhook receiver routes by name, so the value CI sends in its request body has to be unambiguous.

GitLab credentials are shared — Kargo finds them in the kargo namespace via global.sharedResources.namespace and matches by regex against all onyourmarks/k8s/* repos. No per-project secret seeding needed.

Warehouse

One per environment. The tag regex is environment-suffixed:

spec:
  subscriptions:
    - image:
        repoURL: registry.gitlab.com/onyourmarks/k8s/<project>/app
        imageSelectionStrategy: NewestBuild
        discoveryLimit: 20
        allowTagsRegexes:
          - ^[0-9a-f]{40}-staging$

NewestBuild pulls metadata for every matching tag — the regex stays tight so the registry isn't hammered.

Stage promotion template

Each Stage subscribes to its matching Warehouse and runs four steps to commit the new tag into the project's values file:

1. git-clone main of this repo into ./repo
2. yaml-update writes global.image.repository + global.image.tag into k8s/app/platform/<project>/helm/values-<env>.yaml
3. git-commit with the message build: automatic update of <project>-<env>
4. git-push back to main

ArgoCD picks up the commit and rolls out the new image.

End-to-end flow from CI

Sequence:

1. GitLab CI builds the image and the manual tag:image:staging job publishes registry.gitlab.com/onyourmarks/k8s/<project>/app:<sha>-staging
2. Kargo Warehouse polls the registry (default cadence) and notices the tag, creates Freight
3. ProjectConfig has autoPromotionEnabled: true for the matching stage, so Kargo opens a Promotion
4. Promotion runs git-clone → yaml-update → git-commit → git-push against main
5. ArgoCD sees the new commit, syncs the workload Application, the new image rolls out

On-push trigger (cluster-wide, single URL, CI-driven)

One ClusterConfig hosts a single generic webhook receiver. The stage-tag-environment CI component (oym-k8s-deploy-templates ≥ 1.0.6) curls it right after re-tagging the image as <sha>-<env>. No per-project Secrets, one URL across every project, and crucially the trigger fires after the image is in the registry — so no race between "Kargo refreshes" and "image exists".

Cluster-wide pieces (in k8s/app/system/kargo/config/)

File	Purpose
cluster-config.yaml	ClusterConfig/cluster with one generic receiver named gitlab-ci. Action refreshes the Warehouse named in request.body.warehouse.
secret/gitlab-webhook.yaml	Secret/gitlab-ci-webhook in the kargo ns. Kargo bakes the secret value into a hard-to-guess URL; possession of the URL is the auth.

externalWebhooksServer is enabled in k8s/app/system/kargo/helm/values.yaml with Ingress on kargo-webhooks.k8s.instance.onyourmarks.tech, sharing the traefik-ingress-tls cert. Keep that hostname separate from the admin UI.

Why generic and not gitlab

Kargo's gitlab receiver only responds to git push events and only refreshes Warehouses subscribed to the pushed git repo. That fires before CI has built the image, so Kargo refreshes, sees the old tag, and does nothing useful. The generic receiver puts the trigger in CI's hands — fired after docker push, guaranteed the image is there.

Reading the URL

After cluster-config.yaml syncs:

kubectl get clusterconfig cluster \
  -o jsonpath='{.status.webhookReceivers[?(@.name=="gitlab-ci")].url}'

Store that URL as a group-level CI/CD variable in GitLab (ARGOCD_KARGO_WEBHOOK_URL, masked + protected) at https://gitlab.com/groups/onyourmarks/k8s/-/settings/ci_cd. Every project under the group inherits it — adding a new project requires no GitLab changes.

CI side

Projects don't add anything: the call is built into the tag:image:<env> job of the stage-tag-environment component. It runs after docker buildx imagetools create has published the <sha>-<env> tag:

curl -fsS --retry 3 --retry-delay 2 --max-time 15 \
  -X POST "${ARGOCD_KARGO_WEBHOOK_URL}" \
  -H "Content-Type: application/json" \
  -d "{\"warehouse\": \"${CI_PROJECT_NAME}-${DEPLOY_ENVIRONMENT}\"}"

${CI_PROJECT_NAME}-${DEPLOY_ENVIRONMENT} matches the Warehouse naming convention, so the GitLab project name must equal the project slug. Projects still on a vendored copy of the templates (local: includes under .deploy/templates/) don't send it and fall back to Kargo's polling cadence.

Flow

1. CI builds registry.gitlab.com/onyourmarks/k8s/<project>/app:<sha>; the manual tag:image:staging job re-tags it as <sha>-staging.
2. The same job curls the cluster-wide URL with {"warehouse": "<project>-staging"}.
3. Kargo refreshes the named Warehouse → polls the registry → finds the new tag → creates Freight.
4. The matching Stage auto-promotes → runs git-clone/yaml-update/git-commit/git-push against main.
5. ArgoCD sees the commit and rolls.

End-to-end latency: seconds after docker push, not minutes.

Credentials

GitLab access reuses the same glpat-… PAT that ArgoCD already uses for OCI Helm pulls and group-wide repo access (argocd/meta/gitlab-oci-repo-creds.yaml, gitlab-repo-creds.yaml). The Kargo-flavoured copy lives at argocd/meta/kargo-gitlab-creds.yaml and is applied manually as part of bootstrap:

kubectl apply -f argocd/meta/kargo-gitlab-creds.yaml

Two Secrets, both in the kargo namespace (Kargo's shared-resources namespace, configured via global.sharedResources.namespace):

Secret	Label	repoURL regex
gitlab-image-credentials	cred-type: image	^registry\.gitlab\.com/onyourmarks/k8s/.*
gitlab-git-credentials	cred-type: git	^https://gitlab\.com/onyourmarks/k8s/.*

repoURLIsRegex: "true" makes a single Secret cover every project in the group — no per-project credential setup needed when migrating new apps.

When the PAT rotates, update all three meta files in one go.

Common pitfalls

• Tag regex too loose — if allowTagsRegexes matches both env suffixes, the wrong tag will eventually win the NewestBuild race.
• Project ns vs app ns — the Kargo Project creates a namespace <project>-kargo (e.g. ziggo-espn-quiz-be-kargo). The actual workload lives in <project>-<env> namespaces. The -kargo suffix exists so the platform AppProject's destination patterns (*-staging, *-production, *-kargo) cover Kargo's namespace without granting wildcard access.
• Missing credential labels — without kargo.akuity.io/cred-type: image|git Kargo won't find the secret, even if it exists in the right namespace. Check kubectl get secret -n kargo --show-labels if pulls or pushes fail silently.
• Forgot to apply kargo-gitlab-creds.yaml — bootstrap files in argocd/meta/ aren't synced by ArgoCD. Warehouses will sit with ImagePullError-style status until the Secret exists.