Skip to main content

Step 6 — Apply, watch it come up, go to production

Everything is GitOps except one command: telling the cluster that the new xplatformapp.yaml exists. After that, every deploy is a click in GitLab.

6.0 Order matters

  1. The image tag <sha>-staging exists in the registry (step 4). Kargo needs something to pick up; until it finds one the values file says tag: initial and pods sit in ImagePullBackOff.
  2. The files from step 5 are pushed to main. ArgoCD reads values from main; another branch is invisible.
  3. Then apply. Doing it the other way round just means a few minutes of red status while things catch up, but don't confuse yourself.

6.1 Apply the definition

You need the cluster kubeconfig and an Auth0 login; set that up once via Connecting to MySQL (sections "Prerequisites" and "Log in"). Then, from your oym-k8s-cluster checkout on main:

git pull
./argocd/sync-apps.sh

The script re-applies every definition in the repo, which is harmless; your new one is the only change. If you get Forbidden, ask DevOps to add you to the cluster admin group.

You'll need this command again whenever xplatformapp.yaml itself changes (typically a chartVersion bump). Values files never need it; ArgoCD watches those on its own.

6.2 Watch it come up

Within a minute:

kubectl get xplatformapp <slug> # READY True once everything is generated
kubectl -n <slug>-kargo get warehouse,stage,freight # a Freight = Kargo found your image tag

Then open Kargo (https://kargo.k8s.instance.onyourmarks.tech, project <slug>-kargo). The staging stage should show a promotion that ended in a commit on main called build: automatic update of <slug>-staging. That commit is the cluster picking your image.

Then open ArgoCD (https://argocd.k8s.instance.onyourmarks.tech, app <slug>-staging). Wait for Synced / Healthy. Meanwhile:

kubectl -n <slug>-staging get externalsecret,mysqlapp,pods
kubectl -n <slug>-staging logs -l app=app --tail=100

Finally:

curl -i https://<slug>.staging.k8s.instance.onyourmarks.tech/health

6.3 When it doesn't come up

Work top to bottom; each row is one layer.

SymptomWhere to lookCause and fix
kubectl get xplatformapp shows nothing6.1The apply didn't happen or the file has a YAML error. Re-run sync-apps.sh and read its output.
No Freight in <slug>-kargo, warehouse shows an errorKargo UI → warehouseNo tag matching ^[0-9a-f]{40}-staging$ in registry.gitlab.com/onyourmarks/k8s/<slug>/app. Check step 4's registry view; check CI_PROJECT_NAME equals the slug.
Freight exists but no commit on mainKargo UI → stage → promotionPromotion error is shown there. Usually the values file path: it must be k8s/app/platform/<slug>/helm/values-staging.yaml.
ArgoCD app Unknown / ComparisonErrorArgoCD UI → app → conditionsValues file doesn't render: wrong chartVersion (not published), YAML typo, or a wrong chart name. Re-run the helm template check from step 5.
externalsecret shows SecretSyncedErrorkubectl -n <slug>-staging describe externalsecret app-secretsInfisical: project slug not equal to <slug>, environment slug not staging/prod, or machine identity CI/CD not added (step 2).
mysqlapp shows waiting for … key DATABASE_PASSWORDkubectl -n <slug>-staging get mysqlapp databaseThe key named in passwordField isn't in Infisical for this environment, or the name differs. Add it; the user is created within a minute.
Pods ImagePullBackOffkubectl describe podTag in the values file doesn't exist in the registry. Still initial: Kargo hasn't found a <sha>-staging tag yet, see the Freight row. Otherwise re-run the tag job.
Pods CrashLoopBackOffkubectl logs (add --previous)App crashes on start: usually a missing env var. Compare .env.example with Infisical.
Pods Init:Errorkubectl logs <pod> -c app-init.container-init.sh failed (migration error).
Pods Running but never Ready, ArgoCD stuck Progressingkubectl describe pod → EventsProbes fail: wrong port, /health or /ready not answering 200 within 2s, or auth middleware blocking them. Step 1.
503/default certificate in the browserkubectl -n <slug>-staging describe ingressHost in hosts:/tls: doesn't match the URL, or for custom domains DNS isn't pointing at 57.129.55.146 yet.
Site works but robots can index stagingvalues filetraefik-block-robots@kubernetescrd missing from the middlewares.

Everything in that table lives in git or Infisical. Fix the source, push, and ArgoCD reconciles within a few minutes (the External Secret refreshes on every image bump and hourly otherwise).

6.4 Production

Same pipeline, one more click: open the pipeline whose staging build you are happy with and run tag:image:production. Kargo commits build: automatic update of <slug>-production, ArgoCD rolls it out to <slug>-production, and the tag job in GitLab turns green once ArgoCD reports healthy.

Before the first production promotion make sure Infisical prod is complete (step 2) and, for a custom domain, DNS points at the cluster (step 5).

6.5 After go-live

  • Every later deploy: push, wait for build:image, click the tag job. Nothing in this repo changes by hand.
  • Chart upgrades: DevOps bumps chartVersion across all apps periodically and re-applies. If you do it for your own app, remember ./argocd/sync-apps.sh afterwards.
  • Database access from your laptop: Connecting to MySQL. Your database is stg-<slug> / prod-<slug>.
  • Manual jobs and cron: component-tasks.
  • Removing an environment or the app: delete the environment from environments: (or the whole XPlatformApp) and re-apply; follow the teardown order in component-database so the MySQL user is cleaned up.

Done when

  • kubectl get xplatformapp <slug> is READY True
  • Kargo Bot commit build: automatic update of <slug>-staging exists on main
  • ArgoCD <slug>-staging is Synced / Healthy
  • https://<slug>.staging.k8s.instance.onyourmarks.tech serves the app
  • Production tagged and healthy when you are ready