Skip to main content

Step 6 — Watch it come up, go to production

Everything is GitOps. The push from step 5 is what tells the cluster the app exists; after that, every deploy is a click in GitLab.

6.0 Order matters​

  1. The image tag <sha>-staging exists in the registry (step 4). Kargo needs something to pick up; until it finds one the values file says tag: initial, and the chart renders no app pods, workers, cron jobs or migrations Job yet: only the secrets, database, Redis/Varnish, Service and Ingress (which answers 404/503).
  2. The files from step 5 are pushed to main. ArgoCD reads values from main; another branch is invisible. Doing it the other way round just means a few minutes of red status while things catch up, but don't confuse yourself.

6.1 Let ArgoCD pick it up​

Nothing to run. The cluster-apps Application in ArgoCD watches k8s/app/ on main and applies your xplatformapp.yaml within its reconciliation interval (up to five minutes). Open ArgoCD, app cluster-apps, and click Refresh if you don't want to wait.

For the checks below you need the cluster kubeconfig and an Auth0 login; set that up once via Connecting to MySQL (sections "Prerequisites" and "Log in").

6.2 Watch it come up​

Within a minute:

kubectl get xplatformapp <slug> # READY True once everything is generated
kubectl -n <slug>-kargo get warehouse,stage,freight # a Freight = Kargo found your image tag

Then open Kargo (https://kargo.k8s.instance.onyourmarks.tech, project <slug>-kargo). The staging stage should show a promotion that ended in a commit on main called build: automatic update of <slug>-staging. That commit is the cluster picking your image.

Then open ArgoCD (https://argocd.k8s.instance.onyourmarks.tech, app <slug>-staging). Wait for Synced / Healthy. Meanwhile:

kubectl -n <slug>-staging get externalsecret,mysqlapp,pods
kubectl -n <slug>-staging logs -l app=app --tail=100

Finally:

curl -i https://<slug>.staging.k8s.instance.onyourmarks.tech/health

6.3 When it doesn't come up​

Work top to bottom; each row is one layer.

SymptomWhere to lookCause and fix
kubectl get xplatformapp shows nothingArgoCD UI → app cluster-appsNot synced yet (Refresh), or the file has a YAML error: the cluster-apps app shows the error in its conditions.
No Freight in <slug>-kargo, warehouse shows an errorKargo UI → warehouseNo tag matching ^[0-9a-f]{40}-staging$ in registry.gitlab.com/onyourmarks/k8s/<slug>/app. Check step 4's registry view; check CI_PROJECT_NAME equals the slug.
Freight exists but no commit on mainKargo UI → stage → promotionPromotion error is shown there. Usually the values file path: it must be k8s/app/platform/<slug>/helm/values-staging.yaml.
ArgoCD app Unknown / ComparisonErrorArgoCD UI → app → conditionsValues file doesn't render: wrong chartVersion (not published), YAML typo, or a wrong chart name. Re-run the helm template check from step 5.
externalsecret shows SecretSyncedErrorkubectl -n <slug>-staging describe externalsecret app-secretsInfisical: project slug not equal to <slug>, environment slug not staging/prod, or machine identity CI/CD not added (step 2).
mysqlapp shows waiting for … key DATABASE_PASSWORDkubectl -n <slug>-staging get mysqlapp databaseThe key named in passwordField isn't in Infisical for this environment, or the name differs. Add it; the user is created within a minute.
No app pods at allgrep tag: values-staging.yamlStill initial: Kargo hasn't found a <sha>-staging tag yet, see the Freight row.
Pods ImagePullBackOffkubectl describe podTag in the values file doesn't exist in the registry. Re-run the tag job.
Pods CrashLoopBackOffkubectl logs (add --previous)App crashes on start: usually a missing env var. Compare .env.example with Infisical.
Sync Failed, old pods keep servingArgoCD UI → app → sync result, or kubectl -n <slug>-staging logs job/app-migrationsThe migrations Job failed, so ArgoCD did not roll the new version. Fix the migration and re-tag.
Pods Init:Errorkubectl logs <pod> -c app-init.container-init.sh failed.
Pods Running but never Ready, ArgoCD stuck Progressingkubectl describe pod → EventsProbes fail: wrong port, /health or /ready not answering 200 within 2s, or auth middleware blocking them. Step 1.
503/default certificate in the browserkubectl -n <slug>-staging describe ingressHost in hosts:/tls: doesn't match the URL, or for custom domains DNS isn't pointing at 57.129.55.146 yet.
Site works but robots can index stagingvalues filetraefik-block-robots@kubernetescrd missing from the middlewares.

Everything in that table lives in git or Infisical. Fix the source, push, and ArgoCD reconciles within a few minutes (the External Secret refreshes on every image bump and hourly otherwise).

6.4 Production​

Same pipeline, one more click: open the pipeline whose staging build you are happy with and run tag:image:production. Kargo commits build: automatic update of <slug>-production, ArgoCD rolls it out to <slug>-production, and the tag job in GitLab turns green once ArgoCD reports healthy.

Before the first production promotion make sure Infisical prod is complete (step 2) and, for a custom domain, DNS points at the cluster (step 5).

6.5 After go-live​

  • Every later deploy: push, wait for build:image, click the tag job. Nothing in this repo changes by hand.
  • Chart upgrades: DevOps bumps chartVersion across all apps periodically. If you do it for your own app, push to main and ArgoCD rolls it out.
  • Database access from your laptop: Connecting to MySQL. Your database is stg-<slug> / prod-<slug>.
  • Manual jobs and cron: component-tasks.
  • Removing an environment: delete it from environments: and push.
  • Removing the app: follow the teardown order in component-database so the MySQL user is cleaned up, then kubectl delete xplatformapp <slug> and delete the directory from the repo (cluster-apps does not prune on its own).

Done when​

  • kubectl get xplatformapp <slug> is READY True
  • Kargo Bot commit build: automatic update of <slug>-staging exists on main
  • ArgoCD <slug>-staging is Synced / Healthy
  • https://<slug>.staging.k8s.instance.onyourmarks.tech serves the app
  • Production tagged and healthy when you are ready