Monitoring & alerting

Everything lives in the monitoring namespace and is deployed by the kube-prometheus-stack, victoria-logs, victoria-logs-collector, prometheus-blackbox-exporter and kubernetes-event-exporter Applications under k8s/app/system/.

Web UI

https://monitoring.k8s.instance.onyourmarks.tech — Grafana, behind the same Auth0 login as ArgoCD. Prometheus is at /prometheus, Alertmanager at /alertmanager.

What is collected

Source	How	Where it ends up
Node, kubelet, cAdvisor, kube-state-metrics	kube-prometheus-stack defaults	Prometheus, 30 days
Traefik (per router/service requests and latency), ArgoCD, Velero, cert-manager, Kyverno, VictoriaLogs	ServiceMonitors in each app's config/ directory	Prometheus
MySQL (Galera, connections, table engines)	mysqld-exporter sidecar, see MySQL	Prometheus
PXC backup objects	kube-state-metrics custom-resource state (pxc_backup_state, pxc_backup_completed_timestamp_seconds)	Prometheus
Public endpoints	blackbox exporter probes every Ingress labelled monitoring.onyourmarks.tech/probe	Prometheus
Traefik access logs, CrowdSec logs	victoria-logs-collector	VictoriaLogs, 31 days
Application container logs, every *-staging, *-production, *-sprint namespace	victoria-logs-collector; fields kubernetes.pod_namespace, kubernetes.pod_name, kubernetes.container_name, pod labels app/component	VictoriaLogs, 31 days, 50Gi volume
Kubernetes events (all namespaces)	kubernetes-event-exporter → VictoriaLogs JSON-line API, stream {source="kubernetes-event",kind,namespace}	VictoriaLogs, 31 days

System namespaces (kube-system, monitoring, argocd, ...) are not collected; use kubectl logs there. Events are kept for a month instead of the one hour the API server keeps them.

From a graph to the logs

Any Prometheus series that carries namespace and pod labels (Explore, App Overview, the upstream pod dashboards) has two correlations: Pod logs and Pod events. Click a point, pick the link, and VictoriaLogs opens on that pod with the same time range. They are provisioned on the Prometheus datasource in k8s/app/system/kube-prometheus-stack/helm/values.yaml.

Every ArgoCD sync that completes is an event (kind=Application, reason=OperationCompleted); App Overview draws those as purple deploy markers on all graphs for the selected namespaces.

Dashboards

Dashboards are ConfigMaps with the label grafana_dashboard: "1" in k8s/app/system/kube-prometheus-stack/config/dashboards/; the Grafana sidecar loads them. The OYM dashboards (App Overview, Platform Home, Environment Status) are generated: edit k8s/app/system/kube-prometheus-stack/dashboards-src/generate.py and run it, never the YAML. The annotation grafana_folder names the Grafana folder as an absolute sidecar path (/tmp/dashboards/Apps, /tmp/dashboards/Platform); only the last directory becomes the folder, Grafana's file provisioning cannot nest. Dashboards without the annotation, i.e. the upstream kube-prometheus-stack ones, land in Kubernetes. Edit the JSON in the repo, not in the UI (UI edits are lost on the next sync).

Dashboard	Use it for
OYM / Platform Home (Platform)	Grafana's home page. Share of healthy environments, a table of only the environments that are WARN/DOWN (from the oym:namespace_health:score recording rule: ArgoCD state, pods available, 5xx share; click to open App Overview), a timeline of problems in the range, firing alerts, node and volume capacity, backup and certificate ages, recent deploys, policy warnings.
OYM / Environment Status (Platform)	Every environment, one row each, production left and staging right: health status, ArgoCD sync and health, pods ready/desired (Deployments and StatefulSets), restarts in the last hour, requests/s and 5xx share. Sorted so problems come first; click a row to open App Overview. Linked from the home page ("All environments").
OYM / App Overview (Apps)	One or more namespaces: ArgoCD status, traffic and latency, CPU/memory against requests and limits, restarts, failed jobs and cron freshness, Kubernetes events. Start here when an app misbehaves.
OYM / Node Health (Platform)	Node capacity, disk pressure, pods per node.
OYM / Traefik Traffic (Platform)	Who is calling what, status codes, CrowdSec bans. Built on the access logs.
OYM / Varnish Cache (Apps)	Hit ratio and backend health for apps with component-varnish.
Kubernetes/	Upstream Kubernetes dashboards (namespace, pod, workload, PVC usage).

Alerts

Alertmanager sends everything to Slack (#ops); the Watchdog alert feeds an UptimeRobot heartbeat so a dead Alertmanager is itself noticed. Rules are PrometheusRule objects labelled release: kube-prometheus-stack, kept next to the thing they watch:

File	Alerts
k8s/app/system/kube-prometheus-stack/config/prometheusrule-platform.yaml	RolloutStalled, StatefulSetRolloutStalled, PodReadyStateInconsistent
k8s/app/system/argocd-config/config/prometheusrules/argocd.yaml	ArgoAppOutOfSync, ArgoAppDegraded, ArgoAppStuckProgressing
k8s/app/system/velero/config/prometheusrule.yaml	VeleroBackupFailed, VeleroBackupPartiallyFailed, VeleroDailyBackupStale, VeleroBackupLocationUnavailable
k8s/app/database/percona-mysql/config/monitoring/prometheusrule.yaml	MySQL* connection, availability, Galera and backup alerts
k8s/app/system/cert-manager/config/prometheusrule.yaml	CertManagerCertificate*
k8s/app/system/prometheus-blackbox-exporter/config/prometheusrule.yaml	ProbeFailed, ProbeSSLCert*
kube-prometheus-stack defaults	KubePodCrashLooping, KubeContainerWaiting, KubePersistentVolumeFillingUp, node and API server rules

Every alert's description says what to check first. The platform ones encode failure modes this cluster has actually had:

• RolloutStalled / StatefulSetRolloutStalled — a Deployment's generation is ahead of observedGeneration for 15 minutes: the OVH controller-manager has stopped reconciling. Rollouts hang silently (ArgoCD shows Progressing forever). Fix: restart the control plane through the OVH API (POST /cloud/project/{serviceName}/kube/{kubeId}/restart); workloads keep running meanwhile.
• PodReadyStateInconsistent — a pod is Running, every container is ready, yet the pod condition says Ready=False. One API server replica is serving a stale watch cache for that node, and Services have dropped the pod. Fix: kubectl annotate pod <pod> nudge=$(date +%s) makes the kubelet re-report; if it hits several pods on one node, restart the control plane as above.
• ArgoAppOutOfSync for 30 minutes — auto-sync should clear it within minutes; persistent means a failing sync, most often the migrations Job (kubectl -n <ns> logs job/app-migrations).
• MySQLBackupStale / VeleroDailyBackupStale — the 02:00 schedules did not complete. Both have a 26-hour window so a slow backup does not page anyone. For Velero, PartiallyFailed is not Completed: a backup that skips items every night keeps this alert firing until the skipped items are fixed or excluded.

Adding an alert

Put the rule in the config/ directory of the app it concerns, label it release: kube-prometheus-stack, add it to that directory's kustomization.yaml, push. Prometheus reloads rules within a minute of the sync. Use severity: critical only when a site is down or data is at risk; warning for everything that can wait until morning.

Warning events in Slack

kubernetes-event-exporter also posts selected Warning events to #ops-k8s-alertmanager (same webhook as Alertmanager, secret alertmanager-slack-webhook, Infisical /grafana/alertmanager-slack-webhook): OOMKilling, Evicted, FailedScheduling, BackOff, Unhealthy, FailedMount, FailedCreate, NodeNotReady, once an event has repeated 3 times, at most one message per object and reason every 5 minutes. Kyverno policy events and the system namespaces are excluded. The routing lives in k8s/app/system/kubernetes-event-exporter/config/configmap.yaml.

Searching events and logs

In Grafana → Explore, datasource VictoriaLogs:

{source="kubernetes-event",namespace="oym-paddock-fe-production"}                     # everything for one app
{source="kubernetes-event"} type:="Warning" reason:="OOMKilling"                        # OOM kills, cluster-wide
{source="kubernetes-event"} reason:="Evicted" | stats by (namespace) count()           # who got evicted
{source="kubernetes-event",kind="Node"}                                                # node conditions and pressure

Fields: type, reason, _msg (the event message), count, kind, namespace, name, component, host. The App Overview dashboard shows the same stream for the selected namespaces.

Application logs:

{kubernetes.pod_namespace="oym-paddock-fe-production"}                                   # everything for one environment
{kubernetes.pod_namespace="oym-paddock-fe-production"} ~"(?i)error"                      # error lines
{kubernetes.pod_namespace=~".*-production"} kubernetes.pod_labels.component:="worker"     # all production workers
{kubernetes.pod_namespace="circuit-zandvoort-production"} | stats by (kubernetes.container_name) count()
{kubernetes.pod_namespace="traefik"} | format "<DownstreamStatus> <RequestMethod> <RequestHost><RequestPath>" as _msg  # access logs have no message field

Traefik access logs are JSON without a message field, so in Explore they read missing _msg field; see ... until you | format one (the Traefik dashboard does). The App Overview log feed prefixes every line with namespace/pod and hides liveness/readiness probe requests (kube-probe, GET /health, /ready, /ping...), about 20% of all app lines.