Skip to main content

Edge security (Traefik + CrowdSec)

Every HTTP request to the cluster passes through Traefik, and every request is checked by CrowdSec before it reaches an app. This page explains how the pieces fit, which IPs are trusted and why, and how the TransIP proxy box in front of the NN sites is wired in.

Request path​

visitor ──► [TransIP HAProxy box] ──► OVH load balancer ──► Traefik ──► crowdsec middleware ──► app
            (only the NN sites)       PROXY protocol v2      entrypoint
                                      adds the TCP peer IP   web / websecure
  • The OVH load balancer (57.129.55.146) speaks PROXY protocol v2 to Traefik, so Traefik's RemoteAddr is the real TCP peer, not the load balancer.
  • Traefik runs the crowdsec middleware on both entrypoints (k8s/app/system/traefik/config/ingress/middleware/crowdsec.yaml), so no app can opt out.
  • Traefik access logs (JSON) are shipped to VictoriaLogs and read by the CrowdSec agent.

CrowdSec components​

All in namespace crowdsec, Helm values in k8s/app/system/crowdsec/helm/values.yaml.

PartRole
agent (DaemonSet on the system pool)tails Traefik pod logs, runs the crowdsecurity/traefik collection and scenarios, sends alerts to LAPI
LAPIstores alerts and decisions (bans); the bouncer polls it
AppSecinline WAF (crowdsecurity/appsec-virtual-patching + generic rules); the bouncer forwards each request to it
bouncerthe crowdsec-bouncer-traefik-plugin middleware in Traefik: checks the visitor IP against LAPI decisions (stream mode, refreshed every 60s) and asks AppSec

Everything fails open: an unreachable LAPI or AppSec never blocks traffic (updateMaxFailure: -1, crowdsecAppsecFailureBlock/UnreachableBlock: false, httpTimeoutSeconds: 2).

Which IP gets checked​

The bouncer determines the visitor IP like this:

  1. Start from RemoteAddr.
  2. If RemoteAddr is in forwardedHeadersTrustedIPs, walk X-Forwarded-For from right to left, skipping trusted proxies, and use the first other IP. No header → fall back to RemoteAddr.
  3. If the resulting IP is in clientTrustedIPs, the request bypasses LAPI and AppSec entirely.
  4. Otherwise: 403 if LAPI has a ban for it, then AppSec inspection, then the app.

So the two lists mean different things:

  • clientTrustedIPs = "never check this traffic". Only in-cluster callers (pod CIDR 10.2.0.0/16: blackbox probes, oauth2-proxy, app-to-app calls) and OYM's own OVH server belong here.
  • forwardedHeadersTrustedIPs = "this is a reverse proxy; check the visitor behind it". The TransIP boxes belong here. Putting a proxy in clientTrustedIPs instead would waive all checks for every visitor behind it.

CrowdSec detection uses a different field: the agent parser takes the source IP from Traefik's ClientHost, which Traefik sets to the X-Forwarded-For header when one is present. That is why Traefik itself must only accept forwarded headers from known proxies (next section); otherwise any visitor can choose which IP CrowdSec bans.

Trusted proxies in Traefik​

k8s/app/system/traefik/helm/values.yaml, ports.web / ports.websecure:

  • proxyProtocol.trustedIPs: 0.0.0.0/0 — the OVH load balancer's amphora IPs are not stable, and the PROXY header only reaches Traefik through the load balancer anyway.
  • forwardedHeaders.trustedIPs — pod CIDR plus the TransIP boxes. Traefik strips X-Forwarded-* from every other source and rebuilds them from the connection.

The same three entries appear in the bouncer middleware. Keep the two lists in sync when a proxy is added or removed.

Parser whitelist​

config.parsers.s02-enrich.oym-monitoring-whitelist.yaml in the CrowdSec values lists IPs that may never be banned: the blackbox exporter's node, the TransIP boxes and the OVH server. The TransIP boxes forward the visitor IP, so bans normally hit visitors, but whitelisting the boxes' own addresses guarantees that a scenario can never take all sites behind a box offline at once.

The TransIP box: nn.tdebv.nl​

nn.tdebv.nl = 93.119.12.22 (AlmaLinux 8, ssh root@nn.tdebv.nl). DNS for these hosts still points at it, so their traffic reaches the cluster through it:

  • supportervanallehardlopers.nl, www.supportervanallehardlopers.nl
  • hardloopsupporter.nl, www., app.hardloopsupporter.nl
  • hardloopplanner.nl, nnhardloopplanner.nl and their www.
  • runningsupporter.com, www.runningsupporter.com
  • nn-supportervanallehardlopers.nn.tdebv.nl

The box runs HAProxy 1.8 (/etc/haproxy/haproxy.cfg) as an HTTP reverse proxy:

  • :443 terminates TLS with the box's own certbot certificates (/etc/haproxy/certs/*.pem, one per certbot lineage, selected by SNI) and re-encrypts to the load balancer with the visitor's Host as SNI. :80 is forwarded in plain HTTP; Traefik does the HTTPS redirect.
  • Both frontends overwrite X-Forwarded-For and X-Real-IP with the TCP peer (%[src]), so the value Traefik and CrowdSec see cannot be spoofed by the visitor, and set X-Forwarded-Proto.
  • Logs: /var/log/haproxy.log via rsyslog (/etc/rsyslog.d/haproxy.conf listens on the chroot socket /var/lib/haproxy/dev/log).

The previous TCP-passthrough config is kept as /etc/haproxy/haproxy.cfg.tcp-passthrough.bak. Rollback: copy it back and systemctl reload haproxy. In passthrough mode the cluster only sees the box IP, so CrowdSec can then only run AppSec on that traffic, not ban individual visitors.

Certificates on the box​

cert-manager issues certificates for the same hosts inside the cluster, and the box needs its own to terminate TLS. Both use HTTP-01 and both must keep working:

  • certbot on the box uses the standalone authenticator on 127.0.0.1:8402.
  • HAProxy routes /.well-known/acme-challenge/ to that local port only while certbot runs: the pre hook sets the bk_acme_local/certbot server to ready through the HAProxy runtime socket (socat), the post hook sets it back to maint. At every other time the ACME path is forwarded to the cluster, so cert-manager's challenges succeed as before.
  • The deploy hook writes fullchain + privkey to /etc/haproxy/certs/<lineage>.pem, validates the config and reloads HAProxy.
  • Hooks live in /etc/letsencrypt/renewal-hooks/{pre,post,deploy}/10-haproxy-*.sh; the certbot-renew.timer runs daily. certbot renew --dry-run exercises the routing against the Let's Encrypt staging environment without touching the live certificates.

Adding a host behind the box​

  1. Issue a certificate on the box: certbot certonly --standalone --http-01-port 8402 -d <host> with the pre/post hooks (--pre-hook/--post-hook pointing at the scripts above), then run the deploy script once for the new lineage.
  2. Add the host to the app's ingress in this repo as usual; nothing else changes on the box, the frontend forwards every host.

Better still, point the host's DNS at the load balancer directly and skip the box.

Verifying​

  • Who is really talking to a site (VictoriaLogs, LogsQL): RequestHost:"www.supportervanallehardlopers.nl" _time:15m | stats by (ClientHost) count() — ClientHost must be visitor IPs while ClientAddr stays 93.119.12.22:*.
  • Spoofing check: ClientHost:re("[,a-zA-Z ]") _time:1h | stats count() should be zero.
  • Bouncer end to end: in the LAPI pod cscli decisions add -i <your IP> -d 2m, request a site through the box, expect 403; cscli decisions delete -i <your IP> afterwards.
  • cscli metrics in the LAPI pod shows AppSec request counts and per-scenario hits.