Skip to content
diggn.de
← all posts Projects · · 3 minutes

Three bugs only a real deploy could find

This site runs as a Docker stack behind Traefik. The configuration looked right three times over and wasn't, and none of it was visible on my machine.

The preview sits on a Portainer host running Swarm, with a Traefik in front of it that fetches certificates over HTTP-01. I copied the pattern from a stack already running there. The first deploy still failed, for three unrelated reasons.

The label Traefik never saw

Traefik answered with a 404 and its own default certificate. The labels were there, the rules were right, the container was up.

They just sat under deploy.labels:

diggn-de.preview.docker-compose.ymlYAML
deploy:
  replicas: 1
  restart_policy:
    condition: any
  labels:
    - 'traefik.enable=true'

Nothing wrong with that as such; it's just meant for a different Traefik. deploy.labels end up on the Swarm service object, but Traefik runs on this host as an ordinary container, so it uses the Docker provider, and that provider reads container labels. It never looks at the service object. Both spellings are valid Compose, both are accepted without complaint, and exactly one of them is read.

The stack I had copied from gets this right. I'd read past the difference, because it looks like indentation.

An error message as a password

The n8n instance in the same stack is meant to sit behind basic auth, and the stack is meant to refuse to start at all if the credentials are missing. That's what the question mark form is for:

compose.preview.ymlYAML
- 'traefik.http.middlewares.diggn-n8n-auth.basicauth.users=${N8N_UI_BASICAUTH:?set N8N_UI_BASICAUTH}'

My first version had a long German explanation in there, em dash and all. It didn't survive substitution. Instead of stopping the start, what was left of it landed in the label as the supposed htpasswd value, Traefik accepted it, and the interface was open.

The safeguard meant to stop the start opened the hole it was there to guard.

It's a short ASCII string now. A side finding along the way: Portainer passes the variable through literally and doesn't resolve $$, so the bcrypt hash belongs in there with single $ signs — doubled, the login fails.

The deploy that reported success without looking

The third one was the most comfortable. The CI job called git/redeploy on the Portainer API and then reported success. Except this is a file stack: the content comes from this repository, not from a git remote Portainer knows about. The call went through and did nothing.

The right call is PUT /api/stacks/:id with stackFileContent, plus registries and the id of the stored credentials — without those, Swarm won't pull the private image and rejects the tasks with "No such image".

The part I now think matters most is the job's last step. It no longer stops at the API accepting the redeploy. It asks the real URL until it answers 200:

.gitlab-ci.ymlShell
for i in $(seq 1 20); do
  code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 10 https://www.diggn.preview.diggn.de/ || echo 000)
  [ "$code" = "200" ] && { echo "Preview antwortet mit 200."; exit 0; }
  sleep 15
done

All three have the same shape. Nothing on my machine could have shown them: there was no Traefik reading labels and no Portainer substituting variables, and no API to politely ignore a call. Whether a configuration is right only shows once something real answers on the other end. That's what the job checks now.

#docker#traefik#ci

Related