Deploy the test-server stack, and keep it current with Watchtower

[yschimke]

test-server builds and publishes to ghcr.io/yschimke/okhttp-testbed/test-server on every push to main (#28). Nothing runs it. Until something does, the suites can only reach it through Testcontainers, and the two things a hosted instance is for — a real network path, and an endpoint the Android matrix can reach without Docker on the device — stay out of reach.

This is the deployment, plus the part that keeps a deployment from going stale: Watchtower watching the published tag and restarting the container when the digest moves.

The host

Anything that runs Docker and has a name. A 1 vCPU / 1 GB VM is generous — the server is a static Go binary with no state and no disk use beyond logs.

Requirements, in order of how much they constrain the choice:

1. A DNS name. Needed for a publicly trusted certificate, and for the suites to have something to configure other than an IP.
2. Inbound 443. See below.
3. Inbound on the remaining listener ports (or a decision to publish only some of them).
4. IPv6, ideally — Happy Eyeballs against real dual-stack and single-stack names #21 wants a dual-stack name, and GitHub-hosted runners having no IPv6 is already the limiting factor there. A v6-reachable instance doesn't fix the runner, but it means the v6 half is testable from anywhere else.

Ports

test-server/README.md covers this in full; the short version is that a non-standard port changes nothing about the results — every absolute URL the server emits is built from the Host the request arrived with, and the CI smoke test runs the image on non-default ports to keep that true.

The exception worth spending a real port on: 443 is the only port that tests the internet. Interception middleboxes, CDN behaviour and captive portals mostly act on 443, so an instance on 8443 is not being handled the way a user's traffic is. Put https there; the per-version and hostile ports can go wherever.

Listener	Container	Suggested published	Notes
https	8443	443	the one that matters
http	8080	80	plain HTTP, and the cleartext-policy cases
raw	8081	8081	request-head echo; not an HTTP server
tls10–tls13	8410–8413	8410–8413	badssl-style, non-standard by design

Certificates, and the one real conflict

The image has two modes and they are mutually exclusive today:

• Generated CA (default) — mints its own CA at startup, serves it at /ca.pem. This is what lets a test assert a chain is accepted.
• Supplied certificate (TLS_CERT_FILE, TLS_KEY_FILE) — a real, publicly trusted chain. /ca.pem 404s.

A hosted instance wants both, for different suites: a publicly trusted chain exercises the platform's own trust anchors (which is the whole point of a deployment over a container), while the fixture CA is what the positive-chain assertions need. Both currently apply to every TLS listener at once.

Three ways out, in preference order:

1. Per-listener certificates — https gets the real chain, the per-version ports keep the generated one. A small change to tlsServer, and the honest fix.
2. Two instances — one with TLS_CERT_FILE, one without, on different ports or names. No code change; two things to keep running.
3. Generated CA only, for now — the deployment adds a real network path but not real trust anchors. Fine as a first step, and worth being explicit that it is one.

Whichever, the certificate has to renew, so a deployment with a real chain needs an ACME client writing to a volume the container reads, and a restart or reload on renewal. Caddy is already in the compose stack and can do the ACME half.

Watchtower

Watchtower polls the registry, compares digests, and recreates a container when its tag moves. Since test-server:latest is republished on every push to main, that closes the loop: merge, and the deployment follows within the poll interval.

Use the maintained fork, nicholas-fedor/watchtower — published as nickfedor/watchtower on Docker Hub and ghcr.io/nicholas-fedor/watchtower, currently 1.20.3 and actively released. The original containrrr/watchtower has been quiet since 1.7.1. The fork keeps the same com.centurylinklabs.watchtower.* label namespace, so configuration and scoping carry over unchanged.

It must be scoped to test-server and nothing else. httpbin and caddy are pinned deliberately — a moved image is breakage the daily run should catch, not absorb, which is the same reason mockserver is pinned. Watchtower updating those would quietly defeat that. Scope it with WATCHTOWER_LABEL_ENABLE and label only the one service:

  test-server:
    image: ghcr.io/yschimke/okhttp-testbed/test-server:latest
    labels:
      com.centurylinklabs.watchtower.enable: "true"

  watchtower:
    image: nickfedor/watchtower:1.20.3
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      WATCHTOWER_LABEL_ENABLE: "true"      # only labelled containers
      WATCHTOWER_CLEANUP: "true"           # don't accumulate old layers on a small disk
      WATCHTOWER_POLL_INTERVAL: "300"

Things worth knowing before wiring it up:

• It needs the Docker socket, which is root-equivalent on the host. That is the real cost of this convenience. Keep the host doing nothing else, and don't expose Watchtower's own HTTP API.
• The image is public, so no registry credentials are involved. Don't add any.
• Pin Watchtower's own tag. An auto-updater that auto-updates itself is the one container where a bad release takes the recovery mechanism with it.
• Rollback is manual. If latest ships something broken, the fix is to pin the service to the previous commit tag (every build publishes :<sha> as well) and restart. Worth writing down next to the deployment rather than discovering under pressure.
• The alternative to all of this is a webhook from the publish job. Watchtower is less to build and has no inbound attack surface; a webhook is faster and needs no Docker socket. Watchtower first.

Pointing the suites at it

Once it exists, the suites need to find it, and to cope with it being down:

• A single configuration point — TESTBED_SERVER_URL or a Gradle property — defaulting to unset, so a developer with no deployment still runs the Testcontainers path.
• An Endpoint and Probe entry (Endpoint reachability preflight, and endpoint availability on the status page #7) so a suite skips with a reason when the deployment is unreachable, instead of reporting as OkHttp failing. This is exactly the case that machinery exists for, and our own server is no more entitled to a false red than anyone else's.
• The Android matrix is the strongest argument for the whole thing: a device or emulator can reach a hosted URL directly, where today it needs adb reverse and a fixture process on the host (see run-ech-test.sh).

Acceptance

• Host provisioned, with a DNS name and 443 inbound.
• Compose stack running, restart: unless-stopped, surviving a reboot.
• GET /info over the deployed name reports the expected listeners and the base URL the client sees.
• GET /tls reports the negotiated handshake and a populated offered block.
• Each of tls10–tls13 negotiates its version and refuses the others.
• Certificate story decided and documented — which of the three options above, and why.
• Watchtower (the maintained fork, pinned) scoped by label, verified: push to main, confirm the container is recreated within the poll interval and /health recovers.
• Rollback documented: pin to :<sha>, restart.
• Deployment URL wired into the suites behind a property, with a preflight probe (Endpoint reachability preflight, and endpoint availability on the status page #7).
• Something notices when it is down that isn't a failing test run.

Follows #28. Related: #8 (self-hosted fixtures), #7 (preflight), #21 (IPv6).

[yschimke]

Co-tenancy with preview.coo.ee, and which address the testbed gets

Notes from working out how this shares a host with the compose-ai-tools preview stack, which already holds 80 and 443 there behind Caddy.

Why a reverse proxy is the wrong answer here

The obvious move — put the testbed behind the Caddy that is already terminating TLS for preview.coo.ee, routed by hostname — defeats the deployment. Every assertion this server exists to support is about the connection itself:

• /tls would report Caddy's handshake, not the client's. The ClientHello would be the one Caddy saw, re-terminated. Publish the ClientHello OkHttp actually emits #17's whole point evaporates.
• The per-version listeners (TLS version and cipher policy through ConnectionSpec #14) can't work through a proxy that negotiates its own version.
• /hostile/… writes deliberately invalid framing by hijacking the socket. A proxy either normalises it or falls over — either way the client never sees the bytes under test.
• ALPN, SNI and the certificate chain would all be the proxy's.

A proxy is fine for the plain HTTP endpoints and nothing else, which is not worth the split. So the testbed needs a socket of its own.

The shape: two paths, not one

GitHub-hosted runners have no IPv6 (#21 records this). So a v6-only endpoint on 443 is unreachable from the scheduled suites, and the honest arrangement is two paths from one container:

Path	Address	Ports	Serves
Real-internet	the second IPv6 from the /64	443, 80	local runs, Android devices, anything asserting on what real networks do to a connection
CI	the shared IPv4	8081, 8410–8413, 8090	the scheduled suites, which cannot reach v6

This is not a compromise on the "443 is the only port that tests the internet" point in test-server/README.md — it gets a genuine 443 for the traffic where that matters, and keeps a reachable path for CI. If CI should also hit a real 443, that needs a second IPv4 (see pricing below).

What Hetzner gives us

IPv6	a /64 per server, free, all of it usable; only one address is configured by default
Primary IPv4	~€0.00097/hour ≈ €0.70/month, billed attached or not
IPv6-only servers	supported — the ~€0.50/month saving they advertise
Traffic, firewalls, DDoS	included

So the second address is free on v6, or ~€0.70/month if we'd rather have a second IPv4 and give CI a real 443 too.

Sources: IP pricing, Primary IP configuration, Better Stack review.

Confirming on the host, before changing anything

The fourth check decides whether the preview stack needs touching at all — Docker only publishes on IPv6 when the daemon has IPv6/ip6tables enabled, and if it doesn't, [::]:443 is already free and Caddy never sees it.

ip -6 addr show dev eth0                 # the /64 and the configured address
ip -6 route show default                 # expect gateway fe80::1
curl -6 -sS https://ifconfig.co          # outbound v6 works

sudo ss -lntp '( sport = :443 or sport = :80 )'
#   0.0.0.0:443 only        → [::]:443 free, no change to the preview stack
#   [::]:443 as well        → scope Caddy's published ports to the v4 address

docker network inspect bridge --format '{{.EnableIPv6}}'
grep -E 'ipv6|ip6tables' /etc/docker/daemon.json 2>/dev/null
# and: does a Hetzner Cloud Firewall need the new ports opened on v6?

The wiring

Host networking rather than published ports — it sidesteps Docker's IPv6/ip6tables setup, and every listener in this image takes a full listen address, so it binds exactly what it is told:

  test-server:
    image: ghcr.io/yschimke/okhttp-testbed/test-server:latest
    network_mode: host
    restart: unless-stopped
    labels:
      com.centurylinklabs.watchtower.enable: "true"
    environment:
      HTTPS_ADDR: "[2a01:4f8:xxxx:yyyy::2]:443"
      HTTP_ADDR:  "[2a01:4f8:xxxx:yyyy::2]:80"
      RAW_ADDR:   ":8081"
      TLS12_ADDR: ":8412"
      TLS13_ADDR: ":8413"
      CERT_HOSTS: "testbed.coo.ee"

Add the address persistently (cloud-init/netplan on Hetzner's images — ip -6 addr add alone does not survive a reboot, and the reboot test below is what catches that). If ss showed Caddy on [::]:443, scope it: - "203.0.113.5:443:443".

DNS: AAAA only for the testbed name

Give the testbed name an AAAA record and no A record. With an A record pointing at the shared IPv4, v4 clients land on Caddy and get the preview site, and Let's Encrypt may validate over v4 and fail. Keep the CI ports on a name that has A only.

Verification

dig +short AAAA testbed.coo.ee            # and confirm there is no A
curl -6 -sS https://testbed.coo.ee/info | jq '.observed.baseUrl, .listeners'
curl -6 -sS https://testbed.coo.ee/tls | jq '.version, .negotiatedProtocol'
curl -sS --cacert ca.pem https://<v4-name>:8413/health
curl -sS https://preview.coo.ee/readyz    # undisturbed, on both families
sudo reboot                               # the address must survive

observed.baseUrl coming back as https://testbed.coo.ee with no port is the proof it is genuinely on 443 rather than remapped or proxied.

What this forces

The certificate question in this issue stops being hypothetical. CERT_HOSTS alone keeps the fixture CA, which most assertions want; a publicly trusted chain on the same name needs an ACME client writing where TLS_CERT_FILE points, and over v6 that means LE reaching 80 or 443 on the AAAA, or DNS-01. That is the argument for per-listener certificates (option 1 above): real chain on https, fixture CA on the per-version ports, one container.

---

Generated by Claude Code