feat(server): expose watchtower metrics API + sidecar for Uptime Kuma monitoring

Watchtower is headless (no UI), so to track up/down like the other services it
gets a sidecar fronting only its token-gated HTTP metrics API:
WATCHTOWER_HTTP_API_METRICS=true + WATCHTOWER_HTTP_API_TOKEN, served at
https://watchtower.<tailnet>.ts.net/v1/metrics. Metrics-only — NOT the update API
— so periodic polls (the 3am WATCHTOWER_SCHEDULE) keep running.

Uptime Kuma monitors it with an HTTP check + 'Authorization: Bearer <token>'
header (recipe in HTTPS.md). No homepage href — the endpoint is an API, not a UI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: ulises-c

TODO.md

@@ -198,6 +198,11 @@ and full rollout table in [linux-server/HTTPS.md](linux-server/HTTPS.md).
       `host.docker.internal:8088`. Verified 200 with no redirect, real page
       content, confirmed in browser. Don't use port `:5252` — see HTTPS.md
       Gotchas
+- [ ] watchtower — no UI; sidecar scaffolded to front only its token-gated
+      `/v1/metrics` API (`WATCHTOWER_HTTP_API_METRICS=true`). Pending live apply +
+      an Uptime Kuma HTTP monitor on `https://watchtower.<tailnet>.ts.net/v1/metrics`
+      with a Bearer-token header (recipe in HTTPS.md). Metrics-only, so the 3am
+      schedule is unaffected.
 - [x] Decide auth method: OAuth client + `tag:container` (reusing the elevated
       tailscale-proxy client) — resolved during the Forgejo rollout, see HTTPS.md
 - [x] Decide whether to retire NPM or keep it — KEEP, as the non-tailnet HTTPS

linux-server/HTTPS.md

@@ -230,6 +230,7 @@ side of the `ports:` mapping (`host:container`), not the host side.
 | homepage          | 3000  | ✅ done       | host-networked variant — keep `network_mode: host` (reaches localhost widgets), sidecar proxies via `host.docker.internal`; add the domain to `HOMEPAGE_ALLOWED_HOSTS` |
 | cockpit           | 9090  | ✅ done       | host systemd service — **sidecar-only** stack proxies `https+insecure://host.docker.internal:9090`; `cockpit.conf.example`'s `Origins` line turned out to be unnecessary in practice — see Gotchas |
 | tailscale-web     | 8088  | ✅ done       | not in the original rollout — added because the homepage Tailscale tile linked plain HTTP. `tailscale web` is a host **systemd user unit**, not a container; `ExecStart` needs `--listen 0.0.0.0:8088 --origin https://tailscale-web.<tailnet>.ts.net` so it's reachable via `host.docker.internal` and knows it's reverse-proxied. Don't use port `:5252` — see Gotchas |
+| watchtower        | 8080  | todo          | **no UI** — the sidecar fronts only watchtower's token-gated `/v1/metrics` HTTP API (enable `WATCHTOWER_HTTP_API_METRICS=true` + `WATCHTOWER_HTTP_API_TOKEN`); no homepage `href`. Monitor it in Uptime Kuma — see below |
 
 Services that also expose **non-HTTP** ports the LAN/tailnet needs (AdGuard DNS
 `:53`, Syncthing sync `:22000`, Forgejo SSH `:22`) keep those as direct
@@ -244,6 +245,25 @@ non-empty. `ts-state/` is already gitignored for every service
 (`linux-server/*/ts-state/`). Remember `docker compose up -d` (not `restart`)
 for `homepage` afterward — see Homepage links below.
 
+### Monitoring a UI-less service in Uptime Kuma (watchtower)
+
+Watchtower is a headless daemon — no UI, nothing to click. To get the same
+up/down tracking as the other services, enable its HTTP metrics API and point an
+Uptime Kuma HTTP monitor at it (metrics-only, so the `WATCHTOWER_SCHEDULE` keeps
+running — only the *update* API would disable periodic polls):
+
+1. In `watchtower/.env`: set `WATCHTOWER_API_TOKEN` (e.g. `openssl rand -hex 32`)
+   and `TS_AUTHKEY`; `docker compose up -d`.
+2. In Uptime Kuma, add an **HTTP(s)** monitor:
+   - URL: `https://watchtower.<tailnet>.ts.net/v1/metrics`
+   - Header: `Authorization: Bearer <WATCHTOWER_API_TOKEN>`
+   - Accepted status codes: `200` (an unauthenticated probe gets `401`, so the
+     header is what proves it's both up *and* reachable).
+
+The same pattern fits any future no-UI service that exposes a health/metrics
+endpoint. For a daemon with *no* endpoint at all, Uptime Kuma's "Docker Container"
+monitor (via the docker socket) checks the container's running state instead.
+
 ## Gotchas / migration
 
 - **Existing git remotes** pointing at `http://...:3300` must be updated:

linux-server/watchtower/.env.example

@@ -0,0 +1,10 @@
+# Copy this file to .env and set the values. See ../HTTPS.md for the full setup.
+
+# Bearer token guarding watchtower's HTTP API (/v1/metrics). Generate any random
+# string, e.g.: openssl rand -hex 32
+# Uptime Kuma sends it as the header:  Authorization: Bearer <this value>
+WATCHTOWER_API_TOKEN=
+
+# Tailscale auth for the sidecar — same OAuth client secret as the other sidecars
+# (needs the Auth Keys scope + tag:container in the ACL).
+TS_AUTHKEY=

linux-server/watchtower/docker-compose.yml

@@ -1,11 +1,45 @@
 services:
+  # Tailscale sidecar — joins the tailnet as `watchtower`, terminating HTTPS for
+  # https://watchtower.<tailnet>.ts.net. Watchtower has no human UI; this fronts
+  # only its token-gated /v1/metrics API so Uptime Kuma can health-check it.
+  # See ../HTTPS.md.
+  watchtower-ts:
+    image: tailscale/tailscale:latest
+    container_name: watchtower-ts
+    hostname: watchtower
+    environment:
+      - TS_AUTHKEY=${TS_AUTHKEY}
+      - TS_STATE_DIR=/var/lib/tailscale
+      - TS_SERVE_CONFIG=/config/serve.json
+      # Requires TS_AUTHKEY to be allowed to apply tag:container (OAuth client, or
+      # a tag-scoped auth key). Drop this line if using an untagged personal key.
+      - TS_EXTRA_ARGS=--advertise-tags=tag:container
+    volumes:
+      - ./ts-state:/var/lib/tailscale
+      - ./ts-serve.json:/config/serve.json:ro
+    devices:
+      - /dev/net/tun:/dev/net/tun
+    cap_add:
+      - NET_ADMIN
+    restart: unless-stopped
+
   watchtower:
     image: containrrr/watchtower:latest
     container_name: watchtower
+    # Share the sidecar's netns: tailscale serve proxies :443 to the HTTP API on
+    # 127.0.0.1:8080. The docker socket is a file mount, unaffected by the netns.
+    network_mode: service:watchtower-ts
+    depends_on:
+      - watchtower-ts
     restart: unless-stopped
     environment:
       - WATCHTOWER_SCHEDULE=0 0 3 * * *   # 3am daily (6-field cron: sec min hr dom mon dow)
       - WATCHTOWER_CLEANUP=true            # remove old images after updating
       - DOCKER_API_VERSION=1.40            # override default 1.25 — incompatible with Docker 29.x
+      # HTTP API: metrics only (NOT the update endpoint), so the schedule above
+      # keeps running — only the update API disables periodic polls. Serves
+      # GET /v1/metrics on :8080, gated by the Bearer token below.
+      - WATCHTOWER_HTTP_API_METRICS=true
+      - WATCHTOWER_HTTP_API_TOKEN=${WATCHTOWER_API_TOKEN}
     volumes:
       - /var/run/docker.sock:/var/run/docker.sock

linux-server/watchtower/ts-serve.json

@@ -0,0 +1,8 @@
+{
+  "TCP": { "443": { "HTTPS": true } },
+  "Web": {
+    "${TS_CERT_DOMAIN}:443": {
+      "Handlers": { "/": { "Proxy": "http://127.0.0.1:8080" } }
+    }
+  }
+}