fix(tooling): make inventory.sh's derived tag and lookup single-sourced

Follow-up to the input-validation pass, closing the rest of the review:

- `validator` was special-cased in three places (a branch in hastag, plus two
  exemptions in the unknown-tag check) with `devnet-` hardcoded inside the
  branch, so a second derived tag meant touching all three and a fleet that
  names chains differently got a silent empty answer. It is now one table entry
  (prefix + disqualifying tag), and DEVNET_TAG_PREFIX names the prefix.
- A literal `validator` in the file was silently ignored, which is exactly the
  disagreement the derivation exists to rule out: the row said one thing and the
  computation another. It now names the line and refuses.
- `--tag validator` against a file carrying no chain tag at all said only "no
  host matches", when the actionable fact is that the tag can never match there.
- The file-lookup ladder was a copy of devnet-env.sh's, comment included, and had
  already drifted (return-0 vs exit-2 on nothing found). Both now call one
  `devnet_find_file`, so the two config files of a deployment cannot end up with
  two ideas of where they live.
- The known-tag list, whose whole job is to be read next to the operator's typo,
  printed in awk's hash order.

Docs: devnet.inventory is the record of which hosts exist and `SERVERS` is a
working set filled from a query against it, rather than two hand-kept host lists
with no stated precedence. Also notes that the `for h in $SERVERS` workflows rely
on bash word-splitting, which zsh does not do, and drops the em-dashes this
branch added.

Author: MegaRedHand

.claude/skills/multi-server-devnet/SKILL.md

@@ -40,8 +40,10 @@ B owns `k+1..N-1`). Everything below still applies with three changes, and
 
 Nothing about the servers is hardcoded. Establish these from the operator:
 
-- `SERVERS` — the SSH targets (any count, any names), one devnet each.
-- `SSH_USER` — login user. `docker` is invoked with `sudo`.
+- `SERVERS`: the SSH targets for the operation at hand (any count, any names),
+  one devnet each. A working set taken FROM the inventory below, not a second
+  list of hosts to keep in step with it.
+- `SSH_USER`: login user. `docker` is invoked with `sudo`.
 - Per-devnet `NODES` (validators on that server) and `SUBNETS`
   (`ATTESTATION_COMMITTEE_COUNT`); these can differ between servers.
 - A central host for Grafana + the federating Prometheus (often one of the
@@ -53,7 +55,7 @@ gitignored) instead of retyping them: the operator-side scripts source it via
 file. It is the one place a deployment's urls and Grafana ids live.
 
 **The servers themselves live in `scripts/devnet.inventory`** (copy
-`scripts/devnet.inventory.example`; gitignored) — name, ip, tags, and per-devnet
+`scripts/devnet.inventory.example`; gitignored): name, ip, tags, and per-devnet
 `NODES`/`SUBNETS`, one row per host. Query it with `scripts/inventory.sh` rather
 than reading it by hand:
 
@@ -67,9 +69,10 @@ SERVERS=${SERVERS//$'\n'/ }                                 # newlines -> spaces
 
 Tag conventions: a `devnet-*` tag names the chain a host's nodes belong to (two
 hosts sharing one means the split-chain model, *not* two like-named devnets), and
-`aggregator` is a whole-server role. `validator` is **derived** — "has a `devnet-*`
-tag and is not tagged `aggregator`" — so it can never disagree with the aggregator
-tag. An unknown or empty `--tag` exits 2 rather than returning nothing (or, worse,
+`aggregator` is a whole-server role. `validator` is **derived**: "has a `devnet-*`
+tag and is not tagged `aggregator`", so it can never disagree with the aggregator
+tag, and writing it literally in the file is an error rather than a second opinion.
+An unknown or empty `--tag` exits 2 rather than returning nothing (or, worse,
 everything): a typo that yields an empty loop reports success while doing nothing,
 and `--tag "$UNSET"` would act on hosts the caller never named.
 
@@ -151,7 +154,8 @@ redirect must run under sudo).
 ## Workflows
 
 Examples assume `SSH_USER` is set and you iterate over `SERVERS`. Per server you
-pass its own `NODES`/`SUBNETS`.
+pass its own `NODES`/`SUBNETS`. `for h in $SERVERS` splits under bash; zsh does not
+split parameter expansions, so there write `for h in $(echo $SERVERS)`.
 
 ### Pull the latest images on all servers
 ```bash

.claude/skills/multi-server-devnet/scripts/devnet-env.sh

@@ -4,6 +4,9 @@
 #
 #   . "$SCRIPT_DIR/devnet-env.sh"; devnet_load_env
 #
+# Also provides devnet_find_file, the file-lookup ladder both this and
+# inventory.sh use.
+#
 # Lookup order, first hit wins: $DEVNET_ENV, ./devnet.env, <scripts dir>/devnet.env.
 # Copy devnet.env.example -> devnet.env and fill it in (devnet.env is gitignored,
 # since it names your hosts and may point at a webhook file).
@@ -17,21 +20,32 @@
 # fine. A line that looks like an assignment but whose name isn't usable is
 # reported on stderr rather than dropped in silence, because a config line that
 # goes unread is how you deploy against the wrong deployment.
-devnet_load_env() {
-  local dir file line key val
+
+# Resolve one of the deployment's config files. First hit wins: the path in $2 (an
+# env var NAME), ./<name>, <scripts dir>/<name>. Prints the path on stdout.
+# Returns 1 when nothing exists, and 2 when the env var names a path that doesn't
+# -- an explicit path that isn't there is a typo, not a licence to fall back to
+# another deployment's file. Shared with inventory.sh so devnet.env and
+# devnet.inventory can't end up with two ideas of where they live.
+devnet_find_file() {
+  local name=$1 envvar=$2 dir explicit candidate
   dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-  file=""
-  if [ -n "${DEVNET_ENV:-}" ]; then
-    # An explicit path that doesn't exist is a typo, not a reason to silently fall
-    # back to some other devnet.env and deploy against the wrong deployment.
-    [ -f "$DEVNET_ENV" ] || { echo "DEVNET_ENV=$DEVNET_ENV does not exist" >&2; return 1; }
-    file=$DEVNET_ENV
-  else
-    for candidate in "./devnet.env" "$dir/devnet.env"; do
-      [ -f "$candidate" ] && { file=$candidate; break; }
-    done
+  explicit=${!envvar:-}
+  if [ -n "$explicit" ]; then
+    [ -f "$explicit" ] || { echo "$envvar=$explicit does not exist" >&2; return 2; }
+    printf '%s\n' "$explicit"; return 0
   fi
-  [ -n "$file" ] || return 0
+  for candidate in "./$name" "$dir/$name"; do
+    [ -f "$candidate" ] && { printf '%s\n' "$candidate"; return 0; }
+  done
+  return 1
+}
+
+devnet_load_env() {
+  local file line key val rc
+  file=$(devnet_find_file devnet.env DEVNET_ENV); rc=$?
+  [ "$rc" -eq 2 ] && return 1        # DEVNET_ENV names a missing file: reported
+  [ "$rc" -eq 0 ] || return 0        # no env file at all is not an error
 
   while IFS= read -r line || [ -n "$line" ]; do
     line=${line#"${line%%[![:space:]]*}"}          # ltrim, so an indented line is read

.claude/skills/multi-server-devnet/scripts/devnet.env.example

@@ -5,6 +5,9 @@
 # your shell overrides the value here.
 
 # --- fleet (one independent devnet per server) -------------------------------
+# devnet.inventory is the record of which hosts exist; SERVERS is just the working
+# set for these scripts, so fill it FROM a query rather than maintaining a second
+# list: SERVERS=$(bash scripts/inventory.sh --tag devnet-ab --field name) || exit
 SERVERS="host-a host-b"                 # ssh targets, any count/names
 SSH_USER=user                           # login user; docker is invoked with sudo
 

.claude/skills/multi-server-devnet/scripts/devnet.inventory.example

@@ -15,7 +15,7 @@
 #   SERVERS=$(bash scripts/inventory.sh --field name) || exit   # exit 2 = typo'd tag
 #   SERVERS=${SERVERS//$'\n'/ }                                 # newlines -> spaces
 #
-# FORMAT — whitespace-separated columns, aligned however you like. Blank lines and
+# FORMAT: whitespace-separated columns, aligned however you like. Blank lines and
 # lines whose first non-space character is '#' are ignored. Five columns:
 #
 #   name      ssh target / display name. Need not equal the machine's hostname,
@@ -28,7 +28,7 @@
 #   nodes     NODES on this server, or '-' if it runs no devnet.
 #   subnets   ATTESTATION_COMMITTEE_COUNT, or '-'.
 #
-# TAGS — free-form, with two conventions inventory.sh knows about:
+# TAGS: free-form, with two conventions inventory.sh knows about:
 #
 #   devnet-*    names the chain this server's nodes belong to. Two servers sharing
 #               one tag means they share one genesis, i.e. the split-chain model

.claude/skills/multi-server-devnet/scripts/inventory.sh

@@ -15,8 +15,10 @@
 #   --count       print how many rows matched, nothing else.
 #
 # `validator` is a DERIVED tag: "has a devnet-* tag and is not tagged aggregator".
-# It is computed, not read, so it can never disagree with the aggregator tag.
-# `--tag validator` is exactly `--tag <that devnet> --not-tag aggregator`.
+# It is computed from the row, never read from it, so it can never disagree with
+# the aggregator tag; writing it literally in the file is an error rather than a
+# second opinion. `--tag validator` is `--tag <that devnet> --not-tag aggregator`.
+# Set DEVNET_TAG_PREFIX if a fleet names its chains something other than devnet-*.
 #
 # An unknown `--tag` is an ERROR, not an empty result. A typo ('devnet5' for
 # 'devnet-5') that quietly returns no hosts turns `for h in $(inventory.sh ...)`
@@ -82,34 +84,60 @@ case $field in
   *) echo "--field $field: expected name|ip|tags|nodes|subnets|all" >&2; exit 2 ;;
 esac
 
-# An explicit path that doesn't exist is a typo, not a reason to fall back to some
-# other inventory and act on the wrong fleet -- same rule devnet-env.sh applies.
 if [ -n "$file" ]; then
+  # An explicit path that doesn't exist is a typo, not a reason to fall back to
+  # some other inventory and act on the wrong fleet.
   [ -f "$file" ] || { echo "--file $file does not exist" >&2; exit 2; }
-elif [ -n "${DEVNET_INVENTORY:-}" ]; then
-  file=$DEVNET_INVENTORY
-  [ -f "$file" ] || { echo "DEVNET_INVENTORY=$file does not exist" >&2; exit 2; }
 else
+  # Same ladder, from the same helper, that finds devnet.env: two config files of
+  # one deployment should not have two ideas of where they live. Sourcing only
+  # defines functions.
   dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-  for candidate in "./devnet.inventory" "$dir/devnet.inventory"; do
-    [ -f "$candidate" ] && { file=$candidate; break; }
-  done
-  [ -n "$file" ] || {
+  . "$dir/devnet-env.sh"
+  file=$(devnet_find_file devnet.inventory DEVNET_INVENTORY); rc=$?
+  # rc 2 means DEVNET_INVENTORY names a file that isn't there, already reported.
+  [ "$rc" -eq 2 ] && exit 2
+  [ "$rc" -eq 0 ] || {
     echo "no inventory file found (tried \$DEVNET_INVENTORY, ./devnet.inventory, $dir/devnet.inventory)" >&2
     echo "copy $dir/devnet.inventory.example to $dir/devnet.inventory and fill it in" >&2
     exit 2
   }
 fi
 
-awk -v want="${tags[*]-}" -v nowant="${nottags[*]-}" \
+# Only the derived `validator` tag depends on how chains are named, so a fleet
+# that doesn't use devnet-* can say so instead of getting an empty answer.
+chain_prefix=${DEVNET_TAG_PREFIX:-devnet-}
+
+awk -v want="${tags[*]-}" -v nowant="${nottags[*]-}" -v chain="$chain_prefix" \
     -v field="$field" -v docount="$count" -v src="$file" '
-  # Tags are matched against ",a,b," so "agg" never matches "aggregator".
+  # Tags are matched against ",a,b," so "agg" never matches "aggregator". A
+  # DERIVED tag is computed from the row instead: it carries some chain tag and
+  # is not disqualified by a role tag.
   function hastag(t, x) {
-    if (x == "validator")
-      return (index(t, ",devnet-") > 0 && index(t, ",aggregator,") == 0)
+    if (x in DPREFIX)
+      return (index(t, "," DPREFIX[x]) > 0 && !hastag(t, DEXCLUDE[x]))
     return index(t, "," x ",") > 0
   }
-  BEGIN { nw = split(want, W, " "); nn = split(nowant, NW, " ") }
+
+  # `for (k in a)` walks a hash order, and these lists get read by a human next to
+  # their own typo. Insertion sort: the tag list is a few dozen entries at most.
+  function sortkeys(a, out,   i, j, n, v) {
+    n = 0
+    for (i in a) out[++n] = i
+    for (i = 2; i <= n; i++) {
+      v = out[i]
+      for (j = i - 1; j >= 1 && out[j] > v; j--) out[j+1] = out[j]
+      out[j+1] = v
+    }
+    return n
+  }
+
+  BEGIN {
+    nw = split(want, W, " "); nn = split(nowant, NW, " ")
+    # One entry per derived tag: the tag PREFIX a row must carry, and the tag that
+    # disqualifies it. Adding a derived tag is a line here, not a new branch.
+    DPREFIX["validator"] = chain; DEXCLUDE["validator"] = "aggregator"
+  }
 
   { sub(/^[[:space:]]+/, "") }
   /^#/ || /^$/ { next }
@@ -137,9 +165,14 @@ awk -v want="${tags[*]-}" -v nowant="${nottags[*]-}" \
     }
     t = "," tags ","
 
-    # Every literal tag in the file, so a typo can be told from a real absence.
+    # Every literal tag in the file, so a typo can be told from a real absence,
+    # and where it was first written, so both can be pointed at a line.
     n = split(tags, TT, ",")
-    for (i = 1; i <= n; i++) if (TT[i] != "") seen[TT[i]] = 1
+    for (i = 1; i <= n; i++) if (TT[i] != "") {
+      if (!(TT[i] in seen)) seenrow[TT[i]] = FNR
+      seen[TT[i]] = 1
+      if (substr(TT[i], 1, length(chain)) == chain) chainseen = 1
+    }
 
     for (i = 1; i <= nw; i++) if (!hastag(t, W[i])) next
     for (i = 1; i <= nn; i++) if ( hastag(t, NW[i])) next
@@ -161,14 +194,42 @@ awk -v want="${tags[*]-}" -v nowant="${nottags[*]-}" \
     }
 
     bad = 0
-    for (i = 1; i <= nw; i++)
-      if (W[i] != "validator" && !(W[i] in seen)) {
+
+    # A derived tag written as a literal is the exact disagreement the derivation
+    # exists to rule out: the row would claim one thing and the computation
+    # another. Say which line, rather than ignoring what the operator wrote.
+    nd = sortkeys(DPREFIX, D)
+    for (i = 1; i <= nd; i++)
+      if (D[i] in seen) {
+        printf "%s:%d: '\''%s'\'' is a DERIVED tag (%s* and not %s), remove it from the file\n", \
+          src, seenrow[D[i]], D[i], DPREFIX[D[i]], DEXCLUDE[D[i]] > "/dev/stderr"
+        bad = 1
+      }
+
+    for (i = 1; i <= nw; i++) {
+      if (W[i] in DPREFIX) {
+        # Derived, so never "unknown" -- but if no row carries a chain tag it can
+        # never match, and that is a stale inventory, not a query to answer.
+        if (!chainseen) {
+          printf "no %s* tag in %s, so %s matches nothing (set DEVNET_TAG_PREFIX?)\n", \
+            chain, src, W[i] > "/dev/stderr"
+          bad = 1
+        }
+        continue
+      }
+      if (!(W[i] in seen)) {
         printf "no such tag in %s: %s\n", src, W[i] > "/dev/stderr"; bad = 1
       }
+    }
+
     if (bad) {
       printf "known tags: " > "/dev/stderr"
-      for (k in seen) printf "%s ", k > "/dev/stderr"
-      printf "(+ derived: validator)\n" > "/dev/stderr"
+      nk = sortkeys(seen, K)
+      # A derived name that leaked into the file is listed as derived, not twice.
+      for (i = 1; i <= nk; i++) if (!(K[i] in DPREFIX)) printf "%s ", K[i] > "/dev/stderr"
+      printf "(+ derived:" > "/dev/stderr"
+      for (i = 1; i <= nd; i++) printf " %s", D[i] > "/dev/stderr"
+      printf ")\n" > "/dev/stderr"
       exit 2
     }