SEPE-1177: Package the receiver alongside the agent

One release tag ships one package containing both binaries and both
units. They share the dedup key format -- the agent writes it, the
receiver parses it back -- so letting them reach a host at different
versions would break acknowledgements in a way neither side could
detect. Shipping them together makes that impossible rather than
unlikely.

The receiver's unit is staged under /var/lib/pdagent/scripts and
deliberately not enabled. It cannot start without a webhook signing
secret, which only configuration management can place, and a failed start
inside a `set -e` postinstall scriptlet would abort before the agent is
started -- taking down outbound paging to fix nothing. This repo has
already shipped that exact failure once, for a different reason, and the
comment explaining it is still in the scriptlet. Enabling the receiver is
the deploying system's job, which also gives a dark launch: install
everywhere, enable on one host.

The package creates the pdagent-receiver account and its config directory
at 0750. A separate account is the point of a separate process: the
controls that keep an internet-facing listener away from the agent's
routing keys -- a targeted ACL on Naemon's command pipe, an nftables
egress rule matched on uid, the unit's sandbox -- are all per-user.

The archive is now scoped to the agent. The receiver is Linux-only, so
including it would give the tarball two binaries on Linux and one on
macOS, which goreleaser rejects as confusing.

The receiver builds for the same Linux architectures as the agent,
including i386. Nobody runs a monitoring host on 32-bit, but a package
that claims to be this product and silently lacks one of its two binaries
on one architecture is the kind of inconsistency found at install time.
The CI layout check caught exactly that while this was being written.

That check now asserts both binaries against the ExecStart of their own
unit, and that the receiver unit is staged rather than installed. It
exists because a bindir default once shipped a package that installed
cleanly and then failed to start.

Author: Eric Schoeller

.github/workflows/build.yml

@@ -92,9 +92,14 @@ jobs:
         run: |
           set -euo pipefail
           unit_path=$(grep -oP '(?<=^ExecStart=)\S+' init/pdagent.service)
-          echo "Unit invokes: $unit_path"
-
-          docker run --rm -v "$PWD/dist:/dist:ro" -e unit_path="$unit_path" \
+          # The receiver's ExecStart spans continuation lines, so take the
+          # first field rather than the whole invocation.
+          recv_path=$(grep -oP '(?<=^ExecStart=)\S+' init/pdagent-receiver.service)
+          echo "Agent unit invokes:    $unit_path"
+          echo "Receiver unit invokes: $recv_path"
+
+          docker run --rm -v "$PWD/dist:/dist:ro" \
+            -e unit_path="$unit_path" -e recv_path="$recv_path" \
             rockylinux:8 bash -euo pipefail -c '
               shopt -s nullglob
               rpms=(/dist/*.rpm)
@@ -110,13 +115,25 @@ jobs:
                 files=$(rpm -qpl "$rpm")
                 echo "$files"
 
-                for f in "$unit_path" /usr/local/bin/pd-send /usr/local/bin/pd-queue \
-                         /var/lib/pdagent/scripts/pdagent.service; do
+                for f in "$unit_path" "$recv_path" \
+                         /usr/local/bin/pd-send /usr/local/bin/pd-queue \
+                         /var/lib/pdagent/scripts/pdagent.service \
+                         /var/lib/pdagent/scripts/pdagent-receiver.service; do
                   if ! grep -qx "$f" <<<"$files"; then
                     echo "::error::$rpm is missing expected path $f"
                     exit 1
                   fi
                 done
+
+                # The receiver unit must ship staged, not installed. Dropping it
+                # straight into /lib/systemd/system would present a unit that
+                # cannot start until configuration management supplies a signing
+                # secret, and a failed start there would abort the postinstall
+                # scriptlet before the paging daemon is started.
+                if grep -qx "/lib/systemd/system/pdagent-receiver.service" <<<"$files"; then
+                  echo "::error::$rpm installs the receiver unit directly; it must be staged under /var/lib/pdagent/scripts"
+                  exit 1
+                fi
               done
               echo "Package layout OK across ${#rpms[@]} RPM(s)"
             '

.goreleaser.yml

@@ -6,7 +6,8 @@ before:
     - go generate ./...
 
 builds:
-  - binary: pdagent
+  - id: pdagent
+    binary: pdagent
     env:
       - CGO_ENABLED=0
     # Matches the platform set the project has historically released.
@@ -26,10 +27,45 @@ builds:
       - -X 'github.com/PagerDuty/go-pdagent/pkg/common.Commit={{.ShortCommit}}'
       - -X 'github.com/PagerDuty/go-pdagent/pkg/common.Date={{.Date}}'
 
+  # The inbound webhook receiver. Built from the same module and shipped in the
+  # same package as the agent, so the two can never disagree about the dedup key
+  # format they share -- one is the writer and the other the reader, and a
+  # version skew between them would break acknowledgements silently.
+  #
+  # Linux only. It exists to write Naemon's command pipe on a monitoring host,
+  # which is not a thing anyone does on a Mac.
+  - id: pdagent-receiver
+    binary: pdagent-receiver
+    main: ./cmd/receiver
+    env:
+      - CGO_ENABLED=0
+    goos:
+      - linux
+    # The same Linux architectures the agent ships. Not because anyone runs a
+    # monitoring host on i386, but because a package that claims to be this
+    # product and silently lacks one of its two binaries on one architecture is
+    # the kind of inconsistency that is discovered at install time.
+    goarch:
+      - amd64
+      - arm64
+      - "386"
+    ldflags:
+      - -s -w
+      - -X 'github.com/PagerDuty/go-pdagent/pkg/common.Version={{.Version}}'
+      - -X 'github.com/PagerDuty/go-pdagent/pkg/common.Commit={{.ShortCommit}}'
+      - -X 'github.com/PagerDuty/go-pdagent/pkg/common.Date={{.Date}}'
+
 # `replacements` was removed in goreleaser v2; the same artifact names are
 # now produced by templating the OS and architecture directly.
+#
+# Scoped to the agent alone. The receiver is a Linux-only server component
+# deployed from the rpm or deb, so including it here would make the archive
+# hold two binaries on Linux and one on macOS -- which goreleaser rejects, and
+# rightly, as confusing to anyone downloading it.
 archives:
-  - name_template: >-
+  - ids:
+      - pdagent
+    name_template: >-
       {{ .ProjectName }}_{{ .Version }}_
       {{- title .Os }}_
       {{- if eq .Arch "amd64" }}x86_64
@@ -74,8 +110,19 @@ nfpms:
         dst: "/var/lib/pdagent/scripts/pdagent.init"
       - src: "init/pdagent.service"
         dst: "/var/lib/pdagent/scripts/pdagent.service"
+      # Staged, not installed into /lib/systemd/system by the package. The
+      # receiver needs a signing secret the package cannot supply, so it is
+      # enabled by configuration management once that is in place. This also
+      # keeps a receiver that cannot start from aborting the postinstall
+      # scriptlet and leaving the paging daemon down.
+      - src: "init/pdagent-receiver.service"
+        dst: "/var/lib/pdagent/scripts/pdagent-receiver.service"
       - src: "scripts/pd-*"
         dst: "/usr/local/bin/"
+      - dst: /etc/pdagent-receiver
+        type: dir
+        file_info:
+          mode: 0750
       # `empty_folders` was removed in goreleaser v2; directories the agent
       # needs at runtime are now declared as dir-type contents so the package
       # still owns them and the postinstall chown has something to act on.

scripts/deb/postinstall.sh

@@ -11,6 +11,17 @@ install_init () {
 
 install_systemd () {
     cp /var/lib/pdagent/scripts/pdagent.service /lib/systemd/system
+
+    # The receiver's unit is installed but deliberately neither enabled nor
+    # started. It requires a webhook signing secret that only configuration
+    # management can place, so starting it here would fail -- and a failure
+    # under `set -e` would abort this script before the agent below is started,
+    # taking down outbound paging to fix nothing.
+    if [ -f /var/lib/pdagent/scripts/pdagent-receiver.service ]; then
+        cp /var/lib/pdagent/scripts/pdagent-receiver.service /lib/systemd/system
+    fi
+
+    systemctl daemon-reload || :
     systemctl enable pdagent
     systemctl start pdagent
 }
@@ -21,6 +32,17 @@ if [ "$1" = "configure" ]; then
         /usr/sbin/adduser --system --shell /bin/false --no-create-home \
                           --group pdagent
 
+    # The receiver runs as its own account. It listens to the public internet
+    # and needs write access to Naemon's command pipe, while the agent holds
+    # PagerDuty routing keys -- a compromise of either should not reach the
+    # other, and every control that enforces that is per-user.
+    /usr/bin/getent passwd pdagent-receiver >/dev/null || \
+        /usr/sbin/adduser --system --shell /bin/false --no-create-home \
+                          --group pdagent-receiver
+
+    chown pdagent-receiver:pdagent-receiver /etc/pdagent-receiver 2>/dev/null || :
+    chmod 0750 /etc/pdagent-receiver 2>/dev/null || :
+
     # `pdagent init` refuses to overwrite an existing config and exits
     # non-zero. The package manager preserves /etc/pdagent/config.yaml across
     # removal and upgrade, so on any reinstall or upgrade this aborted the

scripts/deb/preremove.sh

@@ -8,6 +8,15 @@ uninstall_init () {
 }
 
 uninstall_systemd () {
+    # Stop the receiver first. It may hold Naemon's command pipe open, and
+    # leaving an enabled unit behind for a binary that is being removed would
+    # make the next boot log a failure for something nobody installed.
+    if systemctl list-unit-files pdagent-receiver.service >/dev/null 2>&1; then
+        systemctl stop pdagent-receiver || :
+        systemctl disable pdagent-receiver || :
+    fi
+    rm -f /lib/systemd/system/pdagent-receiver.service
+
     systemctl stop pdagent
     systemctl disable pdagent
 }

scripts/rpm/postinstall.sh

@@ -11,6 +11,18 @@ install_init () {
 
 install_systemd () {
     cp /var/lib/pdagent/scripts/pdagent.service /lib/systemd/system
+
+    # The receiver's unit is installed but deliberately neither enabled nor
+    # started. It requires a webhook signing secret that only configuration
+    # management can place, so starting it here would fail -- and a failure
+    # under `set -e` would abort this scriptlet before the agent below is
+    # started, taking down outbound paging to fix nothing. Enabling it is the
+    # deploying system's job, once the secret exists.
+    if [ -f /var/lib/pdagent/scripts/pdagent-receiver.service ]; then
+        cp /var/lib/pdagent/scripts/pdagent-receiver.service /lib/systemd/system
+    fi
+
+    systemctl daemon-reload || :
     systemctl enable pdagent
     systemctl start pdagent
 }
@@ -19,6 +31,18 @@ install_systemd () {
 /usr/bin/getent passwd pdagent >/dev/null || \
     /usr/sbin/adduser --system --shell /bin/false --no-create-home pdagent
 
+# The receiver runs as its own account. It listens to the public internet and
+# needs write access to Naemon's command pipe, while the agent holds PagerDuty
+# routing keys -- a compromise of either should not reach the other, and every
+# control that enforces that (a targeted ACL, an nftables egress rule matched on
+# uid, the unit's sandbox) is per-user.
+/usr/bin/getent passwd pdagent-receiver >/dev/null || \
+    /usr/sbin/adduser --system --shell /bin/false --no-create-home pdagent-receiver
+
+# Only the receiver's own account may read its signing secret.
+chown pdagent-receiver:pdagent-receiver /etc/pdagent-receiver 2>/dev/null || :
+chmod 0750 /etc/pdagent-receiver 2>/dev/null || :
+
 # `pdagent init` refuses to overwrite an existing config and exits non-zero.
 # rpm preserves /etc/pdagent/config.yaml across erase and upgrade, so on any
 # reinstall or upgrade this aborted the scriptlet under `set -e` -- before the

scripts/rpm/preremove.sh

@@ -8,6 +8,15 @@ uninstall_init () {
 }
 
 uninstall_systemd () {
+    # Stop the receiver first. It may hold Naemon's command pipe open, and
+    # leaving an enabled unit behind for a binary that is being removed would
+    # make the next boot log a failure for something nobody installed.
+    if systemctl list-unit-files pdagent-receiver.service >/dev/null 2>&1; then
+        systemctl stop pdagent-receiver || :
+        systemctl disable pdagent-receiver || :
+    fi
+    rm -f /lib/systemd/system/pdagent-receiver.service
+
     systemctl stop pdagent
     systemctl disable pdagent
 }