Register(defaultValue) makes systemplane keys leaky — production validators crash on in-code defaults

[ferr3ira-gabriel]

Revision 4: the dual-listener-in-one-process model from revision 3 was actually backwards. The pattern that already works in production in plugin-br-pix-switch is deployment-per-concern: a dedicated systemplane deployment, separate Docker image, separate K8s deployment, sharing the same codebase via a monorepo. The app deployment can crash freely while config is incomplete; the systemplane deployment stays up and serves the admin API. They never deadlock because they're physically isolated K8s objects. Reverted the in-process dual-listener proposal in favour of this proven pattern.

---

Problem 1 (root): Register(defaultValue) makes systemplane keys leaky

Summary

The Register(namespace, key, defaultValue, ...) API forces every consumer to declare an in-code default at startup. That default is used to seed the runtime cache before Client.Start() hydrates from the DB. In practice this:

1. Makes every "systemplane-managed" key a leaky abstraction — the canonical value lives in code, the DB is just a cache override.
2. Breaks production-only bootstrap validators in any consumer that runs cfg.Validate() before Client.Start() — defaults from dev mode reach the validator and crash the boot.
3. Just moves "env defaults" out of helm and into Go code — operators still can't control the value through configuration alone. The problem didn't disappear; it relocated from helm to Go source code, where it's harder to override.

We hit this concretely in plugin-br-bank-transfer (see LerianStudio/plugin-br-bank-transfer#196): a dev-mode CORS = "http://localhost:3000" default flows through Register() → seeds the cache → trips the production validator → CrashLoopBackOff. The systemplane DB is never read because Validate() already aborted bootstrap.

Why this is a platform issue, not an app issue

Seven Lerian apps currently depend on this lib (underwriter, plugin-br-bank-transfer, br-sfn x2, br-spb, br-spi, go-boilerplate-ddd). Every one of them inherits the same shape: declare a defaultValue at registration, ship the app, hope the DB-stored value wins before any validator fires.

The lib's own neutral DDL (ddl/default_seed.sql) actually does the right thing — it seeds cors.allowed_origins as "" with a comment that explicitly says "no app-specific tuning, no dev origins." But that DDL is bypassed because consumers pass cfg.Server.CORSAllowedOrigins (whatever the in-code safety net put there) into Register(), and the runtime cache is seeded from THAT, not from the DDL.

So the lib has good intent, but the API shape forces every consumer to defeat it. Anyone adding a new production-only validator on a systemplane-managed field will hit the same wall — CORS today, rate-limit floors tomorrow, signed-webhook-secrets next week.

Repro

Any consumer:

cfg := &Config{}
applyInCodeDefaults(cfg)              // cfg.X = "dev-default"
client.Register("ns", "x", cfg.X)     // seeds cache with "dev-default"
cfg.Validate()                        // rejects "dev-default" in production -> CRASH
client.Start(ctx)                     // never reached

Reference: internal/client/client.go:218-228 seeds the runtime cache directly from def.defaultValue registered by the consumer, not from the lib's neutral DDL.

Proposal for Problem 1

Stop accepting defaultValue at Register() time, or make it advisory only (recorded in the catalog for documentation/UI, never seeded into the cache).

Concrete shape:

• Register(namespace, key, opts...) — no defaultValue argument. Or keep the signature and ignore the value for cache seeding (zero-code-change migration).
• Required(true) option — marks a key as required for the app to consider itself fully configured.
• Client.Ready() / Client.ReadyKeys() — returns true only when every required key has a DB-stored value.
• Bootstrap validators run AFTER Client.Start() so DB-stored values are visible. Or: validators only fire against the systemplane snapshot, not against cfg.Field.
• ddl/default_seed.sql stays the only source of "starter values" — operator-controlled, environment-aware, never overridden by code.

In this model, the canonical value is always the DB. The catalog records what defaults the consumer suggested (for documentation), but those suggestions never silently take effect.

Why this isn't "just an opinion" — the current shape is structurally broken

The lib registered cors.allowed_origins (and 50+ other keys) as systemplane-managed in plugin-br-bank-transfer, but added the env var to an ignoredSystemplaneEnvVars allowlist. That means there is literally no operator-controlled mechanism to set this value before the bootstrap validator runs. The only way to populate the value is via the admin API, which only exists after Client.Start(), which only runs after Validate() passes. It's a deadlock that no consumer can resolve through configuration.

Backwards compatibility

Two migration paths:

1. Breaking v2 — new Register(namespace, key, opts...) (no defaultValue), deprecate old form, major version bump.
2. Semantic shift (non-breaking) — keep the Register(...) API but change behaviour: defaultValue is recorded in the catalog (for /admin/keys listing and DDL generation) but is never used to seed the cache. Cache is hydrated exclusively from DB. Zero code change required in consumers.

---

Problem 2 (secondary): once defaults are removed, how does the app come up for the first time?

This problem only exists because Problem 1 is being fixed. Today, the app boots with whatever the in-code defaults are (broken in production, but it does boot). If we remove the defaults, the app comes up "unconfigured" on first deploy and we need a sane behaviour for that state instead of CrashLoopBackOff.

Why a separate deployment for the systemplane

The systemplane is not just "another HTTP listener that needs a port." It is a different role in the cluster from the app it configures, with a different blast radius, a different security posture, a different scaling profile, and a different failure mode. Running it as a separate Kubernetes deployment isn't a stylistic preference — it's the only model where each role gets the operational properties it needs.

The arguments below stand on their own. The production precedent in plugin-br-pix-switch only confirms they hold under real load.

1. Survivability — the admin must outlive the app

The single most important property of the systemplane is that it must be reachable when the app is broken. That is exactly the moment configuration changes are needed.

In a single-process model, the admin listener dies with the app. If the app crashes in Validate() because of a missing or invalid config value, the only mechanism for fixing that config also crashes. The operator has no recourse but kubectl edit on the helm release, redeploy, wait, hope. That is a regression compared to today's broken-but-debuggable state.

In a separate-deployment model, an app crash is an app problem. The systemplane deployment keeps running, the admin API keeps serving, the operator fixes the config, the next app restart picks it up. The mechanism for recovery cannot itself depend on the thing being recovered.

2. Blast radius — bugs in one role must not take down the other

A bug in the app's request handlers, a memory leak in the consumer adapters, a goroutine deadlock in a worker, an OOM kill from a transfer storm — all of these are app problems. None of them should be able to take down the runtime configuration plane that other apps also depend on.

In a single-process model, every one of those bugs is also a systemplane outage. The processes are coupled at the OS level, sharing memory, file descriptors, goroutine schedulers.

In a separate-deployment model, the K8s scheduler enforces the isolation for free. Different pods, different cgroups, different OOM groups, different restart cycles. An app crash loop is logged and alerted; the systemplane keeps serving.

3. Security posture — the admin surface must never be on the public network path

The systemplane admin API can mutate every runtime knob the apps depend on: CORS allowlists, rate-limit floors, license gateway URLs, authentication toggles, encryption mode flags. A leak of this surface to the internet is a "game over" event for the platform.

In a single-process model, both the public API and the admin API bind to the same pod. NetworkPolicy can constrain ports but not paths reliably; Service objects map ports 1:1 to backends. Misconfiguration of a single Ingress rule could route public traffic to the admin port. The blast radius of an operator mistake is very high.

In a separate-deployment model, the admin lives in a Service that is physically separate from anything an Ingress controller can reach. ClusterIP only, NetworkPolicy at the deployment boundary, ServiceAccount with its own IAM scope. The pattern is the same one the K8s ecosystem already uses for control-plane components (kube-apiserver, etcd) — and for exactly the same reasons.

4. Scaling profile — admin and app have nothing in common

The app scales with customer traffic. It needs N replicas, autoscalers tuned for request rate or CPU, HPAs that react to spikes. The systemplane serves the operator and an occasional LISTEN/NOTIFY change feed. It needs 1-2 replicas total, no autoscaling, no spike handling.

In a single-process model, the two are coupled. If the systemplane needs an N-replica restart for any reason (rolling update, node maintenance), the app's autoscaler has to absorb that. If the app needs to scale to 30 replicas, the admin scales with it — 30 redundant admin endpoints, 30 RBAC subjects to manage, 30 NetworkPolicy targets.

In a separate-deployment model, each role gets the replica count and autoscaling policy it actually needs. The admin runs lean. The app scales freely. Neither interferes with the other.

5. Identity and permissions — they need different IAM, different secrets, different audit trails

The app needs read access to the systemplane store and the business databases. The systemplane needs write access to the systemplane store, read access to the admin authentication source, and zero access to business data. These are different ServiceAccounts, different IRSA roles, different AWS Secrets Manager paths, different audit log streams.

In a single-process model, both roles share one ServiceAccount. The union of permissions is granted to both. The principle of least privilege is violated by construction.

In a separate-deployment model, each deployment has its own ServiceAccount. The app cannot write to the systemplane store; the systemplane cannot read business data. A compromise of either role is contained to that role.

6. Release cadence — they change for different reasons, on different timelines

App releases happen weekly or daily — new features, bug fixes, dependency bumps. Systemplane releases happen rarely — schema changes, new key registrations, admin API upgrades. Coupling them in a single deployment means every app release also restarts the admin API; every admin API upgrade also rolls all the app pods.

In a separate-deployment model, each rolls on its own cadence. The app team ships features without touching the systemplane. The platform team upgrades the admin API without disturbing customer traffic.

7. Liveness and readiness probes — single /readyz cannot encode both states

K8s probes are simple by design: one probe, one binary decision. The pod is Ready or it isn't. There is no notion of "ready for admin traffic but not for public traffic."

In a single-process model, /readyz has to make a choice. Gate it on systemplane configuration → app crashes in CrashLoopBackOff because helm-set env vars are no longer the source of truth and the DB starts empty. Gate it on process liveness → public traffic flows to a half-configured app and customers see broken responses. Neither answer is correct.

In a separate-deployment model, each deployment has its own /readyz with its own semantics. The app's readyz gates on Client.Ready() (DB has all Required keys). The systemplane's readyz gates on process + DB reachability. K8s probes work as designed; no intermediate state to encode.

8. Operational mental model — one job per workload

This one is more cultural than technical, but it matters. The K8s operational mental model is "one workload, one purpose." Deployments, Services, NetworkPolicies, RBAC bindings, alerts, runbooks — all of these are easier to write, read, and operate when each artifact corresponds to one role.

"The app deployment is broken" and "the systemplane deployment is broken" are different runbooks. They escalate to different people, have different SLAs, get different alerts. Merging them into "the app pod is broken, which might be the API or the admin or both" makes every operational interaction harder.

Summary

Property	Single process, dual listeners	Separate deployments
Admin survives app crash	✗ — dies with the process	✓ — different pod
Bug containment	✗ — shared OS resources	✓ — separate cgroups, separate scheduling
Public/admin network isolation	Partial — port-level only	Full — different Service, different NetworkPolicy
Independent scaling	✗ — N admin replicas with N app replicas	✓ — 1-2 admin, N app
Least-privilege IAM	✗ — union of both roles	✓ — separate ServiceAccount per role
Independent release cadence	✗ — coupled rollouts	✓ — each rolls on its own schedule
K8s probes work cleanly	✗ — /readyz has to encode two states	✓ — one probe per role
Operational simplicity	✗ — one pod, two purposes	✓ — one workload, one purpose

Production precedent

plugin-br-pix-switch already runs three instances of this model in production (COB, DICT, SPI). Each app component has a peer systemplane deployment, built from the same codebase via a separate main.go and Dockerfile. The pattern has survived:

• Real production traffic on the app side
• Admin-mediated runtime config changes on the systemplane side
• App crash loops during early development that did not affect operator access
• Independent release cadences across the three apps and their respective systemplane components

The arguments above are why the pattern works. The pix-switch production deployment is the evidence that it works.

Architecture

                  Lerian repo (monorepo)
                          |
            +-------------+-------------+
            |                           |
            v                           v
  apps/<app>/components/         apps/<app>/components/
       api/cmd/app/main.go            systemplane/api/cmd/app/main.go
            |                           |
            v                           v
  +-------------------+         +-------------------+
  | Docker image      |         | Docker image      |
  | <app>-api         |         | <app>-systemplane |
  +-------------------+         +-------------------+
            |                           |
            v                           v
   +-----------------------------------------------------+
   |                  Kubernetes Cluster                 |
   |                                                     |
   |  +-------------------+   +-----------------------+  |
   |  | Deployment        |   | Deployment            |  |
   |  | <app>-api         |   | <app>-systemplane     |  |
   |  | N replicas        |   | 1-2 replicas          |  |
   |  | port :APP_PORT    |   | port :9091            |  |
   |  | readyz gates on   |   | readyz = 200 once     |  |
   |  |   DB-hydrated     |   |   process + DB are up |  |
   |  |   Required keys   |   | crashes only on infra |  |
   |  | CrashLoopBackOff  |   |   issues (not on app  |  |
   |  | if config missing |   |   config state)       |  |
   |  +---------+---------+   +-----------+-----------+  |
   |            |                         |              |
   |            v                         v              |
   |  +---------+---------+   +-----------+-----------+  |
   |  | Service           |   | Service               |  |
   |  | <app>-api         |   | <app>-systemplane     |  |
   |  | behind Ingress    |   | ClusterIP only        |  |
   |  | public            |   | internal access only  |  |
   |  +-------------------+   +-----------+-----------+  |
   |                                      |              |
   |                                      v              |
   |                          +-----------+-----------+  |
   |                          | NetworkPolicy         |  |
   |                          |   only operator       |  |
   |                          |   namespace +         |  |
   |                          |   cluster-admin       |  |
   |                          +-----------------------+  |
   |                                                     |
   +-----------------------------------------------------+
                          |
                          v
                +---------+---------+
                | Postgres          |
                | systemplane store |
                | (shared between   |
                |  both deployments)|
                +-------------------+

How the apps share configuration

The systemplane deployment is the single source of truth for runtime config. The app deployment connects to the same Postgres systemplane store as a read-only consumer and reacts to changes via the change feed (LISTEN/NOTIFY).

Both deployments are built from the same codebase. They differ only in their main.go and Dockerfile:

• apps/<app>/components/api/cmd/app/main.go — boots the app, validates config, fails fast if Required keys are missing in the DB
• apps/<app>/components/systemplane/api/cmd/app/main.go — boots the systemplane store, runs first-boot env seeding, serves the admin API

First-boot bootstrap — no manual intervention needed

The pix-switch pattern includes a SeedFromEnv step in the systemplane deployment: on first boot the systemplane reads selected env vars from its own pod (LOG_LEVEL, ENV_NAME, DEPLOYMENT_MODE, telemetry knobs, auth toggle, multi-tenant config, etc.) and writes them to the DB if no admin override is already present. This means:

• First deploy: operator sets env vars in helm just like today. The systemplane deployment starts, sees an empty DB, copies env values into the DB. The app deployment starts, reads from the DB, boots successfully. No human in the loop.
• Subsequent deploys: env values change in helm. Systemplane seeder compares each DB row to the registered default; if the row still equals the default (no admin override happened), the new env value is written. If an admin override is present (DB value differs from default), the env value is ignored — admin wins.
• Runtime change: admin hits the systemplane admin API. DB is updated. Change feed notifies the app deployment. App reconciles without a restart.

So helm continues to be the bootstrap source of truth for first boot, the DB takes over as runtime source of truth, and the admin API is the only mechanism for runtime changes. No "operator must run admin API before pod becomes Ready" deadlock.

Standard port across all apps

Pin the systemplane port at :9091 across every Lerian app using lib-systemplane. Rationale:

• Consistent with the existing :9090 Prometheus metrics convention. Both ports are "ops surface."
• One NetworkPolicy template works for every app.
• Helm chart values can default systemplane.port = 9091 and never need to be set.
• Operator tooling (CLI, automation scripts) hardcodes the port — no per-app drift.

The pix-switch components currently use different ports per app (DICT :4009, etc.) — this should be standardized to :9091 going forward.

Concrete shape of the lib API

The lib stays largely as-is for the deployment-per-concern model. What's needed:

• Register(namespace, key, opts...) — Problem 1's fix; no defaultValue in cache seeding.
• Required(true) option — Problem 1; marks keys whose absence in the DB should keep the app deployment Not Ready (and crash on Validate).
• Client.Ready() — Problem 1; app deployments gate /readyz on this.
• A standardized SeedFromEnv(ctx, client, mappings) error — already exists in pix-switch's pkg/systemplane/seed. Upstream into lib-systemplane so every consumer gets first-boot env seeding for free.
• Documented deployment pattern — Helm subchart template or reference manifests for the two-deployment shape, so consumers don't have to invent it.

The lib does NOT need a MountAdmin API — apps already mount the admin routes themselves (see plugin-br-pix-switch/apps/dict/components/systemplane/api/bootstrap/server.go:139-150 calling mount.MountSystemplaneAPI). The lib already exposes the building blocks; what's missing is the deployment-pattern guidance plus the seed package upstreamed.

Boot sequence in the new model (both problems addressed)

Systemplane deployment:

1. load env (deployment-level config: DB DSN, log level, etc.)
2. open Postgres connection to systemplane store
3. start admin listener on :9091
     - /livez = 200 immediately
     - /readyz = 200 once Postgres is reachable
     - /admin/sp/* serves
4. Client.Start(ctx) -> hydrate cache from DB
5. SeedFromEnv -> for each (env, key) mapping, write env value to DB if not overridden
6. ready to serve admin traffic

App deployment:

1. load env (deployment-level config only — no in-code defaults for systemplane keys)
2. connect to systemplane store as read-only consumer
3. Client.Start(ctx) -> hydrate cache from DB
4. if any Required key is missing in DB -> crash (CrashLoopBackOff)
5. Validate() runs against hydrated systemplane cache, not in-code defaults
6. start public listener on :APP_PORT
     - /readyz = 200
     - /v1/* serves

The app deployment can CrashLoopBackOff indefinitely without affecting the systemplane deployment. The operator (or first-boot env seeding) populates the DB; on the next restart cycle the app boots successfully. No deadlock.

Why this works for production AND dev

• Production: systemplane deployment runs first-boot env seeding from helm values. App deployment boots once the DB has all Required keys. No CrashLoopBackOff after the first reconciliation cycle. No "ENV_NAME=staging" workarounds.
• Dev (docker-compose, local): both binaries run as separate processes. Same pattern, smaller scale. The systemplane container can seed from a .env file just like helm.
• First deploy of a new environment: helm/Terraform sets env vars. Both deployments come up. Systemplane writes env → DB. App reads DB → boots. Operator doesn't have to manually call any admin API for a vanilla deploy. Manual admin calls are only needed for runtime changes after deploy.

---

How the two problems relate

Concern	Problem 1 (defaults)	Problem 2 (deployment-per-concern)
What it fixes	In-code defaults silently override the DDL and trip production validators	Naive "awaiting-config" model deadlocks K8s probes and exposes admin to the internet
Could it be solved without the other?	Yes — keep single-deployment if validators move after Client.Start() and a different chicken-and-egg solution is chosen	No — separate deployment doesn't help unless defaults are removed first; otherwise the in-code defaults still beat the DB-stored values
Priority	HIGH — blocking plugin-br-bank-transfer production today	Medium — only matters once Problem 1 is fixed; pix-switch already runs the model in production

Problem 1 is the root cause. Problem 2 is the consequence of fixing Problem 1 correctly. Solving Problem 2 alone (separate deployment with defaults still in code) gains nothing — the in-code defaults still flow into the cache and still crash the validators.

Asks

1. Confirm the platform view: should systemplane be authoritative (DB is source of truth) or advisory (code defaults still win on first boot)? — Problem 1.
2. If authoritative — pick a migration path for Problem 1 (breaking v2 vs semantic shift).
3. Confirm the deployment-per-concern model for Problem 2 (validated by plugin-br-pix-switch production today). Pin the standard systemplane port at :9091.
4. Upstream the pkg/systemplane/seed package from plugin-br-pix-switch into lib-systemplane so first-boot env seeding is reusable.
5. Track the immediate consumer bug in plugin-br-bank-transfer#196; the fix that lands there should be compatible with whatever the lib decides here.

Happy to draft the API change PRs (Register change + Required option + upstreaming the seed package) if direction is agreed.

References

• ddl/default_seed.sql — current neutral DDL (good shape, gets bypassed by Problem 1).
• internal/client/client.go:218-228 — cache seeded from def.defaultValue, not DDL.
• api_client.go:15-19 — current Register(namespace, key, defaultValue, opts...) signature.
• internal/client/catalog.go:38 — DefaultValue field on Definition.
• Production precedent for Problem 2:
  • plugin-br-pix-switch/apps/{cob,dict,spi}/components/systemplane/api/cmd/app/main.go — separate main.go per component
  • plugin-br-pix-switch/apps/dict/components/systemplane/api/Dockerfile — separate Docker image
  • plugin-br-pix-switch/pkg/systemplane/appboot/appboot.go — shared bootstrap wiring for the systemplane components
  • plugin-br-pix-switch/pkg/systemplane/seed/seed.go — first-boot env-to-store seeding
• Concrete consumer failure: LerianStudio/plugin-br-bank-transfer#196.

Other consumers affected (today)

App	lib-systemplane version
underwriter	v0.2.1
plugin-br-bank-transfer	v1.6.0
br-sfn (spb + spi)	v1.3.0
br-spb	v1.0.0
br-spi	v1.1.0
go-boilerplate-ddd	v1.6.0

[ferr3ira-gabriel]

Updated issue body (revision 2). Original proposal had two defects pointed out in review:

1. /readyz gated on systemplane config would deadlock against K8s probes: pod never becomes Ready -> Service has no endpoints -> operator can't reach admin API -> can't set the config -> /readyz never flips.
2. systemplane admin routes must not be exposed to the internet — they mutate production config (CORS allowlist, rate-limit floors, license gateway URL, etc.).

The revised proposal splits the HTTP surface into two listeners (public + admin) on different ports, with K8s probes and ingress rules that respect both constraints. Admin listener defaults to :9091 across all consumer apps so NetworkPolicy templates and operator tooling can rely on a single convention.

Diagram in the issue body. The 'awaiting-config' state now lives on the public listener only; the admin listener is Ready as soon as the process is up.

[ferr3ira-gabriel]

Revision 3 restructured the issue to clearly separate the two problems and rank them:

Problem 1 (root, HIGH priority): Register(defaultValue) makes systemplane keys leaky. In-code defaults silently override the lib's neutral DDL and crash production validators. This is what's blocking plugin-br-bank-transfer today.

Problem 2 (secondary, depends on Problem 1): once defaults are removed, the app needs a way to come up unconfigured without deadlocking K8s probes — hence the dual-listener model with a standardized admin port (:9091).

The dual listener doesn't help unless Problem 1 is solved first. Solving Problem 2 alone leaves in-code defaults still flowing into the cache, still crashing validators. They're independent fixes but only meaningful together — Problem 1 is the priority.

[ferr3ira-gabriel]

Revision 4 — major rewrite of Problem 2.

The previous proposal (dual-listener in one Go process) was backwards. The pattern that already runs in production at LerianStudio/plugin-br-pix-switch for the COB, DICT, and SPI components is deployment-per-concern: same codebase, separate main.go, separate Dockerfile, separate K8s deployment for the systemplane. The app deployment can crash freely while config is incomplete — the systemplane deployment stays up and serves the admin API. They never deadlock because they're physically isolated K8s objects.

Why this beats the in-process model is now argued point-by-point in the body (8 reasons covering survivability, blast radius, security posture, scaling, IAM, release cadence, K8s probe semantics, and operational mental model). The pix-switch production deployment is the empirical evidence that the model works under real load.

Problem 1 (the root cause — Register(defaultValue) making keys leaky) is unchanged from revision 3 and remains the priority. Problem 2 (separate deployment) only matters once Problem 1 is fixed.

Also added: ask to upstream plugin-br-pix-switch/pkg/systemplane/seed into lib-systemplane so first-boot env→DB seeding is reusable across all consumers without copy-pasting.

[ferr3ira-gabriel]

Companion issue opened for the migration strategy of legacy apps adopting systemplane: #30

The architectural decisions in this issue (#29) are prerequisites — #30 assumes the primitives discussed here land first, then proposes how non-consumer apps should adopt them in BYOC environments without downtime or customer coordination.

Single-release strategy is recommended over two-phase migration, with a primitives-based approach that survives helm upgrade / helm rollback in any order. Details and the list of primitives the lib must guarantee are in #30.

[lerian-studio]

This issue has had no activity for 30 days and is now marked stale. It will be closed in 7 days unless updated. Add the no-stale label to keep it open indefinitely.

Learn more about the routine and exempt labels: https://github.com/LerianStudio/github-actions-shared-workflows/blob/main/instructions/repository-routines.md