Jrepp - Backend Config Management #188
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
User request: "let's work on backend modeling for the patterns, we want to create a flat set of backends that have unique names and are known to the admin and shared to everyone in the control plane, it will be neccessary for the pattern runners to have access to the backend configuration on startup so they can map their slot implementation details to specific backend configurations"
Created comprehensive RFC for centralized backend configuration management:
## Key Design Decisions
**Flat Backend Registry**:
- All backends have globally unique names (e.g., kafka-prod, postgres-primary)
- Shared across all namespaces and patterns
- Eliminates config duplication and enables central management
**Admin-Managed with Raft**:
- Backends stored in admin FSM state (replicated via Raft)
- New admin commands: REGISTER_BACKEND, UPDATE_BACKEND, DELETE_BACKEND
- Synced to local storage (SQLite/PostgreSQL) on each admin node
**Pattern Slot Binding**:
- Patterns declare slot_bindings: {registry: "postgres-primary", messaging: "kafka-prod"}
- Pattern runners fetch backend configs from admin at startup
- SlotBinder utility creates type-specific slot implementations
**Type-Specific Configs**:
- BackendType enum: KAFKA, NATS, POSTGRES, REDIS, SQLITE, S3, MEMSTORE
- Structured configs per type (KafkaConfig, PostgresConfig, etc.)
- Credentials, connection pooling, timeouts all captured
## Example Flow
1. Operator registers backend:
`prism-admin backend register kafka-prod --brokers kafka:9092`
2. Pattern references backend:
```yaml
namespace: order-processing
pattern: multicast-registry
slot_bindings:
registry: postgres-primary
messaging: kafka-prod
```
3. Pattern runner binds slots:
- Fetches kafka-prod config from admin
- Creates KafkaMessagingSlot with connection details
- Connects to Kafka and starts processing
## Benefits
- **DRY**: One backend config used by multiple patterns
- **Centralized ops**: Change Kafka URL once, all patterns update
- **Separation of concerns**: Pattern authors don't need connection details
- **Type safety**: Structured configs with validation
- **Observability**: Admin knows which patterns use which backends
## Implementation Plan
6-phase rollout over 4 weeks:
1. Protobuf definitions
2. Admin FSM integration
3. Admin API implementation
4. Pattern runner integration
5. Testing
6. Documentation
## Open Questions
- Secret management (proposed: integrate Vault/K8s Secrets)
- Backend versioning and hot-reload (proposed: require restart initially)
- Multi-region backends (proposed: separate entries per region)
- Health monitoring (proposed: Phase 2 feature)
Builds on RFC-014 (layered patterns), RFC-017 (multicast registry slots),
RFC-035 (pattern launcher), RFC-038 (admin raft), and MEMO-004 (backend guide).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
…ration
User request: "pull in slot configuration of pattern implementations - we also want to define a type of config similar to backend with is a frontend, a frontend is a type of interface binding on the proxy, the default is the grpc pattern interface, this happens by default but can be disabled, additional itnerfaces can be added based on a front end definition - we should use the openapi semantics so that if we define a rest based front end interface that maps for example the mcp rest interface we can then bind it to specific pattern interfaces with some route config that can be consumed by the proxy to map rest interfaces to patterns on the backend, a concrete example is to expose a registry pattern as a confluent schema registry api"
Major additions to RFC-039:
1. Frontend Interface Binding Model (parallel to Backend):
- Frontend resource with globally unique names
- FrontendType enum: REST, GraphQL, gRPC-Web, SSE, WebSocket
- Type-specific configs (RestConfig, GraphQLConfig, etc.)
- RouteMapping for OpenAPI-style REST → gRPC mapping
- ParamMapping: path/query/header/body → protobuf field mapping
- ResponseMapping: protobuf → HTTP response transformation
2. Admin State Integration:
- FrontendEntry in AdminState (Raft-replicated)
- Frontend management commands (Register, Update, Delete)
- Frontend management RPCs in ControlPlane service
- Storage sync to persist frontends
3. Concrete Example: Registry Pattern as Confluent Schema Registry API:
- Complete route mappings for Confluent REST API
- POST /subjects/{subject}/versions → RegisterSchema gRPC
- GET /subjects/{subject}/versions/{version} → GetSchema gRPC
- POST /compatibility/... → CheckCompatibility gRPC
- DELETE /subjects/{subject}/versions/{version} → DeleteSchema gRPC
- Full sequence diagram showing HTTP → gRPC translation
- Python client example using Confluent SDK with Prism backend
- Benefits: API compatibility, backend flexibility, protocol translation
4. Pattern Slot Schema Integration (MEMO-006):
- Slot definitions with required/optional interfaces
- Runtime validation: backend must implement required interfaces
- Backend capability metadata (keyvalue_basic, pubsub_basic, etc.)
- SlotBinder validates interface requirements at pattern startup
- 45 thin interfaces across 10 data models (per MEMO-006)
5. Namespace Configuration Extensions:
- FrontendBinding message for namespace opt-in
- Default gRPC interface (can be disabled)
- Multiple frontends per namespace
- Namespace-specific overrides
6. Expanded Implementation Plan (8 phases, 5 weeks):
- Phase 1: Protobuf definitions for both backend and frontend
- Phase 2: Admin FSM integration for both registries
- Phase 3: Admin API implementation for both
- Phase 4: Pattern runner slot binding with schema validation
- Phase 5: Proxy frontend integration (REST adapter, route matching)
- Phase 6: Confluent Schema Registry concrete example
- Phase 7: Comprehensive testing (backend + frontend)
- Phase 8: Documentation for operators
Key design principles:
- Parallel architecture: Frontends to Backends (same admin management)
- OpenAPI semantics for route mapping (not full codegen)
- Protocol translation at proxy layer (HTTP → gRPC)
- Centralized admin management for both registries
- Default gRPC + optional additional interfaces
References added:
- RFC-020 (HTTP adapter pattern)
- RFC-032 (Confluent API compatibility)
- MEMO-006 (interface decomposition, slot schemas)
- Confluent Schema Registry API documentation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
User request: "the PR is waiting for status check but it's not available for this change, for document only changes can we run the document verification, lint, build and update the CI status?" Problem: CI workflow has paths-ignore for docs-cms/** and *.md files, so documentation-only PRs don't trigger any status checks, leaving PRs without validation. Solution: Created dedicated docs-pr.yml workflow that: - Triggers on PRs with documentation changes only - Runs uv run tooling/validate_docs.py (validates frontmatter, links, MDX) - Runs uv run tooling/build_docs.py (builds Docusaurus site) - Provides docs-status check for PR merge requirements - Uses concurrency groups to cancel stale runs Benefits: - Documentation PRs now get status checks - Validates MDX compilation before merge - Catches broken links and invalid frontmatter - Prevents GitHub Pages build failures - Independent from main CI workflow (doesn't run code tests for doc changes) Workflow triggers on: - docs-cms/** (ADRs, RFCs, MEMOs) - docusaurus/** (Docusaurus config) - **/*.md (all markdown files) - tooling/validate_docs.py, tooling/build_docs.py - .github/workflows/docs-pr.yml (self-test) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR introduces a comprehensive backend and frontend configuration registry system for the Prism data layer platform. The registry enables centralized management of backend connections (Kafka, PostgreSQL, Redis, etc.) and frontend API bindings (REST, GraphQL, gRPC-Web), separating infrastructure concerns from pattern implementation code.
Key Changes:
- Defines protobuf schemas for backend and frontend resources with type-specific configurations
- Extends the admin control plane with registry management APIs and Raft state integration
- Introduces slot binding mechanism for pattern runners to dynamically connect to backends
- Adds REST adapter pattern for exposing gRPC patterns via HTTP/REST APIs with OpenAPI-style route mappings
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| docs-cms/rfcs/RFC-039-backend-configuration-registry.md | Complete RFC defining backend/frontend registry architecture, protobuf schemas, admin integration, pattern slot binding, and Confluent Schema Registry compatibility example |
| .github/workflows/docs-pr.yml | GitHub Actions workflow for validating and building documentation on pull requests |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
mergify
bot
added
documentation
|
This PR has merge conflicts with the base branch. Please resolve them. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User request: "look at all local branches for unmerged commits, create PRs if they are found by first merging origin/main and submitting the commit data"
This branch contains 3 unmerged commit(s). Conflicts resolved automatically with aggressive strategy.
Co-Authored-By: Claude [email protected]