feat: add PyO3 extension modules for heartbeat, mailbox_kernel, and message_bureau#237
Open
agnitum2009 wants to merge 5 commits into
Open
feat: add PyO3 extension modules for heartbeat, mailbox_kernel, and message_bureau#237agnitum2009 wants to merge 5 commits into
agnitum2009 wants to merge 5 commits into
Conversation
added 5 commits
July 1, 2026 02:09
…essage_bureau Introduces Rust-backed Python extension modules to reduce Python orchestration memory for stable subsystems: - ccb-heartbeat, ccb-mailbox, ccb-message-bureau (Rust implementations) - ccb-py-heartbeat (heartbeat Python binding) - ccb-py-mailbox (mailbox_kernel + message_bureau Python binding) Also brings in the supporting crates they depend on: - ccb-project, ccb-storage, ccb-storage-classification, ccb-jobs - ccb-types, ccb-runtime-env, ccb-ui-text Validation: - cargo test -p ccb-py-heartbeat -p ccb-py-mailbox -p ccb-jobs -p ccb-storage passes - cargo clippy -p ccb-py-heartbeat -p ccb-py-mailbox -p ccb-jobs -p ccb-storage clean - Python smoke tests for both extension modules pass
- Route evaluate_heartbeat through ccb_py_heartbeat when the extension is installed and CCB_HEARTBEAT_RUST is not explicitly disabled. - Fall back to the Python implementation automatically in source checkouts that do not have the Rust extension built. - Convert Rust results back to Python dataclasses so existing consumers (job_heartbeat_runtime/tick.py) need no changes. - Add tests for the env gate and backend selection.
…ility - Add record_codec compatibility layer so Rust-written mailbox, inbox, and lease records carry schema_version/record_type headers expected by Python readers. - Wire ccb-mailbox stores through record_codec for cross-language round-trips. - Extend ccb-py-mailbox bindings with apply_incremental_summary_update, upsert_mailbox_summary, rewrite_head, and Ellipsis-aware optional active_inbound_event_id mapping. - Add env-gated mailbox_kernel/__init__.py shim that delegates to the Rust extension when available and falls back to Python otherwise. Relates to PR SeemSeam#237.
Add an opt-in Rust backend for message_bureau via CCB_MESSAGE_BUREAU_RUST=1. The proven Python implementation remains the default, so production is unaffected. Rust side: - Extend ccb-py-mailbox PyO3 bindings for MessageBureauFacade and MessageBureauControlService (recording, query, ack_reply, callback edges). - Add record_codec wrappers for Message/Attempt/Reply/CallbackEdge stores so Python readers see schema_version/record_type compatible JSONL. - Serialize DeliveryScope as single/group/broadcast strings for Python parity. - Add serde aliases for DeliveryScope in ccb-jobs. - Validate target mailbox exists in queue_summary before building detail. Python side: - lib/message_bureau/__init__.py now imports the Rust extension only when CCB_MESSAGE_BUREAU_RUST=1; otherwise it re-exports the existing Python classes. Results from Rust are converted back to Python dataclasses. Validation: - cargo test -p ccb-mailbox -p ccb-py-mailbox: pass - cargo clippy -p ccb-mailbox -p ccb-py-mailbox -p ccb-jobs: clean - test_message_bureau_*.py + test_v2_message_bureau_*.py (85 tests): pass with both Python and Rust backends.
…p RSS Reduce the amount of Python code loaded at ccbd startup by only loading provider backends and request handlers that are actually needed. Provider backend filtering: - Add optional ``providers`` filter to provider_core registry/catalog helpers and provider_execution registry so only configured providers are imported. - ``builtin_backends.build_builtin_backends`` now conditionally imports each backend module instead of importing all 18 providers up front. - ``ccbd.app_runtime.bootstrap`` computes configured providers from ``config.agents`` and passes the set to ``build_default_provider_catalog`` and ``build_default_execution_registry``. - ``ccbd.app_runtime.service_graph`` builds session bindings once for the configured providers and passes them to ``RuntimeService`` and ``HealthMonitor`` instead of letting each service rebuild the full map. Local imports in hot paths: - Move provider_backends imports out of ``cli.services.provider_hooks`` top level and into the per-provider branches of ``_materialize_provider_home``, ``provider_hook_home_root``, and ``_route_claude_binary_cache_if_possible``. - Move ``provider_backends.codex...resolve_codex_home_layout`` and ``provider_hooks.activity`` imports into the functions that use them in ``ccbd.project_view``. Lazy handlers: - ``ccbd.app_runtime.handlers`` no longer imports ``ccbd.handlers`` at the top level. ``register_handlers`` now registers lightweight wrappers that import and construct the real handler on the first request. This defers loading all handler modules (and their heavy dependencies such as reload_apply_graph and provider_backends.pane_log_support) until first use. Measured improvement (PYTHONPATH=lib, import ccbd.app): - RSS delta: ~36 MB -> ~30.4 MB - provider_backends modules loaded: 454 -> 237 - ``ccbd.handlers`` no longer loaded at import time Validation: - test_ccbd*.py + test_provider*.py: 639 passed, 2 skipped - test_message_bureau_*.py + test_v2_message_bureau_*.py: 85 passed - test_ccbd_service_graph.py, test_ccbd_project_view.py, test_v2_provider_core_registry.py, test_v2_provider_catalog.py, test_provider_hook_settings.py: all passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR introduces Rust-backed Python extension modules for stable CCB subsystems to reduce Python orchestration memory:
New Rust crates:
ccb-heartbeat/ccb-mailbox/ccb-message-bureau— the underlying Rust implementations.ccb-py-heartbeat— PyO3 binding exposing the heartbeat state machine.ccb-py-mailbox— PyO3 binding exposingmailbox_kernelandmessage_bureausubmodules.Supporting crates included:
ccb-project,ccb-storage,ccb-storage-classification,ccb-jobsccb-types,ccb-runtime-env,ccb-ui-textThe extension modules return plain Python dicts for models, minimizing per-struct wrapper overhead and making them easy to adopt behind thin Python shims.
Validation:
cargo test -p ccb-py-heartbeat -p ccb-py-mailbox -p ccb-jobs -p ccb-storage -- --test-threads=1passescargo clippy -p ccb-py-heartbeat -p ccb-py-mailbox -p ccb-jobs -p ccb-storage -- -D warningscleanccb_py_heartbeatandccb_py_mailboxpass