[Feature] Platform Management Design

[lterrac]

Summary

Purpose

This document defines the platform-management design and its relationship with the existing PyPTO Serving layers.

The goal is to add a platform-management part to PyPTO Serving without changing the repository's core design principle: keep the shortest path from request scheduling to PyPTO/Simpler execution. Platform management should make the system more scalable and robust, but it should not become another model-execution abstraction layer.

Context

PyPTO Serving currently focuses on a minimal local inference path:

HTTP / CLI
  -> Async serving control plane
  -> Scheduler and KV cache manager
  -> Worker process
  -> PyPTO executor
  -> Simpler runtime
  -> Ascend NPU kernels

The platform proposal adds a separate layer for distributed-system management. It addresses the problems that appear when a local inference path becomes a multi-instance service:

• Workload spikes require dynamic resource scaling.
• Placement should be aware of latency, topology, and application-specific traffic patterns.
• Faults should be handled without restarting the whole service.
• The deployment should be reconfigurable so model partitions and kernels can move to better resources.

Design Split

Serving should be split into two major submodules:

Submodule	Owner	Responsibility
Platform	VNL platform work	Start and scale the distributed system, create communication channels, manage replicas, monitor health, react to faults, and expose topology/resource changes to the model layer.
Model support	PyPTO Serving model layer	Handle LLM-specific behavior: request lifecycle, batching, model partition execution, KV cache policy, prefix reuse, token scheduling, sampling, and model-specific PyPTO executor logic.

The platform submodule owns the process during bootstrap. After the initial deployment is ready, it hands request execution to the model-support submodule and becomes a passive management service. From that point on, it reacts to explicit model-support requests and health/resource events.

Examples of platform requests from the model-support layer:

• Spawn more replicas for a partition when demand exceeds capacity.
• Remove or drain replicas when demand drops.
• Create or delete data channels between partitions.
• Reconfigure model partition placement based on topology or resource availability.
• Start periodic services such as heartbeat, monitoring, and scaling decisions.
• Repair unhealthy instances and publish updated channel/endpoint metadata to the model layer.

Target Architecture

Client / API
  |
  v
Service entry layer
  |
  v
Async serving control plane
  |
  +-------------------- Platform management API --------------------+
  |                                                                 |
  |  Deployment manager                                             |
  |    - owns desired deployment state                              |
  |    - maps model partitions to instances                         |
  |    - asks runtime/cloud/Lingqu backend to spawn or remove nodes  |
  |                                                                 |
  |  Channel manager                                                |
  |    - creates device payload channels between model partitions    |
  |    - creates control channels between coordinators and replicas  |
  |                                                                 |
  |  Health and monitoring                                          |
  |    - heartbeat                                                  |
  |    - replica status                                             |
  |    - load and capacity signals                                  |
  |                                                                 |
  |  Scaling and placement policy                                   |
  |    - ramp up/down replicas                                      |
  |    - topology-aware placement                                   |
  |    - fault response and replacement                             |
  +-----------------------------------------------------------------+
  |
  v
Scheduler / KV cache / model support
  |
  v
Partition coordinators and replicas
  |
  v
PyPTO executor -> Simpler -> Ascend NPU

The platform-management API sits beside the current async serving control plane. It should not sit between the scheduler and executor for every token step. Token-level execution should remain on the current minimal path.

Feature Overview

Platform management has several responsibilities. This section keeps them at overview level so the first implementation can focus on one narrow feature without losing the larger design direction.

Host and Device Boundary

The platform layer must keep a strict boundary between host-side orchestration and device-side execution.

Host-side tasks are control tasks. A task running on the host should not execute model kernels or move tensors through host memory as the steady-state data path. Its responsibility is to launch, configure, supervise, and terminate a Simpler runtime instance for the assigned partition or replica.

Device-side runtime instances do the actual model work:

• The host task starts a Simpler runtime instance with the partition's model subset, device placement, and channel descriptors.
• PyPTO operators execute on the Ascend device through that Simpler runtime instance.
• Tensor payload channels between partitions should be created as device-side channels whenever they move activations, token tensors, logits, KV-related tensors, or other model data.
• Host-side channels should be limited to control-plane traffic: lifecycle commands, readiness, health, monitoring, placement decisions, and error reporting.
• Data should not bounce through host memory between model partitions unless there is an explicit fallback or debug mode.

This means a platform task is not the model executor itself. It is the host-side launcher and supervisor for a device-side executor.

Deployment Model

The platform layer should describe the service as a deployment graph:

• A deployment is the desired distributed application.
• A partition is a model or service stage that owns a subset of work.
• A task is the host-side launcher/supervisor for the partition's Simpler runtime instance.
• An edge is a data/control channel between partitions, with tensor payload channels placed on device when they carry model data.
• A replica is an instance that can execute a partition's work.
• A coordinator owns routing and load balancing for a partition.

This maps to the platform configuration model:

Concept	Platform role	Meaning for PyPTO Serving
Deployment	Top-level desired state	Whole serving application, including partitions, edges, request manager, heartbeat settings, and control-buffer settings.
Partition	Logical execution stage	A shard/stage of model execution, for example a pipeline stage, expert group, prefill stage, or decode stage.
Replica	Concrete runtime instance	A concrete instance assigned to run a partition.
Task	Host-side launcher	The control function that starts and supervises a partition-local Simpler runtime instance.
Edge	Communication link	A channel between producers and consumers. Tensor payload edges should be device-side; control edges can be host-side.

The deployment model needs enough metadata to describe where host control tasks run, which Simpler runtime instance they launch, where model execution happens, and where channels are placed. The detailed API proposal for this is in the deep dive below.

Coordinator and Replica Pattern

The platform uses coordinator/replica roles to reduce deployment complexity when scaling out a partition.

At a high level:

• The coordinator receives jobs from upstream partitions or the serving control plane.
• The coordinator tracks available replicas.
• The coordinator sends ready jobs to replicas over host-side control channels.
• A replica's host-side task launches or references the partition-local Simpler runtime instance.
• The Simpler runtime instance executes the partition-local model function on the device.
• Tensor outputs flow over device-side data channels when another model partition consumes them.
• The coordinator marks the replica available again and forwards completion/status metadata.

The coordinator should not inspect token-level model details or tensor payloads. It only sees requests, readiness, health, placement metadata, and channel metadata.

Channel Management

The platform layer owns channel lifecycle. Model support should request channels by intent rather than constructing distributed transport directly.

Channel placement follows the host/device boundary:

• Tensor payload channels are device-side channels. They move activations, token tensors, logits, KV-related tensors, and partition outputs directly between device-side runtime instances.
• Control channels are host-side channels. They move lifecycle commands, job metadata, readiness, health, monitoring, and errors.
• Host-mediated tensor movement is a fallback/debug path, not the normal serving path.

The channel controller should follow a desired-state reconciliation model: register desired producers and consumers, compare desired channels against actual channels, create missing channels through the selected backend's resource-exchange mechanism, and remove stale channels when they are no longer desired.

Health and Monitoring

Reliability and zero-downtime fault response are core platform goals. Heartbeat provides liveness, while monitoring provides load and capacity signals.

Recovery routing is owned by the platform:

• If a replica becomes unhealthy, the coordinator stops assigning it new jobs.
• The platform computes the replacement placement, requests a new instance if needed, and rebuilds the required host control channels and device tensor channels.
• The platform updates model support with new channel handles, endpoint descriptors, or partition routing metadata.
• Model support may pause, fail, or replay affected in-flight work according to model semantics, but it should not choose the replacement instance or reroute topology.

Useful monitoring signals include queue depth, pending requests, decode-token throughput, prefill latency, NPU memory use, KV-cache pressure, and channel backpressure.

Scaling and Placement

Dynamic scaling is a later capability built on the same deployment and channel metadata.

Scale-up can be triggered by queue depth, latency targets, token-throughput saturation, KV-cache pressure, or failed capacity. Scale-down should be drain based: mark a replica draining, stop assigning new jobs, wait for in-flight work or timeout, unregister from the coordinator, delete channels, and release backend resources.

Topology-aware placement should consider NPU topology, device memory, channel locality, partition type, and KV-cache locality. The model layer can provide pressure and locality hints, but the platform owns placement decisions.

Supported Parallelism

The platform should be able to describe and place several forms of parallelism:

• Pipeline parallelism: each pipeline stage is a partition connected by edges.
• Expert parallelism: expert subsets can be placed as partitions and selected through model-support routing logic.
• Batch parallelism: whole-model or stage replicas can be scaled and load-balanced by coordinators.
• Tensor parallelism: model support owns tensor-level execution details; platform provides placement and channels for tensor-parallel groups.

The boundary is important: platform describes placement, channels, and replica lifecycle; PyPTO Serving model support describes how a partition computes.

Deep Dive: Static Deployment API

The first implementation target should be static deployment description and startup. This is the easiest useful feature because it defines the host/device boundary and the API shape without requiring dynamic scaling, failure replacement, or topology-aware reconfiguration.

This deep dive is a proposed API shape, not an existing schema in the repository.

Goals

• Describe partitions, replicas, host tasks, Simpler runtime instances, and channels.
• Make host/device placement explicit.
• Start a fixed deployment from a desired-state spec.
• Launch one Simpler runtime instance per replica task.
• Create host control channels and device tensor channels.
• Publish runtime and channel descriptors to model support.

Non-Goals for the First API

• No automatic scale-up or scale-down.
• No automatic fault replacement.
• No topology optimizer.
• No model-layer rerouting policy.
• No host-side tensor data path except explicit fallback/debug modes.

Proposed Deployment Shape

A first proposed deployment shape can stay minimal:

{
  "name": "qwen3-14b-serving",
  "partitions": [
    {
      "name": "prefill",
      "model_range": {"layers": [0, 47]},
      "task_placement": "host",
      "runtime": "simpler",
      "execution_placement": "device",
      "parallelism": "batch",
      "replicas": 1
    },
    {
      "name": "decode",
      "model_range": {"layers": [0, 47]},
      "task_placement": "host",
      "runtime": "simpler",
      "execution_placement": "device",
      "parallelism": "batch",
      "replicas": 1
    }
  ],
  "channels": [
    {"name": "api_to_prefill", "producer": "api", "consumer": "prefill", "placement": "host", "kind": "control"},
    {"name": "prefill_to_decode", "producer": "prefill", "consumer": "decode", "placement": "device", "kind": "tensor"},
    {"name": "decode_to_api", "producer": "decode", "consumer": "api", "placement": "host", "kind": "control"}
  ],
  "heartbeat": {"enabled": true, "interval_ms": 1000, "tolerance_ms": 3000}
}

The platform API should eventually support richer partition metadata for pipeline parallelism, expert parallelism, batch parallelism, tensor parallelism, KV-cache locality, and NPU topology.

Proposed Python API

The Python API can mirror the deployment shape with typed data classes. Exact field names can change, but the placement concepts should remain explicit.

from dataclasses import dataclass
from typing import Literal


Placement = Literal["host", "device"]
ChannelKind = Literal["control", "health", "tensor"]


@dataclass(frozen=True)
class ModelRange:
    layers: tuple[int, int]


@dataclass(frozen=True)
class PartitionSpec:
    name: str
    model_range: ModelRange
    task_placement: Literal["host"]
    runtime: Literal["simpler"]
    execution_placement: Literal["device"]
    parallelism: str
    replicas: int


@dataclass(frozen=True)
class ChannelSpec:
    name: str
    producer: str
    consumer: str
    placement: Placement
    kind: ChannelKind
    capacity: int | None = None
    payload_size: int | None = None


@dataclass(frozen=True)
class HeartbeatSpec:
    enabled: bool
    interval_ms: int
    tolerance_ms: int


@dataclass(frozen=True)
class DeploymentSpec:
    name: str
    partitions: tuple[PartitionSpec, ...]
    channels: tuple[ChannelSpec, ...]
    heartbeat: HeartbeatSpec

Platform Manager API

The first platform manager API should focus on startup, shutdown, and publishing resolved descriptors:

The Python-facing PlatformManager can be implemented as bindings over the existing C++ platform runtime, keeping Python as the serving/control API while C++ realizes deployment, host task launch, channel creation, and runtime supervision.

class PlatformManager:
    def start(self, deployment: DeploymentSpec) -> "RuntimePlan": ...
    def stop(self) -> None: ...
    def get_runtime_plan(self) -> "RuntimePlan": ...

start() should validate the deployment, launch host-side tasks, start Simpler runtime instances, create channels with the requested placement, and return a resolved runtime plan.

RuntimePlan is the model layer's read-only view of what the platform created:

@dataclass(frozen=True)
class RuntimeEndpoint:
    partition: str
    replica_id: str
    host_task_id: str
    simpler_instance_id: str
    device_id: str


@dataclass(frozen=True)
class ChannelHandle:
    name: str
    producer: str
    consumer: str
    placement: Placement
    kind: ChannelKind
    handle: object


@dataclass(frozen=True)
class RuntimePlan:
    endpoints: tuple[RuntimeEndpoint, ...]
    channels: tuple[ChannelHandle, ...]

Model support should consume RuntimePlan but should not mutate it. If a later platform version replaces a replica or channel, the platform publishes a new plan or an incremental update.

Channel Creation API

For the static version, channel creation can be internal to PlatformManager.start(). If exposed, it should preserve placement:

create_channel(source_partition, target_partition, channel_kind, placement, capacity, size)
delete_channel(channel_id)
subscribe(channel_id, message_type, handler)
publish(channel_id, message)

Channel kinds and placement should distinguish:

• Device payload channels for activations, token batches, logits, KV-related movement, and partition-to-partition tensor flow.
• Host control channels for coordinator/replica commands and responses.
• Host health channels for heartbeat and status events.

Relation to PyPTO Serving Layers

Service Entry Layer

Current files:

• python/cli/main.py

Expected platform relation:

• Add platform deployment configuration options.
• Choose local-only mode or distributed platform mode.
• Initialize the platform manager before starting model-serving traffic.

The default should remain local and minimal. Distributed platform mode should be explicit.

HTTP API Layer

Current files:

• python/core/server.py

Expected platform relation:

• Continue handling OpenAI-compatible requests and health endpoints.
• Expose high-level service health that includes platform status when enabled.
• Avoid exposing internal platform controls through the public inference API unless explicitly needed.

Async Serving Control Plane

Current files:

• python/core/async_engine.py
• python/core/scheduler.py
• python/core/kv_cache.py

Expected platform relation:

• The async engine should call the platform manager for coarse-grained deployment changes, not for each token step.
• The scheduler should keep deciding request batching and token budgets.
• KV-cache policy should remain model support, but it may provide placement hints such as KV locality and memory pressure.

Model Execution Layer

Current files:

• python/core/serving_worker.py
• python/core/executor.py
• python/core/pypto_executor.py
• examples/model/qwen3_14b/runner/

Expected platform relation:

• Workers become replicas when platform mode is enabled.
• A partition replica owns a model subset and a local Simpler/PyPTO executor on the assigned device.
• The host-side task for a replica starts and supervises the Simpler runtime instance.
• Model execution happens on the Ascend device, not in the host platform task.
• The platform manager starts and stops replicas, but does not own model execution logic.

Backend Runtime Layer

Current files:

• pypto-lib/
• Simpler runtime integration

Expected platform relation:

• Simpler remains the runtime for NPU dispatch.
• Platform code should not introduce a competing model runtime.
• Platform backend adapters can target local processes, MPI-based launch, cloud-style emulation, or Lingqu infrastructure, but the model-execution path should still terminate in PyPTO/Simpler.
• The Python platform API should call into the C++ platform runtime through bindings rather than reimplementing platform orchestration in Python.
• Tensor data channels between model partitions should be created on the device side when they carry model data.
• Host runtime code should only orchestrate Simpler instances and control-plane channels.

Static Bootstrap Flow

Initial startup should follow this sequence:

parse serving and deployment config
  -> initialize platform backend
  -> validate deployment graph
  -> allocate initial instances
  -> start partition coordinators
  -> start initial replicas
  -> create host-side control channels
  -> launch Simpler runtime instances from host-side tasks
  -> create device-side tensor payload channels
  -> initialize heartbeat and monitoring on the host control plane
  -> initialize PyPTO model executors inside device-side Simpler runtime instances
  -> mark deployment ready
  -> start accepting model requests

This matches the issue discussion: the platform submodule takes ownership immediately upon launch and relinquishes request execution to model support once initialization is ready.

Future Platform APIs

Later platform work can extend the static API with coarse-grained management calls. These should remain outside the per-token hot path.

Potential future calls:

• spawn_replica(partition, resources) for dynamic scale-up or failure replacement.
• drain_replica(replica) for controlled scale-down.
• remove_replica(replica) after drain completes.
• get_topology() for topology-aware placement.
• get_health() for liveness snapshots.
• report_load(metrics) for scheduler and worker load signals.

The platform owns these decisions and publishes updated RuntimePlan data to model support after changes are applied.

Future Failure Handling

Failure handling should build on the static deployment API. When a replica fails, the platform should mark it unhealthy, stop assigning work to it, create replacement capacity, rebuild host control channels and device tensor channels, and publish an updated RuntimePlan to model support. Model support can then resume, retry, or fail affected work according to model semantics.

Coordinator failure requires stronger state handling and is a later milestone. A minimal first implementation can treat coordinator failure as partition-level failure and restart the partition's coordinator and replicas.

Non-Goals

• Do not turn PyPTO Serving into a full production serving framework.
• Do not put platform calls in the per-token execution hot path.
• Do not duplicate model-specific scheduling, KV-cache, or sampling policy inside the platform layer.
• Do not introduce a second NPU execution runtime alongside Simpler.
• Do not require distributed platform mode for the current single-node reference path.

References

• GitHub issue: [Discussion] Serving as a mixture of Platform + Model Support Submodules #13
• PyPTO Serving architecture wiki: https://github.com/hw-native-sys/pypto-serving/wiki/PyPTO-Serving-Architecture-Overview
• PyPTO Serving design philosophy wiki: https://github.com/hw-native-sys/pypto-serving/wiki/PyPTO-Serving-%E2%80%94-Design-Philosophy

Area

Executor or runtime

Motivation / Use Case

Enhance PTO serving capabilities to support distributed execution

Proposed API / Behavior

No response

Alternatives Considered

No response

Additional Context

No response

[puddingfjz]

Open Design Questions in PyPTO Serving Platform Management

This note records unresolved design ambiguities in the Platform Management
proposal from #32.

The proposal's direction is clear: add a platform-management layer for
distributed deployment, replica lifecycle, channel management, health
monitoring, scaling, and placement, while preserving the short execution path
from scheduling to PyPTO/Simpler execution.

This note is not a rejection of that direction. It focuses on the ownership and
interface questions that should be resolved before treating the proposal as
implementation-ready.

The current issue text still leaves several ownership and interface boundaries
unclear. These are not implementation details; they affect whether Platform
remains a control-plane component or slowly becomes another model-execution
abstraction layer.

1. DeploymentSpec Ownership Is Ambiguous

The issue proposes a static deployment shape:

{
  "name": "decode",
  "model_range": {"layers": [0, 47]},
  "task_placement": "host",
  "runtime": "simpler",
  "execution_placement": "device",
  "parallelism": "batch",
  "replicas": 1
}

The issue does not state who produces this object.

This is an important ownership ambiguity because the object mixes two kinds of
fields:

• model-semantic fields: decode, model_range, parallelism;
• platform/runtime fields: task_placement, execution_placement,
  runtime, replicas.

If Platform produces or mutates this spec, it may end up deciding model
partitioning and parallelism. If Model Support produces it, Platform must treat
it as desired state and only materialize it. The current issue text does not
make that boundary explicit.

2. Coordinator Semantics Need Clarification Against the Hot Path

The issue says that the Platform Management API should not sit between the
scheduler and executor for every token step.

At the same time, the coordinator/replica pattern says that a coordinator
receives jobs, tracks replicas, and sends ready jobs to replicas.

These statements need a tighter boundary. If the coordinator chooses a replica
for every decode step or every decode batch, then Platform is in the execution
hot path. If the coordinator only manages membership, health, or low-frequency
admission, then "receives jobs" and "sends jobs to replicas" need a narrower
definition.

The issue currently does not define whether the coordinator is a slow
control-plane component, a hot-path scheduling component, or a split design.
That ambiguity directly affects latency, process placement, ownership, and API
boundaries.

3. Request Dispatch and Resource Routing Need Separation

The proposal discusses coordinators, replicas, placement, channels, and job
routing, but it does not separate model-semantic dispatch from resource-level
routing.

Some dispatch decisions require Model Support semantics:

• whether a request is in prefill or decode,
• whether it hits prefix cache,
• where its KV cache currently lives,
• whether it can migrate to another replica,
• whether KV should be reloaded, offloaded, or recomputed.

Other routing decisions are resource-level:

• which replicas exist,
• which replicas are healthy,
• which replicas are draining,
• where endpoints and channels are located,
• what topology or capacity is available.

The issue does not clearly say which side owns the final request-to-replica
binding when both sets of information are needed. For example, assigning a
request to a decode replica may depend on both replica health and KV locality.
Without a clear split, Platform may need to understand prefix cache and KV
state, or Model Support may need to duplicate Platform's resource controller
state.

4. Scaling Policy Ownership Is Under-Specified

The issue assigns scaling and placement policy to Platform, which is reasonable
for resource management. However, the signals that trigger scaling are often
model-specific:

• decode queue depth,
• prefill latency,
• KV-cache pressure,
• prefix-cache locality,
• expert hotspots,
• token throughput,
• request priority.

The issue does not define how much semantic meaning Platform is expected to
understand from these signals.

This is a design risk because autoscaling can easily cross the Model
Support/Platform boundary. A policy such as "scale decode when decode queue
depth and KV pressure are high" depends on model-layer metrics. A policy such
as "place the new decode replica near the KV locality group" also depends on
model-layer state. The issue says Platform owns scaling and placement, but it
does not define the contract by which Model Support supplies the meanings,
constraints, and safety rules behind those decisions.

5. RuntimePlan Update Semantics Are Missing

The proposed API includes:

class PlatformManager:
    def start(self, deployment: DeploymentSpec) -> RuntimePlan: ...
    def stop(self) -> None: ...
    def get_runtime_plan(self) -> RuntimePlan: ...

The basic purpose of RuntimePlan is clear: it is the resolved deployment view
that Model Support consumes.

The issue does not define the consistency model for that view:

• Does RuntimePlan have a version?
• Is get_runtime_plan() a snapshot?
• Are updates full replacements or incremental patches?
• How does Model Support learn that a replica is draining or unhealthy?
• Can Platform delete a channel while Model Support may still be using it?
• What happens to in-flight requests during replacement?
• Is there a watch/subscribe API, or only polling?

This is not just an API polish issue. Scaling, draining, fault replacement, and
channel deletion all require a safe way to update the model layer's view while
requests may still be in flight. Without an update contract, Model Support can
hold stale endpoint or channel handles while Platform believes it has already
replaced or removed them.

6. Host/Device Channel Contract Is Not Concrete Enough

The issue correctly states that tensor payloads should not bounce through host
memory in the normal path. Host channels should carry control traffic, while
device-side channels should carry tensors.

The issue does not define the lower-level channel contract:

• which layer creates device tensor channels,
• what a channel handle contains,
• how Simpler runtime instances consume those handles,
• how cross-node device channels are represented,
• who owns channel deletion,
• how channel updates are synchronized with Model Support.

This missing contract can cause two classes of implementation problems.

First, a nominal "device channel" may become unusable by PyPTO/Simpler, forcing
an accidental host-mediated tensor path. That would violate the host/device
boundary stated in the issue.

Second, channel lifecycle can race with request execution. For example,
Platform may delete or replace a channel during drain or failure recovery while
Model Support still has in-flight work using the old handle.

7. First Milestone Scope Needs Tightening

The issue says the first implementation target should be static deployment.

However, the issue also discusses health monitoring, scaling, placement, fault
recovery, coordinator/replica routing, and reconciliation. These are valid
future directions, but they make the issue broad.

The scope risk is that the issue mixes a concrete first API with future control
plane capabilities. Static deployment has very different correctness
requirements from dynamic scaling and failure replacement. If the first
milestone is not constrained, implementers may make incompatible assumptions
about whether coordinator routing, health state, channel reconciliation, and
RuntimePlan updates must work from day one.

8. Control-Plane Process Boundaries Are Unspecified

The issue describes several Platform responsibilities:

• deployment management,
• channel management,
• health and monitoring,
• scaling and placement policy,
• coordinator/replica management.

It does not say whether these run in the same process, separate processes, or
per-partition/per-node controllers.

This matters because the components have different performance and isolation
requirements. A coordinator that participates in job dispatch is latency
sensitive. A health monitor and autoscaler are slower background control-plane
tasks. A replica host task supervises a local Simpler runtime. Treating all of
these as one "Platform" block hides important implementation constraints.

Summary

The Platform Management proposal is directionally aligned with PyPTO Serving's
goal: keep the model execution path minimal while adding distributed deployment
and lifecycle management.

The unresolved questions are mainly ownership and boundary questions:

• DeploymentSpec mixes model semantics and platform concerns, but its owner
  is not defined.
• Coordinator responsibilities need clarification against the requirement that
  Platform stay out of the per-token hot path.
• Request dispatch and resource routing are not separated clearly enough.
• Scaling policy depends on model-layer signals, but the Platform/Model
  Support contract for those signals is not specified.
• RuntimePlan lacks update and consistency semantics.
• Host/device channel handles and lifecycle rules are not concrete enough.
• The first milestone mixes static deployment with future dynamic-management
  features.
• Process boundaries for coordinator, health, scaling, and host tasks are not
  stated.

These questions do not invalidate the proposal. They mark the boundaries that
should be resolved before implementation starts.

[lterrac]

Hi @puddingfjz , thanks for the review. I will address all the concerns in the next days. In the meantime, as we agreed via meeting with @bumble0918 , I created a PR #34 with an initial implementation and a first set of modules. This has 2 goals:

1. have an idea of what would be the shape of the system
2. indipendently from whether we decide the platform to be a closed control loop or an abstraction layer, these building blocks are needed either way.

[lterrac]

Thanks for the detailed review. I agree these are the right boundary questions, and I do not think the current issue and PR should be read as the final platform-management design. It is a WIP first implementation meant to introduce the initial building blocks and make the shape of the system easier to discuss.

On DeploymentSpec ownership, I do not want to lock the design prematurely. I see three possible models:

1. Platform owns the full spec, including partitioning, parallelism, replicas, placement, and runtime setup. This gives Platform more autonomy, but risks making it model-aware and duplicating Model Support logic.
2. Model Support owns the model-semantic desired state: prefill/decode structure, model ranges, tensor/pipeline/expert parallelism, KV constraints, and batching assumptions. Platform receives that desired state and materializes it by allocating resources, starting tasks, creating channels, and monitoring health.
3. Ownership is split: Model Support produces a model-facing desired-state object, and Platform resolves it into a platform-facing RuntimePlan.
   The current implementation is intended to support experimentation with these boundaries. My current leaning is that Platform should not independently invent model partitioning semantics unless we explicitly decide to make it a model-aware orchestration layer.

On coordinators, I agree Platform should not participate in token-level scheduling. The coordinator is still useful, but its first role should be partition-level replica management rather than per-token or per-decode-step scheduling. It should maintain replica membership, readiness, health, drain state, endpoint/channel metadata, and coarse resource state for a partition. Model Support should consume that state and make the actual request/token scheduling decisions because it owns batching, KV locality, prefix-cache behavior, and decode semantics.

For request dispatch versus resource routing, I agree this needs a separate design pass with Model Support. The hard cases involve both Platform state and model state, for example healthy/draining replicas on one side and KV locality or migration safety on the other. I would keep final request-to-replica binding out of this first milestone.

For scaling and placement, I think the first milestone should expose explicit platform APIs/building blocks rather than require a fully autonomous control loop. Model Support or serving control code can call platform functions to start deployments, create/delete channels, spawn/drain replicas, and query health/runtime state. Autonomous scaling and placement can be layered on top later once the metrics, constraints, and safety contract between Model Support and Platform is stable.

For RuntimePlan, I agree we need concrete update semantics. A reasonable initial direction is an immutable, versioned snapshot returned by Platform. Dynamic updates should use safe state transitions such as active -> draining -> removed, and Platform should not delete channels or endpoints that are still visible to Model Support. Incremental patches, watch APIs, and acknowledgement protocols can be future work.

For host/device channels, the intended model is that Platform owns channel lifecycle from the host/control side. Platform creates/destroys channels, exchanges buffers/descriptors, and publishes handles. Simpler/PyPTO consumes those handles for execution. Platform should not interpret tensor payloads or execute model kernels. If future Ascend/Simpler APIs expose a more device-native channel creation mechanism, the Platform abstraction can adapt, but the ownership boundary should remain: Platform manages lifecycle and descriptors; Simpler/model execution consumes them.

On first milestone scope, I agree it should stay narrow. This PR should be scoped as static platform infrastructure: deployment configuration, module lifecycle, deployment broadcast, channel-controller primitives, service execution, examples, and documentation. Dynamic scaling, failure recovery, topology-aware placement, RuntimePlan update protocol, Python bindings, and model-layer integration are future milestones.

On process boundaries, the initial implementation intentionally introduces the pieces gradually. For now, these components can run inside one MPI/HiCR-based runtime so we can validate the shape of the system. Future work can split deployment management, channel management, health monitoring, autoscaling, coordinators, and replica host tasks into separate processes or per-node/per-partition controllers as needed.

So overall I agree the broader issue needs tighter ownership and interface boundaries. This issue is WIP and should be treated as the first set of platform building blocks, not the complete platform-management design.

[lterrac]

Following up on my previous comment: I have narrowed down #34 based on the feedback that the first milestone scope was too broad.

The PR now covers only the undisputed control-plane foundation — the pieces that are needed regardless of how the open design questions (DeploymentSpec ownership, request dispatch vs resource routing) are resolved.

What is in the PR

• Engine () — the control-plane kernel that manages module lifecycle (initialize / run / await / terminate / finalize) and coordinates deployer/worker startup and shutdown via HiCR RPCs. No model semantics.
• Module interface (include/modules/module.hpp, subscription.hpp) — the pure lifecycle contract that all future platform modules will implement.
• Channel primitives (include/system/channels/) — host-side Input/Output/Message/MessageTypeRegistry channels. These are the control-plane transport layer; tensor payload channels are a later step.
• Build scaffolding — Meson project with HiCR as the only subproject dependency.
• Docs — top-level platform README, system/engine, and system/channels pages.

What is explicitly deferred

• include/modules/configuration/ — the deployment/partition/edge types mix model-semantic fields (model_range, parallelism) with platform fields. Deferring until the DeploymentSpec ownership question from your review is settled.
• The three concrete modules (broadcastDeployment, channelController, service) and their examples — these depend on the configuration types and will come in a follow-up PR.
• TaskR — only needed by the service module, so removed for now.

The goal of this first PR is to land something reviewable that establishes the platform/control-plane boundary without pre-empting any of the open design decisions. Happy to get feedback on the Engine and channel interfaces before we build the modules on top.

[puddingfjz]

On DeploymentSpec ownership, I do not want to lock the design prematurely. I see three possible models:

1. Platform owns the full spec, including partitioning, parallelism, replicas, placement, and runtime setup. This gives Platform more autonomy, but risks making it model-aware and duplicating Model Support logic.
2. Model Support owns the model-semantic desired state: prefill/decode structure, model ranges, tensor/pipeline/expert parallelism, KV constraints, and batching assumptions. Platform receives that desired state and materializes it by allocating resources, starting tasks, creating channels, and monitoring health.
3. Ownership is split: Model Support produces a model-facing desired-state object, and Platform resolves it into a platform-facing RuntimePlan.

One question I still have is about the distinction between option 2 and option 3, because they look largely equivalent to me. In option 2, Platform still needs to derive some internal runtime/deployment plan from the Model Support desired state in order to allocate resources, start tasks, create channels, and track endpoints. Option 3 seems to mainly make that derived plan explicit and visible to Model Support as a RuntimePlan.

I agree that platform should not independently invent model partitioning semantics.

[lterrac]

You are right that both options require Platform to derive the same internal resolved plan. The distinction is about who gets to see it.

In option 2 the resolved plan stays internal to Platform — Model Support provides desired state, Platform materializes it, and Model Support only sees effects.

In option 3 the resolved plan is an explicit, versioned RuntimePlan object that Platform exposes. That makes it useful to more than one consumer: Model Support can use it to make routing and scheduling decisions (KV locality, drain-aware replica selection, channel handle lookup) without duplicating Platform's resource state; but the same object could also be queried externally — by an operator or a higher-level control loop — to inspect what is actually deployed at any point in time.

So option 3 is option 2 plus an explicit contract for the resolved state, which turns the deployment plan into a general observability and coordination surface rather than an internal Platform detail. Thinking about it now, option 2 is probably not ideal precisely because we want visibility into what is actually running — option 3 is the direction I am leaning.