Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 171 additions & 0 deletions docs/proposals/template-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,171 @@
# ROCK Template 管理设计文档

## 1. 概述

Template 管理提供 Sandbox 预热池的创建与查询能力。调用方通过 operator 方法提交镜像和资源规格创建 Template,获得稳定的 `templateID`。该 ID 对应一个 OpenSandbox Pool CRD,由 Pool Controller 自动维护热备 Pod。后续创建 Sandbox 时,`templateID` 作为 `poolRef` 被引用,实现秒级分配。

Template 能力以 operator 方法形式交付,不以 HTTP 接口暴露。上层应用(如 OpenSandbox)可自行包装为 HTTP 接口。

| Operator 方法 | 说明 |
| --- | --- |
| `operator.create_template(spec)` | 提交 Template 创建请求,同步返回 `templateID` 及状态 |
| `operator.get_template_status(template_id)` | 查询 Template 当前状态 |
| `operator.scale_template(template_id, capacity)` | 调整 Template 容量(PATCH 语义) |
| `operator.delete_template(template_id)` | 删除 Template(= 删除 Pool CRD) |

### 设计要点

- **无独立存储**:K8s Pool CRD 即为持久化层,Informer 本地缓存提供高效读取,无需 DB。
- **同步创建**:Pool CRD 创建不耗时,请求内同步完成,无需后台异步任务。
- **天然去重**:相同 spec 生成相同 `templateID`(= Pool Name),K8s API Server 通过 409 Conflict 自动去重。容量字段不参与 `templateID` 计算。
- **模版渲染**:Pool 的 PodTemplateSpec、capacitySpec 等通过 YAML 模版配置 + Jinja2 渲染。

### 概念说明

Template 遵循 E2B 风格概念,以 operator 方法形式交付(不以 HTTP 接口暴露)。Template 是统一抽象——描述 Sandbox Pod 如何被创建的配置。差异只在交付机制:

| | Warm(当前实现) | Cold(未来可选,不实现) |
| --- | --- | --- |
| 底层资源 | Pool CRD | BatchSandbox template config |
| Pod 何时创建 | 预创建热备,引用时秒级分配 | Sandbox 请求时即时创建 |
| templateID 映射 | `tpl-xxx` → Pool Name | template name → config key |
| 状态查询 | 需查 Pool reconcile 进度 | 天然 `ready`(无需预热) |

Operator 内部将 Template 概念转换为 Pool 操作:

| 外部概念 | 内部映射 | K8s 资源 |
| --- | --- | --- |
| Template | Pool | Pool CRD |
| templateID | Pool Name | Pool.metadata.name |
| Template status | Pool status 映射 | Pool.status |

## 2. 整体设计

### 2.1 链路

```
创建 Template
├─ 生成 templateID = tpl- + sha256(非空 spec 字段)[:16]
├─ 从 pool_template 配置渲染 Pool CRD manifest
├─ K8sApiClient(Pool).create_custom_object()
│ └─ 409 → Pool 已存在,返回已有 templateID + status
└─ 返回 templateID + status

查询 Template
├─ templateID = Pool Name
├─ K8sApiClient(Pool).get_custom_object() ← Informer 本地缓存
├─ 映射 Pool status → template status
└─ 返回 templateID + status + timestamps

删除 Template
├─ templateID = Pool Name
├─ K8sApiClient(Pool).delete_custom_object()
│ └─ 404 → not found = already deleted
└─ K8s GC 自动清理 Pool 拥有的 Pod

扩缩容 Template
├─ templateID = Pool Name
├─ 校验容量字段(pool_min/pool_max/buffer_min/buffer_max)
├─ 仅将提供的非空字段 PATCH 到 Pool.spec.capacitySpec
├─ K8sApiClient(Pool).update_custom_object()
└─ 返回更新后的 templateID + status + capacity
```

### 2.2 templateID 生成

覆盖完整 spec 的**非空字段**,忽略 null/空值,保证前向兼容——后续新增字段传入非空值时自动参与哈希,传入 null 时不影响已有映射。

- 格式:`tpl-{sha256(排序后的非空字段)[:16]}`
- 前缀用连字符 `-`(符合 K8s RFC 1123 命名规范)
- 参与哈希的非空字段:`from_image`、`cpu_count`、`memory_mb`;`disk_gb`/`num_gpus`/`accelerator_type`/`os` 为 null 时不参与哈希
- 容量字段(`pool_min`/`pool_max`/`buffer_min`/`buffer_max`)被排除在外,因为容量不影响 Template 身份

### 2.3 状态映射

Pool CRD status 字段为 `available`/`total`(非 `readyReplicas`/`replicas`):

| 条件 | Template status | 说明 |
| --- | --- | --- |
| Pool 不存在 | 404 | templateID 无效 |
| `available > 0` | `ready` | 至少一个热备 Pod 就绪 |
| `available == 0 && total > 0` | `building` | Pod 已创建但未就绪 |
| `available == 0 && total == 0` | `building` | Controller 尚未处理或 Pool 为空 |
| conditions 中 `Ready=False` | `error` | Pool 创建失败 |

返回的 Template status 中还包含容量信息,结构如下:

```json
{
"capacity": {
"spec": {
"poolMin": ...,
"poolMax": ...,
"bufferMin": ...,
"bufferMax": ...
},
"status": {
"available": ...,
"total": ...,
"allocated": ...
}
}
}
```

### 2.4 Pool 标识

Template API 创建的 Pool 携带 Label `rock.sandbox/managed-by: template-api`,与 Nacos 配置的系统 Pool 区分。

### 2.5 关于 Update

不支持修改 Template 的 spec 字段(镜像、CPU、内存等)。templateID 由这些 spec 字段的 hash 生成,改 spec 即产生新 templateID,本质是 create 而非 update。配置变更场景:创建新 Template → 更新引用方配置 → 删除旧 Template。

容量字段(`pool_min`/`pool_max`/`buffer_min`/`buffer_max`)可通过 `operator.scale_template()` 单独调整,采用 PATCH 语义:仅更新提供的非空字段,允许缩容到 0,provider 层校验字段合法性与 `min <= max`。

### 2.6 冒烟测试

Template 不再以 HTTP 接口暴露,原 HTTP 冒烟测试已 skip。后续以 operator 方法调用形式重写冒烟测试。

## 3. 实现要点

### 3.1 Pool 模版配置

`K8sConfig.pool_template` 字段(YAML 配置 + Jinja2 渲染)生成 Pool CRD spec。关键约定:

- capacitySpec 使用 **camelCase** 键(`bufferMin`/`bufferMax`/`poolMin`/`poolMax`),匹配 Pool CRD spec
- Jinja2 变量需加**双引号**(`"{{ from_image }}"`),避免 YAML flow mapping 解析错误
- 渲染上下文:`from_image`、`cpu_count`、`memory_mb`(必需);`disk_gb`/`num_gpus`/`accelerator_type`/`os`(可选,为 null 时不传入 ctx)。容量字段不来自 `TemplateSpec`,由 `pool_template` 默认配置提供

### 3.2 关键常量

- `TEMPLATE_ID_PREFIX = "tpl-"`(RFC 1123 连字符)
- `LABEL_MANAGED_BY = "rock.sandbox/managed-by"`,值为 `"template-api"`
- Pool CRD:plural=`pools`,kind=`Pool`

### 3.3 BatchSandboxProvider 扩展

Provider 新增 Pool informer(复用同一 `ApiClient`,独立 watch `pools` CRD),并提供四组方法:

| 外部方法(template 术语) | 内部方法(pool 术语) | 行为 |
| --- | --- | --- |
| `create_template(spec)` | `_create_pool(name, spec)` | 渲染 manifest → create_custom_object;409 时返回已有 Pool |
| `get_template_status(id)` | `_get_pool(name)` | 从 Informer 缓存读取 Pool → 映射 status;未找到返回 None |
| `scale_template(id, capacity)` | — | 校验字段 → PATCH capacitySpec → 返回更新后的 status |
| `delete_template(id)` | `_delete_pool(name)` | delete_custom_object;404 视为已删除 |

辅助方法:`_build_pool_manifest_from_template`(渲染)、`_map_pool_to_template_status`(状态映射)。

## 4. 涉及文件

| 文件 | 变更 |
| --- | --- |
| `rock/sandbox/operator/k8s/constants.py` | 新增 Pool CRD 常量、Label 常量、`TEMPLATE_ID_PREFIX` |
| `rock/sandbox/operator/k8s/provider.py` | 新增 Pool informer、template/pool 转换方法、`TemplateSpec`(含 `os` 字段)、`scale_template` |
| `rock/sandbox/operator/k8s/template_loader.py` | 新增 `build_pool_manifest`,渲染上下文含 `os` 变量 |
| `rock/sandbox/operator/k8s/operator.py` | 新增 `create_template`/`get_template_status`/`scale_template`/`delete_template` 委托方法 |
| `rock/sandbox/operator/abstract.py` | 新增 `create_template`/`get_template_status`/`scale_template`/`delete_template` 抽象方法默认实现 |
| `rock-conf/rock-junxin.yml` | 新增 `pool_template` 配置段 |
| `tests/unit/sandbox/operator/test_k8s_template_provider.py` | 单元测试:ID 生成、Spec、状态映射、scale、常量 |
| `tests/unit/sandbox/operator/test_k8s_template_loader.py` | 单元测试:`build_pool_manifest` 渲染 |
| `tests/smoke/test_template.py` | 冒烟测试(已 skip,待以方法调用形式重写) |

6 changes: 6 additions & 0 deletions rock/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -389,6 +389,12 @@ class K8sConfig:
# ROCK_IMAGE_AUTH_KEY environment variable if not set here.
image_auth_key: str | None = None

# Pool rendering templates for Template API (Warm path).
# Keyed by name (e.g. "default", "windows"); rendered via Jinja2 to produce
# Pool CRD spec. Selected by TemplateSpec.os at create time, falling back to
# "default".
pool_templates: dict[str, dict] = field(default_factory=dict)

# ============================================================================
# DEPRECATED: The following fields are deprecated and will be removed in a
# future version. Do NOT use them in new code.
Expand Down
45 changes: 45 additions & 0 deletions rock/sandbox/operator/abstract.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Any

from rock.actions.sandbox.sandbox_info import SandboxInfo

if TYPE_CHECKING:
from rock.sandbox.operator.k8s.provider import TemplateSpec
from rock.admin.core.redis_key import alive_sandbox_key
from rock.common.constants import StopReason
from rock.config import RuntimeConfig
Expand Down Expand Up @@ -67,6 +71,47 @@ async def get_remote_status(self, sandbox_id: str, host_ip: str):

return ServiceStatus()

# ========================================================================
# Template API (Warm path) — default: raise NotImplementedError
# ========================================================================

async def create_template(self, spec: Any) -> dict:
"""Create or reuse a template (Pool CRD).

Only K8sOperator supports this; other operators raise BadRequestRockError.
Returns a dict with template_id and status.
"""
from rock.sdk.common.exceptions import BadRequestRockError

raise BadRequestRockError(f"template not supported on {type(self).__name__}")

async def get_template_status(self, template_id: str) -> dict | None:
"""Get template (Pool) status.

Only K8sOperator supports this; other operators raise BadRequestRockError.
"""
from rock.sdk.common.exceptions import BadRequestRockError

raise BadRequestRockError(f"template not supported on {type(self).__name__}")

async def delete_template(self, template_id: str) -> bool:
"""Delete template (Pool CRD).

Only K8sOperator supports this; other operators raise BadRequestRockError.
"""
from rock.sdk.common.exceptions import BadRequestRockError

raise BadRequestRockError(f"template not supported on {type(self).__name__}")

async def scale_template(self, template_id: str, capacity: dict[str, Any]) -> dict:
"""Scale a template's Pool capacity.

Only K8sOperator supports this; other operators raise BadRequestRockError.
"""
from rock.sdk.common.exceptions import BadRequestRockError

raise BadRequestRockError(f"template not supported on {type(self).__name__}")

def set_redis_provider(self, redis_provider: RedisProvider):
self._redis_provider = redis_provider

Expand Down
11 changes: 11 additions & 0 deletions rock/sandbox/operator/k8s/constants.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,4 +31,15 @@ class K8sConstants:
# Nacos config keys
NACOS_POOLS_KEY = "pools"
NACOS_TEMPLATE_RULES_KEY = "template_rules"

# Pool CRD (Warm path: Template API creates Pool CRD)
CRD_PLURAL_POOL = "pools"
CRD_KIND_POOL = "Pool"

# Label: distinguish Template API created Pools from system-configured Pools
LABEL_MANAGED_BY = "rock.sandbox/managed-by"
LABEL_MANAGED_BY_TEMPLATE_API = "template-api"

# templateID prefix
TEMPLATE_ID_PREFIX = "tpl-"
K8S_ALIVE_CHECK_SWITCH = "k8s_alive_check_enabled"
49 changes: 48 additions & 1 deletion rock/sandbox/operator/k8s/operator.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
"""K8s Operator implementation for managing sandboxes via Kubernetes."""

from typing import Any

from rock.actions.sandbox.sandbox_info import SandboxInfo
from rock.common.constants import StopReason
from rock.config import K8sConfig
from rock.deployments.config import DockerDeploymentConfig
from rock.logger import init_logger
from rock.sandbox.operator.abstract import AbstractOperator
from rock.sandbox.operator.k8s.constants import K8sConstants
from rock.sandbox.operator.k8s.provider import BatchSandboxProvider, TemplateFiberPoolLookup
from rock.sandbox.operator.k8s.provider import BatchSandboxProvider, TemplateSpec, TemplateFiberPoolLookup

logger = init_logger(__name__)

Expand Down Expand Up @@ -149,3 +151,48 @@ async def stop(self, sandbox_id: str, reason: StopReason = StopReason.MANUAL) ->
async def delete(self, config: DockerDeploymentConfig, host_ip: str | None = None) -> bool:
"""Treat delete as successful because stop already removes the K8s resource."""
return True

# ========================================================================
# Template API (Warm path)
# ========================================================================

async def create_template(self, spec: TemplateSpec) -> dict:
"""Create or reuse a template (Pool CRD).

Returns a dict with template_id and status.
"""
return await self._provider.create_template(spec)

async def get_template_status(self, template_id: str) -> dict | None:
"""Get template (Pool) status.

Args:
template_id: Template identifier (= Pool name)

Returns:
Status dict or None if not found
"""
return await self._provider.get_template_status(template_id)

async def delete_template(self, template_id: str) -> bool:
"""Delete template (Pool CRD).

Args:
template_id: Template identifier (= Pool name)

Returns:
True if deleted or not found, False if in use
"""
return await self._provider.delete_template(template_id)

async def scale_template(self, template_id: str, capacity: dict[str, Any]) -> dict:
"""Scale a template's Pool capacity.

Args:
template_id: Template identifier (= Pool name).
capacity: Snake-case capacity fields to update.

Returns:
Updated template status dict.
"""
return await self._provider.scale_template(template_id, capacity)
Loading
Loading