feat: add vllm sidecar auto-registration.

vllm_sidecar/.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+*.pyc

vllm_sidecar/README.md

@@ -0,0 +1,105 @@
+# vLLM Sidecar — auto-register a vLLM instance into xllm-service
+
+A vLLM server speaks only HTTP/OpenAI. It does not write etcd, hold a lease, or
+emit heartbeats — so on its own it cannot join an xllm-service cluster. This
+sidecar runs next to a vLLM process and bridges that gap, **with no C++ changes**:
+xllm-service already discovers instances by watching the `XLLM:DEFAULT:` etcd
+prefix.
+
+It replaces the manual `demo/register_vllm.sh` (a one-shot `etcdctl put`) with a
+long-running process that keeps registration in lockstep with vLLM's health.
+
+## Lifecycle
+
+```
+            ┌─────────────┐   /health up    ┌────────────────────────────┐
+  start ──▶ │ wait health │ ───────────────▶│ grant lease(ttl) + put key │
+            └─────────────┘                 └──────────────┬─────────────┘
+                  ▲                                         │ every keepalive-interval:
+                  │  health recovers                        │   probe /health
+                  │  (re-register, new incarnation)         ▼
+            ┌─────┴───────┐  N consecutive fails   ┌──────────────────────┐
+            │ deregistered│ ◀──────────────────────│ healthy → keepalive  │
+            └─────────────┘                        │ unhealthy → count++  │
+                  ▲                                 └──────────────────────┘
+                  │ SIGTERM/SIGINT → revoke lease (immediate deregister)
+```
+
+* **health-gate** — never registers until vLLM `/health` is up.
+* **lease, not heartbeat** — the etcd lease *is* the liveness signal. Keepalive
+  refreshes the TTL **without rewriting the key**, so the master's watcher sees
+  only the initial `PUT` and the final `DELETE` (on revoke or TTL expiry); no
+  watch churn.
+* **fast + safe deregister** — on N failed probes or a signal, the lease is
+  revoked so the key disappears at once; if the sidecar is killed `-9`, the key
+  still expires within `--lease-ttl`.
+* **clean restart** — each (re)registration uses a fresh `incarnation_id`
+  (uuid4), which the master uses to ignore stale deletes for a replaced instance
+  and to clean up the previous incarnation.
+
+## How the master picks it up (no C++ changes)
+
+| step | code |
+|---|---|
+| watch `XLLM:DEFAULT:` etc. | `instance_mgr.cpp:133` `add_watch(prefix, ...)` |
+| `PUT` → register | `instance_mgr.cpp:568` `update_instance_metainfo` → `register_instance` |
+| lease expiry → `DELETE` → probe → suspect/remove | `instance_mgr.cpp:615+` |
+| JSON schema | `common/types.h:248` `parse_from_json` (needs `name`,`rpc_address`,`type`) |
+| key prefix / namespace | `instance_mgr.cpp:45` map, `utils.cpp:105` namespace |
+
+> **Constraint:** a single vLLM instance with no decode peer is routable only as
+> `type=DEFAULT (0)` — keep `--instance-type=DEFAULT`. This mirrors the M2 finding.
+
+## Install
+
+```bash
+pip install -r vllm_sidecar/requirements.txt   # only `requests`
+```
+
+## Run
+
+```bash
+python -m vllm_sidecar.sidecar \
+    --etcd-endpoints 127.0.0.1:2379 \
+    --vllm-url       http://127.0.0.1:18000 \
+    --register-addr  127.0.0.1:18000          # host:port, NO scheme
+```
+
+`--register-addr` defaults to the host:port derived from `--vllm-url`. The master
+prepends `http://` itself (`init_brpc_channel`), so pass it bare.
+
+### Flags
+
+| flag | default | notes |
+|---|---|---|
+| `--etcd-endpoints` | `127.0.0.1:2379` | comma-separated; also `ETCD_ENDPOINTS` |
+| `--etcd-namespace` | `""` | **must match** master's `--etcd_namespace` |
+| `--etcd-username` / `--etcd-password` | `""` | also `ETCD_USERNAME`/`ETCD_PASSWORD` |
+| `--vllm-url` | `http://127.0.0.1:18000` | local vLLM base URL |
+| `--register-addr` | derived | host:port the master dials, no scheme |
+| `--backend-type` | `vllm` | written into InstanceMetaInfo |
+| `--instance-type` | `DEFAULT` | keep DEFAULT for a single instance |
+| `--instance-name` | `vllm` | prefix for incarnation id / logs |
+| `--lease-ttl` | `6` | seconds (≈ 2× keepalive interval) |
+| `--keepalive-interval` | `2` | seconds between probe + lease refresh |
+| `--health-fail-threshold` | `3` | failed probes before deregister |
+
+## Verify
+
+```bash
+# instance appears (with a lease) and is reachable through xllm-service
+etcdctl get --prefix XLLM:DEFAULT:
+etcdctl lease list
+curl http://127.0.0.1:9998/v1/models           # -> backend model list
+
+# kill the sidecar; key disappears within lease-ttl, master logs removal
+kill <sidecar_pid> && sleep 7
+etcdctl get --prefix XLLM:DEFAULT:              # empty
+```
+
+## Scope
+
+This PR provides **liveness-only** auto-registration via the etcd lease. Load
+metrics (queue depth, KV-cache usage), an HTTP `/register`+`/heartbeat` contract
+on the master, and `X-Internal-Token` auth are intentionally left to a follow-up
+PR — none are needed for single-instance routing today.

vllm_sidecar/__init__.py

@@ -0,0 +1,28 @@
+# Copyright 2025 The xLLM Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://github.com/jd-opensource/xllm-service/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""vLLM sidecar: auto-register a vLLM instance into xllm-service via etcd lease.
+
+A vLLM process speaks only HTTP/OpenAI -- it does not write etcd, hold a lease,
+or emit heartbeats. This sidecar runs alongside vLLM and bridges that gap:
+
+  * health-gate  -- only register once vLLM `/health` is up
+  * register     -- write an InstanceMetaInfo JSON under an etcd lease
+  * keep-alive   -- refresh the lease while vLLM stays healthy
+  * deregister   -- revoke the lease on vLLM failure or graceful shutdown,
+                    so xllm-service's etcd watcher removes the instance
+
+This replaces the manual `demo/register_vllm.sh` with no C++ changes: the master
+already discovers instances by watching the `XLLM:DEFAULT:` etcd prefix.
+"""

vllm_sidecar/etcd_registry.py

@@ -0,0 +1,141 @@
+# Copyright 2025 The xLLM Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://github.com/jd-opensource/xllm-service/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Minimal etcd v3 client over the gRPC-gateway JSON/HTTP API.
+
+We deliberately avoid `python-etcd3` (grpcio/protobuf native deps are brittle
+inside the shared GPU container). etcd exposes the full v3 KV/Lease API as JSON
+over the same client port (default 2379), verified against etcd 3.4.30:
+
+    POST /v3/lease/grant      {"TTL": <s>, "ID": 0}      -> {"ID","TTL"}
+    POST /v3/kv/put           {"key","value","lease"}    (key/value base64)
+    POST /v3/lease/keepalive  {"ID": <lease>}            -> {"result":{"TTL"}}
+    POST /v3/lease/revoke     {"ID": <lease>}
+    POST /v3/kv/range         {"key": <b64>}             -> {"kvs":[...]} | {}
+
+Lease keep-alive refreshes the TTL WITHOUT touching the key, so it generates no
+watch event -- the master's watcher only sees the initial PUT and the final
+DELETE (on lease expiry/revoke). Only `requests` is required.
+"""
+
+import base64
+import logging
+
+import requests
+
+logger = logging.getLogger("vllm_sidecar.etcd")
+
+
+class EtcdError(RuntimeError):
+    pass
+
+
+def _b64(s: str) -> str:
+    return base64.b64encode(s.encode("utf-8")).decode("ascii")
+
+
+class EtcdGatewayClient:
+    """Lease-oriented etcd v3 client. Endpoints are tried in order per call."""
+
+    def __init__(
+        self,
+        endpoints: str | list[str],
+        username: str = "",
+        password: str = "",
+        timeout: float = 3.0,
+    ) -> None:
+        # `endpoints` may be a comma-separated string or a list of host:port.
+        if isinstance(endpoints, str):
+            endpoints = [e.strip() for e in endpoints.split(",") if e.strip()]
+        # Keep an explicit scheme if present; only default to http:// otherwise,
+        # so endpoints like "https://host:2379" are not turned into
+        # "http://https://host:2379".
+        self._bases = [
+            e if e.startswith(("http://", "https://")) else "http://" + e
+            for e in (endpoint.rstrip("/") for endpoint in endpoints)
+        ]
+        if not self._bases:
+            raise ValueError("at least one etcd endpoint is required")
+        self._timeout = timeout
+        self._session = requests.Session()
+        if username:
+            self._authenticate(username, password)
+
+    def _authenticate(self, username: str, password: str) -> None:
+        resp = self._post(
+            "/v3/auth/authenticate", {"name": username, "password": password}
+        )
+        token = resp.get("token")
+        if not token:
+            raise EtcdError("etcd authenticate returned no token")
+        # etcd v3 gateway expects the token in the Authorization header.
+        self._session.headers["Authorization"] = token
+
+    def _post(self, path: str, body: dict) -> dict:
+        last_err = None
+        for base in self._bases:
+            try:
+                r = self._session.post(base + path, json=body, timeout=self._timeout)
+                if r.status_code == 200:
+                    try:
+                        data = r.json()
+                    except ValueError as e:
+                        last_err = EtcdError(f"invalid JSON response from {path}: {e}")
+                        continue
+                    # Every v3 gateway endpoint returns a JSON object; coerce any
+                    # other shape (null/list) to {} so callers can rely on .get().
+                    return data if isinstance(data, dict) else {}
+                last_err = EtcdError(f"{path} -> HTTP {r.status_code}: {r.text[:200]}")
+            except requests.RequestException as e:  # connection/timeout
+                last_err = e
+        raise EtcdError(f"all etcd endpoints failed for {path}: {last_err}")
+
+    # --- lease lifecycle ---------------------------------------------------
+
+    def lease_grant(self, ttl_seconds: int) -> str:
+        """Grant a lease; returns the lease id (int64 as a decimal string)."""
+        resp = self._post("/v3/lease/grant", {"TTL": ttl_seconds, "ID": 0})
+        lease_id = resp.get("ID")
+        if not lease_id:
+            raise EtcdError(f"lease grant returned no ID: {resp}")
+        return lease_id
+
+    def lease_keepalive(self, lease_id: str) -> int:
+        """Refresh a lease once; returns the remaining TTL (0 == lease gone)."""
+        resp = self._post("/v3/lease/keepalive", {"ID": lease_id})
+        # "result" may be present but null (proto3 JSON for an empty message),
+        # and "TTL" itself may be missing or null; treat all of these as 0.
+        result = resp.get("result") or {}
+        return int(result.get("TTL") or 0)
+
+    def lease_revoke(self, lease_id: str) -> None:
+        """Revoke a lease -> its key is deleted immediately."""
+        self._post("/v3/lease/revoke", {"ID": lease_id})
+
+    # --- kv ----------------------------------------------------------------
+
+    def put(self, key: str, value: str, lease_id: str) -> None:
+        self._post(
+            "/v3/kv/put", {"key": _b64(key), "value": _b64(value), "lease": lease_id}
+        )
+
+    def get(self, key: str) -> str | None:
+        """Return the string value at ``key`` or None if absent."""
+        resp = self._post("/v3/kv/range", {"key": _b64(key)})
+        kvs = resp.get("kvs")
+        if not kvs:
+            return None
+        # proto3 JSON omits empty values, so "value" may be absent for an
+        # empty string; treat that as "".
+        return base64.b64decode(kvs[0].get("value", "")).decode("utf-8")

vllm_sidecar/health.py

@@ -0,0 +1,53 @@
+# Copyright 2025 The xLLM Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://github.com/jd-opensource/xllm-service/blob/main/LICENSE
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""vLLM health probing.
+
+The lease only proves the sidecar is alive; it says nothing about vLLM. We gate
+registration and lease refresh on vLLM's own `/health` endpoint so that a dead
+vLLM (with the sidecar still running) does not leave a phantom instance routable.
+"""
+
+import logging
+
+import requests
+
+logger = logging.getLogger("vllm_sidecar.health")
+
+
+class VllmHealthProbe:
+    def __init__(self, vllm_url: str, timeout: float = 3.0) -> None:
+        self._base = vllm_url.rstrip("/")
+        self._timeout = timeout
+        # Reuse one connection across the periodic probes (HTTP keep-alive) to
+        # avoid piling up TIME_WAIT sockets on the host.
+        self._session = requests.Session()
+
+    def is_healthy(self) -> bool:
+        """True iff vLLM `/health` returns 2xx within the timeout."""
+        try:
+            r = self._session.get(self._base + "/health", timeout=self._timeout)
+            return 200 <= r.status_code < 300
+        except requests.RequestException as e:
+            logger.debug("vLLM health probe failed: %s", e)
+            return False
+
+    def served_model(self) -> str | None:
+        """Best-effort first model id from `/v1/models`, for logging only."""
+        try:
+            r = self._session.get(self._base + "/v1/models", timeout=self._timeout)
+            data = r.json().get("data", [])
+            return data[0]["id"] if data else None
+        except (requests.RequestException, ValueError, KeyError, IndexError):
+            return None