Skip to content

Fix vllm port toctou #266

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sfc-gh-truwase merged 1 commit into from
Jun 29, 2026
Merged

Fix vllm port toctou #266

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions arctic_inference/server/replica_pool.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,24 @@ async def initialize(
rid = self._bundle_indices[i]
env["VLLM_RAY_BUNDLE_INDICES"] = ",".join(
str(rid * self.tp_size + t) for t in range(self.tp_size))
# Pin a deterministic, per-replica base port. When many colocated
# TP engines initialize concurrently on the same node, vLLM's
# get_open_port() has a TOCTOU window: it binds a probe socket,
# reads the port, closes it, and returns the number, so several
# engines can receive the same port before any binds the TP
# rendezvous TCPStore (tcp://127.0.0.1:<port>) -> EADDRINUSE.
# Giving each replica a distinct VLLM_PORT base makes get_open_port
# start from a unique value per engine; vLLM still increments from
# the base on any residual conflict. The globally-unique replica
# index i guarantees uniqueness across replicas colocated on the
# same node. Disable via ARCTIC_DISABLE_VLLM_PORT_ASSIGN=1; tune the
# range via ARCTIC_VLLM_PORT_BASE / ARCTIC_VLLM_PORT_STRIDE.
if (self.tp_size > 1
and os.environ.get("ARCTIC_DISABLE_VLLM_PORT_ASSIGN") != "1"):
stride = int(os.environ.get("ARCTIC_VLLM_PORT_STRIDE", "100"))
base = int(env.get("VLLM_PORT",
os.environ.get("ARCTIC_VLLM_PORT_BASE", "30000")))
env["VLLM_PORT"] = str(base + i * stride)
Comment on lines +268 to +285

@sfc-gh-mwyatt sfc-gh-mwyatt Jun 29, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Pin a deterministic, per-replica base port. When many colocated
# TP engines initialize concurrently on the same node, vLLM's
# get_open_port() has a TOCTOU window: it binds a probe socket,
# reads the port, closes it, and returns the number, so several
# engines can receive the same port before any binds the TP
# rendezvous TCPStore (tcp://127.0.0.1:<port>) -> EADDRINUSE.
# Giving each replica a distinct VLLM_PORT base makes get_open_port
# start from a unique value per engine; vLLM still increments from
# the base on any residual conflict. The globally-unique replica
# index i guarantees uniqueness across replicas colocated on the
# same node. Disable via ARCTIC_DISABLE_VLLM_PORT_ASSIGN=1; tune the
# range via ARCTIC_VLLM_PORT_BASE / ARCTIC_VLLM_PORT_STRIDE.
if (self.tp_size > 1
and os.environ.get("ARCTIC_DISABLE_VLLM_PORT_ASSIGN") != "1"):
stride = int(os.environ.get("ARCTIC_VLLM_PORT_STRIDE", "100"))
base = int(env.get("VLLM_PORT",
os.environ.get("ARCTIC_VLLM_PORT_BASE", "30000")))
env["VLLM_PORT"] = str(base + i * stride)
# Ensure we start from offset base ports so that vllm workers do not
# collide when attempting to obtain free ports. vLLM increments ports
# until a free one is found, but this can lead to a race condition when
# spinning up many replicas at the same time.
base, stride = 30000, 100
env["VLLM_PORT"] = str(base + i*stride)

@sfc-gh-mwyatt sfc-gh-mwyatt Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need all these env vars and it's safe to apply this for any tp_size. Just simplifying and reducing AI verbosity

init_tasks.append(w.initialize.remote(engine_kwargs, env or None))
await asyncio.gather(*init_tasks)

Expand Down
Loading