Skip to content

Fix vllm port toctou - #266

Merged
sfc-gh-truwase merged 1 commit into
mainfrom
tunji/vllm_port_toctou
Jun 29, 2026
Merged

Fix vllm port toctou#266
sfc-gh-truwase merged 1 commit into
mainfrom
tunji/vllm_port_toctou

Conversation

@sfc-gh-truwase

Copy link
Copy Markdown
Collaborator

No description provided.

Comment on lines +268 to +285
# Pin a deterministic, per-replica base port. When many colocated
# TP engines initialize concurrently on the same node, vLLM's
# get_open_port() has a TOCTOU window: it binds a probe socket,
# reads the port, closes it, and returns the number, so several
# engines can receive the same port before any binds the TP
# rendezvous TCPStore (tcp://127.0.0.1:<port>) -> EADDRINUSE.
# Giving each replica a distinct VLLM_PORT base makes get_open_port
# start from a unique value per engine; vLLM still increments from
# the base on any residual conflict. The globally-unique replica
# index i guarantees uniqueness across replicas colocated on the
# same node. Disable via ARCTIC_DISABLE_VLLM_PORT_ASSIGN=1; tune the
# range via ARCTIC_VLLM_PORT_BASE / ARCTIC_VLLM_PORT_STRIDE.
if (self.tp_size > 1
and os.environ.get("ARCTIC_DISABLE_VLLM_PORT_ASSIGN") != "1"):
stride = int(os.environ.get("ARCTIC_VLLM_PORT_STRIDE", "100"))
base = int(env.get("VLLM_PORT",
os.environ.get("ARCTIC_VLLM_PORT_BASE", "30000")))
env["VLLM_PORT"] = str(base + i * stride)

@sfc-gh-mwyatt sfc-gh-mwyatt Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Pin a deterministic, per-replica base port. When many colocated
# TP engines initialize concurrently on the same node, vLLM's
# get_open_port() has a TOCTOU window: it binds a probe socket,
# reads the port, closes it, and returns the number, so several
# engines can receive the same port before any binds the TP
# rendezvous TCPStore (tcp://127.0.0.1:<port>) -> EADDRINUSE.
# Giving each replica a distinct VLLM_PORT base makes get_open_port
# start from a unique value per engine; vLLM still increments from
# the base on any residual conflict. The globally-unique replica
# index i guarantees uniqueness across replicas colocated on the
# same node. Disable via ARCTIC_DISABLE_VLLM_PORT_ASSIGN=1; tune the
# range via ARCTIC_VLLM_PORT_BASE / ARCTIC_VLLM_PORT_STRIDE.
if (self.tp_size > 1
and os.environ.get("ARCTIC_DISABLE_VLLM_PORT_ASSIGN") != "1"):
stride = int(os.environ.get("ARCTIC_VLLM_PORT_STRIDE", "100"))
base = int(env.get("VLLM_PORT",
os.environ.get("ARCTIC_VLLM_PORT_BASE", "30000")))
env["VLLM_PORT"] = str(base + i * stride)
# Ensure we start from offset base ports so that vllm workers do not
# collide when attempting to obtain free ports. vLLM increments ports
# until a free one is found, but this can lead to a race condition when
# spinning up many replicas at the same time.
base, stride = 30000, 100
env["VLLM_PORT"] = str(base + i*stride)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need all these env vars and it's safe to apply this for any tp_size. Just simplifying and reducing AI verbosity

@sfc-gh-truwase
sfc-gh-truwase merged commit daf3172 into main Jun 29, 2026
2 checks passed
@sfc-gh-truwase
sfc-gh-truwase deleted the tunji/vllm_port_toctou branch June 29, 2026 17:21
@sfc-gh-truwase
sfc-gh-truwase restored the tunji/vllm_port_toctou branch June 29, 2026 17:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants