Fix vllm port toctou - #266
Merged
Merged
Conversation
sfc-gh-truwase
requested review from
sfc-gh-goliaro,
sfc-gh-jrasley and
sfc-gh-reyazda
as code owners
June 29, 2026 16:43
sfc-gh-mhidayetoglu
approved these changes
Jun 29, 2026
sfc-gh-mwyatt
approved these changes
Jun 29, 2026
Comment on lines
+268
to
+285
| # Pin a deterministic, per-replica base port. When many colocated | ||
| # TP engines initialize concurrently on the same node, vLLM's | ||
| # get_open_port() has a TOCTOU window: it binds a probe socket, | ||
| # reads the port, closes it, and returns the number, so several | ||
| # engines can receive the same port before any binds the TP | ||
| # rendezvous TCPStore (tcp://127.0.0.1:<port>) -> EADDRINUSE. | ||
| # Giving each replica a distinct VLLM_PORT base makes get_open_port | ||
| # start from a unique value per engine; vLLM still increments from | ||
| # the base on any residual conflict. The globally-unique replica | ||
| # index i guarantees uniqueness across replicas colocated on the | ||
| # same node. Disable via ARCTIC_DISABLE_VLLM_PORT_ASSIGN=1; tune the | ||
| # range via ARCTIC_VLLM_PORT_BASE / ARCTIC_VLLM_PORT_STRIDE. | ||
| if (self.tp_size > 1 | ||
| and os.environ.get("ARCTIC_DISABLE_VLLM_PORT_ASSIGN") != "1"): | ||
| stride = int(os.environ.get("ARCTIC_VLLM_PORT_STRIDE", "100")) | ||
| base = int(env.get("VLLM_PORT", | ||
| os.environ.get("ARCTIC_VLLM_PORT_BASE", "30000"))) | ||
| env["VLLM_PORT"] = str(base + i * stride) |
Collaborator
There was a problem hiding this comment.
Suggested change
| # Pin a deterministic, per-replica base port. When many colocated | |
| # TP engines initialize concurrently on the same node, vLLM's | |
| # get_open_port() has a TOCTOU window: it binds a probe socket, | |
| # reads the port, closes it, and returns the number, so several | |
| # engines can receive the same port before any binds the TP | |
| # rendezvous TCPStore (tcp://127.0.0.1:<port>) -> EADDRINUSE. | |
| # Giving each replica a distinct VLLM_PORT base makes get_open_port | |
| # start from a unique value per engine; vLLM still increments from | |
| # the base on any residual conflict. The globally-unique replica | |
| # index i guarantees uniqueness across replicas colocated on the | |
| # same node. Disable via ARCTIC_DISABLE_VLLM_PORT_ASSIGN=1; tune the | |
| # range via ARCTIC_VLLM_PORT_BASE / ARCTIC_VLLM_PORT_STRIDE. | |
| if (self.tp_size > 1 | |
| and os.environ.get("ARCTIC_DISABLE_VLLM_PORT_ASSIGN") != "1"): | |
| stride = int(os.environ.get("ARCTIC_VLLM_PORT_STRIDE", "100")) | |
| base = int(env.get("VLLM_PORT", | |
| os.environ.get("ARCTIC_VLLM_PORT_BASE", "30000"))) | |
| env["VLLM_PORT"] = str(base + i * stride) | |
| # Ensure we start from offset base ports so that vllm workers do not | |
| # collide when attempting to obtain free ports. vLLM increments ports | |
| # until a free one is found, but this can lead to a race condition when | |
| # spinning up many replicas at the same time. | |
| base, stride = 30000, 100 | |
| env["VLLM_PORT"] = str(base + i*stride) |
Collaborator
There was a problem hiding this comment.
We don't need all these env vars and it's safe to apply this for any tp_size. Just simplifying and reducing AI verbosity
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.