fix: retry transient init failures in make_factorio_env - #367
Open
rasdani wants to merge 2 commits into
Open
Conversation
When a Factorio container is recycled across samples (e.g. between
Inspect-AI rollouts of an eval-set, or between epochs of a pass@N
solver run), the first ``gym_env.reset`` on the recycled slot
occasionally races with FLE's internal Lua script cache and the RCON
server returns malformed data, surfacing as
RuntimeError: Failed to create Factorio environment: Could not
save research state: { ["technologies"] = { ... } }
The error is transient — a 2-4-second wait followed by a fresh
FactorioInstance build clears it. Wrap the FactorioInstance +
task.setup block in a 3-attempt retry with backoff and best-effort
cleanup of the partially-built instance between tries. The
configuration-error path (no containers / invalid run_idx) is moved
out of the retry loop since those errors are not transient.
Reproduced with::
fle inspect-eval --tasks plastic_bar_throughput,automation_science_pack_throughput \
--model anthropic/claude-sonnet-4-5 --solver controlled \
--pass-n 1 --max-connections 2
Without the retry, both samples fail with 0 model calls and score 0;
with the retry they reach the model and execute normally.
Number of retries is tunable via ``FLE_INIT_RETRIES`` (default 3).
Root cause of "Could not save research state" was deeper than the previous patch handled. cleanup() only closes the RCON socket — it does NOT reset the Factorio server's Lua VM. With cache_scripts=True (the default in make_factorio_env), the LuaScriptManager checksums match between rebuilds and skips re-uploading; the new instance ends up running against stale, corrupted Lua state from the previous run. Real fix: on every retry attempt after the first, force ``cache_scripts=False`` so the Lua scripts are freshly re-uploaded, re-initializing all server-side script state. This is what closes the actual race. Also bump retry budget: 5 attempts (was 3) with 2/5/10/20/30s backoff (was 2/4/6s) — under heavy parallel load, the corruption takes longer than 6s to clear. Budget is configurable via FLE_INIT_RETRIES / FLE_INIT_BACKOFF. Verified: crude_oil_throughput (gpt-4o) which previously bombed at 0 steps (setup race) now succeeds with 6 steps, prod=1438, auto=329.
kiankyars
requested changes
Jun 11, 2026
| # ``cache_scripts=False`` so all scripts are freshly re-uploaded | ||
| # (re-initializing their global Lua state). Plus longer backoff | ||
| # so containers have time to settle. | ||
| max_attempts = int(os.getenv("FLE_INIT_RETRIES", "5")) |
Collaborator
There was a problem hiding this comment.
The docstring says FLE_INIT_RETRIES defaults to 3, but the implementation defaults to "5". Could we make those match so operators know the actual retry/backoff behavior?
| instance=instance, task=task, enable_vision=spec.enable_vision | ||
| ) | ||
|
|
||
| except Exception as e: |
Collaborator
There was a problem hiding this comment.
This retries every exception from instance construction, task setup, and env wrapping, but the PR is targeting transient init failures like Could not save research state. Can we gate retries on known retryable errors/exception types and fail fast for deterministic setup or wrapper errors? Otherwise real task/Lua/env bugs get delayed across the full backoff schedule and may repeat setup side effects.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a Factorio container is recycled across samples (e.g. between
Inspect-AI rollouts of an eval-set, or between epochs of a pass@N
solver run), the first
gym_env.reseton the recycled slotoccasionally races with FLE's internal Lua script cache and the RCON
server returns malformed data, surfacing as
The error is transient — a 2-4-second wait followed by a fresh
FactorioInstance build clears it. Wrap the FactorioInstance +
task.setup block in a 3-attempt retry with backoff and best-effort
cleanup of the partially-built instance between tries. The
configuration-error path (no containers / invalid run_idx) is moved
out of the retry loop since those errors are not transient.
Reproduced with::
fle inspect-eval --tasks plastic_bar_throughput,automation_science_pack_throughput
--model anthropic/claude-sonnet-4-5 --solver controlled
--pass-n 1 --max-connections 2
Without the retry, both samples fail with 0 model calls and score 0;
with the retry they reach the model and execute normally.
Number of retries is tunable via
FLE_INIT_RETRIES(default 3).