What happened?
Tier 3 local evaluation can reject environment variables that SkillEvaluator itself intentionally resets to an empty value.
The generated local runtime environment sets loader/control variables such as BASH_ENV and ENV to "". Later, SkillEvaluatorLocalEnvironment._filter_command_env() rejects those names unconditionally, including when the value is empty. This causes the local command to fail before the candidate agent starts:
environment variable BASH_ENV can execute or alter code before confinement and is not allowed
Expected behavior: empty values for evaluator-managed loader resets are discarded or accepted as inert, while non-empty values remain blocked.
Actual behavior: both empty and non-empty values are rejected.
The relevant interaction is between:
src/skillevaluator/tier3/harbor/adapter.py, where _RUNTIME_LOADER_ENV_RESET and _RUNTIME_ENV_BASE set the loader variables to empty strings.
src/skillevaluator/tier3/harbor/local_environment.py, where _filter_command_env() rejects every blocked name without considering whether the value is empty.
Reproduction steps
On commit ff349e0d9f03868fc27d1e2bbd62eb849cba66c9:
from skillevaluator.tier3.harbor.local_environment import (
SkillEvaluatorLocalEnvironment,
)
SkillEvaluatorLocalEnvironment._filter_command_env(
{"BASH_ENV": "", "ENV": ""},
protected=set(),
)
Observed result:
ValueError: environment variable BASH_ENV can execute or alter code before confinement and is not allowed
A minimal behavior-preserving fix would be to omit blocked loader/control variables when their value is empty, while continuing to reject any non-empty value. Suggested regression coverage:
- Empty
BASH_ENV and ENV are removed from the filtered environment without error.
- Non-empty
BASH_ENV and ENV still raise ValueError.
SkillEvaluator version or commit
ff349e0d9f03868fc27d1e2bbd62eb849cba66c9 (0.2.1)
Environment
- Linux amd64
- Python 3.12
- Installed from the public Git repository with
uv sync --frozen
- Local Tier 3 environment
- Codex CLI 0.147.0
Before submitting
What happened?
Tier 3 local evaluation can reject environment variables that SkillEvaluator itself intentionally resets to an empty value.
The generated local runtime environment sets loader/control variables such as
BASH_ENVandENVto"". Later,SkillEvaluatorLocalEnvironment._filter_command_env()rejects those names unconditionally, including when the value is empty. This causes the local command to fail before the candidate agent starts:Expected behavior: empty values for evaluator-managed loader resets are discarded or accepted as inert, while non-empty values remain blocked.
Actual behavior: both empty and non-empty values are rejected.
The relevant interaction is between:
src/skillevaluator/tier3/harbor/adapter.py, where_RUNTIME_LOADER_ENV_RESETand_RUNTIME_ENV_BASEset the loader variables to empty strings.src/skillevaluator/tier3/harbor/local_environment.py, where_filter_command_env()rejects every blocked name without considering whether the value is empty.Reproduction steps
On commit
ff349e0d9f03868fc27d1e2bbd62eb849cba66c9:Observed result:
A minimal behavior-preserving fix would be to omit blocked loader/control variables when their value is empty, while continuing to reject any non-empty value. Suggested regression coverage:
BASH_ENVandENVare removed from the filtered environment without error.BASH_ENVandENVstill raiseValueError.SkillEvaluator version or commit
ff349e0d9f03868fc27d1e2bbd62eb849cba66c9(0.2.1)Environment
uv sync --frozenBefore submitting