Tier 3 --env-mode local always fails: the evaluator's own managed runtime env injects BASH_ENV, which local mode's command filter rejects

[aunysillyme]

Summary

skillevaluator tier3 evaluate --env-mode local cannot run any trial. Every agent fails at agent-runtime-preflight with exit 126 and:

Local mode command blocked: environment variable BASH_ENV can execute or alter code before confinement and is not allowed

BASH_ENV is not set in the host environment. SkillEvaluator injects it itself, then rejects it.

Versions

• skillevaluator 0.2.1 (uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git", commit ff349e0)
• harbor 0.13.2, skillspector 2.11.2
• macOS (Darwin 25.6.0), arm64, Python 3.13
• agents installed and authenticated: codex 0.153.4, claude (Claude Code) 2.1.226

Reproduce

skillevaluator create-eval-dataset ./my-skill --full
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode local --n-attempts 1

Result:

[00:00:03] agent-runtime-preflight: failed - codex runtime preflight failed: Harbor agent-only job
did not complete successfully: 1 errored; first trial: <case>: NonZeroAgentExitCodeError |
Command failed (exit 126): codex --version stdout: None stderr: Local mode command blocked:
environment variable BASH_ENV can execute or alter code before confinement and is not allowed

It fails on codex --version, so nothing agent-specific is involved.

Root cause

tier3/harbor/adapter.py:

_EVALUATOR_MANAGED_RUNTIME_ENV = {
    **dict.fromkeys(_RUNTIME_LOADER_ENV_NAMES, ""),
    "CLAUDE_CODE_DISABLE_POLICY_SKILLS": "1",
}

_RUNTIME_LOADER_ENV_NAMES contains BASH_ENV, CLASSPATH, LD_PRELOAD, PYTHONPATH and about twenty others. That dict is merged into runtime_env (adapter.py lines 1872 and 4378).

Setting those names to "" is right for the Docker path, where ENV BASH_ENV="" neutralises the variable inside the image. In local mode the same dict reaches LocalEnvironment._exec_env, which calls:

tier3/harbor/local_environment.py

@staticmethod
def _filter_command_env(env: dict[str, str], *, protected: set[str]) -> dict[str, str]:
    for key, value in env.items():
        normalized = key.upper()
        if normalized in _BLOCKED_COMMAND_ENV_NAMES or normalized.startswith(_BLOCKED_COMMAND_ENV_PREFIXES):
            raise ValueError(
                f"environment variable {key} can execute or alter code before confinement and is not allowed"
            )

The check rejects on key presence, regardless of value, and _BLOCKED_COMMAND_ENV_NAMES overlaps _RUNTIME_LOADER_ENV_NAMES (BASH_ENV, CLASSPATH, ENV, GCONV_PATH, LOCPATH, NLSPATH, and more). So the hardening the evaluator applies to itself is what its own guard refuses.

Evidence the host is not the source

$ env | grep -cE '^(BASH_ENV|BASHOPTS)='
0
$ bash -c 'echo "${BASH_ENV-UNSET}"'
UNSET
$ python3 -c "import os; print('BASH_ENV' in os.environ)"
False

The retained Harbor job config also shows every env block empty (/agent/env, /environment/env, /verifier/env all {}), so it is not arriving from the task definition either.

Not a workaround

SKILLEVALUATOR_LOCAL_SANDBOX=off does not help. The command-env filter runs regardless of the sandbox backend, and the failure and message are identical.

Suggested fix

Either:

1. Treat an empty value as absent in _filter_command_env (if normalized in _BLOCKED_... and value != ""), since an empty BASH_ENV cannot execute anything; or
2. Do not pass the Docker-shaped loader-reset keys into the local path, and let the local sandbox apply its own reset.

Option 1 keeps the guard meaningful for a genuinely populated variable inherited from a host.

Impact

--env-mode local is documented as the way to run Tier 3 without a container, and it is the only Tier 3 path for a user whose agent CLIs are authenticated by subscription rather than by an API key. As shipped it produces no trials at all.

Happy to test a patch.

[rng1995]

Fixed on main by #133 (commit 7e189c6), which addresses the same root cause tracked in #132. Empty blocked process-control values are now dropped before local execution, non-empty values remain rejected, and unrelated empty variables are preserved. I also re-ran the focused current-main regressions covering these paths (18 passed). Closing as fixed; thanks for the detailed end-to-end reproduction.