Skip to content

[BUG]: Tier 3 local mode rejects its own empty managed loader variables #132

Description

@RyaliNvidia

What happened?

Tier 3 local evaluation can reject environment variables that SkillEvaluator itself intentionally resets to an empty value.

The generated local runtime environment sets loader/control variables such as BASH_ENV and ENV to "". Later, SkillEvaluatorLocalEnvironment._filter_command_env() rejects those names unconditionally, including when the value is empty. This causes the local command to fail before the candidate agent starts:

environment variable BASH_ENV can execute or alter code before confinement and is not allowed

Expected behavior: empty values for evaluator-managed loader resets are discarded or accepted as inert, while non-empty values remain blocked.

Actual behavior: both empty and non-empty values are rejected.

The relevant interaction is between:

  • src/skillevaluator/tier3/harbor/adapter.py, where _RUNTIME_LOADER_ENV_RESET and _RUNTIME_ENV_BASE set the loader variables to empty strings.
  • src/skillevaluator/tier3/harbor/local_environment.py, where _filter_command_env() rejects every blocked name without considering whether the value is empty.

Reproduction steps

On commit ff349e0d9f03868fc27d1e2bbd62eb849cba66c9:

from skillevaluator.tier3.harbor.local_environment import (
    SkillEvaluatorLocalEnvironment,
)

SkillEvaluatorLocalEnvironment._filter_command_env(
    {"BASH_ENV": "", "ENV": ""},
    protected=set(),
)

Observed result:

ValueError: environment variable BASH_ENV can execute or alter code before confinement and is not allowed

A minimal behavior-preserving fix would be to omit blocked loader/control variables when their value is empty, while continuing to reject any non-empty value. Suggested regression coverage:

  1. Empty BASH_ENV and ENV are removed from the filtered environment without error.
  2. Non-empty BASH_ENV and ENV still raise ValueError.

SkillEvaluator version or commit

ff349e0d9f03868fc27d1e2bbd62eb849cba66c9 (0.2.1)

Environment

  • Linux amd64
  • Python 3.12
  • Installed from the public Git repository with uv sync --frozen
  • Local Tier 3 environment
  • Codex CLI 0.147.0

Before submitting

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions