Skip to content

[Bug]: Graph counterexample task grants public network access despite offline contract #547

Description

@morluto

Problem

mathematical-benchmarks-v1/graph-counterexample is documented as a self-contained offline task, but its agent environment declares network_mode = "public". Harbor resolves that mode to unrestricted network access, so an agent trial can make arbitrary outbound requests even though the task has no external-data requirement and explicitly says web search is disabled.

Evidence

The task contract declares the agent environment as public at benchmarks/datasets/mathematical-benchmarks-v1/graph-counterexample/task.toml:35-39. The same task says the input is offline and self-contained at instruction.md:3 and README.md:27-31, and its metadata tags the task offline at task.toml:22.

Harbor's Docker backend applies public as allow-all (harbor/environments/docker/docker.py:_apply_network_policy), while the job's extra_allowed_hosts list is ignored for a public policy (harbor/trial/network_policy.py:28-35). Thus the committed job configuration does not narrow the task's egress.

Reproduction

  1. Load task.toml for graph-counterexample.
  2. Resolve the task's agent environment policy: it is public.
  3. Run the task under the Docker environment. Harbor sends the egress controller the allow-all policy for the agent phase.
  4. The agent can reach arbitrary external hosts, despite the task's offline/self-contained contract.

The separate verifier environment is correctly no-network at task.toml:41-45; the issue concerns the agent phase.

Expected behavior

The agent environment should use no-network for this self-contained task, or an explicit allowlist if a documented runtime dependency is required. The task metadata, README, instruction, and job policy should agree so that an offline task cannot silently run with unrestricted egress.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions