Chemistry-RL: multi-step organic reaction product prediction

A long-horizon, coding-based RL environment in the verifiers format. The agent writes RDKit/Python code to predict the product of each reaction in a multi-step synthesis sequence; the reward is graded over every intermediate and the final product, and the verifier is built to resist reward hacking.

Capability targeted: chemoselectivity under competing functional groups, and state tracking across a 3–5 step sequence. The named failure mode: applying a reagent to the wrong functional group (e.g. reducing an ester with NaBH₄, which in reality leaves esters untouched). See SUBMISSION.md for the full rationale, contamination argument, and receipts.

Install

# core (standalone, no heavy deps)
pip install -e .

# with the verifiers harness and/or a real-LLM leaderboard slot
pip install -e ".[verifiers,llm,dev]"

RDKit is the only non-trivial dependency. The environment runs fully without the verifiers package; installing verifiers makes load_environment() return a conforming verifiers.MultiTurnEnv instead of the standalone engine.

Quick start

from chemistry_rl import load_environment
from chemistry_rl.engine import run_rollout

env = load_environment("eval")
task = env.tasks[0]

def my_agent(obs):
    # obs has: substrate, reagents, conditions, step_index, n_steps, task_id
    # return Python code that uses RDKit and calls submit(<product_smiles>)
    return "m = Chem.MolFromSmiles(SUBSTRATE)\nsubmit(Chem.MolToSmiles(m))"

result = run_rollout(task, my_agent)
print(result.total, result.predictions)

The agent's code runs in an isolated sandbox: 30 s timeout, memory cap, no network, and the reference answers (chemistry_rl.oracle, *.full.jsonl) are import- and read-blocked.

Reproduce every number

# 1. (re)generate the dataset deterministically from a seed
python -m chemistry_rl.oracle.generator --seed 7 --n 600

# 2. prove every gold answer reproduces from its template path
python -m chemistry_rl.oracle.grader --split eval

# 3. soundness receipt: false-accept / false-reject with Wilson 95% CIs
python -m chemistry_rl.adversarial.run_soundness --split eval

# 4. capability leaderboard (rule-based ladder; add a model with --model)
python -m chemistry_rl.baselines.leaderboard --split eval
ANTHROPIC_API_KEY=... python -m chemistry_rl.baselines.leaderboard --split eval --model claude-sonnet-4-6

# 5. tests (verifier, oracle isolation, soundness, dataset integrity)
pytest -q

What ships

Path	What it is
chemistry_rl/environment.py	load_environment() — verifiers MultiTurnEnv or standalone engine
chemistry_rl/engine.py	framework-free multi-turn rollout engine
chemistry_rl/verifier/	chemical validator, graded reward, isolation sandbox
chemistry_rl/oracle/	answer key (templates, gold, grader) — blocked from the agent
chemistry_rl/data/	*.jsonl agent surface + *.full.jsonl gold (sandbox-blocked)
chemistry_rl/adversarial/	cheat battery + soundness runner
chemistry_rl/baselines/	leaderboard + baseline agents
tests/	verifier correctness, oracle isolation, soundness, dataset integrity

Sandbox manifest (declared, not implied)

• Isolation granularity: process-level (subprocess + RLIMIT_AS/RLIMIT_CPU). For training at scale against adversarial policies, wrap in a container (Docker+seccomp / gVisor / Firecracker). This is a deliberate, documented trade-off — see verifier/sandbox.py.
• Network egress: blocked (socket connect disabled in-process).
• Oracle access: chemistry_rl.oracle.* import-blocked; *.full.jsonl read-blocked. Proven by tests/test_oracle_hidden.py.
• Timeout / memory: 30 s wall, 4 GB virtual address space (BLAS pinned to 1 thread; precise resident-memory limiting needs cgroups in production).
• Determinism: dataset and all reports regenerate from a fixed seed.

Reward (summary)

Per step, against the gold product for that step:

score	condition
0.0	invalid/empty, or unchanged from the substrate (no reaction occurred)
0.1	valid molecule, Tanimoto < 0.3
0.5	Tanimoto 0.3–0.7
0.8	same connectivity, wrong stereo, or Tanimoto > 0.7
1.0	exact canonical (stereo-aware) match

Aggregate: R = 0.4·R_final + 0.6·mean(R_intermediates). Distributing the signal across intermediates means a lucky final answer without correct intermediate work cannot score well — the long-horizon property the task requires.