chore(deps): Bump trl from 1.7.1 to 1.8.0

[dependabot]

Bumps trl from 1.7.1 to 1.8.0.

Release notes

Sourced from trl's releases.

v1.8.0

Features

🎓 KTO is now a stable trainer

After many cycles of KTOTrainer ↔ DPOTrainer alignment work, KTO graduates from trl.experimental.kto to the top-level trl package. Same API as DPO/GRPO/SFT — imports move from experimental, tests move to the main test tree, docs no longer flag it as experimental. The experimental path still works and emits a FutureWarning (removal in v2.0.0).

# Before
from trl.experimental.kto import KTOConfig, KTOTrainer
Now
from trl import KTOConfig, KTOTrainer

Per our telemetry, KTO is the 4th most used trainer in TRL — this graduation was overdue.

by @​albertvillanova in huggingface/trl#6175, huggingface/trl#6287 and huggingface/trl#6345

Environment-owned rewards & multi-environment support

Three interrelated changes make agentic RL training with environments substantially more ergonomic.

Environment-owned reward. If your environment_factory env defines a reserved get_reward() method (no args → float), it's called once per completed rollout and treated as a reward source. reward_funcs becomes optional — no more leaking env state back out to trainer-owned reward funcs.

class WordleEnv:
    def reset(self, **kwargs):
        self._target = sample(words); self._solved = False
def get_reward(self) -> float:       # optional, reserved (not a tool)
    return 1.0 if self._solved else 0.0
def guess(self, word: str) -> str:   # exposed as a tool
self._solved = word == self._target; ...

trainer = GRPOTrainer(
model=model,
train_dataset=dataset,
environment_factory=WordleEnv,       # no reward_funcs needed
)

Multi-environment support. environment_factory now accepts dict[str, factory] in addition to a single callable. Each dataset row selects its environment via an environment column, and only that env's tools are exposed in that row's prompt — so a coding task and a game can train together in one run without leaking each other's tool schemas. Single-callable usage is unchanged.

Same wiring lands in GRPO, AsyncGRPO, DPPO, and GRPO-with-replay-buffer.

Env-owned reward by @​qgallouedec in huggingface/trl#6238; multi-env in huggingface/trl#6001 and huggingface/trl#6002

Entropy regularization for GRPO

... (truncated)

Commits

• 95809b9 Release: v1.8 (#6346)
• 2ee37b8 Docs: treat KTO as a stable trainer (#6345)
• bae8238 Fix incorrect examples in distillation docs (#6334)
• 00d6209 Neuron-friendly chunked_nll via static-shape token packing (#6314)
• 9ffb54e Fix chunked_nll patch hiding VLM kwargs from generate (#6156)
• ce39cbb Raise on quantization_config + already-instantiated model in DPOTrainer/ ...
• 70f95f3 Document that max_steps is required for iterable train datasets (#6333)
• 627a83e Support DatasetDict and IterableDatasetDict as eval_dataset in trainers (#6322)
• 8c75069 Align KTO with DPO: quantization_config trainer argument (#6276)
• cfef01a Environment-owned reward (#6238)
• Additional commits viewable in compare view

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!