Agent-RLlib

Research sandbox for tool-using reinforcement learning agents. The project currently focuses on a simulated customer-support environment, a PPO-style agent interface, built-in tools, evaluation scripts, and a FastAPI serving path.

The goal is to make the agent stack easy to inspect and extend. This repository is an experimental prototype, not a production RL platform.

What Is Implemented

• SupportBotEnv: a Gymnasium-style environment for support conversations with tool-use actions.
• MultiAgentNegotiationEnv: an experimental multi-agent environment for resource allocation scenarios.
• HybridPPOAgent: a PPO-inspired agent wrapper for choosing response and tool actions.
• Tool registry with built-in search, calculator, API, code, and time tools.
• Training and evaluation entrypoints under src/agent_rllib/training/.
• FastAPI server under src/agent_rllib/api/server.py.
• Unit tests for the environment and tool registry.

Repository Layout

configs/
  ppo_default.yaml          Default training configuration

data/
  corpus/articles.jsonl     Small sample corpus
  tasks/eval_tasks.jsonl    Example evaluation tasks

examples/
  basic_training.py         Example RLlib training flow

scripts/
  benchmark.py              Local benchmark helper

src/agent_rllib/
  agents/                   Agent implementations
  api/                      FastAPI server
  envs/                     Simulated RL environments
  llm/                      LLM service wrappers
  tools/                    Tool registry and built-in tools
  training/                 Training and evaluation scripts

src/tests/
  test_env.py
  test_tools.py

Quickstart

git clone https://github.com/zengxiao-he/agent-rllib.git
cd agent-rllib

python3 -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt
pip install -e .

Run the tests:

pytest src/tests -q

Run a local benchmark:

python scripts/benchmark.py --output benchmark_results.json

Run the API server:

uvicorn src.agent_rllib.api.server:app --reload

Minimal Usage

from src.agent_rllib.envs import SupportBotEnv
from src.agent_rllib.agents import HybridPPOAgent

env = SupportBotEnv(difficulty="medium", tools=["search", "calculator"])
agent = HybridPPOAgent(
    observation_space=env.observation_space,
    action_space=env.action_space,
)

obs, info = env.reset()
done = False

while not done:
    action = agent.get_action(obs)
    obs, reward, terminated, truncated, info = env.step(action)
    agent.update(obs, reward, terminated or truncated, info)
    done = terminated or truncated

Training

The training entrypoint is:

python -m src.agent_rllib.training.train_ppo --config configs/ppo_default.yaml

This code path is intended for local experimentation. Before treating results as meaningful, run a fixed-seed benchmark, record the environment version, and compare against a simple baseline.

Evaluation

The evaluation entrypoint is:

python -m src.agent_rllib.training.evaluate --model checkpoints/best_model.pt

Suggested metrics for future project work:

• task completion rate
• average reward
• tool selection accuracy
• failure category distribution
• latency per decision step

Current Limitations

• The environments are simulations, so benchmark results should be interpreted as engineering signals rather than claims about real-world agent performance.
• The PPO stack is still an experimental wrapper and needs more fixed-seed evaluation before publishing headline performance numbers.
• The API server is a development interface; it still needs auth, persistence, tracing, and deployment hardening.
• The example training flow may need adjustment as Ray/RLlib APIs evolve.

Roadmap

• Add a small reproducible benchmark suite with fixed seeds and checked-in results.
• Add baseline agents for random, heuristic, pure LLM, and PPO-only comparisons.
• Add tracing for tool calls, rewards, and policy decisions.
• Add a simple hosted demo for inspecting trajectories.
• Replace qualitative README claims with benchmark tables generated from scripts in this repo.

Why This Project Exists

Most agent demos skip the feedback loop: they call tools, but they do not define an environment, reward signal, or evaluation path. This repo explores the opposite direction: start with an inspectable environment, make tool use measurable, and then improve the agent against that loop.