RL PPO Skeleton -- Knapsack Environment

A minimal, heavily-commented PPO (Proximal Policy Optimization) codebase with a custom Gymnasium environment, designed as a teaching skeleton.

What's Inside

File	Purpose
env.py	Custom 0/1 Knapsack Gymnasium environment. The agent sees items one at a time and decides to pick or skip each.
agent.py	Actor-Critic neural network (PyTorch) + full PPO-Clip algorithm with GAE.
train.py	Training loop, logging, and greedy evaluation. This is the entry point.

Quick Start

uv run train.py

That's it. uv will create a virtual environment, install dependencies, and run the script automatically.

If you don't have uv, install it first: curl -LsSf https://astral.sh/uv/install.sh | sh

Training runs for 500 episodes (~10 seconds on CPU) and prints a greedy evaluation at the end.

How It Works

┌──────────────┐      state       ┌──────────────┐
│   Knapsack   │ ───────────────> │   PPO Agent  │
│ Environment  │ <─────────────── │ (ActorCritic)│
│   (env.py)   │     action       │  (agent.py)  │
└──────────────┘                  └──────────────┘
       │                                 │
       │  reward, done                   │  update policy
       └─────────────────────────────────┘

1. The environment presents items one at a time (weight + value).
2. The agent decides to PICK (1) or SKIP (0) each item.
3. After a full episode, GAE computes advantages and the PPO-Clip objective updates the network.
4. Repeat for many episodes until the agent learns a good picking strategy.

Key Concepts Demonstrated

• Custom Gymnasium environment (reset, step, observation/action spaces)
• Actor-Critic architecture (shared backbone, separate policy + value heads)
• PPO-Clip (surrogate objective with clipping to prevent large updates)
• GAE (Generalized Advantage Estimation for variance reduction)
• Entropy bonus (encourages exploration)

Hyperparameters

All hyperparameters are defined at the top of train.py with comments. Tweak them to experiment!