ManiDreams

ManiDreams: An Open-Source Library for Robust Object Manipulation via Uncertainty-aware World Models

Overview

ManiDreams implements a planning paradigm where a virtual constraint (cage) bounds object states during action selection. At each timestep, the framework generates candidate actions, predicts outcomes in parallel via a forward model (TSIP), evaluates them against cage constraints, and executes the best valid action.

Core abstractions:

• DRIS — Domain-Randomized Instance Set: universal state representation
• TSIP — Task-Specific Intuitive Physics: forward model (simulation or learned)
• Cage — Constraints with cost evaluation and validation
• Solver — Action selection via sampling or trajectory optimization

Installation

git clone https://github.com/Rice-RobotPI-Lab/ManiDreams
cd ManiDreams
pip install -e .

Optional extras: catching, diffusion, ppo, eval, docs, or all:

pip install -e ".[all]"

Quick Start

Object Pushing (Simulation)

Multi-object herding with orbital cage constraints in parallel ManiSkill environments.

python examples/tasks/object_pushing/main.py

Object Pushing (Diffusion World Model)

Cage-constrained planning using a learned diffusion world model. Requires a diffusion model checkpoint. Download push16.pt from Google Drive and place it at:

examples/physics/push_backend_learned/models/push16/model/push16.pt

python examples/tasks/object_pushing/main_pixel.py

With iterative feedback loop between diffusion planning and real execution:

python examples/tasks/object_pushing/main_pixel_feedback.py

Object Picking

MPPI-based trajectory optimization with 3D cage constraints for pick-and-place.

python examples/tasks/object_picking/main.py

Object Catching

Plate-based ball catching with optional CAGE enhancement.

# Baseline (direct policy)
python examples/tasks/object_catching/main.py --num_samples 0

# CAGE mode (8 candidate actions, 16 DRIS copies)
python examples/tasks/object_catching/main.py --num_samples 8 --num_objs_tsip 16

ManiSkill Default Tasks

PushCube, PickCube, and PushT with PPO policies and optional CAGE enhancement.

python examples/tasks/maniskill_defaults/main.py --task PushCube-v1
python examples/tasks/maniskill_defaults/main.py --task PickCube-v1
python examples/tasks/maniskill_defaults/main.py --task PushT-v0

Use --num_samples 0 for baseline (direct policy) or --num_samples 16 for CAGE mode.

Zero-shot Real-to-Sim Demo

Real-time object detection (D415 + stereo) with domain-randomized Newton physics simulation. See setup tutorial and demo videos.

TODO

• Real2sim:
  • Better shape matching
  • Multi-physics support based on newton (linkage, fluid, deformable, etc)
• Policies:
  • Integration with recent VLAs and RL pipelines
• System ID:
  • VLM-based estimation
  • from given interation history (extract from videos)
  • Online refinement

Project Structure

src/manidreams/          # Core package
├── base/                # Abstract interfaces (DRIS, TSIPBase, Cage, SolverBase)
├── cages/               # Cage implementations (Circular, DRIS, Plate, Pixel, etc.)
├── solvers/             # Samplers (PolicySampler) + Optimizers (MPPI, Geometric)
├── physics/             # TSIP implementations (Simulation, Learned)
├── executors/           # Action execution (Simulation, Real)
└── env.py               # ManiDreamsEnv: Gym-compatible interface with dream()

examples/                # Task-specific integration
├── physics/             # Backend wrappers (ManiSkill, Newton, Diamond diffusion)
├── samplers/            # Trained policies and checkpoints
├── tasks/               # Main scripts (pushing, catching, picking, real2sim)
└── executors/           # Task-specific executors (simulation, D415+FFS)

License

MIT License