mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies

Humanoid robot performing the Drill in Box spatial memory task in IsaacLab simulation.

Deep feature map (colored by PCA) that enables spatial memory for the mindmap 3D action policy.

What is mindmap?

mindmap is a 3D diffusion policy that generates robot trajectories based on semantic 3D reconstruction of the environment. Unlike traditional approaches that rely solely on current visual input, mindmap maintains spatial memory through deep feature maps, enabling robots to perform complex manipulation tasks that require remembering the layout of the environment and the locations of objects.

Our mindmap paper demonstrates that mindmap effectively solves manipulation tasks where state-of-the-art approaches without memory mechanisms struggle. Through extensive simulation experiments, we show significant improvements in task success rates for scenarios requiring spatial memory.

mindmap builds upon the nvblox codebase, which provides a GPU-accelerated reconstruction system wrapped in PyTorch. This system is essential for building the deep feature maps that enable spatial memory in mindmap.

Open Source Release

We release our complete codebase including:

• Reconstruction System: Real-time 3D mapping and feature extraction
• Training Code: End-to-end model training pipeline
• Evaluation Tasks: Four benchmark tasks for spatial memory testing
• Pre-trained Models: Ready-to-use checkpoints for immediate evaluation

Getting Started

To get started with mindmap, see our documentation site.

License

This project is licensed under the NVIDIA License (NSCLv1). See LICENSE.md for details.

Papers

If you find mindmap useful for your research, please consider citing our work:

• Remo Steiner, Alexander Millane, David Tingdahl, Clemens Volk, Vikram Ramasamy, Xinjie Yao, Peter Du, Soha Pouya and Shiwei Sheng. "mindmap: Spatial Memory in Deep Feature Maps for 3D Action Policies". CoRL 2025 Workshop RemembeRL. arXiv preprint arXiv:2509.20297 (2025).

• Alexander Millane, Helen Oleynikova, Emilie Wirbel, Remo Steiner, Vikram Ramasamy, David Tingdahl, and Roland Siegwart. nvblox: GPU-Accelerated Incremental Signed Distance Field Mapping. ICRA 2024. arXiv preprint arXiv:2311.00626 (2024).