TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning, Andrea Ziani*, Zicong Fan*, Muhammed Kocabas, Sammy Christen, Otmar Hilliges *Equal Contribution. International Conference on 3D Vision (3DV), 2022
You can find our training code here.
- Sept. 28, 2022: Inference code and pre-trained model are ready.
- Sept. 1, 2022: Project page and arXiv paper are online.
- Download and setup pre-trained models
- Setup inference code
- Run inference code
- Acknowledgements
- Citations
In case you encounter any issue following this README, please Contact Us.
Our release models can be downloaded at this Dropbox.
Once you downloaded our release zip file, extract it in the project directory. The final directory structure should look like:
TempCLR/
├── ...
TempCLR_release/
├── FreiHAND_model/
│ └── checkpoints/
│ ├── latest_checkpoint
│ └── tempclr_freihand
├── HO3D_model/
│ └── checkpoints/
│ ├── latest_checkpoint
│ └── tempclr_ho3d
├── LICENSE
└── TempCLR_utils/
├── data/
│ ├── all_means.pkl
│ └── shape_mean.npy
└── models/
└── mano/
└── README
environment.yml
LICENCE
mano_left_extra_joints.yaml
mano_right_extra_joints.yaml
The repository provides the code to perform inference on the FreiHAND dataset and the HO-3D_v2 dataset.
In order to run the inference code, download the datasets first and extract them in the same directory. The resulting directory structure and directory names should look like:
datasets
├── freihand
└── HO3D_v2
This repository provides the recipe tempclr-env.yaml for a conda environment containing every dependency the code needs in order to work.
If you have not installed anaconda in your device, install Anaconda first and then come back to this guide.
To create the environment, open a shell in the TempCLR project directory and run this command:
conda env create -f environment.yml
Wait until the execution ends. If everything went well you should see a suggestion to activate tempclr-env .
To activate the environment run:
conda activate tempclr-env
Great! The environment has been activated. Now that the environment has been set up proceed to the next section.
Our project is based on MANO. Hence, MANO models are needed to correctly run this codebase.
Before proceeding further, download MANO models (i.e. MANO_RIGHT.pkl and MANO_LEFT.pkl) from the official website
and place them inside TempCLR_utils/models/mano directory.
In the shell where you will run the inference code, run the following commands:
export MODELS=<path_to_TempCLR_utils>/models
export MEAN_POSE_PATH=<path_to_TempCLR_utils>/data
export DATASETS=<path_to_the_datasets_directory>
export INFERENCE=<path_to_TempCLR_release>
Once followed all the previous steps, open a shell in the TempCLR project with the conda environment and environment variables set and use the following command to run the inference code:
python TempCLR/main.py --exp-cfg TempCLR/configs/evaluation_freihand.yaml
for inference on FreiHAND, and
python TempCLR/main.py --exp-cfg TempCLR/configs/evaluation_ho3d.yaml
for inference on HO-3D.
Since the ground-truth labels of the test sets are hidden in CodaLab competitions (see HO-3D and FreiHAND), the code will save a pred.json file in the pre-trained model directory.
This file contains the predictions on the test set to be uploaded on CodaLab. Zip the json file and upload it in the respective CodaLab competition to obtain the results for the different metrics reported in the paper.
To save the re-projection images produced by the pre-trained models set these two flags in the config file:
save_reproj_images: True
create_image_summaries: True
The images will be saved in the pre-trained model directory under a new directory named summary .
Muhammed Kocabas is supported by the Max Planck ETH Center for Learning Systems. The authors would like to thank Vassilis Choutas for providing the code of the baseline model (ExPose) adopted for the project.
If you find this work useful, consider citing:
@inProceedings{ziani2022tempclr,
title={TempCLR: Reconstructing Hands via Time-Coherent Contrastive Learning},
author={Ziani, Andrea and Fan, Zicong and Kocabas, Muhammed and Christen, Sammy and Hilliges, Otmar},
booktitle={International Conference on 3D Vision (3DV)},
year={2022}
}
@inproceedings{hampali2020honnotate,
title={Honnotate: A method for 3d annotation of hand and object poses},
author={Hampali, Shreyas and Rad, Mahdi and Oberweger, Markus and Lepetit, Vincent},
booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition},
pages={3196--3206},
year={2020}
}
@inproceedings{Freihand2019,
author = {Christian Zimmermann, Duygu Ceylan, Jimei Yang, Bryan Russell, Max Argus and Thomas Brox},
title = {FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape from Single RGB Images},
booktitle = {IEEE International Conference on Computer Vision (ICCV)},
year = {2019},
url = "https://lmb.informatik.uni-freiburg.de/projects/freihand/"
}
@inproceedings{ExPose:2020,
title = {Monocular Expressive Body Regression through Body-Driven Attention},
author = {Choutas, Vasileios and Pavlakos, Georgios and Bolkart, Timo and Tzionas, Dimitrios and Black, Michael J.},
booktitle = {European Conference on Computer Vision (ECCV)},
pages = {20--40},
year = {2020},
url = {https://expose.is.tue.mpg.de}
}
@article{MANO:SIGGRAPHASIA:2017,
title = {Embodied Hands: Modeling and Capturing Hands and Bodies Together},
author = {Romero, Javier and Tzionas, Dimitrios and Black, Michael J.},
journal = {ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)},
volume = {36},
number = {6},
series = {245:1--245:17},
month = nov,
year = {2017},
month_numeric = {11}
}