README.md

Unofficial implement of Live3DPortrait

This is an unofficial implement of Live3DPortrait: Real-Time Radiance Fields for Single-Image Portrait View Synthesis in Pytorch.

output.mp4

Pretrained Model

The checkpoints can be downloaded from here.

Place them under ./pretrained_models/ folder.

└── root
    ...
    └── pretrained_models
        └── encoder_render.pt
        └── ffhqrebalanced512-128.pth
        └── model_ir_se50.pth        

Usage

python scripts/inference.py --input ./imgs/input.png --output ./imgs/output.png --checkpoint ./checkpoints/ckpt.pth 

Note that the input image should be preprocessed as per EG3D. The LT model implementation differs from the original, yet achieves nearly identical performance.

TODO

• Upload inference code and checkpoint.
• Release Training code.
• Release LT model and checkpoint.

Acknowledgement

This repo is based on triplanenet, EG3D. Thanks to their great work!

@inproceedings{trevithick2023,
  author = {Alex Trevithick and Matthew Chan and Michael Stengel and Eric R. Chan and Chao Liu and Zhiding Yu and Sameh Khamis and Manmohan Chandraker and Ravi Ramamoorthi and Koki Nagano},
  title = {Real-Time Radiance Fields for Single-Image Portrait View Synthesis},
  booktitle = {ACM Transactions on Graphics (SIGGRAPH)},
  year = {2023}
}

@inproceedings{Chan2022,
  author = {Eric R. Chan and Connor Z. Lin and Matthew A. Chan and Koki Nagano and Boxiao Pan and Shalini De Mello and Orazio Gallo and Leonidas Guibas and Jonathan Tremblay and Sameh Khamis and Tero Karras and Gordon Wetzstein},
  title = {Efficient Geometry-aware {3D} Generative Adversarial Networks},
  booktitle = {CVPR},
  year = {2022}
}

@article{bhattarai2024triplanenet,
  title={TriPlaneNet: An Encoder for EG3D Inversion},
  author={Bhattarai, Ananta R. and Nie{\ss}ner, Matthias and Sevastopolsky, Artem},
  booktitle={IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2024}
}