diff --git a/README.md b/README.md index 25c290c..a576bc1 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,10 @@ -# throttling-demo +# Dynamically Throttleable Neural Networks -Latent AI throttle network demo. - -The demo is an end-to-end trainable throttleable neural network for dynamic hand gesture recognition. It consists of three main conponents: gated 3D Convolutional Neural Networks (C3D) data network for gesture recognition of five classes [Swiping left/right/up/down, no gesture], a small controller network providing utilization parameter for the data network, and single-hand keypoint estimation network as a add-on module. The demo also has a simple OpenCV visualization. +This codebase contains code for the paper "[Dynamically Throttleable Neural Networks](https://hengyueliu.com/assets/files/Dynamically_throttleable_neural_networks.pdf)" published on "Machine Vision and Applications", 2022. ## Usage -Download the pretrained weights from [latentai > sample-models > demo branch](https://gitlab.com/latentai/sample-models/tree/demo/demo). +Download the pretrained weights in release. Put `controller_network_2.pkl.latest` inside folder `ckpt/controller`. @@ -26,8 +24,6 @@ The concepts and implementations take from this paper [Toward Runtime-Throttleab Gesture recognition framework is based on a basic [C3D](https://arxiv.org/abs/1412.0767) network with gated version implemented. Some implementations are from [here](https://github.com/jfzhang95/pytorch-video-recognition). Only five gesture classes are used. -The detailed structure is listed here: (todo) - ## Hand Keypoint Estimation Currently, the single-hand keypoints detection model is just used for hand keypoints visualization, but the contextual information can be used for building a data-driven controller. The implementation is based on a variant [Global Context for Convolutional Pose Machines](https://arxiv.org/pdf/1906.04104.pdf) of the original paper [Convolutional Pose Machines](https://arxiv.org/pdf/1602.00134.pdf). Its implementation can be found [here](https://github.com/Daniil-Osokin/gccpm-look-into-person-cvpr19.pytorch). Part of the implementations can also be found [here](https://github.com/HowieMa/CPM_Pytorch). @@ -78,8 +74,22 @@ The dataset for training the gesture recognition model is from [20BN-jester V1]( ## Demo -![](demo.gif)*Demo application interface (take a little time to load)* - - +Demo from Latent AI [link](https://player.vimeo.com/video/432628321?h=dd1eb87f23). + +## Citations + +If you find our work helpful, please cite: +```bibtex +@article{liu2022dynamically, + title={Dynamically throttleable neural networks}, + author={Liu, Hengyue and Parajuli, Samyak and Hostetler, Jesse and Chai, Sek and Bhanu, Bir}, + journal={Machine Vision and Applications}, + volume={33}, + number={4}, + pages={59}, + year={2022}, + publisher={Springer} +} +``` diff --git a/demo.gif b/demo.gif deleted file mode 100644 index bae1b9c..0000000 Binary files a/demo.gif and /dev/null differ