DFT-net

Dual-Window Frequency Transformer for Rhythmic Motion Prediction rhymthic data period timing annonations of the Human3.6M dataset and dual-windewed attention model source code are provided open source from the CAROUSEL+ EU funded FET PROACT project #101017779

Overview

This is the code repo for our paper submitted at CGVC 2024.

Dependencies

Getting the Data

Human3.6m in exponential map format can be downloaded from here.

After downloading, extract actions walking and walking together for S1...11.

Our re-timed interpolated version of H3.6m dataset in exponential map format for actions walking and walking together can be downloaded from here.

Dataset Directory Structure

H3.6m
|-- S1
|-- S5
|-- S6
|-- ...
`-- S11

OurRetimedInterpolated
|-- S1
|   |-- walking_1.txt  
|   |-- walking_2.txt
|   |-- walkingtogether_1.txt  
|   |-- walkingtogether_2.txt
|-- |-- ...
`-- S11

Training Configuration

All the running args are defined in opt.py. We use following commands to train on Human3.6m datasets and representations.

To train,

HisRepItselfDCT

python main_h36m_3d.py --kernel_size 10 --dct_n 20 --input_n 50 --output_n 10 --skip_rate 1 --batch_size 32 --test_batch_size 32 --in_features 66 --dataset ./path to H3.6M dataset/

OurRe-timedDCT

python main_h36m_3d.py --kernel_size 10 --dct_n 20 --input_n 50 --output_n 10 --skip_rate 1 --batch_size 32 --test_batch_size 32 --in_features 66 --dataset ./path to OurRetimedInterpolated/

Dual-windowDCT

python main_h36m_3d.py --kernel_size 10 --dct_n 20 --input_n_run 140 --output_n 10 --skip_rate 1 --batch_size 32 --test_batch_size 32 --in_features 66 --dataset ./path to OurRetimedInterpolated/ --model_fold

Evaluation

References

D. Sinclair, A. Ademola, B. Koniaris, K. Mitchell: DanceGraph: A Complementary Architecture for Synchronous Dancing Online, 2023 36th International Computer Animation & Social Agents (CASA) .

Wei Mao, Miaomiao Liu, Mathieu Salzmann. History Repeats Itself: Human Motion Prediction via Motion Attention. In ECCV 20.