3D Player Tracking with Multi-View Stream

Project for 3DV 2021 Spring @ ETH Zurich [Report Link]

This repo contains a full pipeline to support 3D position tracking of soccer players, with multi-view calibrated moving/fixed video sequences as inputs.
- In single-camera tracking stage, Tracktor++ is used to get 2D positions.
- In multi-camera tracking stage, 2D positions are projected into 3D positions. Then across-camera association is achieved as an optimization problem with spatial, temporal and visual constraints.
- In the end, visualization in 2D, 3D and a voronoi visualization for sports coaching purpose are provided.

3D Tracking	Sports Coaching

Demo

Check demo scripts as examples

Currently, processed data is under protection due to legal issues.

• Run the demo visualization on the moving cameras

bash script/demo_moving.sh

• Run the demo visualization on the fixed cameras

bash script/demo_fix.sh

Preprocessing

• Split video into image frames

python src/utils/v2img.py --pathIn=data/0125-0135/CAM1/CAM1.mp4 --pathOut=data/0125-0135/CAM1/img --splitnum=1

• Estimate football pitch homography (size 120m * 90m ref)

FIFA official document

python src/utils/computeHomo.py --img=data/0125-0135/RIGHT/img/image0000.jpg --out_dir=data/0125-0135/RIGHT/

• Handle moving cameras

python src/utils/mov2static.py --calib_file=data/calibration_results/0125-0135/CAM1/calib.txt --img_dir=data/0125-0135/CAM1/img --output_dir=data/0125-0135/CAM1/img_static

• Convert ground truth/annotation json to text file

python src/utils/json2txt.py --jsonfile=data/0125-0135/0125-0135.json

Single-camera tracking

• Object Detector: frcnn_fpn
  Train object detector and generate detection results with this Google Colab notebook. [pretrained model]
• Run Tracktor++
  Put trainded object detector model_epoch_50.model into src/tracking_wo_bnw/output/faster_rcnn_fpn_training_soccer/.
  Put data and calibration results into src/tracking_wo_bnw/.

cd src/tracking_wo_bnw
python experiments/scripts/test_tracktor.py

• Run ReID(team id) model

python src/team_classification/team_svm.py PATH_TO_TRACKING_RESULT PATH_TO_IMAGES

• Convert tracking results to coordinates on the pitch

Equation to find the intersection of a line with a plane (ref)

python src/calib.py --calib_path=PATH_TO_CALIB --res_path=PATH_TO_TRACKING_RESULT --xymode --reid

# also plot the camera positions for fixed cameras
python src/calib.py --calib_path=PATH_TO_CALIB --res_path=PATH_TO_TRACKING_RESULT --viz

Across-camera association

• Run two-cam tracker

python src/runMCTRacker.py 

# add team id constraint
python src/runMCTRacker.py --doreid

• Run multi-cam tracker (e.g. 8 cams)

python src/runTreeMCTracker.py --doreid

Evaluation

• Produce quatitative results (visualize results)

visualize 2d bounding box

# if format <x, y, w, h>
python src/utils/visualize.py --img_dir=data/0125-0135/RIGHT/img --result_file=output/tracktor/16m_right_prediction.txt 
# if format <x1, y1, x2, y2>
python src/utils/visualize.py --img_dir=data/0125-0135/RIGHT/img --result_file=output/iou/16m_right.txt --xymode
# if with team id
python src/utils/visualize.py --img_dir=data/0125-0135/RIGHT/img --result_file=output/tracktor/16m_right_prediction.txt --reid
# if 3d mode
python src/utils/visualize.py --img_dir=data/0125-0135/RIGHT/img --result_file=output/tracktor/RIGHT.txt --calib_file=data/calibration_results/0125-0135/RIGHT/calib.txt  --pitchmode

visualize 3d tracking result with ground truth and voronoi diagram

python src/utils/visualize_on_pitch.py --result_file=PATH_TO_TRACKING_RESULT --ground_truth=PATH_TO_GROUND_TRUTH

visualize 3d ground truth on camera frames (reprojection)

python src/utils/visualize_tracab --img_path=PATH_TO_IMAGES --calib_path=PATH_TO_CALIB --gt_path=PATH_TO_TRACAB_GT --output_path=PATH_TO_OUTPUT_VIDEO

• Produce quantitative result

# 2d <frame id, objid, x, y, w, h, .., ...>
python src/motmetrics/apps/eval_motchallenge.py data/0125-0135/ output/tracktor_filtered

# 3d
python src/utils/eval3d.py --pred=output/pitch/EPTS_3_pitch.txt_EPTS_4_pitch.txt.txt --fixcam  --gt=data/fixedcam/gt_pitch_550.txt
python src/utils/eval3d.py --fixcam --boxplot

Acknowledgement

We would like to thank the following Github repos or softwares:

• Supervisely
• Tracktor++
• IoU tracker
• SORT
• Football crunching (footyviz)
• Pymotmetrics

Authors

Yuchang Jiang, Tianyu Wu, Ying Jiao, Yelan Tao