Skip to content

Latest commit

 

History

History
314 lines (265 loc) · 16.2 KB

README.md

File metadata and controls

314 lines (265 loc) · 16.2 KB
Raw

Automatic-sports-commentary-framework

ALL Model Video captioning is known as a method to summarize or explain a video. However, it is very difficult to create sports commentaries, which are running scene-by-scene descriptions, for sports videos by using conventional video captioning. This is the case because sports commentary requires not only specific and varied information about every scene, such as player action descriptions in baseball, but also background knowledge and dynamic at-bats statistics that are not found in the video. We propose a new system to automatically generate commentary for baseball games. In our system, given real-time baseball videos, suitable descriptions are relayed using four deep-learning models (i.e., a scene classifier, player detector, motion recognizer, and pitching result recognizer) integrated with domain ontology. Using these four deep-learning models, pieces of information about “who is doing what in which area of the field” and “what results are expected” are obtained. This approach is used to select an appropriate template, which is combined with baseball ontology knowledge for the generation of commentary. We evaluate our system using baseball games from the KBO (Korea Baseball Organization) League’s 2018 season. As a result of the evaluation, we identify that our system can act as a commentator for baseball games.

Demo

[New] Demo
(with NAO Robot, 5:11)		 [New] Demo
(with English, 6:47)		 [Old] Demo 1
(with pitching result recognition, 2:27)
Onedrive Onedrive Onedrive
Ground to 1st Base Double Play
Noop Noop
Outfield Single Hit Outfield Two-Base Hit
Noop Noop
Please wait a moment because the capacity of gif is large.

Flow

Flow_chart

Type Method Example in English
A Casting using *Web Data, Pitching Result Recognition and Ontology(knowledge) 볼 인가요? 스트라이크! 구석에 꽂히는 직구에 타자가 제대로 속았습니다. 유희관 투수 이번 시즌 높은 확률로 초구 스트라이크 잡아내고 있습니다. Is it Ball? Strike! The fastball in the corner. Yoo Hee-kwan is making his first pitch this season with a high chance.
B Casting using Scene Data and Ontology(knowledge) 오지환 타자와 유희관 투수 사이에 긴장이 흐르는 가운데 오지환 타자 이번 시즌 2할 7푼의 타율을 기록하고 있습니다. Amid tensions between Oh and Yoo, Oh has recorded a .27 batting average this season.
C Casting using Motion and Position of Player, and Ontology(knowledge) 유희관 투수 공을 던졌습니다. Yoo Hee-kwan pitched.
** The referee's judgment(strike, ball, foul, out ...) is casted by web data

Models

Scene Classification

Model

Scene Model

Classes

About. BatterBox
BatterBox Batter Close up
Noop Noop Noop
About. Field
1st. Base 2nd. Base 3rd. Base Right Outfield Center Outfield Left Outfield ShortStop
Noop Noop Noop Noop Noop Noop Noop
Etc.
Coach Gallery etc.
Noop Noop Noop

Data Augmentation

Scene

It is difficult to get a similar amount of scene data for each class because of the relatively infrequent number of plays near third base and right field. To solve this problem, we take advantage of the symmetric features of a baseball stadium. The flipped image of the 1st base area and the image of the 3rd base area look alike, despite the different stadiums. We supplement insufficient data for third base and right field by flipping first base and left field data. This is also applied in the opposite direction.

Player Detection

Model

Yolo tiny v2 model Yolo Model

Classes

Pitcher Batter Catcher Fielder Referee
Noop Noop Noop Noop Noop

Recognize Position

ex) Scene of 1st Base
Detected one field player. Detected many field players.
Noop Noop
Only one player who is in 1st Base scene
=> 1st Baseman		 The player who is in significant area of 1st Base scene
=> 1st Baseman

Motion Recognition

Model

Motion Model Motion Model2

Classes

Pitching Waiting Swing Catching - Catcher
Noop Noop Noop Noop
Throwing Walking Running Catching - Fielder
Noop Noop Noop Noop

Pitching Result Recognition

Model

PRR Model PRR Model2

Classes

  • Strike
  • Ball
  • Foul
  • Hit
  • Ground out
  • Fly out
  • Etc.

Ontology

Schema

schema

Example of retrieve knowledges

Knowledge the recent record between the hitter A and the pitcher B
Query SELECT ?date ?stadium ?inning ?result WHERE {?s ?toHitter ?A . ?s ?fromPitcher ?B . ?s ?hasTimeStamp ?t . ?s ?hasResult ?result . ?s ?inInning ?inning . ?s ?inGame ?g . ?g ?inDate ?date . ?g ?hasHomeTeam ?h . ?h ?hasHomeStadium ?stadium} order by desc(?t)
Template KOR) B 투수와 A 타자의 최근 전적은 {date} {stadium}에 진행한 경기 {inning}회 {result}입니다

ENG) The batter "A" {result} in the {inning} inning of the latest game against the pitcher "B" on {date} at {stadium}
Commentary KOR) B 투수와 A 타자의 최근 전적은 2018년 10월 10일, 잠실에 진행한 경기 8회 스트라이크 아웃입니다

ENG) The batter "A" struck out in the 8th inning of the latest game against the pitcher "B" on October 10th, 2018 at Jamsil

Web Data

web

web2ontology

How to run

#download pretrained four models

mkdir _model
cd _model
wget -O https://1drv.ms/u/s!AtL3vScJgk8Fkcpcdo4SbT09PKJlvg?e=dq4uHV
unzip -j _model.zip

#download the web data files (about result "{videoname}.txt", about ball "{videoname}_ball.txt") to _data/{videoname}/
#download the video "{videoname}.mp4" to _data/{videoname}/

#ex) 2018. 09. 06 LG vs NC game
mkdir _data
cd _data
wget https://1drv.ms/u/s!AtL3vScJgk8FkcpryZAWuKuF18hiDQ?e=cOAJov

#modify GAME_DATE, GAME_NO, FILE_NAME in settings.py to find video and web data
#modify FIRST_BATTERBOX_START_IN_VIDEO, FIRST_BATTERBOX_START_IN_OWL, START_FRAME to make the same time point of the web data and the video
#you can set the FIRST_BATTERBOX_START_IN_VIDEO, FIRST_BATTERBOX_START_IN_OWL using frame number and pitchID of {videoname}.txt of the first pitching judgement 

vi settings.py

#run the Automatic-sports-commentary-framework
python kbo.py

#run the simulation with predicted bounding boxes and labels

vi kbo.py
#modify isSimulation=False to isSimulation=True

python kbo.py

#run the simulation with predicted bounding boxes and labels

vi kbo.py
#modify isSimulation=False and isAPI=True
#modify host and port in run_server method

python kbo.py