Multi-User Video Search: Bridging the Gap Between Text and Embedding Queries

Khai Trinh Xuan, Nguyen Nguyen Khoi, Huy Luong-Quang, Sang Hoa-Xuan, Anh Nguyen-Luong-Nam, Minh-Hung An

Pipeline

Dataset preparation

Dataset structure:

|- dict 
   |- ...
   |- faiss_clip_cosine.bin
   |- faiss_clipv2_cosine.bin
|- frontend
   |- ai
   |   |- public
   |   |   |- data
   |   |   |   |- KeyFrames
   |   |   |   |   |-L01
   |   |   |   |   |-L01_extra
   |   |   |   |   |-....

Dict

Download dict zip file: dict

Vector embeddings

Download bin file:

• faiss_clip_cosine.bin
• faiss_clipv2_cosine.bin

Keyframes

Download keyframes zip file and extract to folder frontend/ai/public/data.
Data part 1:

• AIC_KeyframesB1_Reduced
• AIC_KeyframesB1_Extra_Reduced

Data part 2:

• AIC_KeyframesB2_Reduced
• AIC_KeyframesB2_Extra_Reduced

Data part 3:

• AIC_KeyframesB3_Reduced
• AIC_KeyframesB2_Extra_Reduced_0
• AIC_KeyframesB2_Extra_Reduced_1
• AIC_KeyframesB2_Extra_Reduced_2

Raw video from AIChallenge 2023

Data part 1:

• AIC_VideoB1v1
• AIC_VideoB1v2

Data part 2:

• AIC_VideoB2

Data part 3:

• AIC_VideoB3v1
• AIC_VideoB3v2
• AIC_VideoB3v3

Dataset extraction

Detailed on dataset extraction: data

Installation

• Backend

conda create -n AIChallenge2023
conda activate AIChallenge2023
pip install git+https://github.com/openai/CLIP.git
pip install -r requirements.txt

• Frontend

Install nodejs: https://nodejs.org/en/download

npm install

• DB Sever

pip install flask
pip install flask-cors
pip install flask-socketio
pip install pyngrok==4.1.1
ngrok authtoken your_token # Add your ngrok authentication

Usage

It is recommended to configure the environment using Anaconda. Linux support only.

• Backend

Using local machine, from root of repo:

python3 app.py

Using colaboratory, run appNotebook (App section) for starting the backend.

• Frontend

Change url in frontend/ai/src/helper/web_url.js.

cd frontend/ai/
npm run dev

• DB Sever

Open 2 terminal and run:

python appStorage.py

ngrok http 5000

• Interface