SV-RCNet

SV-RCNet: Workflow Recognition from Surgical Videos using Recurrent Convolutional Network (TMI 2018)

Introduction

The SV-RCNet repository contains the codes used in 2016 M2CAI workflow challenge and our SV-RCNet paper. Our method ranks the first in the M2CAI challenge and achieves a promising performance in one large surgical dataset, i.e., Cholec80 dataset.

The implementation is based on Caffe with Ubuntu 14.04, CUDA 8.0, cuDNN 5.0, Anaconda 2.7.

New!! The Pytorch implementation of this work is available. Please refer to 'train_singlenet_phase.py' in MTRCNet-CL repository.

Installation

1. Clone the SV-RCNet repository

   git clone https://github.com/YuemingJin/SV-RCNet.git

2. Build

   cd SV-RCNet
   cp Makefile.config.example Makefile.config
   # Adjust Makefile.config
   # Or directly use provided 'Makefile.config' file in the folder, which we have adjusted the necessary configurations, such as setting "WITH_PYTHON_LAYER := 1".
   make all -j8
   make pycaffe

Note:

• Please first install Anaconda 2.7 following official instructions. In addition, adjust path in your 'Makefile.config' file.
• For other installation issues, please follow the official instructions of Caffe.

Step by Step Recognition

Most related codes are in surgicalVideo/ folder.

1. Download data

   Cholec80 dataset or M2CAI dataset

2. Preprocess data

• Download ffmpeg and use ffmpeg to split the videos to image. We split the videos in 1 fps for Cholec80 and only split video01 as an example.

  cd surgicalVideo
  sh split_video_to_image.sh 

• Resize the image from 1920 x 1080 to 250 x 250.

Note: may need to modify the ground truth file (gt_file_Cholec80) according to the name of images you created.

3. Training the network

• Download pre-trained ResNet-50 model at https://github.com/KaimingHe/deep-residual-networks. Put it in models/ResNet-50/.
• Enter models/ResNet-50 and modify path in ResNet-50-workflow-train-val.prototxt and pre-trained model name in train_ResNet_50.sh.
• Train ResNet-50

  sh train_ResNet_50.sh 

• The trained ResNet-50 will be saved in snapshot/ folder. Please choose and copy the model to the models/SV-RCNet/ folder as the next step pre-trained model when the loss does not decrease.
• Enter python/ folder and modify paths in set_input_layer.py
• Enter models/SV-RCNet and modify pre-trained model name in train_SVRCNet.sh.
• Train SV-RCNet

  sh train_SVRCNet.sh

4. Testing

   Enter test/ folder to inference all the testing videos. Need to change paths in test.py.

   python test.py

Citation

If the code is helpful for your research, please cite our paper.

@ARTICLE{jin2018sv,  
title={SV-RCNet: Workflow Recognition From Surgical Videos Using Recurrent Convolutional Network},  
author={Jin, Yueming and Dou, Qi and Chen, Hao and Yu, Lequan and Qin, Jing and Fu, Chi-Wing and Heng, Pheng-Ann},  
journal={IEEE Transactions on Medical Imaging},    
year={2018},  
volume={37},  
number={5},  
pages={1114-1126},  
doi={10.1109/TMI.2017.2787657}
}