Detailed code running steps

[kido1412y2y]

Hello, author. Could you please provide a specific code execution process, such as which code to run in the first step. I really want to reproduce your code, but my ability is really insufficient. I sincerely hope you can help me。

[ycmin95]

Thank you for your interest in our work. Currently, I do not have any plan to provide a step-by-step tutorial, but I will consider doing so in the future (maybe after June). You can reproduce the experiment following the README, as all necessary steps are clearly outlined there. If you have any questions, you can post them in an issue, which would be convenient for others to refer to as well.

[kido1412y2y]

Thank you very much for your comments and I will try to use it. Thank you sincerely for your answer.

[kido1412y2y]

Hello, author. I sincerely hope you can give me some help when you have time.

When I am running the following Data Preparation step 3

cd ./preprocess
python data_preprocess.py --process-image --multiprocessing

The terminal display is as follows

(base)user5@com:~/Downloads/VAC_CSL-main/preprocess$
python: can't open file '/home/user5/Downloads/VAC_CSLR-main/preprocess/data_preprocess.py': [Errno 2] No such file or directory

Then, I run dataset_preprocess.py in the pycharm

FileNotFoundError: [Errno 2] No such file or directory: '../dataset/phoenix2014/phoenix-2014-multisigner/annotations/manual/dev.corpus.csv'

I have already done the second step of Data Preparation, I can also see the phienix2014 after the soft link in the dataset.
home/public/ywy1/phoenix-2014-release
home/user5/Downloads/VAC_CSL-main/dataset/phoenix-2014

[ycmin95]

You can check whether this file "dev.corpus.csv" exists, which is the annotation file of phienix2014 dataset.

[kido1412y2y]

Hello, author.Thank you for your answer. I checked the folder and found this file "dev.corpus.csv". I'm sorry to mislead you, my file name is wrong, Ha ha ha.

I see
home/user5/Downloads/VAC_CSL-main/dataset/phoenix-2014

actually
home/user5/Downloads/VAC_CSL-main/dataset/phienix-2014

Author, I have a little comment that you can edit again
README
Data Preparation
step 3

before
ln -s PATH_TO_DATASET/phoenix2014-release ./dataset/phienix2014

after
ln -s PATH_TO_DATASET/phoenix2014-release ./dataset/phoenix2014

[kido1412y2y]

I encountered a problem while running the "python main.py --load-weights resnet18_slr_pretrained.pt --phase test"

File "/home/user5/Downloads/VAC_CSLR-main/utils/device.py"，line 57,in occupy.gpu
   torch.zeros(1).cuda(g)

RuntimeError: CUDA error : invalid device ordinal
CUDA kernel errors might be asynchronously reponted at some other API call, so the stacktrace below might be inconect.For debugging consider passing cUDA_LAUNCH_BLOCKING=1.

I running "print(torch.cuda.device_count())"

show
1

I running
if torch.cuda.is_available():
print('CUDA device is available.')
else:
print("CUDA device is not available.")

show
CUDA device is available.

[ycmin95]

Thank you for pointing out this typo and it has been corrected. The default configuration file use two GPUs. You can modify it according to your actual situation. For example, if you only want to use one card (e.g., cuda:0), you can make the necessary changes accordingly:

python main.py --load-weights resnet18_slr_pretrained.pt --phase test --device 0

[kido1412y2y]

Hello, author. I ran into some problems while running the suggestions you gave me
python main.py --load-weights resnet18_slr_pretrained.pt --phase test --device 0

Terminal display
FileNotFoundError: [Errno 2] No such file or directory: 'resnet18_ sLr_pretrained.pt'

I downloaded the file on Baidu pan. The name of the file is "resnet18_slr_pretrained_distill25.pt" . I put it in this directory.

home/user5/Downloads/VAC_CSL-main/resnet18_slr_pretrained_distill25.pt (Didn't decompress it)

I running
python main.py --load-weights resnet18_slr_pretrained_distill25.pt --phase test --device 0

Terminal display
Loading model
Loading model finished.
Loading data
train 5671
Apply training transform.

train 5671
Apply testing transform.

dev 540
Apply testing transform.

test 629
Apply testing transform.

Loading data finished .
.git does not exist in current dir
[ Thu Mar 23 17:09:44 2023 ] Nodel: slr_network.SLRModel.
[ Thu Mar 23 17:09:44 2023 ] Weights: resnet18_slr_pretrained_distill25.pt.
0%| l 0/68 [00:00<?,?it/s]
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thnead-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc: 98] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
lW pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thead-pool after fork.(function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thnead-pool after fork. (function pthreadpool)
[W pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork.(function pthreadpool)
lW pthreadpool-cpp.cc:90] Warning: Leaking Caffe2 thread-pool after fork.(function pthreadpool)
0%| l 0/68 [00:00<?,?it/s]
Traceback (most recent call last):
File "/home/user5/Downloads/VAC_CSLR-main/main.py"，line 211,in
processor.start( )
File "/home/user5/Downloads/VAC_CSLR-main/main.py",line 62,in start
dev_wer = seq_eval(self.arg,self.data_loader[ " dev"],self.model，self.device,
File "/home/user5/Downloads/VAC_CSLR-main/seq_scripts.py"，line 49， in seq_evalfor
batch_idx, data in enumerate(tqdm(loader)):
File "/hnome/com/anaconda3/envs/ywypytorch/lib/python3.9/site-packages/tqdm/std.py"，line 1178，in .iter
for obj in iterable:
File "/nome/com/anaconda/envs/ywypytonch/lib/python3.9/site-packages/torch/utils/data/dataloader.py",line 521,in .. e.t..
data = self._next_data()
File " /hnome/con/enacondaB/envs/ywypytonch/lit/python3.9/site-packages/toreh/utils/data/dataloader.py",，line 1283，in _next_data
return self.__process_data(data)
File "/hore/con /anaconds/envs/ ymypytonet/1tib/pythons.9/site-packages/tonch/utils/data/dataloader.py"，line 1229，in process
datadata.reraise(
File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/site-packages/torch/_utils.py"，line 425，in reraise
raise self.exc_type(msg)
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
File " /hone/conl/anacondal/envs/ymypytorcth/ltitl/python5.9/site-packages/torch/utils/datal_utils/wonker .py"line 287，in _worker_loop
data = fetcher.fetch(index)
File "/hnone/com/anaconda/envs/ywypytorch/Lio/python3.9/site-packages/torch/utits/datal_utils/fetch.py" ，line 44， in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/hore/con/anaconda/envs/ywypytorch/titb/python3.9/site-packages/torch/utils/data/ uti1s/fetch.py"，line 44， in <listcomp;
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/nome/user5/Downloads/VAC_CSLR-main/dataset/dataloader_video.py" , line 47，in -.getitem.-
input_data，label = self.normalize(input_data,label)
File "/nome/user5/Downloads/VAC_CSLR-main/dataset/dataloader_video.py"，line 78，in normalize
video,label = self.data_aug(video,label, file_id)
File "/home/user5/Downloads/VAC_CSLR-main/utils/video_augmentation.py"，line 24，in ..call-.
image = t(image)
File "/home/user5/Downloads/VAC_CSLR-main/utils/video_augmentation.py", line 157，in ._call..
im_h, im_w, im_c = clip[0].shape
IndexErrOr: list index out of range

[ycmin95]

Hi, @kido1412y2y，similar issue here, you can check whether processed images are found.

[kido1412y2y]

I see it, but I really don't understand.
Is meaning there is an error on line 62 of the "dataloader_video.py"?

How do I check whether processed images are found, Could you please describe it in more detail? What are the areas I need to compare and where do I need to correct them?

[ycmin95]

During preprocessing, frames are extracted from video, the glob function here is adopted to load frames. When it fails (the path is wrong or the frames are not extracted), the img_list is empty and leads to IndexError.

[kido1412y2y]

Hello, author, I found that there seems to be a problem with data preparation, and the original image has not been successfully adjusted to 256x256.

Without this file "/dataset/phoenix2014/phoenix-2014-multisigner/features/fullFrame-256x256px"

Only this file"/dataset/phoenix2014/phoenix-2014-multisigner/features/fullFrame-210x260px"
"/dataset/phoenix2014/phoenix-2014-multisigner/features/trackedRightHand-92x132px"

I running
cd ./preprocess
python data_preprocess.py --process-image --multiprocessing

Terminal display
Generate information dict from ../dataset/phoenix2014/phoenix-2014-multisigner/annotations/manual/dev.corpus.csv
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 540/540 [00:00<00:00, 5993.77it/s]
Resize image to 256x256px
0%| | 0/540 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/user5/Downloads/VAC_CSLR/preprocess/dataset_preprocess.py", line 73, in resize_dataset
os.makedirs(rs_img_dir)
File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/os.py", line 215, in makedirs
makedirs(head, exist_ok=exist_ok)
File "/home/com/anaconda3/envs/ywypytorch/lib/python3.9/os.py", line 225, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '../dataset/phoenix2014/phoenix-2014-multisigner/features/fullFrame-256x256px'

I running
sudo python dataset_preprocess.py --process-image --multiprocessing

Terminal display
[sudo] user5 password：
File "dataset_preprocess.py", line 22
print(f"Generate information dict from {anno_path}")
^
SyntaxError: invalid syntax

I have modified the file permissions through chmod 777
drwxrwxrwx 4 2522 2000 4096 11月 3 2017 phoenix2014- release

[ycmin95]

Perhaps you need to check the permission of features and features/fullFrame-256x256px directories and there is no need to run the script with sudo.

[Dikay1]

Hello , when I am running : python main.py --work-dir /home/user/VAC_CSLR-main/work_dir/baseline_res18/ --config /home/VAC_CSLR-main/configs/baseline.yaml --device 0 , the error will be reported as follows:

Loading model
Loading model finished.
Loading data
train 5671
Apply training transform.

train 5671
Apply testing transform.

dev 540
Apply testing transform.

test 629
Apply testing transform.

Loading data finished.
.git does not exist in current dir
[ Mon Mar 27 08:21:39 2023 ] Model: slr_network.SLRModel.
[ Mon Mar 27 08:21:39 2023 ] Weights: [].
0%| | 0/540 [00:01<?, ?it/s]
Traceback (most recent call last):
File "/home/user/VAC_CSLR-main/main.py", line 211, in
processor.start()
File "/home/user/VAC_CSLR-main/main.py", line 63, in start
"dev", 6667, self.arg.work_dir, self.recoder, self.arg.evaluate_tool)
File "/home/user/VAC_CSLR-main/seq_scripts.py", line 56, in seq_eval
ret_dict = model(vid, vid_lgt, label=label, label_lgt=label_lgt)
File "/home/user/anaconda3/envs/ctc_torch1.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/user/VAC_CSLR-main/slr_network.py", line 97, in forward
else self.decoder.decode(outputs, lgt, batch_first=False, probs=False)
File "/home/user/VAC_CSLR-main/utils/decode.py", line 28, in decode
return self.BeamSearch(nn_output, vid_lgt, probs)
File "/home/user/VAC_CSLR-main/utils/decode.py", line 42, in BeamSearch
beam_result, beam_scores, timesteps, out_seq_len = self.ctc_decoder.decode(nn_output, vid_lgt)
File "/home/user/anaconda3/envs/ctc_torch1.8/lib/python3.6/site-packages/ctcdecode/init.py", line 40, in decode
output, timesteps, scores, out_seq_len)
RuntimeError: Not enough space

After I set batch_size and test_batch_size to 1, this problem will still occur. Is there not enough graphics card memory or something else?(gpu can run normally)
Looking forward to your reply.

[ycmin95]

Hi, @Dikay1. I have not met this problem before, it seems like a problem of ctcdecoder, you can find some information from this issue.

[xxxiaosong]

Hello , when I am running : python main.py --work-dir /home/user/VAC_CSLR-main/work_dir/baseline_res18/ --config /home/VAC_CSLR-main/configs/baseline.yaml --device 0 , the error will be reported as follows:

Loading model Loading model finished. Loading data train 5671 Apply training transform.

train 5671 Apply testing transform.

dev 540 Apply testing transform.

test 629 Apply testing transform.

Loading data finished. .git does not exist in current dir [ Mon Mar 27 08:21:39 2023 ] Model: slr_network.SLRModel. [ Mon Mar 27 08:21:39 2023 ] Weights: []. 0%| | 0/540 [00:01<?, ?it/s] Traceback (most recent call last): File "/home/user/VAC_CSLR-main/main.py", line 211, in processor.start() File "/home/user/VAC_CSLR-main/main.py", line 63, in start "dev", 6667, self.arg.work_dir, self.recoder, self.arg.evaluate_tool) File "/home/user/VAC_CSLR-main/seq_scripts.py", line 56, in seq_eval ret_dict = model(vid, vid_lgt, label=label, label_lgt=label_lgt) File "/home/user/anaconda3/envs/ctc_torch1.8/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/home/user/VAC_CSLR-main/slr_network.py", line 97, in forward else self.decoder.decode(outputs, lgt, batch_first=False, probs=False) File "/home/user/VAC_CSLR-main/utils/decode.py", line 28, in decode return self.BeamSearch(nn_output, vid_lgt, probs) File "/home/user/VAC_CSLR-main/utils/decode.py", line 42, in BeamSearch beam_result, beam_scores, timesteps, out_seq_len = self.ctc_decoder.decode(nn_output, vid_lgt) File "/home/user/anaconda3/envs/ctc_torch1.8/lib/python3.6/site-packages/ctcdecode/init.py", line 40, in decode output, timesteps, scores, out_seq_len) RuntimeError: Not enough space

After I set batch_size and test_batch_size to 1, this problem will still occur. Is there not enough graphics card memory or something else?(gpu can run normally) Looking forward to your reply.

Hello. I had the same problem. Have you solved it yet?

[Dikay1]

Hi, @xxxiaosong .It's been a bit long and I can't remember some of it, I may have reinstalled ctcdecoder.

[xxxiaosong]

Hi, @xxxiaosong .It's been a bit long and I can't remember some of it, I may have reinstalled ctcdecoder.

Thanks for your reply.

[kido1412y2y]

Hello author ! When I run : python main.py --work-dir ./work_dir/vac/ --config ./configs/baseline.yaml --device 0 ，the error occurred:

/(vac) user2@com:~/data/VAC_CSLR-main$ python main.py --work-dir ./work_dir/vac/ --config ./configs/baseline.yaml --device 0
Loading model
/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torchvision/models/_utils.py:209: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
f"The parameter '{pretrained_param}' is deprecated since 0.13 and may be removed in the future, "
/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=ResNet18_Weights.IMAGENET1K_V1. You can also use weights=ResNet18_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
Loading model finished.
Loading data
train 5671
Apply training transform.

train 5671
Apply testing transform.

dev 540
Apply testing transform.

test 629
Apply testing transform.

Loading data finished.
.git does not exist in current dir
[ Thu Jul 13 21:34:29 2023 ] Parameters:
{'work_dir': './work_dir/vac/', 'config': './configs/baseline.yaml', 'random_fix': True, 'device': '0', 'phase': 'train', 'save_interval': 5, 'random_seed': 0, 'eval_interval': 1, 'print_log': True, 'log_interval': 50, 'evaluate_tool': 'sclite', 'feeder': 'dataset.dataloader_video.BaseFeeder', 'dataset': 'phoenix14', 'dataset_info': {'dataset_root': './dataset/phoenix2014/phoenix-2014-multisigner', 'dict_path': './preprocess/phoenix2014/gloss_dict.npy', 'evaluation_dir': './evaluation/slr_eval', 'evaluation_prefix': 'phoenix2014-groundtruth'}, 'num_worker': 10, 'feeder_args': {'mode': 'test', 'datatype': 'video', 'num_gloss': -1, 'drop_ratio': 1.0, 'prefix': './dataset/phoenix2014/phoenix-2014-multisigner', 'transform_mode': False}, 'model': 'slr_network.SLRModel', 'model_args': {'num_classes': 1296, 'c2d_type': 'resnet18', 'conv_type': 2, 'use_bn': 1, 'share_classifier': False, 'weight_norm': False}, 'load_weights': None, 'load_checkpoints': None, 'decode_mode': 'beam', 'ignore_weights': [], 'batch_size': 2, 'test_batch_size': 4, 'loss_weights': {'SeqCTC': 1.0}, 'optimizer_args': {'optimizer': 'Adam', 'base_lr': 0.0001, 'step': [20, 35], 'learning_ratio': 1, 'weight_decay': 0.0001, 'start_epoch': 0, 'nesterov': False}, 'num_epoch': 40}

0%| | 0/2835 [00:00<?, ?it/s]
Traceback (most recent call last):
File "main.py", line 211, in
processor.start()
File "main.py", line 45, in start
self.device, epoch, self.recoder)
File "/home/user2/data/VAC_CSLR-main/seq_scripts.py", line 18, in seq_train
for batch_idx, data in enumerate(tqdm(loader)):
File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/tqdm/std.py", line 1178, in iter
for obj in iterable:
File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 628, in next
data = self._next_data()
File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1333, in _next_data
return self._process_data(data)
File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1359, in _process_data
data.reraise()
File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/_utils.py", line 543, in reraise
raise exception
IndexError: Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
data = fetcher.fetch(index)
File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/opt/anaconda3/envs/vac/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 58, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/user2/data/VAC_CSLR-main/dataset/dataloader_video.py", line 47, in getitem
input_data, label = self.normalize(input_data, label)
File "/home/user2/data/VAC_CSLR-main/dataset/dataloader_video.py", line 78, in normalize
video, label = self.data_aug(video, label, file_id)
File "/home/user2/data/VAC_CSLR-main/utils/video_augmentation.py", line 24, in call
image = t(image)
File "/home/user2/data/VAC_CSLR-main/utils/video_augmentation.py", line 119, in call
if isinstance(clip[0], np.ndarray):
IndexError: list index out of range

Can you help me please, Thank you very much!

I have successfully used you provide the pretrained models to inference, and have also obtained the same results.
python main.py --load-weights resnet18_slr_pretrained_distill25.pt --phase test --device 0