[QUESTION] Confusion about return if args.partitions==1

[qazwsx74269]

When I want to preprocess data for pretraining Bert model, I find this code snippet,

python tools/preprocess_data.py \
       --input my-corpus.json \
       --output-prefix my-bert \
       --vocab-file bert-vocab.txt \
       --tokenizer-type BertWordPieceLowerCase \
       --split-sentences

does not work. It won't generate .bin and .idx files because return after if args.partitions==1 ends the program.

Megatron-LM/tools/preprocess_data.py

Line 362 in 86e2927

return