fix bug in python benchmark script

[thevishalagarwal]

Bug: if we use random token ids (with flag --use_random_token), decoding and encoding generates a different set of tokens which is not equal to the original set. This changes the number of prompt tokens and generates incorrect result during benchmarking. e.g.

original_tokens = np.random.randint(100, size=(1, 50))
prompt = tokenizer.decode(original_tokens )
new_tokens = tokenizer.encode(prompt)

Earlier the number of tokens was 50 but in new_tokens it may not be 50. Updated code to prevent change of prompt length.

[thevishalagarwal]

@baijumeswani Can you please review this? Thanks!

[aciddelgado]

Hello @thevishalagarwal , I spoke with the team and we'd like to keep the tokenization metric, perhaps the original random tokens can be used for the benchmark. The decoded version can be tokenized to benchmark tokenization, then this value can be discarded.

[thevishalagarwal]

@aciddelgado updated my changes without removing the tokenization metric. Please review it again. Thanks

[thevishalagarwal]

BTW, this decoding-encoding thing also changes the prompt length (number of input tokens) when using the default option of generating the prompt using the model itself.

If the initial arg for prompt_length is 300. Then generate_prompt(...) generates 300 tokens which is decoded to some text and then again encoded to tokens. The length of this token is expected to be 300 but I'm getting >300.

IMO, this is a bug and prompt_length should not be changed