Mistake in eval_textvqa

[hedes1992]

When I run the script sh ./scripts/eval/textvqa.sh, I found that the evaluation scripts seems wrong?

The model inference and save question_id for each image case in model_vqa_loader.py
, and compare the result vs ground-truth json in eval_textvqa.py.

I run this but get mistake, I find the reason might be: code A save "(image_id, question)" as key, but use the key in code B by "(question_id, question)". I think the code A and B 'key pair should be consistent

[jiajunlong]

Could you please send your error message?

[hedes1992]

Could you please send your error message?

The original error is :

but I have change to code to annotations = {(annotation['question_id'], annotation['question'].lower()): annotation for annotation in annotations}, so this mistake disappear.

By the way, I haven't find the file 'llava_textvqa_val_v051_ocr.jsonl' for textvqa eval, but only find the file 'TextVQA_0.5.1_val.json'. I just generate the former json from the latter by removing the answer

[jiajunlong]

You can find llava_textvqa_val_v051_ocr.jsonl at eval.zip (after extracting it). You can then re-evaluate it with the new file to see if similar errors still occur.