Evaluator name mismatch

[Zane-ZYQiu]

Thank you for your great work.

I am currently working on this dataset and trying to build something on it, but I found a major problem. The names of the evaluators defined in vbvrevalkit/eval/vbvr_bench/evaluators/init.py is different from the name of the tasks in VBVR-Dataset/tars. Specifically, the names of 50 tasks match perfectly. However, for the remaining 50 tasks, both the IDs and the names are different. When looking at it manually, I can identify which evaluator corresponds to which task based on the names (e.g., 'O-60_symbol_substitute_data-generator': SymbolSubstituteEvaluator is actually the evaluator of 'O-4_symbol_substitution_data-generator.tar'). However, this creates significant difficulties when trying to call the evaluator within the code to calculate scores.

I would like to ask: Is this a mistake with how I am using it? Or is this an actual problem within the code itself?
Could you please look into this? Thank you.

[Zane-ZYQiu]

I got it. The Evaluator and the 100 tasks in VBVR-Bench have a one-to-one correspondence, so 50 of those in-domain tasks can be matched with the training set. Then I'd like to ask if you would release the evaluators for the other 50 training tasks. This would allow us to better monitor the scores during the training process and would also enable further exploration, like reinforcement learning.

[oneScotch]

@Zane-ZYQiu Thanks again for your interest in our work and for taking the time to explore the dataset so thoroughly.

We do have plans to release the remaining evaluators, but it may take some time due to manpower constraints. We’ll notify you once they’re ready.

Appreciate your patience!