The output of mixtral-8x7b/evaluate-accuracy.py:

xx.xx% pass@1
{'typescript': x, 'ruby': x, 'python': x, 'javascript': x, 'php': x, 'cpp': x}
{'typescript': x, 'ruby': x, 'python': x, 'javascript': x, 'php': x, 'cpp': x}
Results
{
    'rouge1': np.float64(x),
    'rouge2': np.float64(x),
    'rougeL': np.float64(x),
    'rougeLsum': np.float64(x),
    'gsm8k': x,
    'mbxp': x,
    'gen_len': np.int64(x),
    'gen_num': x,
    'gen_tok_len': x,
    'tokens_per_sample': x
}

conatins np.float64(...) text wrapped around fp value for given fields:

• rogue1, rogue2, rogueL, rogueLsum

contains np.int64 text around long value for given fields:

• gen_len

this fails the regexes we have defined:

im not entirely sure why this behaves differently from our llama script:

Verified Fix:

1. wrap here with built-in float(round(...))

   inference/language/mixtral-8x7b/evaluate-accuracy.py

   Line 180 in 29edbb0

   result = {k: round(np.mean(v) * 100, 4) for k, v in result.items()}

2. wrap here with built-in int int(np.sum(...))

   inference/language/mixtral-8x7b/evaluate-accuracy.py

   Line 215 in 29edbb0

   'gen_len': np.sum(prediction_lens),