You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"""Compute the correlations between human scores and automated metric scores on batch of samples, each of which is a pair of (doc, sys summ) or (ref summ, sys summ)
File~/git/evalbase/eval_utils.py:85, inbatched_corr(corr_df, human_scores, batch_result_df, corr_metrics, batchID)
83else:
84pass--->85cc=eval(f"scipy.stats.{corr_metric}")(human_score, metric_score)[0]
86corr_df.loc[
87 (corr_metric, aspect_name, approach, model, score_name), # row88batchID89 ] =cc90returncorr_dfFile~/.local/lib/python3.10/site-packages/scipy/stats/_stats_py.py:4411, inpearsonr(x, y, alternative)
4408raiseValueError('x and y must have the same length.')
4410ifn<2:
->4411raiseValueError('x and y must have length at least 2.')
4413x=np.asarray(x)
4414y=np.asarray(y)
ValueError: xandymusthavelengthatleast2.
This issue appears in factCC and qags (which have no reference summary), but not in frank (which has reference summary). Not sure if it is caused by the lack of reference summary.
The text was updated successfully, but these errors were encountered:
This issue appears in factCC and qags (which have no reference summary), but not in frank (which has reference summary). Not sure if it is caused by the lack of reference summary.
If the reason is the lack of reference summaries, the error seems to have appeared too late. Can you try to use a placeholder, e.g., "I am a happy reference summary" as reference summaries for FactCC and QAGS? The placeholder will not be used by consistency metrics anyway.
EvalBase currently does NOT support FactCC, Frank, and QAGS. So, can you also make a PR to push your code?
This
scipy.stats
call:EvalBase/eval_utils.py
Lines 70 to 84 in cef8dbf
will throw error:
when
This issue appears in factCC and qags (which have no reference summary), but not in frank (which has reference summary). Not sure if it is caused by the lack of reference summary.
The text was updated successfully, but these errors were encountered: