Skip to content

Latest commit

 

History

History
14 lines (12 loc) · 741 Bytes

README.md

File metadata and controls

14 lines (12 loc) · 741 Bytes
Raw

G2T LLM Evaluation

Here you can find scripts for LLM evaluation on the WEBNLG-2020 dataset

How to Use

Run python llm_evaluator.py --llm=<NAME OF LLM> --dataset_folder=<PATH TO FOLDER WITH WEBNLG DATASET> --dataset_filename=<FILENAME OF WEBNLG DATASET> --output_path=<WHERE TO STORE GENERATED GRAPH DESCRIPTIONS> to generate graph descriptions Supported LLMs are:

  • llama3:8b
  • gemma2:9b
  • gpt-4o
  • gpt-4o-mini

Run python metrics_evaluator.py --preds_path=<PATH TO FILE WITH GRAPH DESCRIPTIONS FROM LLM> --dataset_folder=<PATH TO FOLDER WITH WEBNLG DATASET> --dataset_filename=<FILENAME OF WEBNLG DATASET> --output_path=<WHERE TO STORE DETAILED METRICS> to evaluate WEBNLG metrics and alignscore_evaluator.py for the AlignScore.