This repository contains the code accompanying our paper, "An LLM-based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation".
In this work, we address the limitations of traditional evaluation metrics for counter-narrative (CN) generation, such as BLEU, ROUGE, and BERTScore. These conventional metrics often overlook a critical factor: the specific hate speech (HS) that the counter-narrative is responding to. Without accounting for this context, it's challenging to truly assess the quality of generated CNs.
To tackle this, we propose a novel automatic evaluation approach. Our method uses a pairwise, tournament-style ranking system to evaluate CNs, relying on JudgeLM—a specialized language model trained to assess text quality. JudgeLM allows us to compare CNs directly without human intervention, transforming the subjective task of CN evaluation into manageable binary classification problems.
For thorough validation, we test our approach on two distinct datasets: CONAN and CONAN-MT, ensuring that our method generalizes well across different corpora.
Below, you can find a visualization of the correlation matrix, which demonstrates the effectiveness of our ranking method in comparison with traditional metrics.
Disclaimer This repository contains the core Python code used in the accompanying paper, but please note that the specific implementation of the "repetition rate" metric is proprietary and has been excluded from the shared code. The provided scripts outline the overall framework and analysis process, but the actual calculation of the repetition rate is not available in this version.
@inproceedings{zubiaga-etal-2024-llm,
title = "A {LLM}-based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation",
author = "Zubiaga, Irune and
Soroa, Aitor and
Agerri, Rodrigo",
editor = "Al-Onaizan, Yaser and
Bansal, Mohit and
Chen, Yun-Nung",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
month = nov,
year = "2024",
address = "Miami, Florida, USA",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.559/",
doi = "10.18653/v1/2024.findings-emnlp.559",
pages = "9572--9585",
abstract = "This paper proposes a novel approach to evaluate Counter Narrative (CN) generation using a Large Language Model (LLM) as an evaluator. We show that traditional automatic metrics correlate poorly with human judgements and fail to capture the nuanced relationship between generated CNs and human perception. To alleviate this, we introduce a model ranking pipeline based on pairwise comparisons of generated CNs from different models, organized in a tournament-style format. The proposed evaluation method achieves a high correlation with human preference, with a {\ensuremath{\rho}} score of 0.88. As an additional contribution, we leverage LLMs as zero-shot CN generators and provide a comparative analysis of chat, instruct, and base models, exploring their respective strengths and limitations. Through meticulous evaluation, including fine-tuning experiments, we elucidate the differences in performance and responsiveness to domain-specific data. We conclude that chat-aligned models in zero-shot are the best option for carrying out the task, provided they do not refuse to generate an answer due to security concerns."
}