[Feature] Support MMLU-CF Benchmark #1775
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
liushz
reviewed
Dec 24, 2024
liushz
reviewed
Dec 24, 2024
liushz
reviewed
Dec 24, 2024
|
@liushz Hi, can MMLU-CF be merged into the main branch? Are there any further modifications required? |
|
Please fix the lint issue. |
liushz
reviewed
Dec 30, 2024
README.md
Outdated
| @@ -57,6 +57,7 @@ Just like a compass guides us on our journey, OpenCompass will guide you through | |||
|
|
|||
| ## 🚀 What's New <a><img width="35" height="20" src="https://user-images.githubusercontent.com/12782558/212848161-5e783dd6-11e8-4fe0-bbba-39ffb77730be.png"></a> | |||
|
|
|||
| - **\[2024.12.27\]** We now support the Microsoft's Contamination-Free Multi-task language Understanding Benchmark [MMLU-CF](https://huggingface.co/datasets/microsoft/MMLU-CF). Feel free to give it a try! 🔥🔥🔥 | |||
There was a problem hiding this comment.
We don't recommend this information appear in README now, we will consider adding this in next version of OpenCompass.
liushz
reviewed
Dec 30, 2024
README_zh-CN.md
Outdated
| @@ -57,6 +57,7 @@ | |||
|
|
|||
| ## 🚀 最新进展 <a><img width="35" height="20" src="https://user-images.githubusercontent.com/12782558/212848161-5e783dd6-11e8-4fe0-bbba-39ffb77730be.png"></a> | |||
|
|
|||
| - **\[2024.12.27\]** 现已支持Microsoft去污染多任务语言理解数据集[MMLU-CF](https://huggingface.co/datasets/microsoft/MMLU-CF)，欢迎尝试! 🔥🔥🔥 | |||
There was a problem hiding this comment.
Seem as the comment above.
liushz
reviewed
Dec 30, 2024
| class MMLUCFDataset(BaseDataset): | ||
|
|
||
| @staticmethod | ||
| def load(path: str, name: str): |
There was a problem hiding this comment.
You can set huggingface as your default loading method, and remove the information in datasets_info.py
liushz
reviewed
Dec 30, 2024
| @@ -181,6 +181,12 @@ | |||
| "hf_id": "opencompass/mmlu", | |||
| "local": "./data/mmlu/", | |||
| }, | |||
| # MMLU_CF | |||
There was a problem hiding this comment.
Seem as the comment above.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This PR introduces support for the MMLU-CF benchmark. The motivation behind this contribution is to enhance the evaluation of large language models by integrating the MMLU-CF dataset, a contamination-free and more challenging multiple-choice question benchmark. This dataset contains 10K questions each for the validation set and test set, covering 14 disciplines. By implementing this, the goal is to provide a more accurate and fair evaluation, ensuring that the test results are more reliable and transparent, following the improved guidelines set by Microsoft Research.
Modification
The modification in this PR includes integrating the MMLU-CF dataset into the evaluation pipeline. This involves:
BC-breaking (Optional)
No, this modification does not break backward compatibility.
Use cases (Optional)
This PR introduces a new feature by supporting the MMLU-CF benchmark.
Checklist
Before PR:
After PR: