-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Description
Hi, thanks for sharing the code. i got into trouble when trying the runing the mmlu_hf task.
as i dived into the code, i found pkl and npy is needed:
Lines 158 to 165 in a01e555
def load_mmlu_multi_run(model_names, dataset_name): | |
data_dir = os.path.join(RESULT_DIR, "mmlu") | |
# find number of samples for each model | |
data = [] | |
choices = ["A", "B", "C", "D"] | |
for model_n in model_names: | |
arr_path = os.path.join(data_dir, model_n, dataset_name, f"latest_predictions.npy") | |
outputs_path = os.path.join(data_dir, model_n, dataset_name, f"latest_outputs.pkl") |
however pkl and npy is not presented in the compressed file ur provided. There is only csv files.
Lines 29 to 35 in a01e555
## Basemodel Outputs | |
The base model outputs of these datasets are can be downloaded from the link below and must be put inside `results/` | |
``` | |
https://drive.google.com/drive/folders/17_5JN5koFKsnyty9klNz1nBGYLKSE4tw?usp=sharing | |
``` |
PS: the gsm8k folder has npy and pkl and was successful
Metadata
Metadata
Assignees
Labels
No labels