Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to upload your model performance to CoIR Leaderboard #2

Open
archersama opened this issue Jul 5, 2024 · 0 comments
Open

How to upload your model performance to CoIR Leaderboard #2

archersama opened this issue Jul 5, 2024 · 0 comments

Comments

@archersama
Copy link
Contributor

archersama commented Jul 5, 2024

  1. use your model to generate all the results. For other tasks, run the coir package to directly obtain the results. For CodeSearchNet and CodeSearchNet-CCR, you will get results for six programming languages, namely Codesearchnet-go,Codesearchnet-java,xxx and CodeSearchNet-ccr-go, CodeSearchNet-ccr-java,xxx respectively. This will result in a total of 12 files. You need to calculate the average for all sub-tasks within these two tasks separately.

2.Create a new issue in the repository with the following format:
title: upload new model performance
content:
{"Model": "Contriever", "Model Size (Million Parameters)": 110, "URL": "https://huggingface.co/facebook/contriever-msmarco",
"Apps": 5.14, "CosQA": 14.21, "Synthetic Text2sql":45.46, "CodeSearchNet": 34.72,
"CodeSearchNet-CCR": 35.74, "CodeTrans-Contest": 44.16, "CodeTrans-DL": 24.21,
"StackOverFlow QA": 66.05, "CodeFeedBack-ST": 55.11, "CodeFeedBack-MT": 39.23, "Avg": 36.40}

3.The repository administrator will update the results on the website.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant