Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement a test step #7

Open
Vectorrent opened this issue Dec 23, 2023 · 1 comment
Open

Implement a test step #7

Vectorrent opened this issue Dec 23, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@Vectorrent
Copy link
Contributor

Probably via the Pythia test harness: https://github.com/EleutherAI/lm-evaluation-harness
@Vectorrent Vectorrent added the good first issue Good for newcomers label Dec 23, 2023
@Vectorrent Vectorrent moved this to Backlog in Development Dec 25, 2023
@Vectorrent Vectorrent moved this from Backlog to In progress in Development Dec 25, 2023
@Vectorrent Vectorrent added enhancement New feature or request and removed good first issue Good for newcomers labels Jan 28, 2024
@Vectorrent
Copy link
Contributor Author

There's a basic shell version of the Pythia harness currently working, but we really need to convert to a Python-native version of it. The reason is for models like ModuleFormer; we need to load a custom model architecture, which hooks into the Huggingface Transformers API, before ever calling the harness. I don't think we can do that from the shell (nor are we able to re-use any of our other code).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
Development
Status: In progress
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Vectorrent