Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unclear "pending evaluation" status - Model Aquila-7B #53

Open
lehors opened this issue Nov 19, 2024 · 3 comments
Open

Unclear "pending evaluation" status - Model Aquila-7B #53

lehors opened this issue Nov 19, 2024 · 3 comments

Comments

@lehors
Copy link
Collaborator

lehors commented Nov 19, 2024

The MOT merely states that the model is pending evaluation without giving any information as to what this means and what it will take to change this.
This actually appears to be rooted in the fact that the distribution uses a unknown license but the system ought to be more informative so that the user can remedy the situation.
See https://mot.isitopen.ai/model/904

@lehors
Copy link
Collaborator Author

lehors commented Nov 19, 2024

Looking into the code I found that this is related to the model using a license that is "pending evaluation".
See

Not sure how the evaluation is supposed to take place though.

@gregcube
Copy link
Collaborator

I believe this was implemented when we were originally importing the spreadsheet data. Any model with a "pending evaluation" license was to behave this way. Perhaps it isn't needed anymore.

@lehors
Copy link
Collaborator Author

lehors commented Nov 26, 2024

Ok. Good to know. Thanks for the explanation. Much appreciated!

We still want to be able to use the spreadsheet import function and I think the same could happen with import from the model YAML files but I think we ought to have a message that's a bit more informative as to what's going on and I can't see in the admin interface any way to manage models that are in that state. I will investigate further but if you know of something, please, let me know.

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants