Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OpenShift AI Caikit+TGIS MLPerf Inference Implementation for Llama2-70b #1

Open
wants to merge 75 commits into
base: master
Choose a base branch
from

Conversation

Maxusmusti
Copy link
Collaborator

No description provided.

nvyihengz and others added 3 commits January 23, 2024 11:55
… compliance tests (mlcommons#1576)

* Fix offline_min_samples in submission checker and mlcommons#1569

* Removed mlperf.conf from llama2 directory to avoid confusion

* Update submission_checker.py

* Fixes for 4.0

* Cleanup compliance dir check for models without compliance tests
mlcommons-bot and others added 20 commits January 25, 2024 19:13
…emote 'compliance/sources_checksums.json' (mlcommons#1582)

Co-authored-by: mlcommons-bot <null>
…'compliance/check.py' (mlcommons#1587)

Co-authored-by: mlcommons-bot <null>
* Ignore trailing whitespace lines in spl.txt files.

* Remove fix from sync'ed power_checker.py.

* Reformat according to black.
…mlcommons#1591)

* Add support to dump 10 compliance images during accuracy run for SDXL

* Fix typo

* Dump caption.txt in the same path
* Fix loadgen token metrics latency constrains

* Update perf constraints check for token metrics

* Add equal issue mode for LLMs models
* Add sample length check to test06

* Remove spaces in token metrics recomendation

* Add important item to Llama readme

* Fix Bug: number of tokens logged before computing them

* Fix typo: lenght -> length
* Enable equal issue mode for LLM benchmarks

* Reduce min_query_count to 1 for server/MS/SS

* Remove scenario

* Remove min_query_count so default is used; revoke padding change for equal issue offline

* Pad min_queries, not samples_per_query for non-offline

* Add documentation to the sample equal issue
* Update README.md

No longer need custom fork as the relevant changes are in the inference repository

* Update dataset.py

---------

Co-authored-by: Miro <[email protected]>
…and dlrmv2 models (mlcommons#1604)

* Update README.md

Add CM commands to download Stable diffusion models

* Update README.md

* Update README.md
* Turn equal issue mode off for Llama2 TEST06

* Add TEST06 to the output dir
* Fix submission checker and TEST06 for Llama2

* Remove redundant line

* Move test_dir check
nv-jinhosuh and others added 2 commits February 12, 2024 14:47
…UNet) (mlcommons#1624)

Currently 3D-UNet is the only workload using equal-issue mode on Offline scenario. 
Recent code change on LLM equal-issue mode caused 3D-UNet accuracy run to run more than 1 queries, causing the accuracy log to bloat and fail the accuracy checking script.
This change fixes the problem described above.
* Hotfix: DLRMv2 Audit Test01 fallback failure 

DLRMv2 Audit TEST01 may go to fallback route and the accuracy check script (accuracy-dlrm.py) didn't expect this to happen. It always expects entire sample set to be in the accuracy log while Audit TEST01 would generate subset only.

This fixes the Audit TEST01 failure described above.

* typo fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants