Add benchmark docker image to support getting metrics among microservices and fixed a missing package for benchmarking with Dockerfile #249
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
louie-tsai
force-pushed
the
fix_req
branch
6 times, most recently
from
1c05edb to
13bf5a9
Compare
lkk12014402
reviewed
Mar 21, 2025
lkk12014402
reviewed
Mar 21, 2025
lkk12014402
reviewed
Mar 21, 2025
louie-tsai
force-pushed
the
fix_req
branch
2 times, most recently
from
21e0d26 to
dc75b00
Compare
louie-tsai
changed the title
louie-tsai
force-pushed
the
fix_req
branch
3 times, most recently
from
ba488ac to
c8e8973
Compare
louie-tsai
commented
Mar 26, 2025
Contributor
Author
|
@lvliang-intel @lkk12014402 please help to review it. |
louie-tsai
force-pushed
the
fix_req
branch
3 times, most recently
from
5cbb1f9 to
e556cf4
Compare
Contributor
Author
|
For Gaudi2, lots of time spent on TEI |
louie-tsai
force-pushed
the
fix_req
branch
2 times, most recently
from
5c15b8c to
0e3ed00
Compare
louie-tsai
force-pushed
the
fix_req
branch
2 times, most recently
from
8cc6b23 to
9931eb5
Compare
Signed-off-by: Tsai, Louie <[email protected]> Signed-off-by: louie-tsai <[email protected]> Signed-off-by: Tsai, Louie <[email protected]>
… compose runs Signed-off-by: louie-tsai <[email protected]> Signed-off-by: Tsai, Louie <[email protected]>
lvliang-intel
approved these changes
Apr 3, 2025
Zhenzhong1
approved these changes
Apr 3, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
fixed a missing package for benchmarking with Dockerfile.
Add benchmark dockerfile to get metrics and do benchmark among micoservices starting by docker compose.
here is the benchmark_putput from ChatQnA on Gaudi
chatqnafixed_20250322_084132.zip
there will be a seperate opea-eval instance along with other docker instances from ChatQnA.
Users basically can run benchmark inside the opea-eval instance directly and all are same gaudi_default network.
Gaudi2 Snapshot running GenAIEval
https://snapshots.raintank.io/dashboard/snapshot/WC02g0ZugBYibM6I177TGsVEu3PktIZ1?orgId=0&refresh=5s
https://snapshots.raintank.io/dashboard/snapshot/SAJFc4xELoEt8417yXsEvTgb0tKQG8Jk?orgId=0
https://snapshots.raintank.io/dashboard/snapshot/heUhMVzHffbuMKGZH9rp39dYohQmDzhA?orgId=0&refresh=1m
https://snapshots.raintank.io/dashboard/snapshot/vv9k6CqWHdSCG7xSxTWDnz1g1wL1RjgZ?orgId=0
https://snapshots.raintank.io/dashboard/snapshot/iMELRy0JH2KT5cXLYaZ3luvKPYjQqrXE?orgId=0&refresh=5s
Gaudi2 snapshot for 1 request output token length 128
https://snapshots.raintank.io/dashboard/snapshot/oTUYdvDaCJSqV6TG730pDuYp97WyPoNH?orgId=0
https://snapshots.raintank.io/dashboard/snapshot/KEkRh8Pau2mgv04j3C20Fg6kQFrO2UNc?orgId=0&refresh=5s
https://snapshots.raintank.io/dashboard/snapshot/1fjlelsleLm4EhrGXzz5IXAvjHH7v2H9?orgId=0&refresh=5s
https://snapshots.raintank.io/dashboard/snapshot/X9q4GaL2KOL12Q7yktRtf7dFPPeFaj44?orgId=0&refresh=5s
Gaudi2 snapshot for 1 request output token length 560
https://snapshots.raintank.io/dashboard/snapshot/9pGI1IItBrFtkPcqZkrtjNNkjqZCciaF?orgId=0
https://snapshots.raintank.io/dashboard/snapshot/OdK4AhQQApS28wYZNv62eM7oLdZtNIc9?orgId=0&refresh=5s
Gaudi2 snapshot for 32 request with 4 concurrency output token length 128
https://snapshots.raintank.io/dashboard/snapshot/WzgByEohhBEd936YcmjuvHhsA6xZ8tbd
https://snapshots.raintank.io/dashboard/snapshot/la2aqQ6weent7QrFchTgD2DWKAkfPsxk
https://snapshots.raintank.io/dashboard/snapshot/9a2Vqiaqq5XdkO1EbrpYYPfMJYcDn5v0
Gaudi2 snapshot for 128 request with 4 concurrency output token length 128
https://snapshots.raintank.io/dashboard/snapshot/NjqIt6hvSPIQiKbtB2JwFGJGZn34m99q
https://snapshots.raintank.io/dashboard/snapshot/a1QMfo16vB9aWoLfYwlShF6YXu66GxtU
https://snapshots.raintank.io/dashboard/snapshot/U8UMdhoi4PjsCf4ecev0FIZFMiOCZScJ
Gaudi2 snapshot for 512 request with 4 concurrency output token length 128
https://snapshots.raintank.io/dashboard/snapshot/c1knBpBywmQ0mgIi12JiNSHEA6w8uoIq
https://snapshots.raintank.io/dashboard/snapshot/BxadppaKHnuKIZvnT1jsiv0iB2MKdWMk
https://snapshots.raintank.io/dashboard/snapshot/9OUecRMBQ8vMkmIkrL7dJFrpccuyV1WF
Issues
n/a.Type of change
List the type of change like below. Please delete options that are not relevant.
Dependencies
NA
Tests
test it on both Gaudi and Xeon