feat: add bench workflow for AWS #1330
Conversation
tschneider-aneo
force-pushed
the
ts/add-bench-aws
branch
from
cdd4d48 to
6e32c34
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ttlSecondsAfterFinished: 0 | ||
| template: | ||
| spec: | ||
| containers: |
Check warning
Code scanning / SonarCloud
Service account permissions should be restricted Medium
| template: | ||
| spec: | ||
| containers: | ||
| - name: bench-session |
Check warning
Code scanning / SonarCloud
Memory limits should be enforced Medium
tschneider-aneo
force-pushed
the
ts/add-bench-aws
branch
4 times, most recently
from
ee84b2d to
e5e9472
Compare
tschneider-aneo
force-pushed
the
ts/add-bench-aws
branch
3 times, most recently
from
701ae60 to
e767835
Compare
.github/workflows/bench-aws.yml
Outdated
| AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} | ||
| AWS_EC2_METADATA_DISABLED: true | ||
| run: | | ||
| aws s3 cp "$BENCH_RESULTS_PATH" "s3://test-armonik-bench-storage/benchclient_benchmark_${EVENT_NAME}_${TYPE}_${GHRUNID}.json" |
There was a problem hiding this comment.
Maybe you should use an environment variable here instead of the putting directly the bucket uri
There was a problem hiding this comment.
I don't get your point. Could you be a bit more explicit ?
There was a problem hiding this comment.
The bucket should be configuration variable that is stored within the repo so that we can change it if needed without modifying the workflow.
See: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables#defining-configuration-variables-for-multiple-workflows
tschneider-aneo
force-pushed
the
ts/add-bench-aws
branch
from
54bb3d7 to
82b14d7
Compare
.github/workflows/bench-aws.yml
Outdated
| run: | | ||
| set -ex | ||
| if [ "$TRIGGER" == 'push' ]; then | ||
| echo '{"include":[{"type": "localhost", "ntasks":1000, "polling-limit": 300}, {"type": "aws", "ntasks":1200000, "polling-limit": 1000, "parameters-file-path": "tools/ci/bench-aws.tfvars"}]}' > matrix.json |
There was a problem hiding this comment.
"ntasks":1200000`
Why such an oddly specific number?
There was a problem hiding this comment.
I had session resume troubles when resuming 1.5M tasks with this infra size. I wanted to ensure we wouldn't encounter this kind of problem in the CI and 1,2M seemed to be fine.
I know it is odd to not have a million tasks but I thought 1M could be a bit short for the bench duration.
tschneider-aneo
force-pushed
the
ts/add-bench-aws
branch
from
e3e1c38 to
4296079
Compare
tschneider-aneo
force-pushed
the
ts/add-bench-aws
branch
from
e1a8b2d to
8ac0e20
Compare
| DATE=$(date +"%Y-%m-%d") | ||
| aws s3 cp "$BENCH_RESULTS_PATH" "s3://armonik-bench-storage/${FILE_PREFIX}/${GHRUNID}_${DATE}/benchclient_benchmark_${EVENT_NAME}_${TYPE}.json" | ||
|
|
||
| - if: ${{ (github.event_name == 'workflow_dispatch' && inputs.destroy-on-session-end) || (github.event_name != 'workflow_dispatch' && always()) }} |
There was a problem hiding this comment.
Is the 'always()' really necessary?
There was a problem hiding this comment.
always() is here to ensure that if the workflow isn't triggered manually the infrastructure on AWS is always destroyed even when a previous step has failed.
| - name: BenchOptions__PayloadSize | ||
| value: "1" | ||
| - name: BenchOptions__ResultSize | ||
| value: "1" |
There was a problem hiding this comment.
Is it possible to put zero here?
There was a problem hiding this comment.
Never tested ! Worth trying
tschneider-aneo
force-pushed
the
ts/add-bench-aws
branch
from
8ac0e20 to
00fc251
Compare
tschneider-aneo
force-pushed
the
ts/add-bench-aws
branch
from
00fc251 to
c5e410a
Compare
|
Motivation
Related to ArmoniK benchmarking automation project, Bench application is the most suited one for testing the ArmoniK's sheer orchestration performance
Description
Use Bench action from Armonik.Action.Deploy to benchmark ArmoniK with Bench application on localhost on each commit on main, and on AWS at each release.
There's also a possibility to launch this workflow manually, with an option to choose whether the infrastructure must be automatically destroyed after the session is finished. This can be useful to perform post-mortem analysis.
It introduces a Python script that is meant to retrieve the throughput of a session and return a JSON containing its value.
It also introduces a infrastructure of reference for ArmoniK benchmarks.
Testing
This workflow was run on every push on the PR's branch.
Impact
This PR implements a milestone for ArmoniK benchmarking capacity, as AK would be benchmarked frequently.
Additional Information
Needs aneoconsulting/Armonik.Action.Deploy#24 to be merged.
Sometimes infrastructure destruction fails on AWS due to a deadlock between a security group and a subnet that happens unpredictably.
When this happens, the destruction must be taken over manually by destroying the security group and the network interface associated with it, and finished with
makerecipesdestroyandbootstrap-destroywith the prefix used by the GitHub workflow (currentlybenchmark).Checklist