CI: set up a CI job for automated benchmarking (Linux, Python 3.9) #129
Conversation
HarshCasper
force-pushed
the
ci-benchmarking
branch
from
50370ba to
1570960
Compare
HarshCasper
force-pushed
the
ci-benchmarking
branch
from
1570960 to
b81e89e
Compare
There was a problem hiding this comment.
This looks good, thanks @HarshCasper. Some localized comments. The benchmarks are quite slow, I'm wondering if it would be a good idea to merge them running a subset of all benchmarks to start with, to keep the time down. Maybe just the stats or sparse ones?
| name: Benchmark CI | ||
|
|
||
| on: | ||
| pull_request: |
There was a problem hiding this comment.
I don't think we want to run this job on PRs - it takes too long.
There was a problem hiding this comment.
Same for branches. I think the best would be a scheduled job, say once every 3 days.
There was a problem hiding this comment.
My idea here was to run this on specific PRs with a specific label, like run-benchmarks. It would be quite useful for significant PRs, especially around releases. Please let me know your thoughts, otherwise, I would just remove this.
There was a problem hiding this comment.
I like that idea. That's even better than regularly scheduled runs, very user-friendly.
Can you make the label ci/run-benchmarks? Then we can later extend the pattern with ci/other-job-to-run. PyTorch has this too, and it's quite nice.
There was a problem hiding this comment.
Sure, I will make this change.
| - master | ||
| # types: [labeled] | ||
|
|
||
| # workflow_dispatch: |
| jobs: | ||
| benchmark: | ||
| name: Benchmark | ||
| # if: ${{ github.event.label.name == 'run_benchmark' && github.event_name == 'pull_request' || github.event_name == 'workflow_dispatch' }} |
There was a problem hiding this comment.
Uncomment, and add the protections for not running on a fork (see the linux_meson job)?
| - name: Install Python packages | ||
| run: | | ||
| python -m pip install numpy setuptools wheel cython asv pytest pytest-xdist pybind11 pytest-xdist mpmath gmpy2 pythran ninja | ||
| python -m pip install git+https://github.com/mesonbuild/meson.git@master |
There was a problem hiding this comment.
This can now be replaced by python -m pip install meson==0.61.1
|
|
||
| - name: Setup build and install scipy | ||
| run: | | ||
| python dev.py -j 2 --build-only --werror |
There was a problem hiding this comment.
Let's remove --werror here; it is best-practice to have that only in a single CI job (at least on the same platform).
| pushd benchmarks | ||
|
|
||
| set -x | ||
| asv machine --yes |
There was a problem hiding this comment.
This block could use a few comments on what the goal is; does it run a comparison, use baseline data, or ... ?
| echo "Contender: ${GITHUB_SHA} (${{ github.event.pull_request.head.label }})" | ||
|
|
||
| ASV_OPTIONS="--split --show-stderr --factor $ASV_FACTOR" | ||
| git fetch origin meson |
There was a problem hiding this comment.
ah, this is the branch to compare against
There was a problem hiding this comment.
No, you can leave it like this, I am keeping meson up to date and am trying out various Meson-specific build changes there. Just adding comments so this whole asv block becomes easier to understand would be great.
rgommers
force-pushed
the
meson
branch
2 times, most recently
from
0704d6a to
7deaf3e
Compare
Description
This PR:
ccachefor faster builds