Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: move benchmarks summary to the top #190

Merged
merged 1 commit into from
Sep 3, 2023
Merged

ci: move benchmarks summary to the top #190

merged 1 commit into from
Sep 3, 2023

Conversation

P403n1x87
Copy link
Owner

No description provided.

@codecov
Copy link

codecov bot commented Sep 3, 2023 •
edited
Loading

Codecov Report

Patch coverage has no change and project coverage change: -0.09% ⚠️

Comparison is base (e58eb83) 67.64% compared to head (505115c) 67.56%.

Additional details and impacted files 
@@            Coverage Diff             @@
##            devel     #190      +/-   ##
==========================================
- Coverage   67.64%   67.56%   -0.09%     
==========================================
  Files          27       27              
  Lines        2485     2485              
  Branches      761      761              
==========================================
- Hits         1681     1679       -2     
- Misses        470      471       +1     
- Partials      334      335       +1

see 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@github-actions
Copy link

github-actions bot commented Sep 3, 2023

Austin Benchmarks

Running Austin benchmarks with Python 3.10.12

Benchmark Summary

Comparison of dev against 3.5.0.

The following scenarios show a statistically significant difference in performance between the two versions.

Sample Rate Saturation Error Rate Sampling Speed
Wall time [sampling interval: 1] 🟢 🟡 🟡 🟡
Wall time [sampling interval: 10] 🔴 🟡 🟡 🟡
CPU time [sampling interval: 1] 🔴 🟡 🟡 🟡
CPU time [sampling interval: 10] 🟡 🟡 🟢 🟡
RSA keygen [sampling interval: 1] 🟡 🟡 🟡 🟢
RSA keygen [sampling interval: 100] 🟡 🟡 🟢 🟡
RSA keygen [sampling interval: 1000] 🔴 🟡 🟡 🔴
Full metrics [sampling interval: 10] 🟢 🟡 🟡 🟢
Full metrics [sampling interval: 100] 🟢 🟡 🟡 🟡
Full metrics [sampling interval: 1000] 🟡 🟡 🟡 🔴
Multiprocess wall time [sampling interval: 10] 🔴 🟡 🟡 🔴

Benchmark Results

Wall time [sampling interval: 1]

Wall time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 123000 ± 3000 1 ± 0 7e-06 ± 4e-06 12.5 ± 0.5
3.5.0 122000 ± 3000 1 ± 0 3e-05 ± 6e-05 12.6 ± 0.5
dev 125000 ± 2000 1 ± 0 8e-06 ± 3e-06 12.4 ± 0.5
Wall time [sampling interval: 10]

Wall time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 121000 ± 5000 0.544 ± 0.001 7e-06 ± 4e-06 12.9 ± 0.6
3.5.0 122000 ± 5000 0.543 ± 0.001 8e-06 ± 3e-06 12.6 ± 0.5
dev 117000 ± 5000 0.543 ± 0.002 7e-06 ± 5e-06 13.0 ± 0.8
Wall time [sampling interval: 100]

Wall time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 10600 ± 300 0.0008 ± 0.0002 3e-05 ± 3e-05 14.3 ± 0.5
3.5.0 10500 ± 300 0.0007 ± 0.0002 4e-05 ± 3e-05 14.6 ± 0.5
dev 10600 ± 400 0.0007 ± 0.0001 3e-05 ± 3e-05 14.2 ± 0.4
Wall time [sampling interval: 1000]

Wall time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1860 ± 30 0.0001 ± 0.0002 4e-05 ± 8e-05 18.0 ± 0.9
3.5.0 1860 ± 20 0.0001 ± 0.0002 0 ± 0 17.9 ± 1.0
dev 1850 ± 30 0.0002 ± 0.0003 0 ± 0 19 ± 1
CPU time [sampling interval: 1]

CPU time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 51000 ± 3000 1 ± 0 1.4e-05 ± 7e-06 23.0 ± 0.9
3.5.0 54000 ± 2000 1 ± 0 1.2e-05 ± 6e-06 22.3 ± 0.7
dev 52000 ± 2000 1 ± 0 1.2e-05 ± 5e-06 22.7 ± 0.7
CPU time [sampling interval: 10]

CPU time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 53000 ± 2000 0.9983 ± 0.0002 1.2e-05 ± 6e-06 22.6 ± 0.7
3.5.0 52000 ± 2000 0.9984 ± 0.0004 1.7e-05 ± 7e-06 22.8 ± 0.6
dev 52000 ± 4000 0.9984 ± 0.0006 1.1e-05 ± 6e-06 22 ± 1
CPU time [sampling interval: 100]

CPU time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 5430 ± 80 0.0011 ± 0.0002 9e-05 ± 7e-05 23.5 ± 0.5
3.5.0 5400 ± 100 0.001 ± 0.0002 6e-05 ± 4e-05 23.6 ± 0.7
dev 5420 ± 90 0.0011 ± 0.0003 4e-05 ± 5e-05 23.5 ± 0.7
CPU time [sampling interval: 1000]

CPU time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 946 ± 3 0.0005 ± 0.0006 0.0 ± 0.0001 31 ± 2
3.5.0 949 ± 10 0.0005 ± 0.0006 0.0 ± 0.0001 32 ± 2
dev 946 ± 3 0.0005 ± 0.0006 0 ± 0 31 ± 3
RSA keygen [sampling interval: 1]

RSA keygen [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 40000 ± 1000 1 ± 0 0.0002 ± 0.0002 24.3 ± 0.7
3.5.0 40000 ± 1000 1 ± 0 0.0002 ± 0.0001 24.7 ± 0.7
dev 40200 ± 900 1 ± 0 0.0002 ± 0.0002 24.0 ± 0.5
RSA keygen [sampling interval: 10]

RSA keygen [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 39000 ± 1000 0.98 ± 0.01 0.0002 ± 0.0002 24.6 ± 0.5
3.5.0 39600 ± 400 0.989 ± 0.006 0.0002 ± 0.0002 24.5 ± 0.5
dev 40000 ± 1000 0.985 ± 0.01 0.0002 ± 0.0001 24.2 ± 0.6
RSA keygen [sampling interval: 100]

RSA keygen [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 6280 ± 20 0.003 ± 0.002 0.001 ± 0.0008 28 ± 2
3.5.0 6290 ± 20 0.002 ± 0.002 0.0007 ± 0.0006 28 ± 2
dev 6280 ± 40 0.002 ± 0.001 0.0001 ± 0.0003 29 ± 1
RSA keygen [sampling interval: 1000]

RSA keygen [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 939 ± 8 0 ± 0 0.001 ± 0.002 35 ± 1
3.5.0 941 ± 2 0 ± 0 0.001 ± 0.002 36 ± 1
dev 937 ± 4 0.0 ± 0.001 0.001 ± 0.002 38 ± 3
Full metrics [sampling interval: 1]

Full metrics [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 50000 ± 2000 1 ± 0 2.2e-05 ± 8e-06 30 ± 1
3.5.0 51000 ± 1000 1 ± 0 2.1e-05 ± 9e-06 29.9 ± 0.7
dev 51800 ± 700 1 ± 0 2.3e-05 ± 7e-06 29.5 ± 0.7
Full metrics [sampling interval: 10]

Full metrics [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 51300 ± 1000 1 ± 0 2.9e-05 ± 9e-06 29.8 ± 0.6
3.5.0 50000 ± 3000 1 ± 0 3e-05 ± 1e-05 31 ± 2
dev 52000 ± 1000 1 ± 0 1.6e-05 ± 8e-06 29.3 ± 0.7
Full metrics [sampling interval: 100]

Full metrics [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 9800 ± 200 0.0023 ± 0.0007 5e-05 ± 4e-05 34.0 ± 0.7
3.5.0 9800 ± 200 0.0023 ± 0.0005 4e-05 ± 4e-05 34.8 ± 0.6
dev 10000 ± 100 0.002 ± 0.0004 5e-05 ± 3e-05 34.4 ± 0.5
Full metrics [sampling interval: 1000]

Full metrics [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 1850 ± 30 0.0002 ± 0.0002 6e-05 ± 9e-05 40 ± 2
3.5.0 1860 ± 30 0.0003 ± 0.0002 0.0001 ± 0.0001 39 ± 1
dev 1860 ± 30 0.0004 ± 0.0004 0 ± 0 41 ± 2
Multiprocess wall time [sampling interval: 1]

Multiprocess wall time [sampling interval: 1]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 2600 ± 200 1 ± 0 0.0003 ± 0.0004 251 ± 9
3.5.0 3200 ± 200 1 ± 0 0.0003 ± 0.0003 300 ± 20
dev 3200 ± 200 1 ± 0 0.00015 ± 5e-05 300 ± 20
Multiprocess wall time [sampling interval: 10]

Multiprocess wall time [sampling interval: 10]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 2400 ± 200 1 ± 0 0.00021 ± 7e-05 260 ± 10
3.5.0 3100 ± 100 1 ± 0 0.00015 ± 5e-05 300 ± 10
dev 3000 ± 200 1 ± 0 0.00014 ± 6e-05 320 ± 20
Multiprocess wall time [sampling interval: 100]

Multiprocess wall time [sampling interval: 100]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 2400 ± 200 0.04 ± 0.02 0.00011 ± 6e-05 260 ± 10
3.5.0 3000 ± 100 0.05 ± 0.01 8e-05 ± 5e-05 320 ± 10
dev 3000 ± 200 0.054 ± 0.006 7e-05 ± 3e-05 320 ± 20
Multiprocess wall time [sampling interval: 1000]

Multiprocess wall time [sampling interval: 1000]

Sample Rate Saturation Error Rate Sampling Speed
3.4.1 2100 ± 200 0.0013 ± 0.0005 3e-05 ± 3e-05 47 ± 4
3.5.0 2800 ± 200 0.012 ± 0.002 2e-05 ± 3e-05 94 ± 10
dev 2800 ± 100 0.013 ± 0.002 2e-05 ± 2e-05 96 ± 6

@P403n1x87 P403n1x87 marked this pull request as ready for review September 3, 2023 22:15
@P403n1x87 P403n1x87 merged commit 55f56fb into devel Sep 3, 2023
29 of 38 checks passed
@P403n1x87 P403n1x87 deleted the ci/benchmarks-summary-first branch September 3, 2023 22:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@P403n1x87