Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use randomly structured Set and Map in benchmarks #1075

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Use randomly structured Set and Map in benchmarks #1075

wants to merge 1 commit into from

Conversation

meooow25
Copy link
Contributor

@meooow25 meooow25 commented Nov 30, 2024 •
edited
Loading

Trees constructed from [1..n] have a specific structure of perfect
binary trees linked together. We shuffle the list so that the tree
is generated from repeated insertions instead, which forms a random
structure that should be more representative of average use cases.

Most benchmarks show an increase in measured times, the amount varying
from 10% to 80%.

@meooow25
Copy link
Contributor Author

This is motivated by #1069 (comment).

@meooow25
Copy link
Contributor Author

Effect on benchmarks, GHC 9.10.1:

Set

Expand 
Name                                  Time - - - - - - - - -    Allocated - - - - - -
                                           A       B       %         A       B       %
<$                                     25 μs   29 μs    +17%    191 KB  191 KB     +0%
<$ really                              56 μs   67 μs    +20%    191 KB  191 KB     +0%
alter absent                          299 μs  407 μs    +36%    1.0 MB  1.1 MB     +4%
alter delete                          291 μs  469 μs    +61%    1.0 MB  1.1 MB     +8%
alter insert                          296 μs  437 μs    +47%    1.1 MB  1.2 MB     +6%
alter update                          244 μs  346 μs    +41%    985 KB  1.0 MB     +5%
alterF alter absent                   168 μs  238 μs    +42%      0 B     0 B
alterF alter delete                   335 μs  528 μs    +57%    1.3 MB  1.4 MB     +8%
alterF alter insert                   345 μs  487 μs    +41%    1.5 MB  1.6 MB     +6%
alterF alter update                   187 μs  282 μs    +50%    1.3 MB  1.3 MB     +5%
alterF delete absent                  154 μs  231 μs    +49%      0 B     0 B
alterF delete present                 325 μs  525 μs    +61%    1.3 MB  1.4 MB     +9%
alterF insert absent                  349 μs  500 μs    +43%    1.6 MB  1.7 MB     +5%
alterF insert present                 188 μs  277 μs    +47%    1.3 MB  1.4 MB     +5%
alterF lookup absent                   79 μs  131 μs    +65%      0 B     0 B
alterF lookup present                  64 μs  114 μs    +78%     32 KB   32 KB     +0%
alterF no rules alter absent           91 μs  142 μs    +56%      0 B     0 B
alterF no rules alter delete          347 μs  597 μs    +72%    1.0 MB  1.1 MB     +5%
alterF no rules alter insert          373 μs  546 μs    +46%    1.1 MB  1.2 MB     +5%
alterF no rules alter update          229 μs  354 μs    +55%    985 KB  1.0 MB     +4%
alterF no rules delete absent          90 μs  140 μs    +56%      0 B     0 B
alterF no rules delete present        350 μs  570 μs    +62%    1.0 MB  1.1 MB     +6%
alterF no rules insert absent         378 μs  551 μs    +45%    1.2 MB  1.2 MB     +5%
alterF no rules insert present        233 μs  336 μs    +44%    1.0 MB  1.1 MB     +5%
alterF no rules lookup absent          86 μs  142 μs    +64%      0 B     0 B
alterF no rules lookup present         73 μs  131 μs    +79%     32 KB   32 KB     +0%
compare                                40 μs   47 μs    +17%    159 KB  159 KB     +0%
delete absent                         156 μs  219 μs    +40%      0 B     0 B
delete present                        271 μs  448 μs    +65%    1.0 MB  1.1 MB     +8%
difference                             79 μs  183 μs   +132%    287 KB  516 KB    +79%
eq                                     39 μs   46 μs    +16%    159 KB  159 KB     +0%
folds with key.foldMap_elem            10 μs   13 μs    +20%      0 B     0 B
folds with key.foldMap_traverseSum     13 μs   14 μs     +6%     32 KB   36 KB    +13%
folds with key.foldl'_sum              12 μs   15 μs    +26%      0 B     0 B
folds with key.foldl_cpsOneShotSum     42 μs   46 μs     +9%    383 KB  383 KB     +0%
folds with key.foldl_cpsSum            87 μs   81 μs     -7%    607 KB  607 KB     +0%
folds with key.foldl_skip              26 μs   32 μs    +22%    191 KB  191 KB     +0%
folds with key.foldl_traverseSum       76 μs   74 μs     -3%    479 KB  479 KB     +0%
folds with key.foldr'_sum              11 μs   15 μs    +33%      0 B     0 B
folds with key.foldr_cpsOneShotSum     42 μs   46 μs    +10%    383 KB  383 KB     +0%
folds with key.foldr_cpsSum            87 μs   82 μs     -6%    607 KB  607 KB     +0%
folds with key.foldr_elem              26 μs   31 μs    +20%    191 KB  191 KB     +0%
folds with key.foldr_traverseSum       76 μs   74 μs     -2%    479 KB  479 KB     +0%
folds.foldMap_elem                    8.4 μs   12 μs    +46%      0 B     0 B
folds.foldMap_traverseSum              17 μs   20 μs    +17%     64 KB   64 KB     +0%
folds.foldl'_sum                       11 μs   14 μs    +24%      0 B     0 B
folds.foldl_cpsOneShotSum              40 μs   46 μs    +15%    320 KB  319 KB     +0%
folds.foldl_cpsSum                     71 μs   68 μs     -4%    414 KB  414 KB     +0%
folds.foldl_skip                       24 μs   29 μs    +21%    160 KB  160 KB     +0%
folds.foldl_traverseSum                59 μs   59 μs     +0%    287 KB  286 KB     +0%
folds.foldr'_sum                       10 μs   12 μs    +20%      0 B     0 B
folds.foldr_cpsOneShotSum              39 μs   43 μs    +11%    320 KB  319 KB     +0%
folds.foldr_cpsSum                     70 μs   67 μs     -4%    414 KB  414 KB     +0%
folds.foldr_elem                       23 μs   29 μs    +27%    159 KB  159 KB     +0%
folds.foldr_traverseSum                58 μs   60 μs     +2%    287 KB  286 KB     +0%
fromAscList                            51 μs   56 μs     +9%    481 KB  481 KB     +0%
fromAscListWithKey                     59 μs   58 μs     -2%    589 KB  592 KB     +0%
fromDescList                           51 μs   50 μs     +0%    481 KB  481 KB     +0%
fromDescListWithKey                    58 μs   58 μs     +0%    592 KB  589 KB     +0%
fromDistinctAscList                    28 μs   27 μs     -5%    271 KB  272 KB     +0%
fromDistinctAscList:fusion             21 μs   21 μs     +0%    336 KB  336 KB     +0%
fromDistinctDescList                   26 μs   26 μs     +0%    272 KB  272 KB     +0%
fromDistinctDescList:fusion            21 μs   22 μs     +1%    336 KB  336 KB     +0%
fromList                               58 μs  880 μs  +1406%    319 KB  2.3 MB   +649%
fromList-desc                         505 μs  507 μs     +0%    2.6 MB  2.6 MB     +0%
insert absent                         285 μs  429 μs    +50%    1.1 MB  1.2 MB     +6%
insert present                        218 μs  201 μs     -7%    899 KB    0 B    -100%
insertLookupWithKey absent            324 μs  441 μs    +36%    1.1 MB  1.2 MB     +6%
insertLookupWithKey present           260 μs  384 μs    +47%    1.0 MB  1.1 MB     +4%
insertLookupWithKey' absent           296 μs  436 μs    +47%    1.1 MB  1.2 MB     +6%
insertLookupWithKey' present          245 μs  362 μs    +47%    1.0 MB  1.0 MB     +3%
insertWith absent                     284 μs  427 μs    +50%    1.2 MB  1.2 MB     +6%
insertWith present                    229 μs  357 μs    +55%    1.0 MB  1.1 MB     +4%
insertWith' absent                    284 μs  423 μs    +48%    1.2 MB  1.2 MB     +6%
insertWith' present                   225 μs  350 μs    +55%    1.0 MB  1.0 MB     +3%
insertWithKey absent                  286 μs  421 μs    +47%    1.1 MB  1.2 MB     +6%
insertWithKey present                 229 μs  355 μs    +55%    1.0 MB  1.1 MB     +6%
insertWithKey' absent                 291 μs  429 μs    +47%    1.1 MB  1.2 MB     +6%
insertWithKey' present                227 μs  363 μs    +59%    985 KB  1.0 MB     +4%
intersection                           33 μs  192 μs   +485%     96 KB  398 KB   +316%
lookup absent                          80 μs  129 μs    +62%      0 B     0 B
lookup present                         64 μs  122 μs    +90%     32 KB   32 KB     +0%
lookupIndex                           159 μs  161 μs     +1%    128 KB  128 KB     +0%
map                                    31 μs   39 μs    +24%    287 KB  287 KB     +0%
map really                             80 μs   91 μs    +13%    351 KB  351 KB     +0%
mapMaybe                               90 μs  105 μs    +17%    247 KB  250 KB     +1%
mapMaybeWithKey                        90 μs  106 μs    +17%    247 KB  250 KB     +1%
mapWithKey                             37 μs   42 μs    +15%    352 KB  352 KB     +0%
minView                                23 ns   22 ns     -5%    135 B   135 B      +0%
split                                 8.5 ns  194 ns  +2194%     47 B   1.1 KB  +2236%
union                                 112 μs  134 μs    +20%    561 KB  579 KB     +3%
update absent                         272 μs  393 μs    +44%    1.0 MB  1.1 MB     +4%
update delete                         269 μs  455 μs    +69%    1.0 MB  1.1 MB     +8%
update present                        211 μs  321 μs    +52%    861 KB  905 KB     +5%
updateLookupWithKey absent            297 μs  404 μs    +36%    1.0 MB  1.1 MB     +4%
updateLookupWithKey delete            283 μs  470 μs    +66%    1.0 MB  1.1 MB     +6%
updateLookupWithKey present           234 μs  334 μs    +42%    893 KB  936 KB     +4%

Map

Expand 
Name                                  Time - - - - - - - - -    Allocated - - - - - -
                                           A       B       %         A       B       %
<$                                     25 μs   29 μs    +17%    191 KB  191 KB     +0%
<$ really                              56 μs   67 μs    +20%    191 KB  191 KB     +0%
alter absent                          299 μs  407 μs    +36%    1.0 MB  1.1 MB     +4%
alter delete                          291 μs  469 μs    +61%    1.0 MB  1.1 MB     +8%
alter insert                          296 μs  437 μs    +47%    1.1 MB  1.2 MB     +6%
alter update                          244 μs  346 μs    +41%    985 KB  1.0 MB     +5%
alterF alter absent                   168 μs  238 μs    +42%      0 B     0 B
alterF alter delete                   335 μs  528 μs    +57%    1.3 MB  1.4 MB     +8%
alterF alter insert                   345 μs  487 μs    +41%    1.5 MB  1.6 MB     +6%
alterF alter update                   187 μs  282 μs    +50%    1.3 MB  1.3 MB     +5%
alterF delete absent                  154 μs  231 μs    +49%      0 B     0 B
alterF delete present                 325 μs  525 μs    +61%    1.3 MB  1.4 MB     +9%
alterF insert absent                  349 μs  500 μs    +43%    1.6 MB  1.7 MB     +5%
alterF insert present                 188 μs  277 μs    +47%    1.3 MB  1.4 MB     +5%
alterF lookup absent                   79 μs  131 μs    +65%      0 B     0 B
alterF lookup present                  64 μs  114 μs    +78%     32 KB   32 KB     +0%
alterF no rules alter absent           91 μs  142 μs    +56%      0 B     0 B
alterF no rules alter delete          347 μs  597 μs    +72%    1.0 MB  1.1 MB     +5%
alterF no rules alter insert          373 μs  546 μs    +46%    1.1 MB  1.2 MB     +5%
alterF no rules alter update          229 μs  354 μs    +55%    985 KB  1.0 MB     +4%
alterF no rules delete absent          90 μs  140 μs    +56%      0 B     0 B
alterF no rules delete present        350 μs  570 μs    +62%    1.0 MB  1.1 MB     +6%
alterF no rules insert absent         378 μs  551 μs    +45%    1.2 MB  1.2 MB     +5%
alterF no rules insert present        233 μs  336 μs    +44%    1.0 MB  1.1 MB     +5%
alterF no rules lookup absent          86 μs  142 μs    +64%      0 B     0 B
alterF no rules lookup present         73 μs  131 μs    +79%     32 KB   32 KB     +0%
compare                                40 μs   47 μs    +17%    159 KB  159 KB     +0%
delete absent                         156 μs  219 μs    +40%      0 B     0 B
delete present                        271 μs  448 μs    +65%    1.0 MB  1.1 MB     +8%
difference                             79 μs  183 μs   +132%    287 KB  516 KB    +79%
eq                                     39 μs   46 μs    +16%    159 KB  159 KB     +0%
folds with key.foldMap_elem            10 μs   13 μs    +20%      0 B     0 B
folds with key.foldMap_traverseSum     13 μs   14 μs     +6%     32 KB   36 KB    +13%
folds with key.foldl'_sum              12 μs   15 μs    +26%      0 B     0 B
folds with key.foldl_cpsOneShotSum     42 μs   46 μs     +9%    383 KB  383 KB     +0%
folds with key.foldl_cpsSum            87 μs   81 μs     -7%    607 KB  607 KB     +0%
folds with key.foldl_skip              26 μs   32 μs    +22%    191 KB  191 KB     +0%
folds with key.foldl_traverseSum       76 μs   74 μs     -3%    479 KB  479 KB     +0%
folds with key.foldr'_sum              11 μs   15 μs    +33%      0 B     0 B
folds with key.foldr_cpsOneShotSum     42 μs   46 μs    +10%    383 KB  383 KB     +0%
folds with key.foldr_cpsSum            87 μs   82 μs     -6%    607 KB  607 KB     +0%
folds with key.foldr_elem              26 μs   31 μs    +20%    191 KB  191 KB     +0%
folds with key.foldr_traverseSum       76 μs   74 μs     -2%    479 KB  479 KB     +0%
folds.foldMap_elem                    8.4 μs   12 μs    +46%      0 B     0 B
folds.foldMap_traverseSum              17 μs   20 μs    +17%     64 KB   64 KB     +0%
folds.foldl'_sum                       11 μs   14 μs    +24%      0 B     0 B
folds.foldl_cpsOneShotSum              40 μs   46 μs    +15%    320 KB  319 KB     +0%
folds.foldl_cpsSum                     71 μs   68 μs     -4%    414 KB  414 KB     +0%
folds.foldl_skip                       24 μs   29 μs    +21%    160 KB  160 KB     +0%
folds.foldl_traverseSum                59 μs   59 μs     +0%    287 KB  286 KB     +0%
folds.foldr'_sum                       10 μs   12 μs    +20%      0 B     0 B
folds.foldr_cpsOneShotSum              39 μs   43 μs    +11%    320 KB  319 KB     +0%
folds.foldr_cpsSum                     70 μs   67 μs     -4%    414 KB  414 KB     +0%
folds.foldr_elem                       23 μs   29 μs    +27%    159 KB  159 KB     +0%
folds.foldr_traverseSum                58 μs   60 μs     +2%    287 KB  286 KB     +0%
fromAscList                            51 μs   56 μs     +9%    481 KB  481 KB     +0%
fromAscListWithKey                     59 μs   58 μs     -2%    589 KB  592 KB     +0%
fromDescList                           51 μs   50 μs     +0%    481 KB  481 KB     +0%
fromDescListWithKey                    58 μs   58 μs     +0%    592 KB  589 KB     +0%
fromDistinctAscList                    28 μs   27 μs     -5%    271 KB  272 KB     +0%
fromDistinctAscList:fusion             21 μs   21 μs     +0%    336 KB  336 KB     +0%
fromDistinctDescList                   26 μs   26 μs     +0%    272 KB  272 KB     +0%
fromDistinctDescList:fusion            21 μs   22 μs     +1%    336 KB  336 KB     +0%
fromList                               58 μs  880 μs  +1406%    319 KB  2.3 MB   +649%
fromList-desc                         505 μs  507 μs     +0%    2.6 MB  2.6 MB     +0%
insert absent                         285 μs  429 μs    +50%    1.1 MB  1.2 MB     +6%
insert present                        218 μs  201 μs     -7%    899 KB    0 B    -100%
insertLookupWithKey absent            324 μs  441 μs    +36%    1.1 MB  1.2 MB     +6%
insertLookupWithKey present           260 μs  384 μs    +47%    1.0 MB  1.1 MB     +4%
insertLookupWithKey' absent           296 μs  436 μs    +47%    1.1 MB  1.2 MB     +6%
insertLookupWithKey' present          245 μs  362 μs    +47%    1.0 MB  1.0 MB     +3%
insertWith absent                     284 μs  427 μs    +50%    1.2 MB  1.2 MB     +6%
insertWith present                    229 μs  357 μs    +55%    1.0 MB  1.1 MB     +4%
insertWith' absent                    284 μs  423 μs    +48%    1.2 MB  1.2 MB     +6%
insertWith' present                   225 μs  350 μs    +55%    1.0 MB  1.0 MB     +3%
insertWithKey absent                  286 μs  421 μs    +47%    1.1 MB  1.2 MB     +6%
insertWithKey present                 229 μs  355 μs    +55%    1.0 MB  1.1 MB     +6%
insertWithKey' absent                 291 μs  429 μs    +47%    1.1 MB  1.2 MB     +6%
insertWithKey' present                227 μs  363 μs    +59%    985 KB  1.0 MB     +4%
intersection                           33 μs  192 μs   +485%     96 KB  398 KB   +316%
lookup absent                          80 μs  129 μs    +62%      0 B     0 B
lookup present                         64 μs  122 μs    +90%     32 KB   32 KB     +0%
lookupIndex                           159 μs  161 μs     +1%    128 KB  128 KB     +0%
map                                    31 μs   39 μs    +24%    287 KB  287 KB     +0%
map really                             80 μs   91 μs    +13%    351 KB  351 KB     +0%
mapMaybe                               90 μs  105 μs    +17%    247 KB  250 KB     +1%
mapMaybeWithKey                        90 μs  106 μs    +17%    247 KB  250 KB     +1%
mapWithKey                             37 μs   42 μs    +15%    352 KB  352 KB     +0%
minView                                23 ns   22 ns     -5%    135 B   135 B      +0%
split                                 8.5 ns  194 ns  +2194%     47 B   1.1 KB  +2236%
union                                 112 μs  134 μs    +20%    561 KB  579 KB     +3%
update absent                         272 μs  393 μs    +44%    1.0 MB  1.1 MB     +4%
update delete                         269 μs  455 μs    +69%    1.0 MB  1.1 MB     +8%
update present                        211 μs  321 μs    +52%    861 KB  905 KB     +5%
updateLookupWithKey absent            297 μs  404 μs    +36%    1.0 MB  1.1 MB     +4%
updateLookupWithKey delete            283 μs  470 μs    +66%    1.0 MB  1.1 MB     +6%
updateLookupWithKey present           234 μs  334 μs    +42%    893 KB  936 KB     +4%

@meooow25
Use randomly structured Set and Map in benchmarks
032ab37 
Trees constructed from [1..n] have a specific structure of perfect
binary trees linked together. We shuffle the list so that the tree
is generated from repeated insertions instead, which forms a random
structure that should be more representative of average use cases.

Most benchmarks show an increase in measured times, the amount varying
from 10% to 80%.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@meooow25