feat: parallelism to speed up conflict search #171
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
letFunny
commented
Nov 7, 2024
| @@ -197,34 +199,60 @@ func (r *Release) validate() error { | |||
| } | |||
| } | |||
|
|
|||
| // Check for glob and generate conflicts. | |||
| // Check for generate and glob conflicts. | |||
There was a problem hiding this comment.
[Comment for reviewer]: This is based on a nitpick by Ed about reading generate as a verb instead of a noun.
| @@ -917,3 +945,22 @@ func packageToYAML(p *Package) (*yamlPackage, error) { | |||
| } | |||
| return pkg, nil | |||
| } | |||
|
|
|||
| type Semaphore struct { | |||
There was a problem hiding this comment.
[Comment for reviewer]: This type mostly exists for documentation because the pattern of using a channel as a semaphore can be difficult to understand when reading the code if you have not seen it before.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Right now all commands (info, find and cut) validate the release as a whole. For big releases such as ubuntu 24.04 this takes up the majority of the execution time. When looking at perf data it clearly shows that the bottleneck is strdist.GlobPath. In the past we already reduced the number of comparisons to a minimum and I have tried to optimize the code but nothing moves the needle. Because this task is CPU bounded and there are no low hanging fruits, the best approach is to parallelize the computation to all available CPUs.
In my computer with 4 logical CPUs, this change halves the total execution time for both the commands and the tests.