-
Notifications
You must be signed in to change notification settings - Fork 233
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Duplicate bundles for repeats before minimizing simulator count #431
Conversation
bluepill/src/BPRunner.m
Outdated
if (self.config.cloneSimulator) { | ||
self.testHostSimTemplates = [bpSimulator createSimulatorAndInstallAppWithBundles:xcTestFiles]; | ||
if ([self.testHostSimTemplates count] == 0) { | ||
return 1; | ||
} | ||
} | ||
|
||
int repeatTestsCount = [self.config.repeatTestsCount intValue]; | ||
if (repeatTestsCount > 1) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
With the goal of handling retris inside of bluepill, I think we can move this logic into the main runloop. e.g. When a test fails, simply add repeatTestsCount
bundles. Then disable retries in the subprocess
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change is not explicitly targeting failure retries. While there might be an option to solve both issues in the same way, this simply enables running a test multiple times, regardless of if the first run passes or fails.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Can you add a test or two?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@RainNapper I took another hard look at the changes and I feel the replication should be done early on.
For example, the Packing step would sort the bundles by test count (or test time estimates, if provided). In the case of multiple test bundles being replicated, packing before bundle replication is would not give the sorted bundles as one would expect.
If you fix this, then you can test this the same way we test the packing step.
Partial resolution of #424
Swap the repeat logic to run before we decide to minimize the simulators to match the bundle size.
If you were running with 10 sims, 1 test, 2 repeats, then the existing logic would cut your sims from 10 to 1, then duplicate bundles from 1 to 2. This left the repeats running in series.
In this change, this would duplicate bundles from 1 to 2, then cut sims from 10 to 2.
Some output of this run:
Notice the timings indicate they are in fact running in parallel.
Also verified by running non-headless, and it showed multiple simulators running the same test in parallel.
I was not sure of a great way to test this since I did not see clear points of abstraction, or how to check the number of simulators running in an integration test. Please advise if you feel I should add tests.