Tests Appear To Be Running Twice When I Execute Using Default Parameters

[bhreinb]

Hi there,

I checked out your repo. It looks great. However it would seem the tests run twice when I run in parallel...

My use case is the following...

1. I have 2 feature files that have a scenario each
2. I use cucumber profiles for my test execution meaning I have a cucumber.js file at the project root folder
3. The main thread I pass flag '--dry-run' so not to repeat running the tests again on the main thread. I set that within the cucumber.js profile file

Please see the output from the test run below...

/usr/local/bin/node /usr/local/lib/node_modules/npm/bin/npm-cli.js run e2e-test-parallel --scripts-prepend-node-path=auto

> testcafe-cucumber@0.1.0 e2e-test-parallel /Users/bhrein.brannick/Documents/src/testcafe-poc/testcafe-cucumber
> node $NODE_DEBUG_OPTION ./node_modules/cuke-skywalker/bin/cuke-skywalker

Cucumber version: 4.2.1
Runner Is Been Sent ./gherkin/**/*.feature --require ./step_definitions/*.js --require ./cucumber/*.js --dry-run
Doling out 2 features across 4 workers as they are available.
1 features remaining
0 features remaining
Parallel Worker Thread Executing Feature File gherkin/github.feature
Runner Is Been Sent gherkin/github.feature --require ./step_definitions/*.js --require ./cucumber/*.js --world-parameters '{"browser":"chrome"}'
World Has Been Called By PID 18517
Running Test Scenario 'Searching for TestCafe on GitHub' Under PID 18517
 Running tests in:
 - Chrome 68.0.3440 / Mac OS X 10.13.6

 fixture
The Test Controller Is Ready Now
..... ✓ test


 1 passed (7s)
.World Has Been Called By PID 18517
Running Test Scenario 'Searching for TestCafe on GitHub' Under PID 18517
 Running tests in:
 - Chrome 68.0.3440 / Mac OS X 10.13.6

 fixture
The Test Controller Is Ready Now
..... ✓ test


 1 passed (5s)
.

1 scenario (1 passed)
4 steps (4 passed)
0m29.826s

Parallel Worker Thread Executing Feature File gherkin/google.feature
Runner Is Been Sent gherkin/google.feature --require ./step_definitions/*.js --require ./cucumber/*.js --world-parameters '{"browser":"chrome"}'
World Has Been Called By PID 18518
Running Test Scenario 'Searching for TestCafe by Google' Under PID 18518
 Running tests in:
 - Chrome 68.0.3440 / Mac OS X 10.13.6

 fixture
The Test Controller Is Ready Now
..... ✓ test (screenshots: reports/screenshots/2018-08-29_18-15-53/test-1)


 1 passed (10s)
.World Has Been Called By PID 18518
Running Test Scenario 'Searching for TestCafe by Google' Under PID 18518
 Running tests in:
 - Chrome 68.0.3440 / Mac OS X 10.13.6

 fixture
The Test Controller Is Ready Now
..... ✓ test (screenshots: reports/screenshots/2018-08-29_18-16-11/test-1)


 1 passed (8s)
.

1 scenario (1 passed)
4 steps (4 passed)
0m33.612s


Process finished with exit code 0

The overall test results at the end list the scenario been run which passes although looking at the test run output unfortunately it does seem to execute the test twice. Observing the browser windows open that was my thinking but the log statements for me prove it beyond reasonable doubt. Thoughts? Let me know if you need any more info and thanks in advance.

[tomatobrown]

can you post the full profile you are using...or as much of it as you can

[bhreinb]

Hi @tomatobrown,

Thanks for your response. I can do better than that, namely I will attach the sample project I'm using your library with 👍 .

testcafe-cucumber.zip

Let me know if you have any trouble running the project. When you run the parallel test you should see more or less the same output as above in my initial comment. Also, within the file 'cucumber.js' you will see how I build the profile plus how I use your library too.

Just on the use of your library it would be super if you exposed a thread id to pid map in the main runner so it could be possibly queried in the worker thread. I'm figuring that out via a hack essentially, you will see that in the code namely within 'cucumber.js' but if you expose that then it reduces the code within that file. Thanks in advance.

[bhreinb]

Hi @tomatobrown,

Was the attachment of any use in reproducing the issue I reported? Many thanks in advance.

[jody-zeitler]

@bhreinb I haven't dug very deeply into this. We tend to lean on our CI server to accomplish the work of profiles, and don't have much exposure to them. From your output though it looks like cuke-skywalker is spawning processes 18517 and 18518 and handing the two features off to the cucumber executable. Any duplicate runs after that point would be the responsibility of cucumber itself.

[bhreinb]

Hi @jody-zeitler

Thanks for your comments. I'm unconvinced as of yet whether this is a an issue with cucumberjs, unless you are using https://github.com/cucumber/cucumber-js/blob/master/docs/cli.md#parallel-experimental to do parallelisation?

Below I include a test run output that doesn’t use your module, namely it goes directly via cucumberjs which uses profiles also and executes as expected. This execution just runs on one browser on the main thread.

/usr/local/bin/node /usr/local/lib/node_modules/npm/bin/npm-cli.js run e2e-test --scripts-prepend-node-path=auto

> testcafe-cucumber@0.1.0 e2e-test /Users/bhrein.brannick/Documents/TEST/testcafe-cucumber
> node $NODE_DEBUG_OPTION ./node_modules/cucumber/bin/cucumber-js --format json:reports/report.json

Main Thread Executing Feature File ./gherkin/github.feature
Runner Is Been Sent ./gherkin/github.feature --require ./step_definitions/*.js --require ./cucumber/*.js  --tags @chrome and (@auto_regression) --world-parameters '{"browser":"chrome"}'
World Has Been Called By PID 24740
Running Test Scenario 'Searching for TestCafe on GitHub' Under PID 24740
 Running tests in:
 - Chrome 69.0.3497 / Mac OS X 10.13.6

 fixture
The Test Controller Is Ready Now
..... ✓ test


 1 passed (8s)
.

1 scenario (1 passed)
4 steps (4 passed)
0m16.806s

Process finished with exit code 0

Let me know if you need any more detail. Btw the sample project includes 2 sample scripts

e2e-test
e2e-test-parallel

The output for this comment was generated using e2e-test.

[tomatobrown]

Hey @bhreinb, I didn't completely dig into your problem either, but I believe it has something to do with that cucumber.js file you were using. I'm not 100% sure, but I did refactor your example project into one that works correctly with cuke-skywalker. The google test was failing, so I ignored it and copied the github one that was passing, you'll see.

Attached:
testcafe-cucumber.zip

Sample Run:
`
tmader-brown@RBH12588:~/github/testcafe-cucumber (master):/usr/local/bin/node node_modules/.bin/cuke-skywalker -t "not @mute"
Cucumber version: 4.2.1
Doling out 2 features across 4 workers as they are available.
1 features remaining
0 features remaining
World Has Been Called By PID 80915
Running Test Scenario 'Searching for TestCafe on GitHub2' Under PID 80915
Running tests in:

• Chrome 68.0.3440 / Mac OS X 10.13.6

fixture
The Test Controller Is Ready Now
..... ✓ test

1 passed (5s)
.

1 scenario (1 passed)
4 steps (4 passed)
0m12.579s

World Has Been Called By PID 80914
Running Test Scenario 'Searching for TestCafe on GitHub' Under PID 80914
Running tests in:

• Chrome 68.0.3440 / Mac OS X 10.13.6

fixture
The Test Controller Is Ready Now
..... ✓ test

1 passed (5s)
.

1 scenario (1 passed)
4 steps (4 passed)
0m13.888s
`

[bhreinb]

Hi @tomatobrown

Thanks for looking into this. I downloaded the sample program you provided and I can confirm it works out of the box. However, that is because the file 'cucumber.js' is not picked up by cuke-skywalker. You can see more about profiles here

https://github.com/cucumber/cucumber-js/blob/master/docs/cli.md#profiles

I'm of the belief that the module here isn't handling profiles (albeit the main thread cuke-skywalker does seem to acknowledge it...hence why I add cli argument --dry-run so tests aren't run in the main thread again). The title of the bug should mention profiles so that's my bad. In addition to this when I run script "e2e-test" (which doesn't use cuke-skywalker this works as expected every time for me).

If it helps in debugging I can refactor cucumber.js profile file to be class design so to make it is easier to diagnose if that helps? It's on my todo list to do anyway.

[tomatobrown]

I understand how profiles work, but your cucumber.js is attempting to do much much more.

I assume you've seen the example in the cucumber project:
https://github.com/cucumber/cucumber-js/blob/master/cucumber.js

What functionality are you looking for the cucumber.js file to do that doesn't already exist in either cucumber-js or cuke-skywalker?

[bhreinb]

I simplified the example again and it is attached. Apologies this is what I should have done the first time around.

testcafe-cucumber.zip

Here is what I'm doing

1. We examine argv arguments for presence of path to cuke-skywalker
2. If so its deemed a main thread so we add cucumber cli argument '--dry-run' to the default profile so tests are not run in the main thread along with the original gherkin pattern passed in
3. As the path isn't cuke-skywalker then its deemed to be a worker thread and a cucumberjs executor so then we pass tags and world parameters to the worker threads
4. In the worker thread we replace the original gherkin pattern we got from the main thread with the one passed to the worker thread via cuke-skywalker which is sent onto cucumber-js as a default profile

Hope all the above makes sense? I'm essentially doing profile functionality that cucumber-js supports.

[tomatobrown]

Yea, I understand what you're trying to do. It seems like most of this should be handled with the CI tool and test tags though

[bhreinb]

Fair point. It's just we are after very lean CI times hence why we are running the tests in parallel this way. Your library pretty much does what we want only it runs it twice unfortunately as mentioned above.

[bhreinb]

Hi @tomatobrown

Were you able to reproduce the problem I attached in the logs above. Let me know in any way I can to help solve this. Many thanks in advance.

[bhreinb]

Hi Folks,

Any update on this. I'd love a fix for this if possible. Many thanks in advance.

[jody-zeitler]

@bhreinb we don't have the bandwidth to look into this right now. If you discover anything, a pull request would be most welcome!