Skip to content

Commit

Permalink
QE 14606 make hook fails into scenario errors (#428)
Browse files Browse the repository at this point in the history 
Issue: When a scenario hooks fail, make the scenario status as 'errored'
instead of 'failed'.
All hooks on a scenario should run (even when first hook fails the rest
should run)
Jira: [QE-14606](https://dominodatalab.atlassian.net/browse/QE-14606)



[QE-14606]:
https://dominodatalab.atlassian.net/browse/QE-14606?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
  • Loading branch information
@ddl-kavya
ddl-kavya authored Feb 2, 2024
1 parent 79b3ab5 commit b780d0a
Show file tree
Hide file tree
Showing 18 changed files with 280 additions and 69 deletions.
7 changes: 5 additions & 2 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,18 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project closely adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## 0.185.0
- Change - hook failures will report as 'errored' instead of 'failed' to testrail and html reports

## 0.184.0
- Change - Added step duration to failure step

## 0.183.0
- Change - Added ability to take multiple screenshots per step

## 0.182.0
Chore - Update docker README for ARM64 based CPU
Chore - Add seleniarm_hub bash file
- Chore - Update docker README for ARM64 based CPU
- Chore - Add seleniarm_hub bash file

## 0.181.0
- Change - only obfuscate values in config yaml file
Expand Down
11 changes: 10 additions & 1 deletion data/features/feature_with_mixed_results.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,15 @@ Feature: Feature with mixed results
Given I fail
And I echo "should never see this"

Scenario: Scenario and after-hook both fail
Given I error after-scenario hook
And I fail
And I echo "should never see this"

Scenario: Scenario with after-hook error
Given I error after-scenario hook
And I echo "should never see this"

Scenario: Scenario that also passes
Given I echo "passing"

Expand All @@ -16,4 +25,4 @@ Feature: Feature with mixed results

@disabled
Scenario: Scenario that is skipped
Given I echo "should never see this"
Given I echo "should never see this"
16 changes: 15 additions & 1 deletion data/features/steps/cucu_steps.py
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
from cucu import StopRetryException, retry, run_steps, step
from cucu import (
StopRetryException,
register_after_this_scenario_hook,
retry,
run_steps,
step,
)


@step("I use a step with substeps that log")
Expand Down Expand Up @@ -63,6 +69,14 @@ def fail():
retry(fail)()


@step("I error after-scenario hook")
def i_error_after_hook(_):
def after_hook_fail(_):
raise RuntimeError("after-hook errors on purpose")

register_after_this_scenario_hook(after_hook_fail)


@step('I use a step with "{nth:nth}" usage')
def uses_nth_step(ctx, nth):
print("just a step that nth behave argument type")
Expand Down
2 changes: 1 addition & 1 deletion features/cli/ipdb.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@ Feature: ipdb
Then I strip ansi codes from "{STDOUT}" and save to the variable "STDOUT"
And I should see "{STDOUT}" contains the following
"""
---> 55 raise RuntimeError("step fails on purpose")
---> 61 raise RuntimeError("step fails on purpose")
"""
39 changes: 23 additions & 16 deletions features/cli/report_basics.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,15 +86,17 @@ Feature: Report basics
And I start a webserver at directory "{CUCU_RESULTS_DIR}/mixed-results-report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
Then I should see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Feature with mixed results | 4 | 2 | 2 | 0 | failed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Feature with mixed results | 6 | 2 | 3 | 0 | 1 | failed | .* |
When I click the button "Feature with mixed results"
Then I should see a table that matches the following:
| Offset | Scenario | Steps | Status | Duration |
| .* | Scenario and after-hook both fail | 3 | failed | .* |
| .* | Scenario that also passes | 1 | passed | .* |
| .* | Scenario that fails | 2 | failed | .* |
| .* | Scenario that has an undefined step | 1 | failed | .* |
| .* | Scenario that passes | 1 | passed | .* |
| .* | Scenario with after-hook error | 2 | errored | .* |
And I click the button "Scenario that fails"
Then I should see the text "RuntimeError: step fails on purpose"
When I click the button "Index"
Expand All @@ -109,16 +111,19 @@ Feature: Report basics
And I start a webserver at directory "{CUCU_RESULTS_DIR}/mixed-results-report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
Then I should see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Feature with mixed results | 5 | 2 | 2 | 1 | failed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Feature with mixed results | 7 | 2 | 3 | 1 | 1 | failed | .* |
When I click the button "Feature with mixed results"
Then I should see a table that matches the following:
| Offset | Scenario | Steps | Status | Duration |
| .* | Scenario and after-hook both fail | 3 | failed | .* |
| .* | Scenario that also passes | 1 | passed | .* |
| .* | Scenario that fails | 2 | failed | .* |
| .* | Scenario that has an undefined step | 1 | failed | .* |
| .* | Scenario that is skipped | 1 | skipped | .* |
| .* | Scenario that passes | 1 | passed | .* |
| .* | Scenario with after-hook error | 2 | errored | .* |

And I click the button "Scenario that fails"
Then I should see the text "RuntimeError: step fails on purpose"
When I click the button "Index"
Expand All @@ -132,8 +137,8 @@ Feature: Report basics
And I start a webserver at directory "{CUCU_RESULTS_DIR}/feature_with_background-report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
Then I should see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Feature with background | 1 | 1 | 0 | 0 | passed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Feature with background | 1 | 1 | 0 | 0 | 0 | passed | .* |
When I click the button "Feature with background"
Then I should see a table that matches the following:
| Offset | Scenario | Steps | Status | Duration |
Expand All @@ -146,8 +151,8 @@ Feature: Report basics
And I start a webserver at directory "{CUCU_RESULTS_DIR}/feature_with_background-report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
Then I should see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Feature with background | 2 | 1 | 0 | 1 | passed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Feature with background | 2 | 1 | 0 | 1 | 0 | passed | .* |
When I click the button "Feature with background"
Then I should see a table that matches the following:
| Offset | Scenario | Steps | Status | Duration |
Expand Down Expand Up @@ -218,15 +223,17 @@ Feature: Report basics
And I start a webserver at directory "{CUCU_RESULTS_DIR}/report_without_skips_report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
Then I should see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Feature with mixed results | 4 | 2 | 2 | 0 | failed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Feature with mixed results | 6 | 2 | 3 | 0 | 1 | failed | .* |
When I click the button "Feature with mixed results"
Then I should see a table that matches the following:
| Offset | Scenario | Steps | Status | Duration |
| .* | Scenario and after-hook both fail | 3 | failed | .* |
| .* | Scenario that also passes | 1 | passed | .* |
| .* | Scenario that fails | 2 | failed | .* |
| .* | Scenario that has an undefined step | 1 | failed | .* |
| .* | Scenario that passes | 1 | passed | .* |
| .* | Scenario with after-hook error | 2 | errored | .* |
And I click the button "Scenario that fails"
Then I should see the text "RuntimeError: step fails on purpose"
When I click the button "Index"
Expand All @@ -239,8 +246,8 @@ Feature: Report basics
And I start a webserver at directory "{CUCU_RESULTS_DIR}/report_without_skips_background_report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
Then I should see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Feature with background | 1 | 1 | 0 | 0 | passed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Feature with background | 1 | 1 | 0 | 0 | 0 | passed | .* |
When I click the button "Feature with background"
Then I should see a table that matches the following:
| Offset | Scenario | Steps | Status | Duration |
Expand Down Expand Up @@ -305,10 +312,10 @@ Feature: Report basics
And I start a webserver at directory "{CUCU_RESULTS_DIR}/report_only_failures_report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
Then I should see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Feature with failing scenario | 1 | 0 | 1 | 0 | failed | .* |
| .* | Feature with failing to find a table | 1 | 0 | 1 | 0 | failed | .* |
| .* | Feature with failing scenario with web | 1 | 0 | 1 | 0 | failed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Feature with failing scenario | 1 | 0 | 1 | 0 | 0 | failed | .* |
| .* | Feature with failing to find a table | 1 | 0 | 1 | 0 | 0 | failed | .* |
| .* | Feature with failing scenario with web | 1 | 0 | 1 | 0 | 0 | failed | .* |
When I click the button "Feature with failing scenario with web"
Then I should see a table that matches the following:
| Offset | Scenario | Steps | Status | Duration |
Expand Down
50 changes: 25 additions & 25 deletions features/cli/report_search_and_sorting.feature
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,34 +9,34 @@ Feature: Report search and sorting
And I start a webserver at directory "{CUCU_RESULTS_DIR}/sorting-in-reports-report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
Then I wait to see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | passed | .* |
| .* | First tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | First tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | 0 | passed | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |

Scenario: User can sort by various fields in the HTML test report and share the exact state via the URL
Given I run the command "cucu run data/features/tagged_features --tags @feature2 --show-skips --results {CUCU_RESULTS_DIR}/sorting-in-reports-results" and expect exit code "0"
And I run the command "cucu report {CUCU_RESULTS_DIR}/sorting-in-reports-results --show-skips --output {CUCU_RESULTS_DIR}/sorting-in-reports-report" and expect exit code "0"
And I start a webserver at directory "{CUCU_RESULTS_DIR}/sorting-in-reports-report/" and save the port to the variable "PORT"
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
When I wait to see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | passed | .* |
| .* | First tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | First tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | 0 | passed | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |
And I click the table header "Status"
And I wait to see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | First tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | passed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | 0 | passed | .* |
| .* | First tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |
And I save the current url to the variable "CURRENT_URL"
And I refresh the browser
Then I wait to see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | First tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | passed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | 0 | passed | .* |
| .* | First tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |

Scenario: User can search by various fields in the HTML test report
Given I run the command "cucu run data/features/tagged_features --tags @feature2 --show-skips --results {CUCU_RESULTS_DIR}/searching-in-reports-results" and expect exit code "0"
Expand All @@ -45,13 +45,13 @@ Feature: Report search and sorting
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
When I write "passed" into the input "Search:"
Then I wait to see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | passed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | 0 | passed | .* |
When I write "skipped" into the input "Search:"
Then I wait to see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | First tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | skipped | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | First tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |
| .* | Third tagged feature | 1 | 0 | 0 | 1 | 0 | skipped | .* |

Scenario: User can search by various fields in the HTML test report and share the exact state via the URL
Given I run the command "cucu run data/features/tagged_features --tags @feature2 --results {CUCU_RESULTS_DIR}/searching-in-reports-results" and expect exit code "0"
Expand All @@ -60,10 +60,10 @@ Feature: Report search and sorting
And I open a browser at the url "http://{HOST_ADDRESS}:{PORT}/index.html"
When I write "passed" into the input "Search:"
Then I wait to see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | passed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | 0 | passed | .* |
When I save the current url to the variable "CURRENT_URL"
And I refresh the browser
Then I wait to see a table that matches the following:
| Started at | Feature | Total | Passed | Failed | Skipped | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | passed | .* |
| Started at | Feature | Total | Passed | Failed | Skipped | Errored | Status | Duration |
| .* | Second tagged feature | 1 | 1 | 0 | 0 | 0 | passed | .* |
Loading

0 comments on commit b780d0a

Please sign in to comment.