Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

simplify configuration and improve UI Session handling implementation #13

Merged
merged 56 commits into from
May 18, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
bf745c1
chaging core aftconfig design
bicarbon8 May 6, 2023
50a1243
changing aftconfig core design
bicarbon8 May 6, 2023
80f756d
adding documentation on new code
bicarbon8 May 7, 2023
6fecd39
code checkpoint
bicarbon8 May 7, 2023
a0a5604
checkpoint updating logging plugins
bicarbon8 May 7, 2023
a2d91d9
code checkpoint
bicarbon8 May 8, 2023
0823641
fixed aft-core and unit tests
bicarbon8 May 8, 2023
2c454d8
checkpoint on core fixes
bicarbon8 May 8, 2023
64f6d66
updating documentation
bicarbon8 May 9, 2023
fc4d572
code checkpoint trying to fix plugin loading
bicarbon8 May 9, 2023
e5d32db
aft-core fixed and all tests passing
bicarbon8 May 9, 2023
5e58fce
aft-core unit tests passing
bicarbon8 May 10, 2023
971ec25
hiding internal functions of verifier in nested object
bicarbon8 May 10, 2023
336d9b8
updating plugins and extension packages
bicarbon8 May 10, 2023
8657e0b
checkpoint updating ui packages
bicarbon8 May 11, 2023
41529b6
code checkpoint
bicarbon8 May 12, 2023
9c1d937
fixed browsers and ui packages
bicarbon8 May 12, 2023
a948791
removing direct integrations with browserstack and saucelabs and inst…
bicarbon8 May 13, 2023
9eaf505
updated to allow passing in addional options to session generator in …
bicarbon8 May 13, 2023
e0b41da
updating examples project
bicarbon8 May 13, 2023
95b231d
unit test fixes
bicarbon8 May 14, 2023
6651baa
additional fixes
bicarbon8 May 14, 2023
f8e5e40
minor fixes
bicarbon8 May 14, 2023
e37a884
updating upload process for mobile apps
bicarbon8 May 14, 2023
f6fd5b5
removing uiplatform
bicarbon8 May 14, 2023
659ddc2
code checkpoint
May 15, 2023
7750da9
functional test fixes
May 15, 2023
d1fcd0f
fixing accidental circular reference on getRoot when calling parent
May 15, 2023
5187a0e
updating example project readme
May 15, 2023
af76da2
simplifying config
May 16, 2023
a4bd771
split out console logging from logging plugin management
May 16, 2023
bcae0f7
fixing reporter implementations for jasmine and mocha integration
May 16, 2023
2bcef05
updating mocha docs
May 16, 2023
0fb9a5e
functional test fixes
May 16, 2023
cf651a6
updated mobile tests to use webdriverio to show an example of how it …
May 16, 2023
fbd970a
adding convenience modules to web-services package
May 17, 2023
1a513a0
updating docs
May 17, 2023
dd6ffd7
fixes and moving code around
May 17, 2023
64218ad
chaging LogManager to Reporter and re-merging with ResultsManager
May 17, 2023
fcfe997
updating package names and references
May 17, 2023
71ad6c7
fixing references and plugin names
May 17, 2023
51fb8a7
cleanup
May 17, 2023
872d6ed
fixing unit tests and documentation
bicarbon8 May 17, 2023
1aeab7e
fixing readme
bicarbon8 May 17, 2023
696e05d
fixing html report filters
bicarbon8 May 17, 2023
00d37c1
updating readme files
bicarbon8 May 17, 2023
74967c1
updating documentation to remove defect references
May 18, 2023
02f2928
updating generated docs
May 18, 2023
0817514
ensure build before testing to generate .js files used for plugin loa…
May 18, 2023
1354783
unit test fixes
May 18, 2023
50d0c66
fixing coverage execution process
May 18, 2023
6b2c554
simplifying process used to generate updated docs
May 18, 2023
bececf4
v10.0.0
May 18, 2023
4f0afee
fixed all functional tests
May 18, 2023
6b8f774
restore parallel running
May 18, 2023
bb105bc
fix workflow
May 18, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
14 changes: 0 additions & 14 deletions .devcontainer/Dockerfile

This file was deleted.

17 changes: 0 additions & 17 deletions .devcontainer/base.Dockerfile

This file was deleted.

45 changes: 14 additions & 31 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,42 +1,25 @@
// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.245.2/containers/typescript-node
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
// README at: https://github.com/devcontainers/templates/tree/main/src/typescript-node
{
"name": "Node.js & TypeScript",
"build": {
"dockerfile": "Dockerfile",
// Update 'VARIANT' to pick a Node version: 18, 16, 14.
// Append -bullseye or -buster to pin to an OS version.
// Use -bullseye variants on local on arm64/Apple Silicon.
"args": {
"VARIANT": "18"
}
},
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
"image": "mcr.microsoft.com/devcontainers/typescript-node:0-20",
"features": {
"ghcr.io/devcontainers/features/git:1": {}
}

// Configure tool-specific properties.
"customizations": {
// Configure properties specific to VS Code.
"vscode": {
// Add the IDs of extensions you want installed when the container is created.
"extensions": [
"dbaeumer.vscode-eslint"
]
}
},
// Features to add to the dev container. More info: https://containers.dev/features.
// "features": {},

// Use 'forwardPorts' to make a list of ports inside the container available locally.
// "forwardPorts": [],

// Use 'postCreateCommand' to run commands after the container is created.
// "postCreateCommand": "npm install",
// "postCreateCommand": "yarn install",

// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
"remoteUser": "node",
"features": {
"git": "os-provided"
},
// Configure tool-specific properties.
// "customizations": {},

"containerEnv": {
"browserstack_user": "${localEnv:browserstack_user}",
"browserstack_key": "${localEnv:browserstack_key}"
}
// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
// "remoteUser": "root"
}
2 changes: 1 addition & 1 deletion .github/workflows/build-test-pull-requests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
# only run functional tests on internal PRs
if: github.repository == 'bicarbon8/automated-functional-testing'
id: functionaltest
run: npm run test-examples
run: npm run test:examples
env:
browserstack_user: ${{ secrets.BROWSERSTACK_USER }}
browserstack_key: ${{ secrets.BROWSERSTACK_KEY }}
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,5 @@ coverage/
.nyc_output/
FileSystemMap/
testresults.html
logs/
logs/
.env
1 change: 1 addition & 0 deletions .npmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
registry=https://registry.npmjs.org/
124 changes: 57 additions & 67 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,59 +5,55 @@ library providing a framework for creating Functional Test Automation supporting
### Example Jasmine Test:
```typescript
describe('Sample Test', () => {
it('can perform a demonstration of AFT', async () => {
let feature: FeatureObj = new FeatureObj();
it('[C1234] can perform a demonstration of AFT', async () => {
const aft = new AftTest();
const feature: FeatureObj = new FeatureObj();
/**
* the `verify(assertion).returns(expectation)` function
* checks any specified `AbstractTestCasePlugin`
* and `AbstractDefectPlugin` implementations
* checks any specified `TestExecutionPolicyPlugin` implementations
* to ensure the test should be run. It will then
* report to any `AbstractLoggingPlugin` implementations
* with an `ITestResult` indicating the success,
* report to any `ReportingPlugin` implementations
* with an `TestResult` indicating the success,
* failure or skipped status
*/
await verify(async () => await feature.performAction())
.withTestId('C1234')
.and.withKnownDefectId('DEFECT-123')
.and.withDescription('expect that performAction will return \'result of action\'')
.returns('result of action');
await aft.verify(async () => await feature.performAction())
.returns('result of action');
});
});
```
the above results in the following console output if the expectation does not return false or throw an exception:
```
5:29:55 PM - expect that performAction will return 'result of action' - PASS - C1234
5:29:55 PM - [[C1234] can perform a demonstration of AFT] - PASS - C1234
```
in more complex scenarios you can perform multiple actions inside the _expectation_ like in the following example:
```typescript
describe('Sample Test', () => {
it('can perform a more complex demonstration of AFT', async () => {
it('[C2345][C3344] can perform a more complex demonstration of AFT', async () => {
const aft = new AftTest();
/**
* the passed in expectation can accept a `Verifier` which can be used
* during more complex actions
*/
await verify(async (v: Verifier) => {
await v.logMgr.step('creating instance of FeatureObj');
await aft.verify(async (v: Verifier) => {
await v.reporter.step('creating instance of FeatureObj');
let feature: FeatureObj = new FeatureObj();
await v.logMgr.step('about to call performAction');
await v.reporter.step('about to call performAction');
let result: string = await feature.performAction();
await v.logMgr.info(`result of performAction was '${result}'`);
await v.logMgr.trace('successfully executed expectation');
await v.reporter.info(`result of performAction was '${result}'`);
await v.reporter.trace('successfully executed expectation');
return result;
}).withTestId('C2345').and.withTestId('C3344')
.and.withDescription('more complex expectation actions')
.returns(containing('result of action'));
}).returns(containing('result of action'));
});
});
```
which would output the following logs:
```
5:29:54 PM - more complex expectation actions - STEP - 1: creating instance of FeatureObj
5:29:55 PM - more complex expectation actions - STEP - 2: about to call performAction
5:29:55 PM - more complex expectation actions - INFO - result of performAction was 'result of action'
5:29:56 PM - more complex expectation actions - TRACE - successfully executed expectation
5:29:56 PM - more complex expectation actions - PASS - C2345
5:29:56 PM - more complex expectation actions - PASS - C3344
5:29:54 PM - [[C2345][C3344] can perform a more complex demonstration of AFT] - STEP - 1: creating instance of FeatureObj
5:29:55 PM - [[C2345][C3344] can perform a more complex demonstration of AFT] - STEP - 2: about to call performAction
5:29:55 PM - [[C2345][C3344] can perform a more complex demonstration of AFT] - INFO - result of performAction was 'result of action'
5:29:56 PM - [[C2345][C3344] can perform a more complex demonstration of AFT] - TRACE - successfully executed expectation
5:29:56 PM - [[C2345][C3344] can perform a more complex demonstration of AFT] - PASS - C2345
5:29:56 PM - [[C2345][C3344] can perform a more complex demonstration of AFT] - PASS - C3344
```
> WARNING: Jasmine's _expect_ calls do not return a boolean as their type definitions would make you think and failed `expect` calls will only throw exceptions if the stop on failure option is enabled:
```typescript
Expand All @@ -72,61 +68,55 @@ verify(() => {throw new Error('failure');}) // AFT will report as 'failed'
- [`aft-core`](./packages/aft-core/README.md) - base library containing helpers and configuration and plugin managers
- [`aft-examples`](./packages/aft-examples/README.md) - provides real-world examples of how the AFT libraries can be used in functional tests
- [`aft-jasmine-reporter`](./packages/aft-jasmine-reporter/README.md) - a Jasmine Reporter Plugin that integrates with AFT to simplify logging and test execution via AFT
- [`aft-logging-awskinesis`](./packages/aft-logging-awskinesis/README.md) - logging plugin supporting logging to AWS Kinesis Firehose
- [`aft-logging-filesystem`](./packages/aft-logging-filesystem/README.md) - logging plugin supporting logging to .log files for all log output
- [`aft-logging-html`](./packages/aft-logging-html/README.md) - logging plugin supporting logging to a HTML results file
- [`aft-reporting-aws-kinesis-firehose`](./packages/aft-reporting-aws-kinesis-firehose/README.md) - reporting plugin supporting logging to AWS Kinesis Firehose
- [`aft-reporting-filesystem`](./packages/aft-reporting-filesystem/README.md) - reporting plugin supporting logging to .log files for all log output
- [`aft-reporting-html`](./packages/aft-reporting-html/README.md) - reporting plugin supporting logging to a HTML results file
- [`aft-mocha-reporter`](./packages/aft-mocha-reporter/README.md) - provides Mocha Reporter Plugin that integrates with AFT to simplify logging and test execution via AFT
- [`aft-testrail`](./packages/aft-testrail/README.md) - logging and test case management plugins supporting logging test results and filtering test execution based on TestRail Projects, Suites and Plans
- [`aft-testrail`](./packages/aft-testrail/README.md) - reporting and test execution policy plugins supporting logging test results and filtering test execution based on TestRail Projects, Suites and Plans
- [`aft-ui`](./packages/aft-ui/README.md) - base library supporting development of UI testing packages
- [`aft-ui-browsers`](./packages/aft-ui-browsers/README.md) - adds support for Selenium-based UI testing using BrowserStack, Sauce Labs or your own Selenium Grid
- [`aft-ui-mobile-apps`](./packages/aft-ui-mobile-apps/README.md) - adds support for Appium-based UI testing using BrowserStack, Sauce Labs or your own Appium Grid
- [`aft-ui-selenium`](./packages/aft-ui-selenium/README.md) - adds support for Selenium-based UI testing
- [`aft-ui-webdriverio`](./packages/aft-ui-webdriverio/README.md) - adds support for WebdriverIO-based UI testing
- [`aft-web-services`](./packages/aft-web-services/README.md) - adds support for testing REST-based services

## Plugins
the primary benefit of using AFT comes from the plugins and the `Verifier`. Because logging using AFT's `LogManager` will also send to any registered logging plugins, it is easy to create logging plugins that send to any external system such as TestRail or to log results to Elasticsearch. Additionally, before running any _assertion_ passed to a `verify(assertion)` function, AFT will confirm if the _assertion_ should actually be run based on the results of queries to any supplied `TestCasePlugin` implementations and a subsequent queries to any supplied `DefectPlugin` implementations.
the primary benefit of using AFT comes from the plugins and the `Verifier`. Because logging using AFT's `Reporter` will also send to any registered logging plugins, it is easy to create logging plugins that send to any external system such as TestRail or to log results to Elasticsearch. Additionally, before running any _assertion_ passed to a `verify(assertion)` function, AFT will confirm if the _assertion_ should actually be run based on the results of queries to any supplied `TestExecutionPolicyPlugin` implementations.

### Logging Plugins
`aft-core` provides a `LoggingPlugin` abstract class which can be extended to create custom loggers which are then loaded by adding their filenames to the `plugins` array under the `logmanager` section of your `aftconfig.json`
### ReportingPlugin
`aft-core` provides a `ReportingPlugin` class which can be extended from to create custom loggers which are then loaded by adding their filenames to the `pluginNames` array under in your `aftconfig.json`
```json
// aftconfig.json
{
"LogManager": {
"plugins": [
{
"name": "testrail-logging-plugin",
"searchDirectory": "../node_modules",
"options": {
"level": "info",
"enabled": false
}
},
"html-logging-plugin"
]
"pluginsSearchDir": "../node_modules",
"pluginNames": [
"testrail-reporting-plugin",
"html-reporting-plugin"
],
"TestRailConfig": {
"url": "https://your.testrail.io",
"user": "[email protected]",
"accessKey": "yourTestRailApiKey",
"projectId": 123,
"suiteIds": [1234, 5678],
"planId": 123456,
"policyEngineEnabled": true,
"logLevel": "error"
},
"HtmlReportingPluginConfig": {
"outputDir": "../Results",
"logLevel": "debug"
}
}
```
> NOTE: you can either specify a `string` containing the plugin filename or an `object` containing the `name`, `searchDirectory` and `options` fields within the `plugins` array configuration to specify a root directory to use when searching for logging plugin implementations

### Test Case Plugin
the purpose of a `TestCasePlugin` implementation is to provide execution control over any expectations by way of supplied _Test IDs_. to specify an implementation of the plugin to load you can add the following to your `aftconfig.json` (where plugins `testrail-test-case-plugin.js` is contained within the test execution directory or a subdirectory of it):
### TestExecutionPolicyPlugin
the purpose of a `TestExecutionPolicyPlugin` implementation is to provide execution control over any expectations by way of supplied _Test IDs_. to specify an implementation of the plugin to load you can add the following to your `aftconfig.json` (where plugin `testrail-test-execution-policy-plugin.js` is contained within the test execution directory or a subdirectory of it):
```json
// aftconfig.json
{
"TestCaseManager": {
"plugins": ["testrail-test-case-plugin"]
}
}
```
> NOTE: if no plugin is specified then external Test Case Management integration will be disabled and _assertions_ will be executed without checking their status before execution

### Defect Plugin
the purpose of a `DefectPlugin` implementation is to provide execution control over any expectations by way of supplied _Test IDs_ referenced in an external ticket tracking system like Bugzilla or Jira. to specify an implementation of the plugin to load you can add the following to your `aftconfig.json` (where plugins `defect-plugin.js` is contained within the test execution directory or a subdirectory of it):
```json
{
"DefectManager": {
"plugins": ["defect-plugin"]
}
"pluginNames": ["testrail-test-execution-policy-plugin"]
}
```
> NOTE: if no plugin is specified then external Defect Management integration will be disabled and _assertions_ will be executed without checking their status before execution, however if a Defect Management plugin is specified, the execution of any _assertions_ passed into a `verify(assertion)` function will be halted if any non-closed defects are found when searching for defects that contain reference to the values passed in via `withTestId(caseId)` or via direct reference to defect using `withKnownDefectId(defectId)`
> NOTE: if no plugin is specified then external Policy Engine integration will be disabled and _assertions_ will be executed without first checking that they should be run based on associated Test IDs

## Example Test Project
- [`aft-examples`](./packages/aft-examples/README.md) - a demonstration of how to develop UI and REST based functional test automation using AFT is located under `./packages/aft-examples`
Expand All @@ -148,4 +138,4 @@ the purpose of a `DefectPlugin` implementation is to provide execution control o

> use `npx lerna version` to automatically update the version of all projects at once (all changes must be committed first)

> generate documentation using `npx typedoc --entryPointStrategy packages ./packages/* --out ./docs`
> generate documentation `npm run docs`
35 changes: 14 additions & 21 deletions docs/assets/highlight.css
Original file line number Diff line number Diff line change
Expand Up @@ -7,30 +7,28 @@
--dark-hl-2: #CE9178;
--light-hl-3: #0000FF;
--dark-hl-3: #569CD6;
--light-hl-4: #001080;
--dark-hl-4: #9CDCFE;
--light-hl-4: #0070C1;
--dark-hl-4: #4FC1FF;
--light-hl-5: #267F99;
--dark-hl-5: #4EC9B0;
--light-hl-6: #008000;
--dark-hl-6: #6A9955;
--light-hl-7: #AF00DB;
--dark-hl-7: #C586C0;
--light-hl-8: #EE0000;
--dark-hl-8: #D7BA7D;
--light-hl-8: #001080;
--dark-hl-8: #9CDCFE;
--light-hl-9: #098658;
--dark-hl-9: #B5CEA8;
--light-hl-10: #0070C1;
--dark-hl-10: #4FC1FF;
--light-hl-11: #0451A5;
--dark-hl-11: #9CDCFE;
--light-hl-12: #CD3131;
--dark-hl-12: #F44747;
--light-hl-13: #811F3F;
--dark-hl-13: #D16969;
--light-hl-14: #D16969;
--dark-hl-14: #CE9178;
--light-hl-15: #000000;
--dark-hl-15: #D7BA7D;
--light-hl-10: #0451A5;
--dark-hl-10: #9CDCFE;
--light-hl-11: #CD3131;
--dark-hl-11: #F44747;
--light-hl-12: #811F3F;
--dark-hl-12: #D16969;
--light-hl-13: #D16969;
--dark-hl-13: #CE9178;
--light-hl-14: #000000;
--dark-hl-14: #D7BA7D;
--light-code-background: #FFFFFF;
--dark-code-background: #1E1E1E;
}
Expand All @@ -51,7 +49,6 @@
--hl-12: var(--light-hl-12);
--hl-13: var(--light-hl-13);
--hl-14: var(--light-hl-14);
--hl-15: var(--light-hl-15);
--code-background: var(--light-code-background);
} }

Expand All @@ -71,7 +68,6 @@
--hl-12: var(--dark-hl-12);
--hl-13: var(--dark-hl-13);
--hl-14: var(--dark-hl-14);
--hl-15: var(--dark-hl-15);
--code-background: var(--dark-code-background);
} }

Expand All @@ -91,7 +87,6 @@
--hl-12: var(--light-hl-12);
--hl-13: var(--light-hl-13);
--hl-14: var(--light-hl-14);
--hl-15: var(--light-hl-15);
--code-background: var(--light-code-background);
}

Expand All @@ -111,7 +106,6 @@
--hl-12: var(--dark-hl-12);
--hl-13: var(--dark-hl-13);
--hl-14: var(--dark-hl-14);
--hl-15: var(--dark-hl-15);
--code-background: var(--dark-code-background);
}

Expand All @@ -130,5 +124,4 @@
.hl-12 { color: var(--hl-12); }
.hl-13 { color: var(--hl-13); }
.hl-14 { color: var(--hl-14); }
.hl-15 { color: var(--hl-15); }
pre, code { background: var(--code-background); }
Loading