diff --git a/README.md b/README.md index ec6c9ed6..f5f84eb4 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,12 @@ automatically navigate the app and reach your desired outcome. **Boost your mobile app testing efficiency with ScriptIQ:** -- **Intuitive Testing:** Describe your test goals in plain English, making mobile app testing accessible to everyone. -- **AI-powered Automation:** ScriptIQ leverages AI to intelligently interact with your mobile app, mimicking real user behavior. -- **Real Device Testing:** Validate your app's functionality directly on real mobile devices, ensuring a true-to-life testing experience. +- **Intuitive Testing:** Describe your test goals in plain English, making + mobile app testing accessible to everyone. +- **AI-powered Automation:** ScriptIQ leverages AI to intelligently interact + with your mobile app, mimicking real user behavior. +- **Real Device Testing:** Validate your app's functionality directly on real + mobile devices, ensuring a true-to-life testing experience. **Who is ScriptIQ for?** @@ -22,18 +25,92 @@ automatically navigate the app and reach your desired outcome. 🚦 ScriptIQ is currently in closed beta! -1. Download and install the ScriptIQ extension (`.vsix`) from [the release page](https://github.com/saucelabs/vscode-scriptiq/releases). +1. Download and install the ScriptIQ extension (`.vsix`) from + [the release page](https://github.com/saucelabs/vscode-scriptiq/releases). 2. From VSCode's main menu, select "Extensions". -3. Click the `...` (View and More Actions) button at the top of the middle panel. +3. Click the `...` (View and More Actions) button at the top of the middle + panel. 4. Select "Install from VSIX..." and follow the prompts. ![installation-instruction](installation-instruction.png) -5. Upload your mobile app file via [Sauce Labs App Management](https://app.saucelabs.com/app-management). -6. Describe your test goal in plain English. -7. Click "Generate Test" to let ScriptIQ navigate your app on a Sauce Labs real device. -8. Review the test results and iterate on your test goals as needed. +5. Upload your mobile app file via + [Sauce Labs App Management](https://app.saucelabs.com/app-management). +6. Click ScriptIQ icon in Activity bar. -For more installation details, please check [here](https://code.visualstudio.com/docs/editor/extension-marketplace#_install-from-a-vsix). + ![activity-bar-icon](activity-bar-icon.png) -ScriptIQ brings the power of AI to mobile app testing, making it easier, and more accessible than ever. +7. Input and save credentials (do this once, unless new credentials are + necessary). +8. Describe your test goal in plain English. +9. Click "Generate Test" to let ScriptIQ navigate your app on a Sauce Labs real + device. +10. Review the test results and iterate on your test goals as needed. -For more information and documentation, please visit the ScriptIQ documentation (coming soon). +For more installation details, please check +[here](https://code.visualstudio.com/docs/editor/extension-marketplace#_install-from-a-vsix). + +ScriptIQ brings the power of AI to mobile app testing, making it easier, and +more accessible than ever. + +For more information and documentation, please visit the ScriptIQ documentation +(coming soon). + +## Prompt Generation Hints + +ScriptIQ is powered by Large Language Models (LLMs). LLMs are very powerful, +but sometimes it takes a couple of tries to create a prompt that the model can +understand and interpret correctly. Therefore it might take multiple attempts +to write a goal which generates the expected test. If your first attempt does +not result in the desired outcome, and you are unsure why this is, you can refer +to these potential issues with your prompt: + +- **Unclear word choice**: Consider whether the goal you created could be + interpreted in multiple ways. If so, consider rewriting or adding clarifying + statements. +- **Incorrect path taken**: ScriptIQ uses common sense reasoning to generate the + test using the intent of your goal. But sometimes ScriptIQ takes the wrong path, + making an assumption about the best way to accomplish your goal which is + incorrect. When an incorrect step is taken, consider defining the correct action + explicitly in your goal. For example, if ScriptIQ attempts to login when you do + not require it to login, you can add "do not login" to your goal. +- **Context provided in assertions**: The assertion statements do not guide + ScriptIQ towards accomplishing the goal, they only check whether the + descriptions provided in the assertion match seen screens. If you've only + included necessary context in the assertion fields you'll need to add it to the + goal. +- **Non-journey based goal**: ScriptIQ is based on GUI interactions, it cannot + reference the source code, so goals which refer to the source code might not be + understood. + +Here is an example of updating your goal: + +1. Create your first, general goal: + ![ScriptIQ example goal which did not work](media/imgs/example_scriptiq_incorrect.png) +2. You can see that it did not accomplish the goal of logging in because it + scrolled down when the option to login is brought up when you click "More". +3. Update the goal with this additional context and run again: + ![ScriptIQ example goal which did work](media/imgs/example_scriptiq_correct.png) +4. The updated goal is now accurately finding the login button and clicking it. + +Those are just some examples of why a goal might not work and how to fix it. +Feel free to be creative! + +## How Do Assertions work? + +Once you've started generating tests, you might want to more quickly determine +if your test succeeded or failed, this is where the _Assert Inputs_ fields come +in. Assert Inputs are descriptions of screens which are used to evaluate the +test generated. When assertions are provided, the LLM will determine if the +assertion matches what is on the screen for each assertion provided against +every screen visited. These assertions can describe the entire screen (e.g. +login screen) or describe an element on the screen (e.g. page with checkout +button). These assertions are _not_ used to generate test steps, any intent +needed to drive the test generation must be included in the _Test Goal_ field. +Below you can see examples of assertions and an example output. + +Example of Assertion Inputs: + + + +Example of Assertion Results: + + diff --git a/activity-bar-icon.png b/activity-bar-icon.png new file mode 100644 index 00000000..a50c6014 Binary files /dev/null and b/activity-bar-icon.png differ diff --git a/media/imgs/assertion_input.png b/media/imgs/assertion_input.png new file mode 100644 index 00000000..d74ff44d Binary files /dev/null and b/media/imgs/assertion_input.png differ diff --git a/media/imgs/assertion_output.png b/media/imgs/assertion_output.png new file mode 100644 index 00000000..b9feab92 Binary files /dev/null and b/media/imgs/assertion_output.png differ diff --git a/media/imgs/example_scriptiq_correct.png b/media/imgs/example_scriptiq_correct.png new file mode 100644 index 00000000..58de8042 Binary files /dev/null and b/media/imgs/example_scriptiq_correct.png differ diff --git a/media/imgs/example_scriptiq_incorrect.png b/media/imgs/example_scriptiq_incorrect.png new file mode 100644 index 00000000..38545528 Binary files /dev/null and b/media/imgs/example_scriptiq_incorrect.png differ