-
Notifications
You must be signed in to change notification settings - Fork 30
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
review of practices #30
base: main
Are you sure you want to change the base?
Changes from 2 commits
6e0b591
f94a984
166d8d7
1fb3bf5
88d0741
497d515
2f24e61
1815a54
3e9b9aa
eab7416
c7f7570
8124fa0
3398f94
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,18 +1,23 @@ | ||
# State clearing | ||
Whenever we execute UI tests, it is likely that we read/write some data locally. | ||
These changes can affect the execution of the subsequent tests, for example: | ||
|
||
This question appears as soon as you need to run more than 1 ui test. | ||
* We run `Test1`, it performs some http requests, saves some data to files and databases. | ||
* When `Test1` is finished, `Test2` will be launched. | ||
* However, `Test1` left some data on the device which can be a reason of `Test2` failing. | ||
|
||
## Problem | ||
That's where *state clearing* comes to the rescue: clear the data before each test | ||
|
||
We run `Test1`, it performs some http requests, saves some data to files and databases. | ||
<br/>When `Test1` is finished, `Test2` will be launched. | ||
<br/>However, `Test1` left some data on the device which can be a reason of `Test2` failing. | ||
## Strategies for state clearing | ||
|
||
Solution — clear the data before each test | ||
There are a few strategies to deal with this: | ||
|
||
## 1. Clearing within a process | ||
1. Clearing within a process | ||
2. Clearing package data | ||
|
||
In this case, we don't kill our application process, and we have 2 options here: | ||
### 1. Clearing within a process | ||
|
||
The state clearing happens *without killing the application process*. We have 2 options here: | ||
|
||
##### Use component from a real code base <br/> | ||
|
||
|
@@ -34,10 +39,10 @@ Databases, Files, Preferences and Runtime cache, and should be executed before e | |
|
||
##### Clear internal storage <br/> | ||
|
||
All cache in an android application is stored in the internal storage: `/data/data/packagename/` | ||
All cache data (e.g.local databases, shared preferences and some files) in any android application is written in the internal storage: `/data/data/packagename/` | ||
<br/>This storage is our application sandbox and can be accessed without any permission. | ||
|
||
Basic idea is to avoid using components from a real code base. Instead of them, use some tests rules which do the job | ||
In order to avoid any issues, the basic idea is to avoid using components from a real code base. Instead of them, use some tests rules which do the job | ||
for us. | ||
|
||
```kotlin | ||
|
@@ -58,32 +63,33 @@ them [here](https://github.com/AdevintaSpain/Barista/tree/master/library/src/mai | |
|
||
!!! warning | ||
|
||
This solution won't in 100% of cases: | ||
This solution won't work in 100% of cases: | ||
|
||
1. You may have runtime cache, which can also affect your tests | ||
2. Test or application process may crash and prevent the launch of next tests | ||
2. The test or the application process may crash and prevent the launch of next tests | ||
|
||
##### Conclusion<br/> | ||
|
||
These are pros/cons for both solutions which don't kill the process: | ||
|
||
➕ Fast implementation<br/> | ||
➕ Easy implementation. Simply add the corresponding TestRules<br/> | ||
➕ Fast execution in the same process<br/> | ||
<br/> | ||
➖ Don't give you any guarantee that your app will be cleared properly<br/> | ||
➖ Application or Test process killing will break tests execution <br/> | ||
➖ Can be a bottleneck<br/> | ||
➖ Can be a bottleneck<br/> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this needs some short hint of why |
||
|
||
Use these solutions only as a temp workaround, because it won't work on perspective in huge projects | ||
Use these solutions only as a temp workaround, because it won't work in long-term for huge projects | ||
|
||
## 2. Clearing package data | ||
### 2. Clearing package data | ||
|
||
Our aim is to simulate the same behavior as when user presses the `clear data` button in application settings. | ||
<br/>Application process will be cleared in that case, our application will be started in a cold start. | ||
Our aim is to simulate the same behavior as when the user presses the `clear data` button in application settings. | ||
<br/>Application process will be cleared in that case, our application will be initialized in a cold start. | ||
|
||
##### Orchestrator | ||
|
||
Basically, you can achieve an isolated state, if you execute your tests like this: | ||
The Android Orchestrator aims to isolate the state of each test by running each of them in a separate process: | ||
That can be achieved by executing your tests like this | ||
|
||
```bash | ||
adb shell am instrument -c TestClass#method1 -w com.package.name/junitRunnerClass | ||
|
@@ -92,22 +98,17 @@ adb shell am instrument -c TestClass#method2 -w com.package.name/junitRunnerClas | |
adb pm clear | ||
``` | ||
|
||
Each test should be executed in an isolated instrumented process and junit reports should be merged into a big one | ||
report when all tests are finished. | ||
|
||
That's the common idea of `Orchestrator`. | ||
That's the idea behind of `Orchestrator`. | ||
<br/> | ||
It's just an `apk` which consist of | ||
only [several classes](https://github.com/android/android-test/tree/master/runner/android_test_orchestrator/java/androidx/test/orchestrator) | ||
and runs tests and clears data, as described above. | ||
It's an `apk` which only consists of [several classes](https://github.com/android/android-test/tree/master/runner/android_test_orchestrator/java/androidx/test/orchestrator) | ||
that run tests and clear data, as described above. | ||
|
||
You should install an `orchestrator` along with `application.apk` and `instrumented.apk` on the device. | ||
It is necessary to install an `orchestrator` along with the `application.apk` and the `instrumented.apk` on the device. | ||
|
||
However, it's not the end. | ||
But that's not all. | ||
<br/> | ||
Orchestrator should somehow execute adb commands. Under the hood, it | ||
uses [special services.](https://github.com/android/android-test/tree/master/services) | ||
It's just a shell client and should be installed to the device. | ||
The Orchestrator also needs to execute adb commands. For that it uses [special services.](https://github.com/android/android-test/tree/master/services) under the hood. | ||
It's just a shell client and should be installed on the device. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. After reading this, I was doubting whether I have to install it on my own, or the Orchestrator does it for me (this one right?). This should be clarified |
||
|
||
 | ||
|
||
|
@@ -118,37 +119,41 @@ It's just a shell client and should be installed to the device. | |
Despite the fact that it does the job, this solution looks overcomplicated: | ||
|
||
1. We need to install +2 different apk to each emulator | ||
2. We delegate this job to the device instead of host machine. | ||
2. We delegate this job to the device instead of the host machine. | ||
<br/>Devices are less reliable than host pc | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Less reliably in the sense of "they do not lose connection to adb server, issues with battery, internet, system dialogs," or sth like that? |
||
|
||
##### Other solutions | ||
|
||
It's also possible to clear package data by | ||
using [3rd party test runners](https://android-ui-testing.github.io/Cookbook/practices/test_runners_review/), like | ||
Marathon, Avito-Runner or Flank. Marathon and Avito-Runner clear package data without an orchestrator. They delegate | ||
this logic to a host machine | ||
this logic to a host machine. | ||
|
||
##### Conclusion<br/> | ||
|
||
These are pros/cons for an `orchestrator` and 3rd party test runners solution: | ||
|
||
➕ Does the job for us in 100% <br/> | ||
➕ Does 100% of the job for us<br/> | ||
➕ Avoid test failing in cascade due to the application process being killed<br/> | ||
<br/> | ||
➖ Slow execution _(can take 10+ seconds and depends on apk size)_ <br/> | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 10+ was for ALL the tests together due to adb pm clear right? This I did not understand very well |
||
➖ Orchestrator — over-complicated <br/> | ||
➖ Slow execution <br/> | ||
➖ Requires to install extra components — over-complicated: <br/> | ||
|
||
Each `adb pm clear` takes some time and depends on apk size. Below you may see some gaps between the tests which | ||
represent such a delay | ||
The slow execution has 2 sources: | ||
1. The killing and restarting of the process where the test runs. | ||
2. Executing `adb pm clear` after each test takes some time and depends on apk size. Below you may see some gaps between the tests which represent such a delay. | ||
|
||
 | ||
|
||
|
||
## Suggestion | ||
!!! success | ||
|
||
Only package clear can guarantee that your data will be celared properly. | ||
Only package clearing can guarantee that the data will be cleared properly between test executions. | ||
Marathon and Avito-Runner provide the easiest way to clear application data. | ||
|
||
1. You can set them just by one flag in configuration | ||
2. They don't use orchestrator under the hood | ||
1. One simply needs to set a flag in their configuration | ||
2. They don't use orchestrator under the hood, avoiding its caveats | ||
|
||
|
||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I did not get this point with the runtime cache. What is stored in there and how it affects? Maybe an example would help
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Imagine if we store our data in databases and in runtime cache, like hashmap
Clearing of internal storage won't clear it. We have to clear it somehow, the only way -- use some production code, which is a bottleneck