-
Notifications
You must be signed in to change notification settings - Fork 74
Battery Tests
Battery tests are a special kind of detect test that simulate a run of detect with known files and verify the same bdio file is generated every time. They require more setup than a regular test but if you have a detect diagnostic zip you can easily create a new battery test. There are also gradle tasks available that can automatically save changes to the bdio file, so when changes do occur the cost is minimal.
A battery test creates a new source directory in a temporary system location (defined by the BATTERY_TESTS_PATH environment variable). Next a battery test creates files in this source directory as defined by the test. Then it executes detect against this source directory. Finally it compares the generated bdio files against known bdio files in resources.
Battery tests are varied, but most battery tests require one or more of the following:
- Creating empty files with known names that simply trigger a detectable into running.
- Copying resource files into the source directory so the detectable runs with known file contents.
- Creating executables that emit the contents of resource files when called.
- Creating executables that copy resource files into known locations when called.
Because battery tests require that every piece of external communication be mocked, it can be difficult to create a new test from scratch. If you have a diagnostic zip, all external communication is recorded and can be use to create a battery test.
Each test instantiates the BatteryTest class, defines the external communications it is mocking, and runs the battery test. The battery test sets up the environment, runs detect and verifies the generated bdio files match known bdio files.
Each battery tests must have a unique name and specify which directory it's resource files are loaded from. Additionally if it's bdio files are inside a "bdio" folder in the given resource directory it can register those bdio automatically.
final BatteryTest test = new BatteryTest("unique-test-name", "test-resource-directory");
test.expectBdioResources();
test.run();
Resource files are always relative to "src/test/resources/battery" and in this case the expected bdio resource files would be located in "src/test/resources/battery/test-resource-directory/bdio". The test verifies the expected number of bdio files are present and that each bdio file's contents matches the generated bdio file.
In many cases the name of the source directory is important and will influence the names of the generated bdio files. You can specify the source folders name.
test.sourceDirectoryNamed("directory-name");
There are also many cases where the diagnostic zip or the bdio file was generated from a git repository and this too will influence the name of the generated bdio files. You can specify if git information should be found.
test.git("https://github.com/BlackDuckCoPilot/example-gradle-travis", "master");
This configures the git detectable to find the given url and branch. It is not necessary but in many cases your bdio names will be influenced by this information.
Some detectables require files to be present but do not actually use the contents of the file. The battery test will create these files by name with no contents.
test.sourceFileNamed("build.gradle");
An empty file names "build.gradle" will be created in the source directory before the battery test runs.
Some detectables do require files to be present. The test can copy resource files by name into the source directory.
test.sourceFileFromResource("myfile.txt");
In this case the resource file "src/test/resources/battery/test-resource-directory/myfile.txt" will be copied into the source directory with the name "myfile.txt".
Some detectables require executables that return a given output. If the executable has a path override property, you can provide an array of resource files. Each time the given executable is called it returns the next resource file. To identify files meant to be returned in this way it is convention to use the "xout" extension (but this is not required).
test.executableFromResourceFiles(DetectProperties.Companion.getDETECT_MAVEN_PATH(), "first-call.xout", "second-call.xout");
The first time detect calls MAVEN the contents of "src/test/resources/battery/test-resource-directory/first-call.xout" are written to stdout (returned by the executable). The second time detect calls MAVEN the contents of "src/test/resources/battery/test-resource-directory/second-call.xout" are written to stdout.
Some detectables execute source files and these can be defined similarly. The names of these executables might change between linux and windows and you should specify what each command will be called on each platform.
test.executableSourceFileFromResourceFiles("mvnw.cmd", "mvnw", "mvn.txt");
In this case when on windows the file "mvnw.cmd" will be created but when on linux the file "mvnw" will be created. Both are created in the source directory and when called write the contents of "mvn.txt" to stdout.
Some detectables require that executables create files in known locations. In this case we can define an executable that copies files from resources and puts them into known locations. Further complicating this, the windows and linux versions of these commands may be different. If you are mocking an executable that copies files you must determine which argument corresponds to the output location on windows and on linux. Additionally you can provide a prefix to remove from the argument before the mock executable copies the file.
test.executableThatCopiesFiles(DetectProperties.Companion.getDETECT_DOTNET_PATH(), "myfile.txt")
.onWindows(5, "")
.onLinux(3, "--output_directory=");
This says that when the DOTNET executable is called; copy the file "src/test/resources/battery/test-resource-directory/myfile.txt". When on windows it copies the file to the folder specified by the 5th argument and when on Linux it copies to the folder specified by the third argument (after trimming "--output_directory=" from that argument).
You should only define each executable once. In the case that none of these patterns fits your specific executable requirements, additional types can be added.
Before battery tests: Complex real world projects generate large and complex dependency graphs. It has historically been unreasonable to add meaningful assertions to graphs this large - in fact original tests of this size generally only verified that one or two known dependencies were present and that the size of the graph stayed the same (same number of dependencies were found). These tests rarely broke and provided little value when they did. Then new tests were added that checked that the exact same json file was generated. When the output format changed, or any tiny variation was introduced the tests would break. When the test broke, you had to make a call on if it was ok to replace the test json file or not - and even if you did want to regenerate the test json file it was difficult to do so.
Enter battery tests: These provide clarification that these tests are expected to break when changes to graph are made. This way when you break a battery test you can take meaningful action - if you expected the break as you modified a generated graph, simply update the battery tests - and if you did not expect the graph to break, you should make a fix. This keeps the intention and expectations explicit and obvious.
Other benefits: They provide tooling to help with updates so when bdio format or dependency graph internals change, they can be easily updated. In general they are immune to refactoring changes - they still require changes when the external system changes or detect's input/output change but that is less frequent than refactoring.
Some drawbacks: Setting up a battery test takes more effort than setting up a regular test. You have to provide detect with everything it needs to run your detectable. This often means the output of multiple commands and/or large resource files. Additionally care has to be taken to ensure the test works across all operating systems.
Verifying bdio: Bdio are json files and as such we can't just simply assert the files contents are exactly the same. Unsorted arrays may appear in different orders and bdio contains some automatically generated UUIDs which cannot be known ahead of time. Thus the battery tests use a JSON compare algorithm to verify the two json files are the same. All dependencies must be present and must have the same relationships. UUIDs are ignored as are things like order of array elements.