Pairwise testing

[CLOVIS-AI]

Pairwise testing seems to be a technique of reducing the combinatiorial explosion of parameterized tests while still attempting to test all unique-enough situations.

It is documented in more details here.

I wonder if Parameterized could implement support with a new DSL builder;

parameterizedPairwise {
    // Exactly the same DSL as within the 'parameterized' block
}

that would run combinatorial requests using the pairwise method instead of running all cases.

[BenWoodworth]

I can tell you I've had a set of changes to make parameterize more flexible that I've had to put on hold because I've been pretty busy. One of them is changing ParameterizeScope to an interface, letting 3rd party code intercept provideDelegate, which in this case I think could be enough to support this on your own. A custom scope could swap out the original arguments iterators from the DSL, and then delegate different arguments accordingly to the default ParameterizeScope implementation.

I don't think it would be too hard to have this:

parameterizePairwise {
    val abc by parameterOf('a', 'b', 'c')
    val xyz by parameterOf('x', 'y', 'z')
    // ...
}

change provideDelegate to appear the same as:

parameterize {
    val abc by parameter(abcArguments)
    val xyz by parameterOf(xyzArgumentsIterator.next()) // One argument at a time
    // ...
}

I've been thinking of pushing ParameterizeScope interface change on its own in a new minor release, so this could be the reason I need to go forward with it, especially if you think this would be enough to do this on your end!

[BenWoodworth]

I will say that I've thought about exploding combinations a good few times and have had a few ideas for behavior or API could work.

---

Straight up pairwise, like parameterOf(Triple(a, b, c), Triple(x, y, z), ...), except separate parameters that iterate in lock step with each other. This assumes that the parameters all have the same number of arguments, and that parameters aren't declared conditionally.

---

Having some API to mark that a parameter should iterate pairwise with an earlier one:

val abc by parameterOf(...)
val xyz by parameterOf(...).pairwiseWith(::abc) // though local property references aren't currently supported

Or maybe even a .pairwise() that makes it iterate pairwise with whatever previous parameter

---

Instead of pairwise, parameters could iterate one at a time:

With parameters abc, ijk, and xyz, we have 7 cases instead of 27:

a, i, x  // The "control" iteration, with all parameters at their first argument

b, i, x  // change only the 1st parameter
c, i, x  //

a, j, x  // only the 2nd parameter
a, k, x  //

a, j, y // only the 3rd parameter
a, j, z //

I think this makes sense when the parameters are independent, when changing one shouldn't affect the result. This also solves parameters needing to have the same number of arguments. And might solve the problem of conditionally declared parameters?

That, and I like how this is similar to scientific experiments, having a control case, and then testing for only one variable at a time.

[BenWoodworth]

And aside from those possible ways to go about limiting exploding combinations, these are a couple of the approaches I currently use.

---

Using one parameter with a pair or data class:

data class TestCase(val abc: Char, val xyz: Char)

val testCase by parameterOf(
    TestCase('a', 'x'),
    TestCase('b', 'y'),
    TestCase('c', 'z')
)

val (abc, xyz) = testCase

An extra line or two for the class and optionally destructuring, but it's easy enough to do when I test on edge cases whose relations are fairly arbitrary.

---

However, if the values have a relationship to one another that can be encoded in a test, I like declaring the second parameter with one argument. The extra parameter is redundant if you're not using parameterize's multi-failure reporting which lists all the parameter values for failures.

val abc by parameterOf('a', 'b', 'c')
val xyz by parameterOf(abc + 23)

[BenWoodworth]

I'm planning on releasing Parameterize v0.4.0 once Kotlin 2.2.0 is out (which I think will be soon). But in the next release I reworked the API a little bit to make it more extensible, e.g. changing ParameterizeScope into an interface. You can peek at the interface-parameterize-scope branch if you want to play around with it.

But using that, it's possible to make something like this, even from outside the library:

parameterize {
    val case = object : PairwiseParameters(this) {
        val abc by parameterOf("a", "b", "c")
        val xyz by parameterOf("x", "y", "z")
    }

    println(case.abc + case.xyz) // "ax", "by", "cz"
}

You can play around with it on my issue-38-PairwiseParameters branch if you'd like. (permalink)

Object expressions seem like the only way to nest/group parameters together while allowing the properties to be accessed from outside.

The PairwiseParameters class also requires you to pass in the ParameterizeScope, that way it can be wrapped with calls mapped in a way that parameters appear to have been declared pairwise manually. In the future it might be possible to add a context parameter to the provideDelegate function in PairwiseParameters, getting ahold of the scope that way.

[CLOVIS-AI]

Kotlin 2.2.0 is out 🎉

[CLOVIS-AI]

Your example doesn't generate the cases in the same way as the article I linked, but indeed this new version should make it possible.

Object expressions seem like the only way to nest/group parameters together while allowing the properties to be accessed from outside.

That's my conclusion as well. I've had the same issue with Spine, I wanted it to be a DSL, but it's not possible.

[BenWoodworth]

Kotlin 2.2.0 is out 🎉

Looks like my intuition is spot-on 😂

Your example doesn't generate the cases in the same way as the article I linked, but indeed this new version should make it possible.

You're right! I ran with the assumption that pairwise meant iterating all the parameters in parallel. I'll have to sit on it to grasp what going through the pairwise combinations would actually look like.

Just to make sure I understand. After parameterizePairwise runs, for any given pair of parameters, each combination should've apprared exactly once, correct?

[BenWoodworth]

^I just wrote up the spec for this behavior in the issue-38-parameterizePairwise branch

[CLOVIS-AI]

Just to make sure I understand. After parameterizePairwise runs, for any given pair of parameters, each combination should've apprared exactly once, correct?

I think so, yes. To be honest, I haven't fully grasped the sequence myself.

[BenWoodworth]

I haven't delved into the literature just yet, that way I can think through this with a fresh perspective first, but I took a quick stab at pairwise iteration.

You can read my commit message for details here, but it was pretty straightforward to write a simple pairwise implementation, tackling one new pair combination each iteration: 7e7a6f9

There's also the idea of covering many pairs per iteration, but I'm not sure that's possible yet (or even desirable for testers, since it might be hard to identify a failure's cause if there were multiple new combinations that iteration). For reference though, with 10 parameters that have 10 arguments:

Approach	Iterations
Exhaustive	1410065408
Pairwise (implemented)	3736
Pairwise (minimal)	100