Marginal differences shouldn't fail the test (there should be an option for this - sometimes a marginal failure is important)

[phorn-NI]

Caraya supports a margin of error when comparing floating point numbers (as 10.0 != 10.0 exactly)

I found a similar issue when generating png images and trying to compare them. The two attached images were generated by exactly the same configuration script and look identical, but not when you put them through a compare tool - however for my purposes they're close enough (its the ends of the error bars that are 1 or 2 pixels different).

Its not urgent as the work around is to compare the configuration script, which is generated from a list of parameters - and that's what I'm really trying test

[stagg54]

Hmm... I imagine this is fixable, but it will require some research. There is probably a png compare tool out there that might work for something like this. Right now IIRC (without looking at the code) I'm pretty sure it's just checking the file hashes to see if they match. That would have to tolerance builtin. That comparison is overrideable though - for VI and CTL files, it uses the LVCompare function.

[phorn-NI]

Interesting research project then 😄
As I said, it's not a big problem for my immediate use case - testing the generated json is robust and gives me the same info.

[stagg54]

Yeah I did a very brief look. ImageMagick seems to have a compare tool. Not sure how well that would work.