Is qsv dedup stable?

[jqnatividad]

[tmtmtmtm]

@jqnatividad Is the behaviour of qsv dedup stable in terms of which rows get removed? My use case is that I have an ID, a bunch of other columns, and a timestamp, and I'd like to remove all but the latest entry for each ID. At the moment it seems that if I ensure that the CSV is ordered by timestamp (ascending) before a dedup -s id, this works — i.e. it's always keeping the last entry it sees for the ID. This doesn't seem to be documented, though, so I'm not sure if I can rely on it.

[jqnatividad]

@tmtmtmtm ...and now we have a specific test for this scenario - 69c854b.

[tmtmtmtm]

Woohoo. It's great to know I can rely on this! I'm afraid I'm very deep in some other stuff atm (for which qsv is proving amazingly useful), so don't really have the time/headspace to add docs for this right now, but I'll try to remember to circle back around to it in a couple of weeks if it's still useful then.

[tmtmtmtm]

@jqnatividad Did something change with this behaviour at some point, or this is still meant to be working the same way? I appear to be seeing different results in some cases since switching to a much newer version of qsv in the last few days, and I'm not sure if I missed an announcement somewhere (I skipped a lot of versions in one go!), or if I should investigate further. If this is still meant to work the same, I can look into producing a proper example case.

[jqnatividad]

@tmtmtmtm I created a test case specifically for your scenario and it's still working.

https://github.com/jqnatividad/qsv/blob/master/tests/test_combos.rs

But yeah, there have been a lot of changes to dedup - as qsv averages about a release a week 😄 ...
https://github.com/jqnatividad/qsv/commits/master/src/cmd/dedup.rs

Most of these changes were under-the-hood performance changes though, so functionally, it should still produce the same result.

So please do share test cases that worked on an older version of qsv, and one that doesn't with the latest release...

[tmtmtmtm]

Thanks. I've added an example at #774

I'm happy to produce other examples if needed: I'm seeing quite a few unexpected results from this today.

[jqnatividad]

It is now stable again with the release of 0.88.2.

It became unstable when we parallelized the sort using par_sort_unstable_by in an attempt to make dedup faster in 0.47.0.
https://github.com/jqnatividad/qsv/blob/master/CHANGELOG.md#0470---2022-05-12

With 0.88.2, dedup still does parallelized sort, but in a stable way using par_sort_by instead.
#776