Changepoint detection may be senstive to extreme outliers

[MichaelClerx]

https://www.cs.ox.ac.uk/projects/PINTS/functional-testing/#mcmc-banana-populationmcmc-1

It works really well on other cases, but here I think I can see a change that isn't picked up. Perhaps because a while earlier there was a KL-divergence of 1e15??

Todo:

• Remove this point in some ad-hoc way, see if the changepoint detection improves
• If so, find a good way to remove these very extreme outliers

[MichaelClerx]

Alternatively, should be fixed by #906

[ben18785]

@abhidg Can we have a look at this please? The method I had before here (using R) worked fine in this regard.

[MichaelClerx]

Not sure there's any point picking off small issues on the functional testing @ben18785 ! We need someone to sit down for it in earnest for a few days and work out what to do + how best to do it. See e.g. the other tickets open on "functional testing" https://github.com/pints-team/pints/issues?q=is%3Aissue+is%3Aopen+label%3Afunctional-testing

[ben18785]

Yep, but think one of the main reason it's failing is due to the way in which we are doing this changepoint detection. I did have a method in R that seemed to work ok

…

On Sat, Mar 21, 2020 at 4:12 PM Michael Clerx ***@***.***> wrote: Not sure there's any point picking off small issues on the functional testing @ben18785 <https://github.com/ben18785> ! We need someone to sit down for it in earnest for a few days and work out what to do + how best to do it. See e.g. the other tickets open on "functional testing" https://www.instagram.com/p/B9-HYxFD4sh/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#998 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCILKFQNWQ45II4FTXTT33RITRNBANCNFSM4I6YMLRA> .

[MichaelClerx]

Have you tested it on this specific data set?

The code we have now works well on lots of cases, but this ticket was about a situation where it didn't give the answer I expected, so thought it was worth debugging (at the time).

The main reason it's "not working" at the moment is that

• we're doing things like looking at the wrong metric Change changepoint metric to number of passes per commit #906
• we're only plotting changepoints, not actually doing anything with hose results (e.g. notifying developers)
• we're not using the x-axis in a sensible way (closely related to 1), it's currently something like commit, or test-run, but commit history isn't linear, and if we start doing things like "number of fails" then it has to be an aggregate of multiple tests etc.

So the whole thing needs thinking out

[MichaelClerx]

I still think the whole changepoint thing is a really great idea btw, and possibly novel for software engineering? Would be a paper if we do it right I'll bet. Just requires some proper thinking which isn't going to come from me for the next few months at least!

[MichaelClerx]

(having said that I'm now thinking about it quite a lot...)