Look at Geweke paper "Getting it right"

[MichaelClerx]

https://doi.org/10.1198/016214504000001132

Comes highly recommended as a way of testing MCMC methods

[martinjrobins]

This looks really useful. It would require that all the samplers in PINTS expose their transition kernel q in a consistent fashion (i.e. the kernel that goes from the theta sample at n, to the next theta sample at n+1), but I think this would be the ask and tell interface, right?

I can have a go at putting this into the functional testing

[MichaelClerx]

That would be great @martinjrobins !

[martinjrobins]

@ben18785, or anyone else: what is a good test to check that a series of samples come from a normal distribution with zero mean and unit variance? I'd like to have as few samples as possible....

[ben18785]

Jarque-Bera test or Kolmogorov-Smirnov...not sure what’s best with small sample sizes

…

On 28 Nov 2018, at 13:30, Martin Robinson ***@***.***> wrote: @ben18785, or anyone else: what is a good test to check that a series of samples come from a normal distribution with zero mean and unit variance? I'd like to have as few samples as possible.... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[MichaelClerx]

If it helps any, the statistician who recommended this test said he'd never bothered with quantifying the mismatch, said it was obvious in the plots

[ben18785]

This paper by Donald Rubin (a statistical big kahuna) offers another testing framework for MCMC software which may be worth implementing alongside Geweke...

[ben18785]

@martinjrobins I have now had a long look at the Geweke testing and think that the reason the test is failing is that the second parameter dimension (the carrying capacity) is very slow to converge for the Metropolis sampler on the logistic model. This is based on the following:

• The test results tend to improve (slightly) when shifting from a uniform prior to a more constrained normal one. This change in the model constrains the joint distribution p(theta, y) more, resulting in a more pronounced distribution of p(theta) (which, at the moment, is the only distribution we compare for inference) meaning MCMC is faster to converge. Although still quite slow!
• The test passes for the logistic growth parameter relative to the carrying capacity. This is evident in the trace plots for each, where the carrying capacity is much slower to mix.

To backup these assertions, see the Geweke-and-Rubin notebook in the studies directory of this branch this branch

Things to consider trying:

• In all cases, I would suggest using a normal prior on parameters opposed to the uniform. This is to make the joint distribution p(theta, y) have a lower variance than if using a uniform.
• Try the test for the adaptive covariance algorithm (after the initial phase). The reason for suggesting this is that the bog standard Metropolis algorithm may converge too slowly for this test.
• Since we know the carrying capacity dimension is slow to converge, don't include this in the test. Not sure how I feel about this given that, if MCMC fails in one dimension, it can cause problems in others. However, we need to be pragmatic.
• Use the simulated data as well as the parameters in g(). From the plots that I've done, it looks like the simulated data for the logistic case may be much better behaved than the parameters.
• Try other dynamic problems. It may be that the logistic is an odd case although I suspect not.
• If the other dynamic problems don't work, we may consider making some more trivial Bayesian problems. For example, a normal prior and normal likelihood, resulting in a normal posterior (due to conjugacy). The more I think of this the more sense it makes to have such a thing in PINTS.

Later consider also trying:

• Rubin's method in the paper I link to above. It seems to be a complementary test based on quite similar principles.

[ben18785]

The only other paper in the citations to Rubin that discusses other tests. This is also worth looking into as they indicate more specific tests looking at failures in the numerical solving (which is likely an issue for our types of system).

[MichaelClerx]

pints-team/change-point-testing#22