cmdstanpy_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CmdStanPy Tutorial\n",
    "\n",
    "\n",
    "## Workflow Outline\n",
    "\n",
    "Given a dataset and a model specification written as a Stan program, the CmdStanPy workflow is:\n",
    "\n",
    "### Assemble input data as either:\n",
    "  + A Python `dict` object consisting of key-value pairs where the key corresponds\n",
    " to Stan data variables and the value is of the correct type and shape.\n",
    "  + An existing data file on disk in either JSON or Rdump format.\n",
    "\n",
    "### Compile the model\n",
    "  + The `CmdStanModel` object stores the filepath of the Stan program.\n",
    "  + The `compile` method translates the Stan program to C++ then calls the C++ compiler which creates the executable program.\n",
    "\n",
    "### Fit the model conditioned on the data to obtain an estimate of the posterior\n",
    "\n",
    "#### Run the NUTS-HMC sampler to obtain the full posterior density\n",
    "\n",
    "  + The `CmdStanModel` class method `sample` invokes Stan's NUTS-HMC sampler which conditions the model on the input data and returns a `CmdStanMCMC` object. The `CmdStanMCMC` object contains the set of draws from the posterior plus metadata.\n",
    "  + Runs any number of chains - default is 4 chains.\n",
    "  + The output of each chain is stored on disk as a Stan csv file.\n",
    "\n",
    "#### Estimate the mode of the posterior density via optimization\n",
    "\n",
    "  + The `CmdStanModel` class method `optimize` runs one of Stan's [optimization](https://mc-stan.org/docs/reference-manual/optimization-algorithms-chapter.html) algorithms to obtains an estimate of the mode of the posterior density.\n",
    "\n",
    "### Summarize and check the sample\n",
    "\n",
    "   + The `CmdStanMCMC` class method `summary` invokes CmdStan's `stansummary` utility. Returns a Pandas DataFrame with estimates of posterior means, standard deviations, Monte-Carlo standard error, effective sample size, and convergence diagnostic statistic for all parameters in the model.\n",
    "   + The `CmdStanMCMC` class method `diagnose` invokes CmdStan's `diagnose` utility which checks for the following problems:\n",
    "    + transitions that hit the maximum treedepth\n",
    "    + divergent transitions\n",
    "    + low E-BFMI values (sampler transitions HMC potential energy)\n",
    "    + low effective sample sizes\n",
    "    + high R-hat values\n",
    "   + See the Stan reference manual section on [posterior analysis](https://mc-stan.org/docs/reference-manual/analysis-chapter.html) for further details.\n",
    "\n",
    "###  Assemble the sample in-memory\n",
    "  + The resulting sample is accessed via the `CmdStanMCMC` object:\n",
    "    + `sample`  - all draws from all chains, stored as a 3-D numpy.ndarray.\n",
    "    + `chains` - number of chains run by sampler\n",
    "    + `draws` - draws per chain\n",
    "    + `column_names` - names of the parameters, transformed parameters, and generated quantities variables returned in each draw\n",
    "    + `csv_files` - list of Stan csv output files which comprise the sample\n",
    "  + The method `get_drawset` flattens the 3-D sample array into a 2-D pandas.DataFrame for downstream analysis.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "* Install Python package from PyPI or directly from GitHub:\n",
    "\n",
    "  + `pip install --upgrade cmdstanpy`\n",
    "  + `pip install -e git+https://github.com/stan-dev/cmdstanpy`\n",
    "\n",
    "\n",
    "* CmdStanPy uses CmdStan directly to compile and run Stan programs, therefore CmdStan must be installed locally.\n",
    "\n",
    "  + if you have a working installation of CmdStan, set environment variable `CMDSTAN` to the full path to the top-level CmdStan directory.\n",
    "  \n",
    "  + if you don't already have CmdStan installed, run Python script `install_cmdstan` which downloads and compiles the latest release from https://github.com/stan-dev/cmdstan/releases.  By default this installs the latest version of CmdStan in the location `~/.cmdstanpy`.  Flags -d and -v are used to specify the directory and version, respectively.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hello World:  Compile and run CmdStan example model `bernoulli.stan`\n",
    "\n",
    "Every CmdStan release has an `examples/bernoulli` directory which contains the Stan model and test data files.  In this example we compile the model and run the sampler on the model and data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import relevant classes and methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import os.path\n",
    "from cmdstanpy import CmdStanModel, cmdstan_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The CmdStanPy directory `examples/bernoulli` contains the model and data files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bernoulli_dir = os.path.join(cmdstan_path(), 'examples', 'bernoulli')\n",
    "bernoulli_path = os.path.join(bernoulli_dir, 'bernoulli.stan')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we create a `CmdStanModel` object for the specified Stan program file, then we call the `compile` method to create the corresponding C++ executable program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bernoulli_model = CmdStanModel(stan_file=bernoulli_path)\n",
    "print(bernoulli_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the HMC-NUTS sampler on the model and data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Input data is either a Python `Dict` with entries corresponding to input data values, or it can be a file in JSON or Rdump format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_json = os.path.join(bernoulli_dir, 'bernoulli.data.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a `Dict` is specified, CmdStanPy writes it to a temp file in JSON format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_data = { \"N\" : 10, \"y\" : [0,1,0,0,0,0,0,0,0,1] }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `CmdStanModel` method `sample` runs the NUTS-HMC sampler and returns a `CmdStanMCMC` object.  By default, `sample` runs 4 sampler chains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_fit = bernoulli_model.sample(data=bern_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Access the sample: the `CmdStanMCMC` object attributes and methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Summarize the results\n",
    "\n",
    "The `CmdStanMCMC` method `summary` returns output of CmdStan bin/stansummary as pandas.DataFrame.  The `summary` report provides estimates of both the parameter value and the goodness of fit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_fit.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Run sampler diagnostics\n",
    "\n",
    "The `CmdStanMCMC` method `diagnose` prints the output of CmdStan bin/stansummary to the console.  This is useful when the summary report shows `R_hat` values outside of the range of .99 to 1.01, or the number of effective samples (`N_eff` value) is below 5% of the total draws in the sample. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_fit.diagnose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assemble all draws from all chains as an in-memory pandas DataFrame\n",
    "\n",
    "The `CmdStanMCMC` method `get_drawset` returns a pandas.DataFrame, one draw per row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_drawset = bern_fit.get_drawset()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `get_drawset` returns a DataFrame which contains all columns from the sampler's csv output file, i.e., it contains both the sampler state and the values for all parameter, transformed parameter, and generated quantities variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_drawset.shape, bern_drawset.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `get_drawset` function argument `params` takes a list of parameter or column names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "thetas = bern_fit.get_drawset(params=['theta'])\n",
    "thetas.shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "thetas[0:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_drawset.theta.plot.density()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `CmdStanMCMC` property `sample` is a 3-D numpy ndarray which contains all draws across all chains.  This array is created only as needed; therefore the first time that this property is accessed CmdStanPy will read in the contents of the sampler's csv output files.  Because the csv output files also contain stepsize and metric information, the `stepsize` and `metric` arrays will also be created.\n",
    "\n",
    "The numpy ndarray is stored column major format so that values for each parameter are stored contiguously in memory, likewise all draws from a chain are contiguous.  Thus the dimensions of the ndarray are arranged as follows:  (draws, chains, columns):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_fit.sample.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python's index slicing operations can be used to access the information by chain.\n",
    "For example, to select all draws and all output columns from the first chain,\n",
    "we specify the chain index (2nd index dimension).  As arrays indexing starts at 0,\n",
    "the index '0' corresponds to the first chain in the ``RunSet``.\n",
    "\n",
    "The following expression selects the first 3 draws from chain 1 for the parameter `theta`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_fit.column_names[7], bern_fit.sample[0:3,0,7]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get HMC sampler tuning parameters\n",
    "\n",
    "##### stepsize\n",
    "\n",
    "The `CmdStanMCMC` property `stepsize` property is a 1-D numpy ndarray which contains the stepsize used by the sampler for each chain.  This array is created at the same time as the `sample` and `metric` arrays are created.\n",
    "\n",
    "At the end of adaptation, the stepsize for the 4 chains in this example is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_fit.stepsize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### metric_type, metric\n",
    "\n",
    "The `metric` property is an numpy ndarray which contains the metric used by the sampler for each chain.  This array is created at the same time as the `sample` and `stepsize` arrays are created.\n",
    "\n",
    "At the end of adaptation, the metric for the 4 chains in this example is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bern_fit.metric_type,  bern_fit.metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}