Skip to content

Commit

Permalink
Updated the Latex syntax in Variational_Inference_and_Joint_Distribut…
Browse files Browse the repository at this point in the history
…ions.ipynb

Updated the Latex syntax in Variational_Inference_and_Joint_Distributions.ipynb
  • Loading branch information
tilakrayal authored Jan 18, 2024
1 parent 17cc66c commit c9ce54c
Showing 1 changed file with 5 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -85,26 +85,27 @@
"## Overview of Bayesian Variational Inference\n",
"\n",
"Suppose we have the following generative process, where $\\theta$ represents random parameters, $\\omega$ represents deterministic parameters, and the $x_i$ are features and the $y_i$ are target values for $i=1,\\ldots,n$ observed data points:\n",
"\\begin{align*}\n",
"\n",
"$$\\begin{align*}\n",
"&\\theta \\sim r(\\Theta) && \\text{(Prior)}\\\\\n",
"&\\text{for } i = 1 \\ldots n: \\nonumber \\\\\n",
"&\\quad y_i \\sim p(Y_i|x_i, \\theta, \\omega) && \\text{(Likelihood)}\n",
"\\end{align*}\n",
"\\end{align*}$$\n",
"\n",
"VI is then characterized by:\n",
"$\\newcommand{\\E}{\\operatorname{\\mathbb{E}}}\n",
"\\newcommand{\\K}{\\operatorname{\\mathbb{K}}}\n",
"\\newcommand{\\defeq}{\\overset{\\tiny\\text{def}}{=}}\n",
"\\DeclareMathOperator*{\\argmin}{arg\\,min}$\n",
"\n",
"\\begin{align*}\n",
"$$\\begin{align*}\n",
"-\\log p(\\{y_i\\}_i^n|\\{x_i\\}_i^n, \\omega)\n",
"&\\defeq -\\log \\int \\textrm{d}\\theta\\, r(\\theta) \\prod_i^n p(y_i|x_i,\\theta, \\omega) && \\text{(Really hard integral)} \\\\\n",
"&= -\\log \\int \\textrm{d}\\theta\\, q(\\theta) \\frac{1}{q(\\theta)} r(\\theta) \\prod_i^n p(y_i|x_i,\\theta, \\omega) && \\text{(Multiply by 1)}\\\\\n",
"&\\le - \\int \\textrm{d}\\theta\\, q(\\theta) \\log \\frac{r(\\theta) \\prod_i^n p(y_i|x_i,\\theta, \\omega)}{q(\\theta)} && \\text{(Jensen's inequality)}\\\\\n",
"&\\defeq \\E_{q(\\Theta)}[ -\\log p(y_i|x_i,\\Theta, \\omega) ] + \\K[q(\\Theta), r(\\Theta)]\\\\\n",
"&\\defeq ``\\text{expected negative log likelihood\"} + ``\\text{kl regularizer\"}\n",
"\\end{align*}\n",
"\\end{align*}$$\n",
"\n",
"(Technically we're assuming $q$ is [absolutely continuous](https://en.wikipedia.org/wiki/Absolute_continuity#Absolute_continuity_of_measures) with respect to $r$. See also, [Jensen's inequality](https://en.wikipedia.org/wiki/Jensen%27s_inequality).)\n",
"\n",
Expand Down

0 comments on commit c9ce54c

Please sign in to comment.