Week2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Second week notes and exercises from the Linear algebra module of the course: Mathematics for Machine Learning, avaliable at Coursera.\n",
    "### Note that only a few exercises are avaliable to help students understand some of the algebra of the course, full disclosure of all exercises isn't possible due to coursera police. Contact me if you have any questions, i will be glad to help you process of learning!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Size of a vector\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given a vectors r with components [r1,r2...rn]:\n",
    "$$r = \\begin{bmatrix}r1\\\\\n",
    "r2\\\\\n",
    ".\\\\\n",
    ".\\\\\n",
    "rn\n",
    "\\end{bmatrix}$$\n",
    "\n",
    "The size of the vector is definied as:\n",
    "\n",
    "$|r| = \\sqrt{r1^2 + r2^2 + ... + rn^2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dot product"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given two vectors, r and s:\n",
    "$$r = \\begin{bmatrix}r1\\\\\n",
    "r2\\\\\n",
    ".\\\\\n",
    ".\\\\\n",
    "rn\n",
    "\\end{bmatrix}$$\n",
    "\n",
    "$$s = \\begin{bmatrix}s1\\\\\n",
    "s2\\\\\n",
    ".\\\\\n",
    ".\\\\\n",
    "sn\n",
    "\\end{bmatrix}$$\n",
    "\n",
    "The dot product between r and s is definied as:\n",
    "\n",
    "$r . s = r1 . s1 + r2 . s2 + .... + rn . sn$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cosine rule"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given two vectors, r and s, the cosine rule in vector notation is:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$|r - s|^2 = |r|^2 + |s|^2 - 2|r||s|cos\\theta$\n",
    "\n",
    "Using notable products we can decompose $|r - s|^2$ in $|r|^2 - 2|r||s| + |s|^2$\n",
    "\n",
    "Replancing in the first equation we have that :\n",
    "$|r|^2 - 2|r||s| + |s|^2 = |r|^2 + |s|^2 - 2|r||s|cos\\theta$\n",
    "\n",
    "Simplyfing this expression gives us that $r.s = |r||s|cos\\theta$\n",
    "\n",
    "This means that the Dot product gives us information about the angle of the vectors, if the vectors are orthogonal the cosine would be 0 and the dot product 0 for example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scalar projections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The projection of a vector s in another vector r is given by $|s|cos\\theta$ where $cos\\theta$ is the angle between s and r.\n",
    "\n",
    "We know that the dot product $r.s = |r||s|cos\\theta$, so the dot product is actually giving us the projection of s $(|s|cos\\theta$) times the size of r ($|r|$).\n",
    "\n",
    "So if we want to find the scalar projection, it is easily done by using this formula: $ \\frac{r.s}{|r|} = |s|cos\\theta$\n",
    "\n",
    "vector projection: $r\\frac{rs}{|r||r|}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practice exercises - dot products of vectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2 -\n",
    "\n",
    "For two n component vectors, $\\mathbf{a}\\cdot\\mathbf{b} = a_1b_1 + a_2b_2 + \\dots + a_nb_na⋅b$\n",
    "\n",
    "What is the dot product of the vectors $\\mathbf{r} = \\begin{bmatrix} -5 \\\\ 3 \\\\ 2 \\\\ 8 \\end{bmatrix}$\n",
    "and $\\mathbf{s} = \\begin{bmatrix} 1 \\\\ 2 \\\\ -1 \\\\ 0 \\end{bmatrix}$\n",
    " ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$-5.1 + 3.2 + (2.-1) + 8.0 = -1$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3 - "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following diagram shows the projection of $\\mathbf{s}$ onto $\\mathbf{r}$ when the vectors are in two dimensions:\n",
    "\n",
    "\n",
    "Remember that the scalar projection is the size of the green vector. If the angle between $\\mathbf{s}$ and $\\mathbf{r}$ is greater than $\\pi/2$, the projection will also have a minus sign.\n",
    "\n",
    "We can do projection in any number of dimensions. Consider two vectors with three components, $\\mathbf{r} = \\begin{bmatrix} 3 \\\\ -4 \\\\ 0 \\end{bmatrix}$\n",
    " and  $\\mathbf{s} = \\begin{bmatrix} 10 \\\\ 5 \\\\ -6 \\end{bmatrix}$\n",
    "\n",
    "What is the scalar projection of $\\mathbf{s}$ onto $\\mathbf{r}$?\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$r.s = 3.10 + (-4*5) + 0 = 10$\n",
    "\n",
    "and\n",
    "\n",
    "$|r| = \\sqrt{3^2 + {-4}^2} = 5$\n",
    "\n",
    "and we know that \n",
    "\n",
    "$ \\frac{r.s}{|r|} = |s|cos\\theta$\n",
    "\n",
    "since the projection of s onto r is $|s|cos\\theta$\n",
    "\n",
    "then\n",
    "\n",
    "$\\frac{r.s}{|r|} = 10/5 = 2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4 - \n",
    "\n",
    "Consider two vectors with three components, $\\mathbf{r} = \\begin{bmatrix} 3 \\\\ -4 \\\\ 0 \\end{bmatrix}$\n",
    " and  $\\mathbf{s} = \\begin{bmatrix} 10 \\\\ 5 \\\\ -6 \\end{bmatrix}$\n",
    "\n",
    "What is the vector projection of $\\mathbf{s}$ onto $\\mathbf{r}$?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The vector projection simple is the scalar projection times the vector(normalized by its length), so since we got the scalar projection in exercise 3 (scalar projection = 2)\n",
    "\n",
    "we normalize the vector r by its length:\n",
    "$\\begin{bmatrix} 3 \\\\ -4 \\\\ 0 \\end{bmatrix} / 5 = \\begin{bmatrix} 3/5 \\\\ -4/5 \\\\ 0 \\end{bmatrix}$\n",
    "\n",
    "and multiply the normalized vector by the scalar projection of s:\n",
    "\n",
    "$ \\begin{bmatrix} 6/5 \\\\ -8/5 \\\\ 0 \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basis change\n",
    "\n",
    "First we have to make sure our new basis are orthogonal (for simplicity), to do this we just calculate the dot product of the basis and check if they are 0 (orthogonal).\n",
    "\n",
    "To change basis from e ($r_{e}$) to b ($r_{b}$):\n",
    "\n",
    "$\\frac{r_{e} . b_{1}}{|b_{1}|^2}$ and $\\frac{r_{e} . b_{2}}{|b_{2}|^2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practice exercise 1 - Changing basis\n",
    "\n",
    "Given vectors $\\mathbf{v} = \\begin{bmatrix} 5 \\\\ -1 \\end{bmatrix}$, $\\mathbf{b_1} = \\begin{bmatrix} 1 \\\\ 1 \\end{bmatrix}$ and $\\mathbf{b_2} = \\begin{bmatrix} 1 \\\\ -1 \\end{bmatrix}$ all written in the standard basis, what is $\\mathbf{v}$ in the basis defined by $\\mathbf{b_1}$?\n",
    "You are given that $\\mathbf{b_1}$ and $\\mathbf{b_2}$ are orthogonal to each other.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\frac{v . b_{1}}{|b_{1}|^2} =  \\frac{5*1 + (-1*1)}{1^2 + 1^2}  = 4/2 = 2$\n",
    "\n",
    "$\\frac{v . b_{2}}{|b_{2}|^2} =  \\frac{5*1 + (-1*-1)}{1^2 + -1^2}  = 6/2 = 3$\n",
    "\n",
    "$v_{b} = \\begin{bmatrix} 2 \\\\ 3 \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Practice exercise 1 - Linear dependency\n",
    "\n",
    "Are the following vectors linearly dependent?\n",
    "\n",
    "$\\mathbf{a} = \\begin{bmatrix} 1 \\\\ 1 \\end{bmatrix}$ and $\\mathbf{b} = \\begin{bmatrix} 2 \\\\ 2 \\end{bmatrix}$\n",
    "\n",
    "Yes, because b can be described as $2 \\times a = 2 \\times \\begin{bmatrix} 1 \\\\ 1 \\end{bmatrix} = \\begin{bmatrix} 2 \\\\ 2 \\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}