01_Numpy and Pandas Intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Why python for data analysis, machine learning?\n",
    "There are lots of reasons that we want to use python for doing data science. It is certainly one of the younger programming languages used in the data science ecosystem (compared to say R and SAS) but it is used just as frequently for analysis as SAS and R. Having a good foundation in python and R, (and SAS or SPSS) should be a *must* for **every data scientist** and machine learning enthusiast. \n",
    "\n",
    "Here is a latex equation $\\lambda=5$ \n",
    "\n",
    "In this course, python allows for an open source method of performing machine learning that runs from just about any machine. So let's start with looking at Numpy and Pandas pachages for analyzing data. \n",
    "\n",
    "With that in mind, let's go over the following:\n",
    "- Numpy matrices\n",
    "- Simple operations on arrays and matrices\n",
    "- Indexing with numpy\n",
    "- Pandas for tabular data\n",
    "- Representing categorical data (discussion point)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3.11.9 (main, Apr 19 2024, 11:43:47) [Clang 14.0.6 ]\n",
      "1.26.4\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import numpy as np\n",
    "\n",
    "print(sys.version)\n",
    "print(np.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.97757361, 0.6074737 , 0.32272259],\n",
       "       [0.44369354, 0.7519841 , 0.61923266],\n",
       "       [0.88949607, 0.63783361, 0.53441279],\n",
       "       [0.529084  , 0.56230195, 0.43445827],\n",
       "       [0.94105557, 0.76364299, 0.35492271]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = np.random.rand(5,3)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5, 3)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "operands could not be broadcast together with shapes (5,3) (3,4) ",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[5], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# will this work?\u001b[39;00m\n\u001b[1;32m      2\u001b[0m y \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39mrandom\u001b[38;5;241m.\u001b[39mrand(\u001b[38;5;241m3\u001b[39m,\u001b[38;5;241m4\u001b[39m)\n\u001b[0;32m----> 3\u001b[0m z \u001b[38;5;241m=\u001b[39m \u001b[43mx\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43my\u001b[49m\n\u001b[1;32m      4\u001b[0m z\n",
      "\u001b[0;31mValueError\u001b[0m: operands could not be broadcast together with shapes (5,3) (3,4) "
     ]
    }
   ],
   "source": [
    "# will this work?\n",
    "y = np.random.rand(3,4)\n",
    "z = x*y\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.24041657, 0.33631319, 1.52497986, 0.94332881],\n",
       "       [1.08697819, 0.44616084, 1.24193673, 0.79875978],\n",
       "       [1.31308337, 0.42982364, 1.56169473, 0.9912235 ],\n",
       "       [0.94452416, 0.33871886, 1.10851523, 0.70420692],\n",
       "       [1.3134251 , 0.37311822, 1.6097747 , 0.99041632]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we can designate what matrix multiplication is directly using objects\n",
    "z = np.dot(x,y)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1.24041657, 0.33631319, 1.52497986, 0.94332881],\n",
       "       [1.08697819, 0.44616084, 1.24193673, 0.79875978],\n",
       "       [1.31308337, 0.42982364, 1.56169473, 0.9912235 ],\n",
       "       [0.94452416, 0.33871886, 1.10851523, 0.70420692],\n",
       "       [1.3134251 , 0.37311822, 1.6097747 , 0.99041632]])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# or we can use the overloaded matrix multiplication operator\n",
    "z = x @ y\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 2, 3],\n",
       "       [4, 5, 6],\n",
       "       [7, 8, 9]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1 = np.array([[1,2,3],\n",
    "               [4,5,6],\n",
    "               [7,8,9]])\n",
    "x1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "5\n",
      "8\n"
     ]
    }
   ],
   "source": [
    "for row in range(x1.shape[0]):\n",
    "    print(x1[row,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2 5 8]\n",
      "[False  True  True]\n",
      "[[4 5 6]\n",
      " [7 8 9]]\n"
     ]
    }
   ],
   "source": [
    "print(x1[:,1])\n",
    "print(x1[:,1]>3)\n",
    "# slicing\n",
    "print(x1[ x1[:,1]>3 ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 2 3 4 5 6 7 8 9]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(10,)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2 = np.array(range(10))\n",
    "print(x2)\n",
    "x2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0],\n",
       "       [1],\n",
       "       [2],\n",
       "       [3],\n",
       "       [4],\n",
       "       [5],\n",
       "       [6],\n",
       "       [7],\n",
       "       [8],\n",
       "       [9]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_tmp = x2[:,np.newaxis]\n",
    "x_tmp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False False False False False False  True  True  True  True]\n",
      "[6 7 8 9]\n"
     ]
    }
   ],
   "source": [
    "idx = x2>5\n",
    "print(idx)\n",
    "print(x2[idx])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6, 7, 8, 9])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2[x2>5] # rows of x2 where x2 is greater than 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Named columns\n",
    "So what if we have a matrix of data where each row is some observation of features and the feature values are represented in each column?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  64, 2100,    1],\n",
       "       [  50, 2200,    4],\n",
       "       [  48, 2300,    3],\n",
       "       [  34,    0,    2],\n",
       "       [  30,  100,    5]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col_names = ['temperature','time','day']\n",
    "data = np.array([[64,2100,1],\n",
    "                 [50,2200,4],\n",
    "                 [48,2300,3],\n",
    "                 [34,0,   2],\n",
    "                 [30,100, 5]])\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  64, 2100,    1],\n",
       "       [  50, 2200,    4],\n",
       "       [  48, 2300,    3]])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data2 = data[data[:,1]>1500]\n",
    "data2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.2.2\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>temperature</th>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>64</td>\n",
       "      <td>2100</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>2200</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>48</td>\n",
       "      <td>2300</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30</td>\n",
       "      <td>100</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   temperature  time  day\n",
       "0           64  2100    1\n",
       "1           50  2200    4\n",
       "2           48  2300    3\n",
       "3           34     0    2\n",
       "4           30   100    5"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# pandas to the rescue\n",
    "import pandas as pd\n",
    "print(pd.__version__)\n",
    "\n",
    "df = pd.DataFrame(data,columns=col_names)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'numpy.ndarray'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[  64, 2100,    1],\n",
       "       [  50, 2200,    4],\n",
       "       [  48, 2300,    3],\n",
       "       [  34,    0,    2],\n",
       "       [  30,  100,    5]])"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# can always access the backend numpy with .values\n",
    "print(type(df.to_numpy()))\n",
    "df.to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>temperature</th>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>64</td>\n",
       "      <td>2100</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>2200</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>48</td>\n",
       "      <td>2300</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   temperature  time  day\n",
       "0           64  2100    1\n",
       "1           50  2200    4\n",
       "2           48  2300    3"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.time>1500]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 5 entries, 0 to 4\n",
      "Data columns (total 3 columns):\n",
      " #   Column       Non-Null Count  Dtype\n",
      "---  ------       --------------  -----\n",
      " 0   temperature  5 non-null      int64\n",
      " 1   time         5 non-null      int64\n",
      " 2   day          5 non-null      int64\n",
      "dtypes: int64(3)\n",
      "memory usage: 252.0 bytes\n"
     ]
    }
   ],
   "source": [
    "# lets get a description of the data\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/8f/szs_zmy96hd71pg9d355wt9h0000gp/T/ipykernel_87038/3346712547.py:2: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. Value 'Mon' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.\n",
      "  df.loc[df.day==1,'day'] = 'Mon'\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>temperature</th>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>64</td>\n",
       "      <td>2100</td>\n",
       "      <td>Mon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>2200</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>48</td>\n",
       "      <td>2300</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30</td>\n",
       "      <td>100</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   temperature  time  day\n",
       "0           64  2100  Mon\n",
       "1           50  2200    4\n",
       "2           48  2300    3\n",
       "3           34     0    2\n",
       "4           30   100    5"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# FutureWarning: Setting an item of incompatible dtype is deprecated and will raise an error in a future version of pandas. \n",
    "# Value 'Mon' has dtype incompatible with int64, please explicitly cast to a compatible dtype first.\n",
    "df.loc[df.day==1,'day'] = 'Mon'\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>temperature</th>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>64</td>\n",
       "      <td>2100</td>\n",
       "      <td>Mon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>2200</td>\n",
       "      <td>Th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>48</td>\n",
       "      <td>2300</td>\n",
       "      <td>Wed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>Tues</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30</td>\n",
       "      <td>100</td>\n",
       "      <td>Fri</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   temperature  time   day\n",
       "0           64  2100   Mon\n",
       "1           50  2200    Th\n",
       "2           48  2300   Wed\n",
       "3           34     0  Tues\n",
       "4           30   100   Fri"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# there is almost always a more efficient built in pandas function\n",
    "df.day = df.day.replace(to_replace=range(7),\n",
    "               value=['Su','Mon','Tues','Wed','Th','Fri','Sat'])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 5 entries, 0 to 4\n",
      "Data columns (total 3 columns):\n",
      " #   Column       Non-Null Count  Dtype \n",
      "---  ------       --------------  ----- \n",
      " 0   temperature  5 non-null      int64 \n",
      " 1   time         5 non-null      int64 \n",
      " 2   day          5 non-null      object\n",
      "dtypes: int64(2), object(1)\n",
      "memory usage: 252.0+ bytes\n"
     ]
    }
   ],
   "source": [
    "# notice how the type of the column has changed to an object \"categorical\"\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fri</th>\n",
       "      <th>Mon</th>\n",
       "      <th>Th</th>\n",
       "      <th>Tues</th>\n",
       "      <th>Wed</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Fri    Mon     Th   Tues    Wed\n",
       "0  False   True  False  False  False\n",
       "1  False  False   True  False  False\n",
       "2  False  False  False  False   True\n",
       "3  False  False  False   True  False\n",
       "4   True  False  False  False  False"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# one hot encoding example\n",
    "pd.get_dummies(df.day)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Some Pandas Syntax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     Mon\n",
      "1      Th\n",
      "2     Wed\n",
      "3    Tues\n",
      "4     Fri\n",
      "Name: day, dtype: object\n",
      "0     Mon\n",
      "1      Th\n",
      "2     Wed\n",
      "3    Tues\n",
      "4     Fri\n",
      "Name: day, dtype: object\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temperature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Mon</td>\n",
       "      <td>64</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Th</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Wed</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tues</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fri</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    day  temperature\n",
       "0   Mon           64\n",
       "1    Th           50\n",
       "2   Wed           48\n",
       "3  Tues           34\n",
       "4   Fri           30"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# slicing into a pandas dataframe\n",
    "print(df.day)\n",
    "print(df['day'])\n",
    "df[['day','temperature']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wed\n",
      "2     Wed\n",
      "3    Tues\n",
      "4     Fri\n",
      "Name: day, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(df.day[2]) # print the value\n",
    "print(df.day[2:]) # print as pandas series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>temperature</th>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>Tues</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30</td>\n",
       "      <td>100</td>\n",
       "      <td>Fri</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   temperature  time   day\n",
       "3           34     0  Tues\n",
       "4           30   100   Fri"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# index location\n",
    "df.iloc[3:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temperature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tues</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fri</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    day  temperature\n",
       "3  Tues           34\n",
       "4   Fri           30"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[3:][['day','temperature']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>day</th>\n",
       "      <th>temperature</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Tues</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Fri</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    day  temperature\n",
       "3  Tues           34\n",
       "4   Fri           30"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[['day','temperature']].iloc[3:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "temperature      45.2\n",
       "time           1340.0\n",
       "dtype: float64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.mean(numeric_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "temperature      13.608821\n",
       "time           1180.254210\n",
       "dtype: float64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.std(numeric_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "temperature    3.321375\n",
       "time           1.135349\n",
       "dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.mean(numeric_only=True)/df.std(numeric_only=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2100, 2200, 2300,    0,  100])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.time.unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pandas Block Manager\n",
    "Let's take a look at some important points from the following post:\n",
    " - https://uwekorn.com/2020/05/24/the-one-pandas-internal.html\n",
    "\n",
    "This is the pandas BlockManager, which tries to group internal structures together to make things fast:\n",
    "<img src=\"https://uwekorn.com/images/pd-df-perception.002.png\" width=200 height=200 />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>temperature</th>\n",
       "      <th>time</th>\n",
       "      <th>day</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>64</td>\n",
       "      <td>2100</td>\n",
       "      <td>Mon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>50</td>\n",
       "      <td>2200</td>\n",
       "      <td>Th</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>48</td>\n",
       "      <td>2300</td>\n",
       "      <td>Wed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>Tues</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>30</td>\n",
       "      <td>100</td>\n",
       "      <td>Fri</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   temperature  time   day\n",
       "0           64  2100   Mon\n",
       "1           50  2200    Th\n",
       "2           48  2300   Wed\n",
       "3           34     0  Tues\n",
       "4           30   100   Fri"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/8f/szs_zmy96hd71pg9d355wt9h0000gp/T/ipykernel_87038/2037560386.py:1: DeprecationWarning: DataFrame._data is deprecated and will be removed in a future version. Use public APIs instead.\n",
      "  print(df._data.nblocks)\n",
      "/var/folders/8f/szs_zmy96hd71pg9d355wt9h0000gp/T/ipykernel_87038/2037560386.py:2: DeprecationWarning: DataFrame._data is deprecated and will be removed in a future version. Use public APIs instead.\n",
      "  df._data\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "BlockManager\n",
       "Items: Index(['temperature', 'time', 'day'], dtype='object')\n",
       "Axis 1: RangeIndex(start=0, stop=5, step=1)\n",
       "NumpyBlock: slice(0, 2, 1), 2 x 5, dtype: int64\n",
       "NumpyBlock: slice(2, 3, 1), 1 x 5, dtype: object"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(df._data.nblocks)\n",
    "df._data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advantages and disadvantages:\n",
    "This can speed up operations because it inhenertly can apply operations along columns in a single pass over the data (like sums, etc.) and therefore is using c++ for much of the heavy lifting.\n",
    "\n",
    "But, **it might be bad** when you are adding columns to the data because it can trigger consolidation of columns, which means copying over data in numpy to creata new matrix. The slow down also doesn't show up until a needed column is accessed (lazy data copying). Let's do an example from:  https://uwekorn.com/2020/05/24/the-one-pandas-internal.html\n",
    "\n",
    "**Block consolidation is triggered after 100 blocks of data are reached.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>int64</th>\n",
       "      <th>float64</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048571</th>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048572</th>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048573</th>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048574</th>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048575</th>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1048576 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           int64    float64\n",
       "0              0        0.0\n",
       "1              1        1.0\n",
       "2              2        2.0\n",
       "3              3        3.0\n",
       "4              4        4.0\n",
       "...          ...        ...\n",
       "1048571  1048571  1048571.0\n",
       "1048572  1048572  1048572.0\n",
       "1048573  1048573  1048573.0\n",
       "1048574  1048574  1048574.0\n",
       "1048575  1048575  1048575.0\n",
       "\n",
       "[1048576 rows x 2 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# we will start with a 2 column dataframe\n",
    "# one column is an int and the other a float\n",
    "# becasue there are two datatypes this has two blocks\n",
    "df_example = pd.DataFrame({\n",
    "    'int64': np.arange(1024 * 1024, dtype=np.int64),\n",
    "    'float64': np.arange(1024 * 1024, dtype=np.float64),\n",
    "})\n",
    "df_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<timed exec>:7: DeprecationWarning: DataFrame._data is deprecated and will be removed in a future version. Use public APIs instead.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "98\n",
      "CPU times: user 50.6 ms, sys: 68.8 ms, total: 119 ms\n",
      "Wall time: 153 ms\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>int64</th>\n",
       "      <th>float64</th>\n",
       "      <th>new_0</th>\n",
       "      <th>new_1</th>\n",
       "      <th>new_2</th>\n",
       "      <th>new_3</th>\n",
       "      <th>new_4</th>\n",
       "      <th>new_5</th>\n",
       "      <th>new_6</th>\n",
       "      <th>new_7</th>\n",
       "      <th>...</th>\n",
       "      <th>new_86</th>\n",
       "      <th>new_87</th>\n",
       "      <th>new_88</th>\n",
       "      <th>new_89</th>\n",
       "      <th>new_90</th>\n",
       "      <th>new_91</th>\n",
       "      <th>new_92</th>\n",
       "      <th>new_93</th>\n",
       "      <th>new_94</th>\n",
       "      <th>new_95</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048571</th>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571.0</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>...</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "      <td>1048571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048572</th>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572.0</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>...</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "      <td>1048572</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048573</th>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573.0</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>...</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "      <td>1048573</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048574</th>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574.0</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>...</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "      <td>1048574</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1048575</th>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575.0</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>...</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "      <td>1048575</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1048576 rows × 98 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           int64    float64    new_0    new_1    new_2    new_3    new_4  \\\n",
       "0              0        0.0        0        0        0        0        0   \n",
       "1              1        1.0        1        1        1        1        1   \n",
       "2              2        2.0        2        2        2        2        2   \n",
       "3              3        3.0        3        3        3        3        3   \n",
       "4              4        4.0        4        4        4        4        4   \n",
       "...          ...        ...      ...      ...      ...      ...      ...   \n",
       "1048571  1048571  1048571.0  1048571  1048571  1048571  1048571  1048571   \n",
       "1048572  1048572  1048572.0  1048572  1048572  1048572  1048572  1048572   \n",
       "1048573  1048573  1048573.0  1048573  1048573  1048573  1048573  1048573   \n",
       "1048574  1048574  1048574.0  1048574  1048574  1048574  1048574  1048574   \n",
       "1048575  1048575  1048575.0  1048575  1048575  1048575  1048575  1048575   \n",
       "\n",
       "           new_5    new_6    new_7  ...   new_86   new_87   new_88   new_89  \\\n",
       "0              0        0        0  ...        0        0        0        0   \n",
       "1              1        1        1  ...        1        1        1        1   \n",
       "2              2        2        2  ...        2        2        2        2   \n",
       "3              3        3        3  ...        3        3        3        3   \n",
       "4              4        4        4  ...        4        4        4        4   \n",
       "...          ...      ...      ...  ...      ...      ...      ...      ...   \n",
       "1048571  1048571  1048571  1048571  ...  1048571  1048571  1048571  1048571   \n",
       "1048572  1048572  1048572  1048572  ...  1048572  1048572  1048572  1048572   \n",
       "1048573  1048573  1048573  1048573  ...  1048573  1048573  1048573  1048573   \n",
       "1048574  1048574  1048574  1048574  ...  1048574  1048574  1048574  1048574   \n",
       "1048575  1048575  1048575  1048575  ...  1048575  1048575  1048575  1048575   \n",
       "\n",
       "          new_90   new_91   new_92   new_93   new_94   new_95  \n",
       "0              0        0        0        0        0        0  \n",
       "1              1        1        1        1        1        1  \n",
       "2              2        2        2        2        2        2  \n",
       "3              3        3        3        3        3        3  \n",
       "4              4        4        4        4        4        4  \n",
       "...          ...      ...      ...      ...      ...      ...  \n",
       "1048571  1048571  1048571  1048571  1048571  1048571  1048571  \n",
       "1048572  1048572  1048572  1048572  1048572  1048572  1048572  \n",
       "1048573  1048573  1048573  1048573  1048573  1048573  1048573  \n",
       "1048574  1048574  1048574  1048574  1048574  1048574  1048574  \n",
       "1048575  1048575  1048575  1048575  1048575  1048575  1048575  \n",
       "\n",
       "[1048576 rows x 98 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "# but now lets start to add columns one by one\n",
    "# to be fast, pandas adds each as a new block \n",
    "# so we will have 99 blocks (2+97 new ones)\n",
    "for i in range(96):\n",
    "    df_example[f'new_{i}'] = df_example['int64'].to_numpy()\n",
    "    \n",
    "print(df_example._data.nblocks)\n",
    "df_example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.87 ms, sys: 9.1 ms, total: 11 ms\n",
      "Wall time: 11.1 ms\n",
      "Number of blocks in data: 99\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/8f/szs_zmy96hd71pg9d355wt9h0000gp/T/ipykernel_87038/2687686915.py:2: DeprecationWarning: DataFrame._data is deprecated and will be removed in a future version. Use public APIs instead.\n",
      "  print('Number of blocks in data:',df_example._data.nblocks)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 133 ms, sys: 825 ms, total: 957 ms\n",
      "Wall time: 1.38 s\n",
      "CPU times: user 775 μs, sys: 3.15 ms, total: 3.93 ms\n",
      "Wall time: 3.23 ms\n",
      "Number of blocks in data: 3\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/8f/szs_zmy96hd71pg9d355wt9h0000gp/T/ipykernel_87038/2687686915.py:8: DeprecationWarning: DataFrame._data is deprecated and will be removed in a future version. Use public APIs instead.\n",
      "  print('Number of blocks in data:',df_example._data.nblocks)\n"
     ]
    }
   ],
   "source": [
    "%time df_example['dummy_name5'] = df_example['int64'].to_numpy() # copy over some new columns\n",
    "print('Number of blocks in data:',df_example._data.nblocks)\n",
    "\n",
    "# force consolidation \n",
    "%time df_example = df_example.reindex()\n",
    "\n",
    "%time df_example['dummy_name6'] = df_example['int64'].to_numpy() # copy over some new columns\n",
    "print('Number of blocks in data:',df_example._data.nblocks)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1048576 entries, 0 to 1048575\n",
      "Data columns (total 100 columns):\n",
      " #   Column       Non-Null Count    Dtype  \n",
      "---  ------       --------------    -----  \n",
      " 0   int64        1048576 non-null  int64  \n",
      " 1   float64      1048576 non-null  float64\n",
      " 2   new_0        1048576 non-null  int64  \n",
      " 3   new_1        1048576 non-null  int64  \n",
      " 4   new_2        1048576 non-null  int64  \n",
      " 5   new_3        1048576 non-null  int64  \n",
      " 6   new_4        1048576 non-null  int64  \n",
      " 7   new_5        1048576 non-null  int64  \n",
      " 8   new_6        1048576 non-null  int64  \n",
      " 9   new_7        1048576 non-null  int64  \n",
      " 10  new_8        1048576 non-null  int64  \n",
      " 11  new_9        1048576 non-null  int64  \n",
      " 12  new_10       1048576 non-null  int64  \n",
      " 13  new_11       1048576 non-null  int64  \n",
      " 14  new_12       1048576 non-null  int64  \n",
      " 15  new_13       1048576 non-null  int64  \n",
      " 16  new_14       1048576 non-null  int64  \n",
      " 17  new_15       1048576 non-null  int64  \n",
      " 18  new_16       1048576 non-null  int64  \n",
      " 19  new_17       1048576 non-null  int64  \n",
      " 20  new_18       1048576 non-null  int64  \n",
      " 21  new_19       1048576 non-null  int64  \n",
      " 22  new_20       1048576 non-null  int64  \n",
      " 23  new_21       1048576 non-null  int64  \n",
      " 24  new_22       1048576 non-null  int64  \n",
      " 25  new_23       1048576 non-null  int64  \n",
      " 26  new_24       1048576 non-null  int64  \n",
      " 27  new_25       1048576 non-null  int64  \n",
      " 28  new_26       1048576 non-null  int64  \n",
      " 29  new_27       1048576 non-null  int64  \n",
      " 30  new_28       1048576 non-null  int64  \n",
      " 31  new_29       1048576 non-null  int64  \n",
      " 32  new_30       1048576 non-null  int64  \n",
      " 33  new_31       1048576 non-null  int64  \n",
      " 34  new_32       1048576 non-null  int64  \n",
      " 35  new_33       1048576 non-null  int64  \n",
      " 36  new_34       1048576 non-null  int64  \n",
      " 37  new_35       1048576 non-null  int64  \n",
      " 38  new_36       1048576 non-null  int64  \n",
      " 39  new_37       1048576 non-null  int64  \n",
      " 40  new_38       1048576 non-null  int64  \n",
      " 41  new_39       1048576 non-null  int64  \n",
      " 42  new_40       1048576 non-null  int64  \n",
      " 43  new_41       1048576 non-null  int64  \n",
      " 44  new_42       1048576 non-null  int64  \n",
      " 45  new_43       1048576 non-null  int64  \n",
      " 46  new_44       1048576 non-null  int64  \n",
      " 47  new_45       1048576 non-null  int64  \n",
      " 48  new_46       1048576 non-null  int64  \n",
      " 49  new_47       1048576 non-null  int64  \n",
      " 50  new_48       1048576 non-null  int64  \n",
      " 51  new_49       1048576 non-null  int64  \n",
      " 52  new_50       1048576 non-null  int64  \n",
      " 53  new_51       1048576 non-null  int64  \n",
      " 54  new_52       1048576 non-null  int64  \n",
      " 55  new_53       1048576 non-null  int64  \n",
      " 56  new_54       1048576 non-null  int64  \n",
      " 57  new_55       1048576 non-null  int64  \n",
      " 58  new_56       1048576 non-null  int64  \n",
      " 59  new_57       1048576 non-null  int64  \n",
      " 60  new_58       1048576 non-null  int64  \n",
      " 61  new_59       1048576 non-null  int64  \n",
      " 62  new_60       1048576 non-null  int64  \n",
      " 63  new_61       1048576 non-null  int64  \n",
      " 64  new_62       1048576 non-null  int64  \n",
      " 65  new_63       1048576 non-null  int64  \n",
      " 66  new_64       1048576 non-null  int64  \n",
      " 67  new_65       1048576 non-null  int64  \n",
      " 68  new_66       1048576 non-null  int64  \n",
      " 69  new_67       1048576 non-null  int64  \n",
      " 70  new_68       1048576 non-null  int64  \n",
      " 71  new_69       1048576 non-null  int64  \n",
      " 72  new_70       1048576 non-null  int64  \n",
      " 73  new_71       1048576 non-null  int64  \n",
      " 74  new_72       1048576 non-null  int64  \n",
      " 75  new_73       1048576 non-null  int64  \n",
      " 76  new_74       1048576 non-null  int64  \n",
      " 77  new_75       1048576 non-null  int64  \n",
      " 78  new_76       1048576 non-null  int64  \n",
      " 79  new_77       1048576 non-null  int64  \n",
      " 80  new_78       1048576 non-null  int64  \n",
      " 81  new_79       1048576 non-null  int64  \n",
      " 82  new_80       1048576 non-null  int64  \n",
      " 83  new_81       1048576 non-null  int64  \n",
      " 84  new_82       1048576 non-null  int64  \n",
      " 85  new_83       1048576 non-null  int64  \n",
      " 86  new_84       1048576 non-null  int64  \n",
      " 87  new_85       1048576 non-null  int64  \n",
      " 88  new_86       1048576 non-null  int64  \n",
      " 89  new_87       1048576 non-null  int64  \n",
      " 90  new_88       1048576 non-null  int64  \n",
      " 91  new_89       1048576 non-null  int64  \n",
      " 92  new_90       1048576 non-null  int64  \n",
      " 93  new_91       1048576 non-null  int64  \n",
      " 94  new_92       1048576 non-null  int64  \n",
      " 95  new_93       1048576 non-null  int64  \n",
      " 96  new_94       1048576 non-null  int64  \n",
      " 97  new_95       1048576 non-null  int64  \n",
      " 98  dummy_name5  1048576 non-null  int64  \n",
      " 99  dummy_name6  1048576 non-null  int64  \n",
      "dtypes: float64(1), int64(99)\n",
      "memory usage: 800.0 MB\n"
     ]
    }
   ],
   "source": [
    "df_example.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}