Code Efficiency Modules.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "40eac52f",
   "metadata": {},
   "source": [
    "# Code efficiency modules\n",
    "    - time\n",
    "    - timeit\n",
    "    - profiling\n",
    "    - line profiling\n",
    "    - memory profiling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70269ad4",
   "metadata": {},
   "source": [
    "## time module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d95c7c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb12f792",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(20):\n",
    "    starttime = time.time()\n",
    "    myList_Sum = sum([random.randint(1,10)**2 for i in range(10000)])\n",
    "    stoptime = time.time()\n",
    "    timetaken_1 = stoptime-starttime\n",
    "    print('Time to taken to run code, ', timetaken_1)\n",
    "\n",
    "    starttime = time.time()\n",
    "    randList = []\n",
    "    for i in range(10000):\n",
    "        randList.append(random.randint(1,10)**2)\n",
    "    myList_Sum = sum(randList)    \n",
    "    stoptime = time.time()\n",
    "    timetaken_2 = stoptime-starttime\n",
    "    print('Time to taken to run code, ', timetaken_2)\n",
    "\n",
    "    if min(timetaken_2,timetaken_1) == timetaken_1:\n",
    "        print('Comprehension took lesser time')\n",
    "    else:\n",
    "        print('Regular for loop took lesser time')\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d09b8cba",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mymean(iterable):\n",
    "    return sum(iterable)/len(iterable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7248ca18",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = lambda iterable:sum(iterable)/len(iterable)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96811439",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f463dc73",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "starttime = time.time()\n",
    "myList_Average = mymean([random.randint(1,10) for i in range(10000)])\n",
    "stoptime = time.time()\n",
    "timetaken1 = stoptime-starttime\n",
    "print('Time to taken to run code, ', timetaken1)\n",
    "\n",
    "\n",
    "starttime = time.time()\n",
    "myList_Average = x([random.randint(1,10) for i in range(10000)])\n",
    "stoptime = time.time()\n",
    "timetaken2 = stoptime-starttime\n",
    "print('Time to taken to run code, ', timetaken2)\n",
    "\n",
    "if min(timetaken2,timetaken1) == timetaken2:\n",
    "    print('lambda took lesser time')\n",
    "else:\n",
    "    print('Regular function took lesser time')\n",
    "print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a159ebf",
   "metadata": {},
   "source": [
    "## timeit module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fde7b32d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import timeit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b0bbf20",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = np.arange(4).reshape((2,2))\n",
    "a.sum()\n",
    "sum(a)\n",
    "np.sum(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7997afc2",
   "metadata": {},
   "source": [
    "The module function timeit.timeit(stmt, setup, timer, number) accepts four arguments: \n",
    "\n",
    "- stmt\n",
    "\n",
    "- setup (some requirements like imports)\n",
    "\n",
    "- number (number of executions) \n",
    "\n",
    "timeit.timeit() function returns the number of seconds it took to execute the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b092fff5",
   "metadata": {},
   "outputs": [],
   "source": [
    "timeit.timeit(\"np.arange(4).reshape((2,2)).sum()\",setup = \"import numpy as np\",number = 10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47cbdb22",
   "metadata": {},
   "outputs": [],
   "source": [
    "timeit.timeit(\"sum(np.arange(4).reshape((2,2)))\",setup = \"import numpy as np\",number = 10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7520d9ce",
   "metadata": {},
   "outputs": [],
   "source": [
    "timeit.timeit(\"np.sum(np.arange(4).reshape((2,2)))\",setup = \"import numpy as np\",number = 10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f0b7b53",
   "metadata": {},
   "source": [
    "#### Proper profiling of codes can be done using cProfile and profile modules (inbuilt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df2c8a88",
   "metadata": {},
   "source": [
    "## cProfile module"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bdf0398",
   "metadata": {},
   "source": [
    "primarily a timing module\n",
    "\n",
    "cProfile - extension of c, low overhead and fast\n",
    "profile - written in python, adds overhead \n",
    "\n",
    "Both add deterministic profiling - monitoring number of function calls, timeing of each, etc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "6052381f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cProfile\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04d354e0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "048da430",
   "metadata": {},
   "source": [
    "#### profiling single line code - `cProfile.run`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "bf5ec481",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         56061 function calls in 0.045 seconds\n",
      "\n",
      "   Ordered by: standard name\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "        1    0.007    0.007    0.045    0.045 <string>:1(<listcomp>)\n",
      "        1    0.000    0.000    0.045    0.045 <string>:1(<module>)\n",
      "    10000    0.014    0.000    0.028    0.000 random.py:200(randrange)\n",
      "    10000    0.010    0.000    0.038    0.000 random.py:244(randint)\n",
      "    10000    0.009    0.000    0.013    0.000 random.py:250(_randbelow_with_getrandbits)\n",
      "        1    0.000    0.000    0.045    0.045 {built-in method builtins.exec}\n",
      "        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}\n",
      "    10000    0.002    0.000    0.002    0.000 {method 'bit_length' of 'int' objects}\n",
      "        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
      "    16056    0.003    0.000    0.003    0.000 {method 'getrandbits' of '_random.Random' objects}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cProfile.run('sum([random.randint(1,10) for i in range(10000)])')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff59a97d",
   "metadata": {},
   "source": [
    "- run only runs with single line statements\n",
    "- profile constrcutor has enable() disable() methods which will profile a set of lines in a code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a9af846",
   "metadata": {},
   "source": [
    "#### profile constructor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b1045994",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x\n",
    "\n",
    "myprof = cProfile.Profile()\n",
    "\n",
    "def my_map(func = cube, arg_list = list(range(10000))):\n",
    "    myprof.enable()\n",
    "    result = []\n",
    "    for i in arg_list:\n",
    "        result.append(func(i))\n",
    "    sumresult = sum(result)    \n",
    "    myprof.disable()\n",
    "    return sumresult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a7c021c4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2499500025000000"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "my_map()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d8f0d444",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8173c46f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         20002 function calls in 0.003 seconds\n",
      "\n",
      "   Ordered by: standard name\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "    10000    0.002    0.000    0.002    0.000 1289081230.py:1(cube)\n",
      "        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}\n",
      "    10000    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}\n",
      "        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "myprof.print_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da9a90e3",
   "metadata": {},
   "source": [
    "#### save to a profile file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "de9ee2fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "myprof.dump_stats(\"mymap.prof\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9635fc11",
   "metadata": {},
   "source": [
    "#### open a profile file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7308a730",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pstats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a00de11a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "afc42d17",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = pstats.Stats('mymap.prof')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2836301e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tue May  3 18:30:49 2022    mymap.prof\n",
      "\n",
      "         20002 function calls in 0.003 seconds\n",
      "\n",
      "   Random listing order was used\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "    10000    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}\n",
      "        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}\n",
      "        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
      "    10000    0.002    0.000    0.002    0.000 /var/folders/64/86wm1bvj5tdbj8tcrs97120w0000gn/T/ipykernel_9256/1289081230.py:1(cube)\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pstats.Stats at 0x7fa879dcd190>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.print_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeabd9ab",
   "metadata": {},
   "source": [
    "pstats module also helps modify results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "17646a25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tue May  3 18:30:49 2022    mymap.prof\n",
      "\n",
      "         20002 function calls in 0.003 seconds\n",
      "\n",
      "   Ordered by: cumulative time\n",
      "   List reduced from 4 to 3 due to restriction <3>\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "    10000    0.002    0.000    0.002    0.000 /var/folders/64/86wm1bvj5tdbj8tcrs97120w0000gn/T/ipykernel_9256/1289081230.py:1(cube)\n",
      "    10000    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}\n",
      "        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "a.sort_stats(pstats.SortKey.CUMULATIVE).print_stats(3);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52ee0174",
   "metadata": {},
   "source": [
    "sort the results of profiling based on cumulative time and then have taken the first 3 results from it when printing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "1d7eb20f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tue May  3 18:30:49 2022    mymap.prof\n",
      "\n",
      "         20002 function calls in 0.003 seconds\n",
      "\n",
      "   Ordered by: internal time, cumulative time\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "    10000    0.002    0.000    0.002    0.000 /var/folders/64/86wm1bvj5tdbj8tcrs97120w0000gn/T/ipykernel_9256/1289081230.py:1(cube)\n",
      "    10000    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}\n",
      "        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}\n",
      "        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "a.sort_stats(pstats.SortKey.TIME, pstats.SortKey.CUMULATIVE).print_stats();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "815098a2",
   "metadata": {},
   "source": [
    "sort the results based on total time and cumulative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "22e3d2ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "p = pstats.Stats(\"mymap.prof\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "8207a538",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tue May  3 18:30:49 2022    mymap.prof\n",
      "\n",
      "         20002 function calls in 0.003 seconds\n",
      "\n",
      "   Random listing order was used\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "    10000    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}\n",
      "        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}\n",
      "        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
      "    10000    0.002    0.000    0.002    0.000 /var/folders/64/86wm1bvj5tdbj8tcrs97120w0000gn/T/ipykernel_9256/1289081230.py:1(cube)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "p.print_stats();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "36f957d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tue May  3 18:30:49 2022    mymap.prof\n",
      "\n",
      "         20002 function calls in 0.003 seconds\n",
      "\n",
      "   Random listing order was used\n",
      "   List reduced from 4 to 1 due to restriction <'append'>\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "    10000    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "p.print_stats(\"append\"); # only results containing \"append\" method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "72d346e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tue May  3 18:30:49 2022    mymap.prof\n",
      "\n",
      "         20002 function calls in 0.003 seconds\n",
      "\n",
      "   Random listing order was used\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "    10000    0.001    0.000    0.001    0.000 {method 'append' of 'list' objects}\n",
      "        1    0.000    0.000    0.000    0.000 {built-in method builtins.sum}\n",
      "        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
      "    10000    0.002    0.000    0.002    0.000 1289081230.py:1(cube)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "p = pstats.Stats(\"mymap.prof\")\n",
    "\n",
    "p.strip_dirs().print_stats();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "63666184",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Random listing order was used\n",
      "\n",
      "Function                                          was called by...\n",
      "{method 'append' of 'list' objects}               <- \n",
      "{built-in method builtins.sum}                    <- \n",
      "{method 'disable' of '_lsprof.Profiler' objects}  <- \n",
      "1289081230.py:1(cube)                             <- \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "p = pstats.Stats(\"mymap.prof\")\n",
    "\n",
    "p.strip_dirs().print_callers() # which function was called by which other function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "45b7efd6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Random listing order was used\n",
      "\n",
      "Function                                                                               called...\n",
      "{method 'append' of 'list' objects}                                                    -> \n",
      "{built-in method builtins.sum}                                                         -> \n",
      "{method 'disable' of '_lsprof.Profiler' objects}                                       -> \n",
      "/var/folders/64/86wm1bvj5tdbj8tcrs97120w0000gn/T/ipykernel_9256/1289081230.py:1(cube)  -> \n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<pstats.Stats at 0x7fa879dcd5b0>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = pstats.Stats(\"mymap.prof\")\n",
    "\n",
    "p.print_callees()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f949104",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b81a253f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cbbaa7fa",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c9023152",
   "metadata": {},
   "source": [
    "#### recursive function profiling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "f872b22c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def factorial(x):\n",
    "    \"\"\"This is a recursive function\n",
    "    to find the factorial of an integer\"\"\"\n",
    "\n",
    "    if x == 1:\n",
    "        return 1\n",
    "    else:\n",
    "        return (x * factorial(x-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "59102459",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         23 function calls (4 primitive calls) in 0.000 seconds\n",
      "\n",
      "   Ordered by: standard name\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "     20/1    0.000    0.000    0.000    0.000 3861038260.py:1(factorial)\n",
      "        1    0.000    0.000    0.000    0.000 <string>:1(<module>)\n",
      "        1    0.000    0.000    0.000    0.000 {built-in method builtins.exec}\n",
      "        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "cProfile.run(\"factorial(20)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "f85e7b42",
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'int' object is not iterable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/64/86wm1bvj5tdbj8tcrs97120w0000gn/T/ipykernel_9256/3021054072.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmy_map\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcube\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m50000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0msum\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmap\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcube\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m50000\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'int' object is not iterable"
     ]
    }
   ],
   "source": [
    "sum(my_map(cube, list(range(1,50000))))\n",
    "sum(list(map(cube,list(range(1,50000)))))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c399e421",
   "metadata": {},
   "source": [
    "## Profile module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "fb199273",
   "metadata": {},
   "outputs": [],
   "source": [
    "import profile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bc00520",
   "metadata": {},
   "outputs": [],
   "source": [
    "profile.run(\"mymean([random.randint(1,10) for i in range(10000)])\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d04d2509",
   "metadata": {},
   "outputs": [],
   "source": [
    "profile.run(\"mymean([1,2,3,4])\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7be93c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "profile.run(\"mymean([random.randint(1,10) for i in range(10000)])\", filename = 'out1.prof')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "810ce471",
   "metadata": {},
   "outputs": [],
   "source": [
    "profile.run(\"mymean([1,2,3,4])\", filename = 'out2.prof')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4986324",
   "metadata": {},
   "source": [
    "### `profile()` constructor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d6ba7b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def square(x):\n",
    "    \"\"\"Square of x.\"\"\"\n",
    "    return x*x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48b91941",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09b35b7a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a83927e",
   "metadata": {},
   "outputs": [],
   "source": [
    "my_num = [4,3,4,6,8,1]\n",
    "\n",
    "\n",
    "def myfunc(my_list):\n",
    "    squarelist = []\n",
    "    cubelist = []\n",
    "    for i in my_list:\n",
    "        if i %2 == 0:\n",
    "            squarelist.append(square(i))\n",
    "        else:\n",
    "            cubelist.append(cube(i))\n",
    "        \n",
    "    print(squarelist)\n",
    "    print(cubelist)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f2f810d",
   "metadata": {},
   "outputs": [],
   "source": [
    "myprofile = profile.Profile()\n",
    "\n",
    "myprofile.run('myfunc(my_num)')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f0f6a13",
   "metadata": {},
   "outputs": [],
   "source": [
    "myprofile.print_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a223431c",
   "metadata": {},
   "source": [
    "### Calibrated profiling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "2aa961d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "elapsed time without profiling = 0.027554999999999552\n",
      "elapsed time with profiling = 0.10639499999999913\n",
      "'CPU seconds' profiler reported = 0.060390999999881956\n",
      "total # calls = 10001.0\n",
      "mean stopwatch overhead per profile event = 1.6416358364104792e-06\n",
      "elapsed time without profiling = 0.02788100000000071\n",
      "elapsed time with profiling = 0.1069899999999997\n",
      "'CPU seconds' profiler reported = 0.05953600000061954\n",
      "total # calls = 10001.0\n",
      "mean stopwatch overhead per profile event = 1.5825917408568556e-06\n",
      "elapsed time without profiling = 0.02738899999999944\n",
      "elapsed time with profiling = 0.10306899999999786\n",
      "'CPU seconds' profiler reported = 0.05735299999971133\n",
      "total # calls = 10001.0\n",
      "mean stopwatch overhead per profile event = 1.4980501949660977e-06\n",
      "elapsed time without profiling = 0.028126999999997793\n",
      "elapsed time with profiling = 0.10902400000000156\n",
      "'CPU seconds' profiler reported = 0.06072400000075007\n",
      "total # calls = 10001.0\n",
      "mean stopwatch overhead per profile event = 1.6296870313344803e-06\n",
      "elapsed time without profiling = 0.029862000000001387\n",
      "elapsed time with profiling = 0.10810099999999778\n",
      "'CPU seconds' profiler reported = 0.06138099999955671\n",
      "total # calls = 10001.0\n",
      "mean stopwatch overhead per profile event = 1.5757924207356924e-06\n"
     ]
    }
   ],
   "source": [
    "pr = profile.Profile()\n",
    "\n",
    "cal = []\n",
    "for i in range(5):\n",
    "    cal.append(pr.calibrate(10000, verbose=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "7e3b6f49",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.5855514448607212e-06"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bias = sum(cal)/5\n",
    "bias"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4256c71",
   "metadata": {},
   "source": [
    "This bias can be reduced from my profilinf events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "b484e4a7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<profile.Profile at 0x7fa879ebf8b0>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prof = profile.Profile(bias=bias)\n",
    "\n",
    "prof.run(\"my_map()\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "90537a7d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "         5 function calls in 0.017 seconds\n",
      "\n",
      "   Ordered by: standard name\n",
      "\n",
      "   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\n",
      "        1    0.000    0.000    0.016    0.016 1289081230.py:7(my_map)\n",
      "        1    0.016    0.016    0.016    0.016 :0(enable)\n",
      "        1    0.000    0.000    0.016    0.016 :0(exec)\n",
      "        1    0.000    0.000    0.016    0.016 <string>:1(<module>)\n",
      "        1    0.000    0.000    0.017    0.017 profile:0(my_map())\n",
      "        0    0.000             0.000          profile:0(profiler)\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "prof.print_stats()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "5d10bdaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " "
     ]
    }
   ],
   "source": [
    "%prun factorial(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc24e94",
   "metadata": {},
   "source": [
    "# Line profiling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18efefa3",
   "metadata": {},
   "source": [
    "Previous modules only generate time taken information which cannot identify bottle necks in codes\n",
    "\n",
    "`line_profiler` helps profile the code line by line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "93adf142",
   "metadata": {},
   "outputs": [],
   "source": [
    "## lineprof1.py\n",
    "\n",
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x\n",
    "\n",
    "@profile\n",
    "def my_map(func = cube, arg_list = list(range(1,50000))):\n",
    "    result = []\n",
    "    for i in arg_list:\n",
    "        result.append(func(i))\n",
    "    sumresult = sum(result)    \n",
    "    return sumresult\n",
    "\n",
    "my_map()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc811fa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "kernprof -l lineprof1.py\n",
    "\n",
    "python -m line_profiler lineprof1.py.lprof"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bdfe1052",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x\n",
    "\n",
    "def my_map(func = cube, arg_list = list(range(1,50000))):\n",
    "    result = []\n",
    "    for i in arg_list:\n",
    "        result.append(func(i))\n",
    "    sumresult = sum(result)    \n",
    "    return sumresult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04f1fa2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(my_map(cube, list(range(1,50000))))\n",
    "print(sum(list(map(cube,list(range(1,50000))))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "863dba77",
   "metadata": {},
   "outputs": [],
   "source": [
    "# conda install -c anaconda line_profiler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6311f7c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext line_profiler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1aa28fe8",
   "metadata": {},
   "outputs": [],
   "source": [
    "%lprun -f my_map my_map()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d75427d",
   "metadata": {},
   "source": [
    "## `pprofile`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df2cd58c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pprofiler similar to line-profiler\n",
    "\n",
    "deterministic profiling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14997a7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "## mypprofile1.py\n",
    "\n",
    "def add(a, b):\n",
    "    return a+b\n",
    "\n",
    "\n",
    "def get_sum_of_list():\n",
    "    final_list = []\n",
    "    for i in range(100):\n",
    "        out = add(i, i)\n",
    "        final_list.append(out)\n",
    "    return final_list\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    l = get_sum_of_list()\n",
    "    print(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bacec825",
   "metadata": {},
   "outputs": [],
   "source": [
    "pprofile mypprofile1.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27c0bd20",
   "metadata": {},
   "outputs": [],
   "source": [
    "pprofile -o mypprofile1.out mypprofile.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e3f91cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "##mypprofile2.py\n",
    "\n",
    "import random\n",
    "\n",
    "def add(a, b):\n",
    "    return a+b\n",
    "\n",
    "\n",
    "def get_sum_of_list():\n",
    "    final_list = []\n",
    "    for i in range(10):\n",
    "        rand1 = random.randint(1,100)\n",
    "        rand2 = random.randint(1,100)\n",
    "        out = add(rand1, rand2)\n",
    "        final_list.append(out)\n",
    "    return final_list\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    l = get_sum_of_list()\n",
    "    print(l)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "369f4151",
   "metadata": {},
   "outputs": [],
   "source": [
    "pprofile -o mypprofilerand.out mypprofile2.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d87d526",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Excluding modules from being profiled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fb57a98",
   "metadata": {},
   "outputs": [],
   "source": [
    "pprofile --exclude random example_1.py\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab70af19",
   "metadata": {},
   "source": [
    "# Memory profiling"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94227d05",
   "metadata": {},
   "source": [
    "Part of the code that take up lot of memory\n",
    "\n",
    "- Fully written in python\n",
    "- Monitors line by line memory usage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0a2edf5",
   "metadata": {},
   "source": [
    "example1 - memprof_1.py:\n",
    "    \n",
    "    A simple deocrator @profile will give memory usage information\n",
    "    Let us use the my_map function as always"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0f2e511",
   "metadata": {},
   "outputs": [],
   "source": [
    "## memprof_1.py\n",
    "\n",
    "from memory_profiler import profile\n",
    "\n",
    "\n",
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x\n",
    "\n",
    "@profile\n",
    "def my_map(func = cube, arg_list = list(range(1,50000))):\n",
    "    result = []\n",
    "    for i in arg_list:\n",
    "        result.append(func(i))\n",
    "    sumresult = sum(result)    \n",
    "    return sumresult\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    my_map()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71a805f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "python -m memory_profiler my_map.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d4bcdde",
   "metadata": {},
   "outputs": [],
   "source": [
    "## memprof_2.py\n",
    "\n",
    "from memory_profiler import profile\n",
    "\n",
    "\n",
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x\n",
    "\n",
    "@profile\n",
    "def my_map(func = cube, arg_list = list(range(1,50000))):\n",
    "    result = []\n",
    "    myres = []\n",
    "    for i in arg_list:\n",
    "        result.append(func(i))\n",
    "        myres.append(func(i)*2)\n",
    "    sumresult = sum(result)  \n",
    "    del result\n",
    "    del myres\n",
    "    return sumresult\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    my_map()\n",
    "    \n",
    "## deleting will deallocate memory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a77c99c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "## memprof_3.py\n",
    "\n",
    "from memory_profiler import profile\n",
    "\n",
    "\n",
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x\n",
    "\n",
    "@profile(precision = 3)\n",
    "def my_map(func = cube, arg_list = list(range(1,50000))):\n",
    "    result = []\n",
    "    for i in arg_list:\n",
    "        result.append(func(i))\n",
    "    sumresult = sum(result)    \n",
    "    del result\n",
    "    return sumresult\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    my_map()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4344c1c",
   "metadata": {},
   "source": [
    "### `mprof` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1460fdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "mprof is a executable from command line; comes with memory_profiler\n",
    "gives information about memory usage over time\n",
    "to identify times when memory usage peaks \n",
    "data is stored in a .dat file which can be accessed for plotting later\n"
   ]
  },
  {
   "cell_type": "raw",
   "id": "8766da32",
   "metadata": {},
   "source": [
    "mprof run random_number_generator.py\n",
    "\n",
    "mprofile_20201006095840.dat\n",
    "\n",
    "mprof plot (plots he latest .dat file)\n",
    "\n",
    "mprof plot filename"
   ]
  },
  {
   "cell_type": "raw",
   "id": "94debaff",
   "metadata": {},
   "source": [
    "mprof list - To list all .dat files generated by mprof in the folder\n",
    "\n",
    "mprof clean - To clean all .dat files\n",
    "\n",
    "mprof rm - To remove any particular .dat file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1a738af",
   "metadata": {},
   "outputs": [],
   "source": [
    "Streaming an output to a logfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97e02aab",
   "metadata": {},
   "outputs": [],
   "source": [
    "## memprof_4.py\n",
    "\n",
    "from memory_profiler import profile\n",
    "\n",
    "fp = open(\"report.log\", \"w+\")\n",
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x\n",
    "\n",
    "@profile(stream = fp)\n",
    "def my_map(func = cube, arg_list = list(range(1,50000))):\n",
    "    result = []\n",
    "    for i in arg_list:\n",
    "        result.append(func(i))\n",
    "    sumresult = sum(result)    \n",
    "    return sumresult\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    my_map()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d35fb45e",
   "metadata": {},
   "outputs": [],
   "source": [
    "python -m memory_profiler memprof_4.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab421121",
   "metadata": {},
   "outputs": [],
   "source": [
    "# conda install -c anaconda memory_profiler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13c71c9f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import memory_profiler\n",
    "import psutil\n",
    "from memprof_1 import my_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dd2f5ee4",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext memory_profiler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7ee70b1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cube(x):\n",
    "    \"\"\"Cube of x.\"\"\"\n",
    "    return x * x * x\n",
    "\n",
    "def my_map(func = cube, arg_list = list(range(1,50000))):\n",
    "    result = []\n",
    "    for i in arg_list:\n",
    "        result.append(func(i))\n",
    "    sumresult = sum(result)    \n",
    "    return sumresult"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "756ab367",
   "metadata": {},
   "outputs": [],
   "source": [
    "from memprof_1 import my_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a5fcf704",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filename: /Users/siddhart/Documents/NMIMS/NMIMS-April2022/Advanced Python/memprof_1.py\n",
      "\n",
      "Line #    Mem usage    Increment  Occurences   Line Contents\n",
      "============================================================\n",
      "     8     55.4 MiB     55.4 MiB           1   @profile\n",
      "     9                                         def my_map(func = cube, arg_list = list(range(1,50000))):\n",
      "    10     55.4 MiB      0.0 MiB           1       result = []\n",
      "    11     55.7 MiB      0.0 MiB       50000       for i in arg_list:\n",
      "    12     55.7 MiB      0.2 MiB       49999           result.append(func(i))\n",
      "    13     55.7 MiB      0.0 MiB           1       sumresult = sum(result)    \n",
      "    14     55.7 MiB      0.0 MiB           1       return sumresult\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%%mprun -f my_map \n",
    "\n",
    "my_map()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "434231f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Filename: /Users/siddhart/Documents/NMIMS/NMIMS-April2022/Advanced Python/memprof_1.py\n",
      "\n",
      "Line #    Mem usage    Increment  Occurences   Line Contents\n",
      "============================================================\n",
      "     8     55.7 MiB     55.7 MiB           1   @profile\n",
      "     9                                         def my_map(func = cube, arg_list = list(range(1,50000))):\n",
      "    10     55.7 MiB      0.0 MiB           1       result = []\n",
      "    11     55.7 MiB      0.0 MiB       50000       for i in arg_list:\n",
      "    12     55.7 MiB      0.0 MiB       49999           result.append(func(i))\n",
      "    13     55.7 MiB      0.0 MiB           1       sumresult = sum(result)    \n",
      "    14     55.7 MiB      0.0 MiB           1       return sumresult\n",
      "\n",
      "\n",
      "peak memory: 55.71 MiB, increment: 0.02 MiB\n"
     ]
    }
   ],
   "source": [
    "%memit my_map()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f057ea9c",
   "metadata": {},
   "source": [
    "## Memory management in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f52e05ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "Python variables are references to objects\n",
    "Variables do not have dat types attached to them\n",
    "id() gives memory location\n",
    "\n",
    "Every Python ojbects maintains 3 things internally.\n",
    "\n",
    "Type - Type of object\n",
    "Value - Value of Object\n",
    "RefCount - Reference Count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ab7a26f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4528999040 4528999040 4528999040\n"
     ]
    }
   ],
   "source": [
    "x = 10\n",
    "y = x\n",
    "t = 10\n",
    "print(id(x), id(y), id(t)) # 10 has three reference counts as three variables point to that object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "690ad155",
   "metadata": {},
   "outputs": [],
   "source": [
    "z = 'hello'\n",
    "l = 'hello'\n",
    "k = l\n",
    "i = 'Hello World'\n",
    "j = 'Hello World'\n",
    "print(id(z), id(l), id(k), id(i), id(j)) # objects with space are created as two separate objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7449fec3",
   "metadata": {},
   "outputs": [],
   "source": [
    "lst = ['hello',10, 'Hello World']\n",
    "print(id(lst[0]), id(lst[1]), id(lst[2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10df6bd9",
   "metadata": {},
   "source": [
    "Decreasing reference count of an object"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fda8cba8",
   "metadata": {},
   "source": [
    "There are 3 ways to decrease the reference count of an object:\n",
    "\n",
    "- Using del statement\n",
    "- Assigning None to a variable name.\n",
    "- Pointing reference variable to some other object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0d438a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "del x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04a8d34e",
   "metadata": {},
   "outputs": [],
   "source": [
    "y, t, id(y), id(t) # object still accessible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "dfc0ff06",
   "metadata": {},
   "outputs": [],
   "source": [
    "y = None\n",
    "y # again object accessible; but reference to obj removed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5011a55f",
   "metadata": {},
   "outputs": [],
   "source": [
    "t, lst[1], id(t), id(lst[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4bd1651",
   "metadata": {},
   "source": [
    "only reference count for object 10 is reduced; similar to def scope"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c331e03",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test():\n",
    "    title = 'Hello'\n",
    "    print(title)\n",
    "\n",
    "test()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "149ecbb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "Garbage collection:\n",
    "    Reference counting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e54bcb4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "Reference counting:\n",
    "    counts references to an object and deletes it when final reference count becomes empty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bfcab0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 40      # Create object <40>\n",
    "b = a       # Increase ref. count  of <40>\n",
    "c = [b]     # Increase ref. count  of <40>\n",
    "del a       # Decrease ref. count  of <40>\n",
    "b = 100     # Decrease ref. count  of <40>\n",
    "c[0] = -1   # Decrease ref. count  of <40>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fe3d8ba",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "83a1296d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b90d5f79",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b4c1175",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cb3ab765",
   "metadata": {},
   "source": [
    "### guppy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "2f3519d9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Heap Status At Starting : \n",
      "Heap Size :  42611547  bytes\n",
      "\n",
      "Partition of a set of 330443 objects. Total size = 42611859 bytes.\n",
      " Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)\n",
      "     0 103969  31 12795163  30  12795163  30 str\n",
      "     1  75730  23  5450744  13  18245907  43 tuple\n",
      "     2  21982   7  3904770   9  22150677  52 types.CodeType\n",
      "     3  43615  13  3320871   8  25471548  60 bytes\n",
      "     4   2844   1  2821400   7  28292948  66 type\n",
      "     5  19894   6  2705584   6  30998532  73 function\n",
      "     6   7467   2  2583096   6  33581628  79 dict (no owner)\n",
      "     7   1020   0  1524440   4  35106068  82 dict of module\n",
      "     8   2844   1  1405712   3  36511780  86 dict of type\n",
      "     9    748   0   664864   2  37176644  87 set\n",
      "<1007 more rows. Type e.g. '_.more' to view.>\n",
      "\n",
      "Heap Status After Setting Reference Point : \n",
      "Heap Size :  3373  bytes\n",
      "\n",
      "Partition of a set of 32 objects. Total size = 3373 bytes.\n",
      " Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)\n",
      "     0      3   9     1360  40      1360  40 types.FrameType\n",
      "     1      4  12      256   8      1616  48 types.MethodType\n",
      "     2      5  16      248   7      1864  55 tuple\n",
      "     3      1   3      176   5      2040  60 types.CodeType\n",
      "     4      2   6      160   5      2200  65 functools.partial\n",
      "     5      1   3      136   4      2336  69 _io.StringIO\n",
      "     6      1   3      136   4      2472  73 function\n",
      "     7      2   6      128   4      2600  77 builtins.Context\n",
      "     8      2   6      128   4      2728  81 dict (no owner)\n",
      "     9      2   6      120   4      2848  84 list\n",
      "<8 more rows. Type e.g. '_.more' to view.>\n",
      "\n",
      "Heap Status After Creating Few Objects : \n",
      "Heap Size :  41297  bytes\n",
      "\n",
      "Partition of a set of 777 objects. Total size = 41297 bytes.\n",
      " Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)\n",
      "     0    745  96    20860  51     20860  51 int\n",
      "     1      3   0     9136  22     29996  73 list\n",
      "     2      1   0     8104  20     38100  92 numpy.ndarray\n",
      "     3      3   0     1360   3     39460  96 types.FrameType\n",
      "     4      4   1      256   1     39716  96 types.MethodType\n",
      "     5      5   1      248   1     39964  97 tuple\n",
      "     6      1   0      176   0     40140  97 types.CodeType\n",
      "     7      2   0      160   0     40300  98 functools.partial\n",
      "     8      1   0      136   0     40436  98 _io.StringIO\n",
      "     9      1   0      136   0     40572  98 function\n",
      "<9 more rows. Type e.g. '_.more' to view.>\n",
      "\n",
      "Memory Usage After Creation Of Objects :  37924  bytes\n"
     ]
    }
   ],
   "source": [
    "from guppy import hpy\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "heap = hpy()\n",
    "\n",
    "print(\"Heap Status At Starting : \")\n",
    "heap_status1 = heap.heap()\n",
    "print(\"Heap Size : \", heap_status1.size, \" bytes\\n\")\n",
    "print(heap_status1)\n",
    "\n",
    "\n",
    "heap.setref()\n",
    "\n",
    "print(\"\\nHeap Status After Setting Reference Point : \")\n",
    "heap_status2 = heap.heap()\n",
    "print(\"Heap Size : \", heap_status2.size, \" bytes\\n\")\n",
    "print(heap_status2)\n",
    "\n",
    "a = [i for i in range(1000)]\n",
    "b = \"A\"\n",
    "c = np.random.randint(1,100, (1000,))\n",
    "\n",
    "print(\"\\nHeap Status After Creating Few Objects : \")\n",
    "heap_status3 = heap.heap()\n",
    "print(\"Heap Size : \", heap_status3.size, \" bytes\\n\")\n",
    "print(heap_status3)\n",
    "\n",
    "print(\"\\nMemory Usage After Creation Of Objects : \", heap_status3.size - heap_status2.size, \" bytes\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ee12c17",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}