02-generators-revisited.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# The basics of generators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A [``Generator``][generator] is a type of iterator which can be constructed natively inside a Python ``def`` function. A ``def`` function is a generator function if it contains any ``yield`` statements. And the immediate return value of calling such a function is a generator iterator, or just a generator.\n",
    "\n",
    "A generator doesn't execute all at once like a regular function does. In fact, calling a generator function doesn't execute any of that function's code immediately, whereas calling a regular function executes all of its code immediately until it returns. Whenever a generator is run, it runs until the end of the function OR until the next ``yield`` statement. At that point, the value in the ``yield`` expression is returned to the caller, and the execution of the generator is suspended until it is next called.\n",
    "\n",
    "While execution is suspended, the full local state of the execution of the generator function closure is stored in memory. This allows it to be resumed later, as if it had never paused.\n",
    "\n",
    "A generator is most commonly used as a normal iterator. This means that it can be automatically iterated over with a for-loop, or manually iterated over with ``next()``. It can also be passed to any function that expects an iterator or an iterable. ``next()`` returns the value of the first ``yield`` statement it encounters, or raises ``StopIteration`` if the generator function completes.\n",
    "\n",
    "[generator]: <https://docs.python.org/3.6/reference/expressions.html#generator-iterator-methods> \"Generator iterator methods\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def range_generator_function(stop):\n",
    "    \"\"\"Naive implementation of builtins.range generator.\"\"\"\n",
    "    # This function runs immediately, since it has no `yield` statements.\n",
    "    # It is a normal function, which happens to return a generator iterator.\n",
    "    print(\"Running line 1\")\n",
    "    if not isinstance(stop, int):\n",
    "        raise TypeError('stop must be an int')\n",
    "    if stop < 0:\n",
    "        raise ValueError('stop must be >= 0')\n",
    "    print(\"Running line 2\")\n",
    "    range_generator = _range_generator_function(stop=stop)\n",
    "    print(\"Running line 3\")\n",
    "    return range_generator\n",
    "\n",
    "def _range_generator_function(stop):\n",
    "    # This function does not run immediately, since it has `yield` statements.\n",
    "    # It is a generator function, and returns a generator iterator.\n",
    "    index = 0\n",
    "    print(\"Running line 4\")\n",
    "    while index < stop:\n",
    "        print(\"Running line 5 with index\", index)\n",
    "        yield index\n",
    "        print(\"Running line 6 with index\", index)\n",
    "        index += 1\n",
    "    print(\"Running line 7 with index\", index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running line 1\n",
      "Running line 2\n",
      "Running line 3\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<generator object _range_generator_function at 0x106322e08>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "range_generator = range_generator_function(2)  # Executes all prints in `range_generator_function()`,\n",
    "range_generator                                # but none in `_range_generator_function()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import collections\n",
    "isinstance(range_generator, collections.Iterable), isinstance(range_generator, collections.Iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isinstance(range_generator, collections.Generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running line 4\n",
      "Running line 5 with index 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(range_generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running line 6 with index 0\n",
      "Running line 5 with index 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(range_generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running line 6 with index 1\n",
      "Running line 7 with index 2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-7-94c725fdbb96>\", line 4, in <module>\n",
      "    next(range_generator)\n",
      "StopIteration\n"
     ]
    }
   ],
   "source": [
    "import traceback\n",
    "\n",
    "try:\n",
    "    next(range_generator)\n",
    "except StopIteration:\n",
    "    traceback.print_exc()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "next(range_generator, 2)  # Generator is exhausted, nothing more will get printed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running line 1\n",
      "Running line 2\n",
      "Running line 3\n",
      "Running line 4\n",
      "Running line 5 with index 0\n",
      "yielded 0\n",
      "Running line 6 with index 0\n",
      "Running line 5 with index 1\n",
      "yielded 1\n",
      "Running line 6 with index 1\n",
      "Running line 5 with index 2\n",
      "yielded 2\n",
      "Running line 6 with index 2\n",
      "Running line 5 with index 3\n",
      "yielded 3\n",
      "Running line 6 with index 3\n",
      "Running line 7 with index 4\n"
     ]
    }
   ],
   "source": [
    "range_generator = range_generator_function(4)\n",
    "for item in range_generator:\n",
    "    print('yielded', item)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced uses of generators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above covers the majority of use cases of generators. It is probably the case that most Python programmers have used the above, or at least encountered it while reading other Python code.\n",
    "\n",
    "However, generators are more than just a special syntax for iterators. There is more to the generator protocol, though it tends to be less widely used and known."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The full generator protocol: send, throw, and close"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generators implement three extra methods besides the two (``__iter__()`` and ``__next__()``) that it needs to be an iterator. These are the ``send()``, ``throw()``, and ``close()`` methods. Together, they allow for even more control over how a generator executes its body when it resumes execution.\n",
    "\n",
    "Generators haven't always had this functionality. It was added in Python 2.5, via [PEP 342][]. \n",
    "\n",
    "[PEP 342]: <https://www.python.org/dev/peps/pep-0342/> \"PEP 342 -- Coroutines via Enhanced Generators\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generator.send"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``send()`` is a slight generalization of ``__next__()``. It accepts a single value, and that value becomes the result of the ``yield`` expression as the generator resumes. The function returns the next value to be yielded, or raises ``StopIteration`` if the generator completes or returns. ``__next__()`` is actually equivalent to ``send(None)``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "print this\n",
      "yielded 1\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "def generator_function():\n",
    "    print((yield 0))\n",
    "    print((yield 1))\n",
    "    \n",
    "generator = generator_function()\n",
    "print(next(generator))  # Advance generator to first `yield` statement.\n",
    "item = generator.send('print this')\n",
    "print('yielded', item)\n",
    "try:\n",
    "    next(generator)  # Same as `generator.send(None)`\n",
    "except StopIteration:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generator.throw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``throw()`` accepts an exception, and raises it inside the generator, at the ``yield`` where the generator is currently paused. If the generator manages to ``yield`` another value, that will get returned from the function. Otherwise, any exceptions raised out of the generator will be propagated out (the same is true for `send()` and `__next__()`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Caught exception ExpectedError()\n",
      "yielded 1\n",
      "Did not catch exception KeyError('key',)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-11-2c9c9b5b897b>\", line 22, in <module>\n",
      "    generator.throw(KeyError('key'))\n",
      "  File \"<ipython-input-11-2c9c9b5b897b>\", line 8, in generator_function\n",
      "    yield i\n",
      "KeyError: 'key'\n"
     ]
    }
   ],
   "source": [
    "import traceback\n",
    "\n",
    "class ExpectedError(Exception): pass\n",
    "\n",
    "def generator_function():\n",
    "    for i in range(2):\n",
    "        try:\n",
    "            yield i\n",
    "        except ExpectedError as exc:\n",
    "            print('Caught exception', repr(exc))\n",
    "            continue\n",
    "        except Exception as exc:\n",
    "            print('Did not catch exception', repr(exc))\n",
    "            raise\n",
    "    return i\n",
    "        \n",
    "generator = generator_function()\n",
    "next(generator)\n",
    "item = generator.throw(ExpectedError)\n",
    "print('yielded', item)\n",
    "try:\n",
    "    generator.throw(KeyError('key'))\n",
    "except KeyError:\n",
    "    traceback.print_exc()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Caught exception ExpectedError()\n",
      "yielded 1\n",
      "Caught exception ExpectedError()\n",
      "StopIteration(1,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-12-edf97ebf6adb>\", line 6, in <module>\n",
      "    generator.throw(ExpectedError)\n",
      "StopIteration: 1\n"
     ]
    }
   ],
   "source": [
    "generator = generator_function()\n",
    "next(generator)\n",
    "item = generator.throw(ExpectedError)\n",
    "print('yielded', item)\n",
    "try:\n",
    "    generator.throw(ExpectedError)\n",
    "except StopIteration as exc:\n",
    "    traceback.print_exc()\n",
    "    print(repr(exc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generator.close"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "``close()`` instructs the generator to stop yielding elements and exit. It is similar, though not identical, to ``throw(GeneratorExit)``.\n",
    "\n",
    "The ``GeneratorExit`` class is a subclass of ``BaseException``, but is not a subclass of ``Exception``. This makes it less likely that a generator function will catch and ignore it by mistake.\n",
    "\n",
    "``close()`` normally returns ``None``, if the generator function returns without any errors, or if the ``GeneratorExit`` exception is propagated out of the generator function. If the generator function raises a new exception while handling ``GeneratorExit``, then that exception is raised by ``close()``. It is illegal for the generator to yield a new value while handling ``close()``, so doing so causes ``RuntimeError`` to be raised instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-13-05ae44296ab2>\", line 3, in generator_function\n",
      "    yield\n",
      "GeneratorExit\n"
     ]
    }
   ],
   "source": [
    "def generator_function():\n",
    "    try:\n",
    "        yield\n",
    "    except:\n",
    "        traceback.print_exc()\n",
    "        raise\n",
    "    print('About to yield 1')\n",
    "    yield 1\n",
    "    \n",
    "generator = generator_function()\n",
    "next(generator)\n",
    "generator.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-14-ddf0738c9bf7>\", line 3, in generator_function\n",
      "    yield 0\n",
      "GeneratorExit\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-14-ddf0738c9bf7>\", line 10, in <module>\n",
      "    generator.close()\n",
      "  File \"<ipython-input-14-ddf0738c9bf7>\", line 5, in generator_function\n",
      "    raise KeyError('key')\n",
      "KeyError: 'key'\n"
     ]
    }
   ],
   "source": [
    "def generator_function():\n",
    "    try:\n",
    "        yield 0\n",
    "    except:\n",
    "        raise KeyError('key')\n",
    "\n",
    "generator = generator_function()\n",
    "next(generator)\n",
    "try:\n",
    "    generator.close()\n",
    "except KeyError:\n",
    "    traceback.print_exc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generators as suspendable/resumable program state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When executing a normal function, it begins execution as soon as it is called, continues (without giving up control) until it completes (by returning or throwing), and cleans up its call stack when exiting.\n",
    "\n",
    "But with a generator, it gives up control every time it hits a `yield`, even if the generator isn't done executing yet. When it does so, then generator becomes suspended. So that its execution can be continued later, its `locals()` are saved on the generator object. Later, it can be resumed (by calling `__next__()`, `send()`, `throw()`, or `close()`), and its `locals()` will be restored before execution resumes from the `yield` point.\n",
    "\n",
    "This has been implicitly assumed in previous examples. For example, in the section [The basics of generators](#The-basics-of-generators), the generator function `_range_generator_function()` relies on this mechanism in order to remember the values of `stop` and `index` between steps. There will be a more advanced example of this in the section [Generators as coroutines](#Generators-as-coroutines).\n",
    "\n",
    "There are other clever ways in which this can be utilized. One great usage of generators is in the implementation of `@contextlib.contextmanager`. This decorator combines the power of generators with the power of the context manager protocol, in order to produce something quite powerful."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `@contextlib.contextmanager`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Code sample from <https://github.com/python/cpython/blob/v3.6.1/Lib/contextlib.py>  \n",
    "Copyright (c) 2001-2017 Python Software Foundation.  \n",
    "All Rights Reserved.  \n",
    "License: Python license, <https://www.python.org/3.6/license.html>  \n",
    "Some modifications made so as to only highlight the interesting parts.\n",
    "\n",
    "```python\n",
    "def contextmanager(func):\n",
    "    \"\"\"@contextmanager decorator.\n",
    "\n",
    "    Typical usage:\n",
    "\n",
    "        @contextmanager\n",
    "        def some_generator(<arguments>):\n",
    "            <setup>\n",
    "            try:\n",
    "                yield <value>\n",
    "            finally:\n",
    "                <cleanup>\n",
    "\n",
    "    This makes this:\n",
    "\n",
    "        with some_generator(<arguments>) as <variable>:\n",
    "            <body>\n",
    "\n",
    "    equivalent to this:\n",
    "\n",
    "        <setup>\n",
    "        try:\n",
    "            <variable> = <value>\n",
    "            <body>\n",
    "        finally:\n",
    "            <cleanup>\n",
    "\n",
    "    \"\"\"\n",
    "    @functools.wraps(func)\n",
    "    def helper(*args, **kwds):\n",
    "        return _GeneratorContextManager(func, args, kwds)\n",
    "    return helper\n",
    "    \n",
    "class _GeneratorContextManager(ContextDecorator, AbstractContextManager):\n",
    "\n",
    "    def __init__(self, func, args, kwds):\n",
    "        self.gen = func(*args, **kwds)\n",
    "\n",
    "    def __enter__(self):\n",
    "        try:\n",
    "            return next(self.gen)\n",
    "        except StopIteration:\n",
    "            raise RuntimeError(\"generator didn't yield\") from None\n",
    "\n",
    "    def __exit__(self, type, value, traceback):\n",
    "        if type is None:\n",
    "            try:\n",
    "                next(self.gen)\n",
    "            except StopIteration:\n",
    "                return\n",
    "            else:\n",
    "                raise RuntimeError(\"generator didn't stop\")\n",
    "        else:\n",
    "            try:\n",
    "                self.gen.throw(type, value, traceback)\n",
    "                raise RuntimeError(\"generator didn't stop after throw()\")\n",
    "            except StopIteration as exc:\n",
    "                return exc is not value\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `@contextlib.contextmanager` decorator makes it really easy for any developer to create their own modular and reusable setup/cleanup logic, or enter/exit logic, without needing to create a new class with `__enter__()` and `__exit__()` methods. `@contextlib.contextmanager` handles that, with the help of the generator that the developer defines.\n",
    "\n",
    "When `__enter__()` is called upon entering the block, `__next__()` is called in order to perform the setup logic, which is everything up until the `yield`. When the `yield` is encountered, the generator suspends its execution and saves its state, the `__enter__()` call returns, and the Python interpretter begins executing the block.\n",
    "\n",
    "When `__exit__()` is called upon exiting the block, either `__next__()` or `throw()` will be called in order to perform the cleanup logic. Upon resuming the generator, its internal state is restored, and the cleanup can proceed as if it were a normal function.\n",
    "\n",
    "When the cleanup logic is placed inside a `finally` block inside the generator function, we get an awesome combination of Python features (generators, context managers, and `finally` blocks) which guarantees that your cleanup code will always be executed whenever the block exits (assuming no bugs in `@contextlib.contextmanager`).\n",
    "\n",
    "This shows how flexible the yielding execution model is. When a generator yields, the caller doesn't have to immediately resume the generator. It can go off and do something else, and then resume it at any point in the future. The really neat thing here, is that the `yield` is essentially replaced with the execution of an arbitrary block of Python code. In future chapters, we'll look at other clever uses of suspending execution which can produce very powerful models of program execution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generators as coroutines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the full generator protocol, a generator is equivalent to a semicoroutine. It is also possible to build a custom dispatcher/trampoline, in order to implement a coroutine system. Reference: <https://en.wikipedia.org/wiki/Coroutine#Comparison_with_generators>. This is used in various concurrency libraries, and is demonstrated in David Beazley's PyCon 2015 talk [\"Python Concurrency From the Ground Up: LIVE!\"][Python Concurrency Talk]. See also the examples given in [PEP 342][].\n",
    "\n",
    "Here's a silly example of a semicoroutine in action.\n",
    "\n",
    "[Python Concurrency Talk]: <https://www.youtube.com/watch?v=MCs5OvhV9S4> \"David Beazley: Python Concurrency From the Ground Up: LIVE!\"\n",
    "[PEP 342]: <https://www.python.org/dev/peps/pep-0342/> \"PEP 342 -- Coroutines via Enhanced Generators\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "At start of adder loop, current total is 0\n",
      "Sending () into adder\n",
      "Adder received ()\n",
      "At start of adder loop, current total is 0\n",
      "Current total in adder is 0\n",
      "\n",
      "Sending (10,) into adder\n",
      "Adder received (10,)\n",
      "At start of adder loop, current total is 10\n",
      "Current total in adder is 10\n",
      "\n",
      "Sending (1, 2, 3) into adder\n",
      "Adder received (1, 2, 3)\n",
      "At start of adder loop, current total is 16\n",
      "Current total in adder is 16\n",
      "\n",
      "Sending (range(0, 8),) into adder\n",
      "Adder received range(0, 8)\n",
      "At start of adder loop, current total is 44\n",
      "Current total in adder is 44\n",
      "\n",
      "Sending StopAdder into adder\n",
      "Adder received exception StopAdder() and is returning with final total 44\n",
      "Final total from adder is 44\n"
     ]
    }
   ],
   "source": [
    "import collections\n",
    "\n",
    "class StopAdder(Exception): pass\n",
    "\n",
    "def adder_function():\n",
    "    total = 0\n",
    "    while True:\n",
    "        print('At start of adder loop, current total is', total)\n",
    "        try:\n",
    "            integers = (yield total)\n",
    "        except (Exception, GeneratorExit) as exc:\n",
    "            print('Adder received exception', repr(exc), 'and is returning with final total', total)\n",
    "            return total\n",
    "        if not isinstance(integers, (list, tuple)):\n",
    "            integers = [integers]\n",
    "        if integers and isinstance(integers[0], collections.Iterable):\n",
    "            integers = integers[0]\n",
    "        print('Adder received', integers)\n",
    "        total += sum(integers)\n",
    "\n",
    "def send_values_into_adder(adder, *integers):\n",
    "    print('Sending', integers, 'into adder')\n",
    "    current_total = adder.send(integers)\n",
    "    print('Current total in adder is', current_total)\n",
    "    return current_total\n",
    "\n",
    "adder = adder_function()\n",
    "next(adder)\n",
    "\n",
    "send_values_into_adder(adder)\n",
    "print()\n",
    "send_values_into_adder(adder, 10)\n",
    "print()\n",
    "send_values_into_adder(adder, 1, 2, 3)\n",
    "print()\n",
    "send_values_into_adder(adder, range(8))\n",
    "print()\n",
    "print('Sending StopAdder into adder')\n",
    "try:\n",
    "    adder.throw(StopAdder)\n",
    "except StopIteration as exc:\n",
    "    print('Final total from adder is', exc.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generators that return values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following example, the generator will yield once, and then raise ``StopIteration`` when it hits the ``return`` statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def generator_function():\n",
    "    yield 0\n",
    "    return\n",
    "\n",
    "generator = generator_function()\n",
    "next(generator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-17-ef31004ccce8>\", line 4, in <module>\n",
      "    next(generator)\n",
      "StopIteration\n"
     ]
    }
   ],
   "source": [
    "import traceback\n",
    "\n",
    "try:\n",
    "    next(generator)\n",
    "except StopIteration:\n",
    "    traceback.print_exc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It might be thought that generators will only pass value via ``yield``, and will always have empty ``return`` statements. This is true up through Python 3.2, but that changed in Python 3.3. Now, just like in normal functions, a generator function can return arbitrary values with ``return`` statements (but note that this is illegal in earlier versions of Python, and will cause a ``SyntaxError``)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "StopIteration('return_value',)\n",
      "'return_value'\n",
      "StopIteration()\n",
      "None\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-18-a206ca87066f>\", line 11, in <module>\n",
      "    next(generator)\n",
      "StopIteration: return_value\n",
      "Traceback (most recent call last):\n",
      "  File \"<ipython-input-18-a206ca87066f>\", line 18, in <module>\n",
      "    next(generator)\n",
      "StopIteration\n"
     ]
    }
   ],
   "source": [
    "import traceback\n",
    "\n",
    "def generator_function_that_returns_a_value():\n",
    "    yield 0\n",
    "    return 'return_value'\n",
    "\n",
    "generator = generator_function_that_returns_a_value()\n",
    "next(generator)\n",
    "\n",
    "try:\n",
    "    next(generator)\n",
    "except StopIteration as exc:\n",
    "    traceback.print_exc()\n",
    "    print(repr(exc))\n",
    "    print(repr(exc.value))\n",
    "    \n",
    "try:\n",
    "    next(generator)\n",
    "except StopIteration as exc:\n",
    "    traceback.print_exc()  # Subsequent calls to `next()` do not use the return value\n",
    "    print(repr(exc))       # when raising `StopIteration`.\n",
    "    print(repr(exc.value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the Python interpretter converted the ``return 'return_value'`` into a ``raise StopIteration('return_value')``. Also notice that the exception object has a ``.value`` attribute, which holds this value.\n",
    "\n",
    "This behavior is hidden when using for-loops or other functionality that catches and ignores ``StopIteration``."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yielded 0\n"
     ]
    }
   ],
   "source": [
    "for item in generator_function_that_returns_a_value():\n",
    "    print('yielded', item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(generator_function_that_returns_a_value())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom generator classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generators aren't required to be implemented using generator functions. If you really wanted to, you could also implement one yourself. Just make sure to define the ``send()`` and ``throw()`` methods correctly, and to correctly define ``__next__()``, ``__iter__()``, and ``close()`` in terms of those.\n",
    "\n",
    "Starting in Python 3.5, the ``collections`` module defines an abstract [``Generator``][Generator class] baseclass. When subclassing this, you only need to define ``send()`` and ``throw()``. The other required methods are already implemented for you correctly.\n",
    "\n",
    "[Generator class]: <https://docs.python.org/3.5/library/collections.abc.html#collections.abc.Generator> \"Generator abstract baseclass\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Adder: total=0 stopped=False>\n",
      "\n",
      "At start of <bound method Adder.send of <Adder: total=0 stopped=False>>, current total is 0\n",
      "<bound method Adder.send of <Adder: total=0 stopped=False>> received []\n",
      "At end of <bound method Adder.send of <Adder: total=0 stopped=False>>, returning current total 0\n",
      "\n",
      "At start of <bound method Adder.send of <Adder: total=0 stopped=False>>, current total is 0\n",
      "<bound method Adder.send of <Adder: total=0 stopped=False>> received [10]\n",
      "At end of <bound method Adder.send of <Adder: total=10 stopped=False>>, returning current total 10\n",
      "\n",
      "At start of <bound method Adder.send of <Adder: total=10 stopped=False>>, current total is 10\n",
      "<bound method Adder.send of <Adder: total=10 stopped=False>> received [1, 2, 3]\n",
      "At end of <bound method Adder.send of <Adder: total=16 stopped=False>>, returning current total 16\n",
      "\n",
      "At start of <bound method Adder.send of <Adder: total=16 stopped=False>>, current total is 16\n",
      "<bound method Adder.send of <Adder: total=16 stopped=False>> received range(0, 8)\n",
      "At end of <bound method Adder.send of <Adder: total=44 stopped=False>>, returning current total 44\n",
      "\n",
      "Sending StopAdder into adder\n",
      "At start of <bound method Adder.throw of <Adder: total=44 stopped=False>>, current total is 44\n",
      "<bound method Adder.throw of <Adder: total=44 stopped=True>> received exception (<class '__main__.StopAdder'>, None, None) and is returning with final total 44\n",
      "Final total from adder is 44\n"
     ]
    }
   ],
   "source": [
    "import collections\n",
    "\n",
    "class Adder(collections.Generator):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.total = 0\n",
    "        self.stopped = False\n",
    "        \n",
    "    def __repr__(self):\n",
    "        return f\"<{self.__class__.__name__}: total={self.total!r} stopped={self.stopped!r}>\"\n",
    "    \n",
    "    def send(self, integers):\n",
    "        if self.stopped:\n",
    "            raise StopIteration\n",
    "        print(f\"At start of {self.send}, current total is\", self.total)\n",
    "        if not isinstance(integers, (list, tuple)):\n",
    "            integers = [integers]\n",
    "        if integers and isinstance(integers[0], collections.Iterable):\n",
    "            integers = integers[0]\n",
    "        print(f\"{self.send} received\", integers)\n",
    "        self.total += sum(integers)\n",
    "        print(f\"At end of {self.send}, returning current total\", self.total)\n",
    "        return self.total\n",
    "    \n",
    "    def throw(self, exc_type, exc_value=None, exc_traceback=None):\n",
    "        if self.stopped:\n",
    "            raise StopIteration\n",
    "        exc_info = (exc_type, exc_value, exc_traceback)\n",
    "        print(f\"At start of {self.throw}, current total is\", self.total)\n",
    "        self.stopped = True\n",
    "        print(f\"{self.throw} received exception\", exc_info, \"and is returning with final total\", self.total)\n",
    "        raise StopIteration(self.total)\n",
    "        \n",
    "\n",
    "def send_values_into_adder(adder, *integers):\n",
    "    print('Sending', integers, 'into', adder)\n",
    "    current_total = adder.send(integers)\n",
    "    print('Current total in', adder, 'is', current_total)\n",
    "    return current_total\n",
    "\n",
    "\n",
    "adder = Adder()\n",
    "print(adder)\n",
    "print()\n",
    "\n",
    "adder.send([])\n",
    "print()\n",
    "adder.send(10)\n",
    "print()\n",
    "adder.send([1, 2, 3])\n",
    "print()\n",
    "adder.send(range(8))\n",
    "print()\n",
    "print('Sending StopAdder into adder')\n",
    "try:\n",
    "    adder.throw(StopAdder)\n",
    "except StopIteration as exc:\n",
    "    print('Final total from adder is', exc.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Copyright Attributions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This chapter includes a code sample from <https://github.com/python/cpython/blob/v3.6.1/Lib/contextlib.py>.  \n",
    "Copyright (c) 2001-2017 Python Software Foundation.  \n",
    "All Rights Reserved.  \n",
    "License: Python license, <https://www.python.org/3.6/license.html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "License: [Apache License, Version 2.0][Apache License]  \n",
    "[Jordan Moldow][], 2017\n",
    "\n",
    ">     Copyright 2017 Jordan Moldow\n",
    ">\n",
    ">     Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    ">     you may not use this file except in compliance with the License.\n",
    ">     You may obtain a copy of the License at\n",
    ">\n",
    ">         http://www.apache.org/licenses/LICENSE-2.0\n",
    ">\n",
    ">     Unless required by applicable law or agreed to in writing, software\n",
    ">     distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    ">     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    ">     See the License for the specific language governing permissions and\n",
    ">     limitations under the License.\n",
    "\n",
    "[Jordan Moldow]: <https://github.com/jmoldow> \"Jordan Moldow\"\n",
    "[Apache License]: <http://www.apache.org/licenses/LICENSE-2.0> \"Apache License, Version 2.0\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}