03-yield-from-syntax-for-delegating-to-subgenerator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Introduction to ``yield from``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To my knowledge, Python 3.3 was the first in the Python 3.x to introduce brand new functionality / language syntax extensions for generators. This was the ability to delegate to subgenerators, via the new [``yield from``][yield from] construct. This was proposed in [PEP 380][].\n",
    "\n",
    "[PEP 380]: <https://www.python.org/dev/peps/pep-0380/> \"PEP 380 -- Syntax for Delegating to a Subgenerator\"\n",
    "[yield from]: <https://docs.python.org/3.3/reference/expressions.html#yield-expressions> \"yield from expressions\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simple interpretation of ``yield from``"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At its most basic, ``yield from`` can be used to make it easier to correctly iterate through, and yield values from another iterable.\n",
    "\n",
    "Any time the iterable in the ``yield from`` expression yields a value, the generator immediately yields that same value. Execution continues inside the iterable, until it is exhausted. After that point, further execution may continue inside the outer generator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def chain_with_yield(*iterables):\n",
    "    for iterable in iterables:\n",
    "        for item in iterable:\n",
    "            yield item\n",
    "            \n",
    "list(chain_with_yield(range(4), range(4, 8)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def chain_with_yield_from(*iterables):\n",
    "    for iterable in iterables:\n",
    "        yield from iterable\n",
    "        \n",
    "list(chain_with_yield_from(range(4), range(4, 8)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Delegating the generator methods to the subgenerator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The naive substitute for ``yield from``\n",
    "\n",
    "```python\n",
    "for item in iterable:\n",
    "    yield item\n",
    "```\n",
    "\n",
    "is only sufficient if the outer generator is being treated as a normal iterator. If you want to use ``send()`` or ``throw()``, they will only have an effect on the outer generator. Correctly delegating to the in-progress subgenerator requires [very tricky and ugly code][yield from semantics].\n",
    "\n",
    "But when using ``yield from``, the Python interpretter handles that delegation for you. Calling ``send()`` or ``throw()`` delegate through all ``yield from`` expressions in the current stack, all the way to the deepest ``yield`` expression, and then perform the ``send()`` or ``throw()`` at that location.\n",
    "\n",
    "An interesting consequence is that the outer generator loses all control of execution until the subgenerator is exhausted. Calls to ``__next__()``, ``send()``, and ``throw()`` will always be proxied directly to the subgenerator. The outer generator has no say in this behavior, and has no way to resume control until the subgenerator raises ``StopIteration`` or some other exception.\n",
    "\n",
    "[yield from semantics]: <https://www.python.org/dev/peps/pep-0380/#formal-semantics> \"yield from semantics\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "yielding None\n",
      "received 3\n",
      "yielding 3\n",
      "3\n",
      "received 4\n",
      "yielding 4\n",
      "4\n",
      "Received IgnorableError() and will re-raise\n",
      "Caught IgnorableError from subgenerator, restarting subgenerator\n",
      "yielding None\n",
      "None\n",
      "Received StopSubGenerator, returning\n",
      "Subgenerator stopped cleanly, restarting subgenerator\n",
      "yielding None\n",
      "None\n",
      "Received GeneratorExit, returning\n",
      "Received GeneratorExit from subgenerator, returning\n"
     ]
    }
   ],
   "source": [
    "import traceback\n",
    "\n",
    "class IgnorableError(Exception): pass\n",
    "\n",
    "class StopSubGenerator(Exception): pass\n",
    "\n",
    "def yield_sent_values():\n",
    "    value = None\n",
    "    while True:\n",
    "        print('yielding', value)\n",
    "        try:\n",
    "            value = (yield value)\n",
    "        except (StopSubGenerator, GeneratorExit) as exc:\n",
    "            print(f'Received {exc.__class__.__name__}, returning')\n",
    "            return\n",
    "        except Exception as exc:\n",
    "            print('Received', repr(exc), 'and will re-raise')\n",
    "            raise\n",
    "        print('received', value)\n",
    "        \n",
    "def delegator_function():\n",
    "    while True:\n",
    "        try:\n",
    "            yield from yield_sent_values()\n",
    "        except IgnorableError:\n",
    "            print('Caught IgnorableError from subgenerator, restarting subgenerator')\n",
    "        except GeneratorExit:\n",
    "            print('Received GeneratorExit from subgenerator, returning')\n",
    "            return\n",
    "        except Exception as exc:\n",
    "            print('Received', repr(exc), 'from subgenerator, re-raising')\n",
    "            raise\n",
    "        else:\n",
    "            print('Subgenerator stopped cleanly, restarting subgenerator')\n",
    "    \n",
    "delegator = delegator_function()\n",
    "next(delegator)\n",
    "print(delegator.send(3))\n",
    "print(delegator.send(4))\n",
    "print(delegator.throw(IgnorableError))\n",
    "print(delegator.throw(StopSubGenerator))\n",
    "delegator.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Value of the ``yield from`` expression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like ``yield``, the ``yield from`` construct can be used as an expression. It can be assigned to a variable, passed to another function, etc.\n",
    "\n",
    "The value is the object that gets returned as the subgenerator exits, or the ``.value`` attribute of the raised ``StopIteration``.\n",
    "\n",
    "Again ignoring the semantics of ``send()`` and ``throw()``, this means that ``RESULT = yield from EXPR`` is similar to the following code.\n",
    "\n",
    "```python\n",
    "iter = (EXPR).__iter__()\n",
    "running = True\n",
    "while running:\n",
    "    try:\n",
    "        yield iter.__next__()\n",
    "    except StopIteration as exc:\n",
    "        RESULT = exc.value\n",
    "        running = False\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generator yielded yielded_value\n",
      "generator yielded 1\n",
      "generator yielded 2\n",
      "generator received final return value returned_value from subgenerator\n"
     ]
    }
   ],
   "source": [
    "def subgenerator():\n",
    "    yield 'yielded_value'\n",
    "    yield '1'\n",
    "    yield '2'\n",
    "    return 'returned_value'\n",
    "\n",
    "def generator():\n",
    "    value = yield from subgenerator()\n",
    "    print('generator received final return value', value, 'from subgenerator')\n",
    "    \n",
    "for item in generator():\n",
    "    print('generator yielded', item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generator received final return value returned_value from SubGenerator\n"
     ]
    }
   ],
   "source": [
    "import collections\n",
    "\n",
    "\n",
    "class SubGenerator(collections.Generator):\n",
    "    def send(self, value):\n",
    "        raise StopIteration('returned_value')\n",
    "    def throw(self, *exc_info):\n",
    "        next(self)\n",
    "        \n",
    "        \n",
    "def generator():\n",
    "    value = yield from SubGenerator()\n",
    "    print('generator received final return value', value, 'from SubGenerator')\n",
    "    \n",
    "    \n",
    "for item in generator():\n",
    "    pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generators as threads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before [PEP 380][] and Python 3.3, there was no `StopIteration.value` attribute, and generators could not return values. Thus, the main purposes of generators were to serve as a special syntax for defining an iterator, and in advanced cases to serve as a general mechanism for suspending and resuming execution of a block of code.\n",
    "\n",
    "Now that generators can return values, they are able to take on a new meaning. A generator which returns a value can be thought of as a computation that produces a result, but which suspends itself in the middle of that computation. It might do this in order to\n",
    "1. `yield` an intermediate value;\n",
    "2. communicate some information to the top-level executor;\n",
    "3. receive values or instructions via `send()`, `throw()`, and `close()`;\n",
    "4. or simply to allow other computations to be interleaved with it.\n",
    "\n",
    "We already showed an example of 1., with the generator-based adders that we defined in the previous chapter. The concept of 2. is a bit more complex, so we will save that for an upcoming chapter. The generator-based adder also shows off 3., though for another example see <<http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yf_current/Examples/Parser/parser.txt>>. And we'll show off an example 4. in the very next chapter.\n",
    "\n",
    "In the case of 3. and 4., the outer executor might not even care about the values that get yielded from the generator, and the generator may just be able to `yield` without a value.\n",
    "\n",
    "The language of [PEP 380][] refers to these behaviors in an interesting way. The proposal document has a section titled \"[Generators as Threads][]\". Here is a quote of the most interesting part of that section:\n",
    "\n",
    "[PEP 380]: <https://www.python.org/dev/peps/pep-0380/> \"PEP 380 -- Syntax for Delegating to a Subgenerator\"\n",
    "[Generators as Threads]: <https://www.python.org/dev/peps/pep-0380/#generators-as-threads> \"Generators as Threads\"\n",
    "\n",
    "> A motivation for generators being able to return values concerns the use of generators to implement lightweight threads. When using generators in that way, it is reasonable to want to spread the computation performed by the lightweight thread over many functions. One would like to be able to call a subgenerator as though it were an ordinary function, passing it parameters and receiving a returned value.\n",
    ">\n",
    "> Using the proposed syntax, a statement such as\n",
    ">\n",
    "> `y = f(x)`\n",
    ">\n",
    "> where f is an ordinary function, can be transformed into a delegation call\n",
    ">\n",
    "> `y = yield from g(x)`\n",
    ">\n",
    "> where g is a generator. One can reason about the behaviour of the resulting code by thinking of g as an ordinary function that can be suspended using a yield statement.\n",
    "\n",
    "When they say \"thread\", they don't mean a `threading.Thread` or an operating system thread. Rather, they are referring to a computation, procedure, or program that can be run concurrently with other such threads. This concept will be explored when we talk about event loops and async tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generators as futures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A related line of thinking is to view these generators as futures. Suppose again that `g` is a generator function which returns a value. Then `y = yield from g(x)` assigns that return value to `y`. However, since the computation of `g(x)` involves `yield` and possibly `yield from` statements, it will suspend itself one or more times, and the final result will not be available right away. But when the final result is available, it is immediately assigned to `y`, and computation of that function resumes. This isn't quite the same as a future, since futures are objects that can be passed around, mapped, assigned callbacks, etc.\n",
    "\n",
    "When a function returns a future, the caller's execution is not suspended until/unless the caller decides to explicitly wait on the result. When thinking of generators as futures, `y = yield from g(x)` is analagous to `future = g(x); y = future.block_and_get_result()`.\n",
    "\n",
    "When we explore event loops, we'll see an actual future that takes advantage of `yield from`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Caveats to generators as threads/futures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately, not everything is sunshine and rainbows, even with the introduction of `yield from`. Say you are writing a function `f`, and you want to call a function `g` that produces some return value. However, `g` is a generator function, whereas `f` is currently a normal function. Here, you have three options:\n",
    "1. Use `y = yield from g()` to get the return value. However, this will turn `f` into a generator function, which means that its caller(s) need to run `f` or must themselves `yield from f()`, and so on and so forth.\n",
    "2. Run `g()` through to completion, perhaps manually, or perhaps using a helper function like\n",
    "   ```python\n",
    "   def run_until_complete(generator):\n",
    "       try:\n",
    "           while True:\n",
    "               next(generator)\n",
    "       except StopIteration as exc:\n",
    "           return exc.value\n",
    "   ```\n",
    "   This has the downside that your program loses out on the ability to possibly interleave other computations and do concurrent programming.\n",
    "\n",
    "3. Reimplement `g` as a non-generator function, and call that function instead.\n",
    "\n",
    "Going with 1. isn't necessarily a bad thing. But it does show how using generators to represent suspendable computations can be infectious. Later we'll see how this is necessary in order to gain the ability to do async programming."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "License: [Apache License, Version 2.0][Apache License]  \n",
    "[Jordan Moldow][], 2017\n",
    "\n",
    ">     Copyright 2017 Jordan Moldow\n",
    ">\n",
    ">     Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    ">     you may not use this file except in compliance with the License.\n",
    ">     You may obtain a copy of the License at\n",
    ">\n",
    ">         http://www.apache.org/licenses/LICENSE-2.0\n",
    ">\n",
    ">     Unless required by applicable law or agreed to in writing, software\n",
    ">     distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    ">     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    ">     See the License for the specific language governing permissions and\n",
    ">     limitations under the License.\n",
    "\n",
    "[Jordan Moldow]: <https://github.com/jmoldow> \"Jordan Moldow\"\n",
    "[Apache License]: <http://www.apache.org/licenses/LICENSE-2.0> \"Apache License, Version 2.0\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}