tutorials/01 Cyclist Detection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rekall Tutorial: Cyclist Detection\n",
    "\n",
    "In this tutorial, you'll learn how to use Rekall to detect a new class of objects (cyslists) from existing person and bicycle detections from Mask R-CNN.\n",
    "\n",
    "Let's first import Rekall and a few of its important classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "from rekall import Interval, IntervalSet, IntervalSetMapping, Bounds3D\n",
    "from rekall.predicates import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial, we'll provide some helpers to handle data loading videos and pre-computed object detections from our servers. Run this cell to load in those helpers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from cyclist_tutorial_helpers import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's load up the pre-computed bounding box detections:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "bboxes = get_maskrcnn_bboxes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `visualize_helper` function to visualize these bounding boxes. Click on the video to expand it, and play the video by hovering over it and using `;`. You can navigate through the video by clicking through the timeline, and using the `+` and `-` buttons to zoom in or out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9eca39ddee04874a2bb0b395c87dde1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VGridWidget(vgrid_spec={'compressed': True, 'data': b'x\\x9c\\xcc\\xbd\\xdb\\xae&K\\x92\\x9c\\xf7*D_\\x0bB\\x9c\\x0f\\xba\\…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_helper([bboxes])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Filtering on Payload\n",
    "Let's give a preview of some of the things you'll be able to do with Rekall. In the above two cells we've loaded up bounding box detections over two videos, and visualized them for you.\n",
    "\n",
    "Let's start by filtering the bounding box detections by class to look at bicycle and person detections:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "bikes = bboxes.filter(lambda interval: interval['payload']['class'] == 'bicycle')\n",
    "person = bboxes.filter(lambda interval: interval['payload']['class'] == 'person')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's visualize them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e9e178339528420e987dcaac55918eb0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VGridWidget(vgrid_spec={'compressed': True, 'data': b'x\\x9c\\xd4\\x9d\\xcb\\xae,Ir]\\x7f\\x85\\xa81\\xd1\\xf0\\xf7CC\\xfd…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_helper([bikes, person])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try some payload filtering functions yourself here!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Filtering on Bounds\n",
    "\n",
    "We are using Rekall's [Bounds3D](https://rekallpy.readthedocs.io/en/latest/source/rekall.bounds.html#rekall.bounds.Bounds3D) to represent these intervals. The intervals all have co-ordinates `t1`, `t2` (seconds), `x1`, `x2` (frame relative, between 0 and 1), and `y1`, `y2` (frame relative, between 0 and 1).\n",
    "\n",
    "We can filter on the bounds co-ordinates as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51475b4d273e4aa084f12f4a0d11490b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VGridWidget(vgrid_spec={'compressed': True, 'data': b'x\\x9c\\xd4\\x9d\\xcb\\xae,Ir]\\x7f\\x85\\xa81\\xd1\\xf0\\xf7CC\\xfd…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_helper([\n",
    "    bikes.filter(lambda interval: interval['t1'] < 300),\n",
    "    person.filter(lambda interval: interval['t1'] < 300)\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try some bounds filtering functions yourself here!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Rekall's Data Model\n",
    "\n",
    "Now that we have a flavor of what we can do with Rekall, let's build our understanding of the data representation from the ground up. Let's first understand what `Interval`s are - these are the fundamental data structure that we use to represent any annotations in videos.\n",
    "\n",
    "Here's a figure demonstrating what these Intervals can look like:\n",
    "\n",
    "![video_volume_v2.png](https://storage.googleapis.com/esper/dan_olimar/rekall_tutorials/videovolume_v2.png)\n",
    "\n",
    "Intervals are parameterized by a Bounds object (`Bounds3D` in all the intervals above), and an optional payload (face identities, word in the caption, or nested Intervals in the figure above):\n",
    "\n",
    "```Python\n",
    "# This interval has time bounds from 0 to 10 seconds, X bounds from 0.5 to 0.7 (frame-relative),\n",
    "# and Y bounds from 0.6 to 0.9 (frame-relative)\n",
    "new_interval = Interval(Bounds3D(\n",
    "    t1 = 0,\n",
    "    t2 = 10,\n",
    "    x1 = 0.5,\n",
    "    x2 = 0.7,\n",
    "    y1 = 0.6,\n",
    "    y2 = 0.9\n",
    "))\n",
    "\n",
    "# This interval has time bounds from 5 to 15 seconds, and default X and Y bounds of the whole\n",
    "# frame (0 to 1 for both X and Y)\n",
    "new_interval2 = Interval(Bounds3D(5, 15))\n",
    "\n",
    "# This interval has a payload. The payload can be an arbitrary object.\n",
    "new_interval3 = Interval(Bounds3D(0, 1), payload={ 'class': 'my first payload' })\n",
    "                         \n",
    "# We can access the co-ordinates of payload and an Interval directly\n",
    "print(new_interval['t1'], new_interval['t2'], new_interval['x1'])\n",
    "print(new_interval2['t1'], new_interval2['x1'])\n",
    "print(new_interval3['payload'])\n",
    "print(new_interval3['payload']['class'])\n",
    "```\n",
    "\n",
    "Try it yourself below!\n",
    "\n",
    "**NB: If you're coming from the paper/tech report, the words \"Label\" are \"Interval\" are interchangeable in the code.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 10 0.5\n",
      "5 0.0\n",
      "{'class': 'my first payload'}\n",
      "my first payload\n"
     ]
    }
   ],
   "source": [
    "# This interval has time bounds from 0 to 10 seconds, X bounds from 0.5 to 0.7 (frame-relative),\n",
    "# and Y bounds from 0.6 to 0.9 (frame-relative)\n",
    "new_interval = Interval(Bounds3D(\n",
    "    t1 = 0,\n",
    "    t2 = 10,\n",
    "    x1 = 0.5,\n",
    "    x2 = 0.7,\n",
    "    y1 = 0.6,\n",
    "    y2 = 0.9\n",
    "))\n",
    "\n",
    "# This interval has time bounds from 5 to 15 seconds, and default X and Y bounds of the whole\n",
    "# frame (0 to 1 for both X and Y)\n",
    "new_interval2 = Interval(Bounds3D(5, 15))\n",
    "\n",
    "# This interval has a payload. The payload can be an arbitrary object.\n",
    "new_interval3 = Interval(Bounds3D(0, 1), payload={ 'class': 'my first payload' })\n",
    "\n",
    "# We can access the co-ordinates of payload and an Interval directly\n",
    "print(new_interval['t1'], new_interval['t2'], new_interval['x1'])\n",
    "print(new_interval2['t1'], new_interval2['x1'])\n",
    "print(new_interval3['payload'])\n",
    "print(new_interval3['payload']['class'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create some Intervals yourself here!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Associating Intervals with Events\n",
    "In Rekall, we use *sets* of Intervals to represent events in videos. A single `IntervalSet` contains all occurrences of an event in a single video (all the bounding box detections, all the cyclist annotations, etc).\n",
    "\n",
    "We can create an `IntervalSet` by passing in a list of `Interval`s:\n",
    "\n",
    "```Python\n",
    "# This IntervalSet represents all occurrences of a \"made up\" event in a video\n",
    "my_first_intervalset = IntervalSet([\n",
    "    Interval(Bounds3D(0, 10), payload = { 'class': 'made up'} ),\n",
    "    Interval(Bounds3D(20, 30), payload = { 'class': 'made up'} ),\n",
    "])\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_first_intervalset = IntervalSet([\n",
    "    Interval(Bounds3D(0, 10), payload = { 'class': 'made up'} ),\n",
    "    Interval(Bounds3D(20, 30), payload = { 'class': 'made up'} ),\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One last thing - we want to associate each `IntervalSet` with the right video. We might have detected bikes in one video, but not the other!\n",
    "\n",
    "We use `IntervalSetMapping` to associate `IntervalSet`s with different videos by keys. We create an `IntervalSetMapping` by passing in a `dict` from \n",
    "\n",
    "```Python\n",
    "my_first_ism = IntervalSetMapping({\n",
    "    0: IntervalSet(...), # the IntervalSet for video 0\n",
    "    2: IntervalSet(...) # the IntervalSet for video 2\n",
    "})\n",
    "```\n",
    "\n",
    "`bboxes` is an `IntervalSetMapping` object that we pre-loaded with `Interval`s containing object detections from Mask-RCNN.\n",
    "\n",
    "Similarly, `bikes` and `person` are `IntervalSetMapping` objects representing the event that a bicycle object or a person object was detected in the video.\n",
    "\n",
    "We can look at the keys of one of the `IntervalSetMapping` objects to see what the keys in our videos are!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys([0, 3])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bboxes.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The videos that we were looking at before have keys 0 and 3. The visualization shows the videos in sorted order, so we know that the first video is video 0, and the second video is video 3.\n",
    "\n",
    "Now we can create a simple `IntervalSetMapping` and visualize it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_first_ism = IntervalSetMapping({\n",
    "    0: my_first_intervalset,\n",
    "    3: IntervalSet([Interval(Bounds3D(50, 60, 0.5, 0.8, 0.1, 0.3)), Interval(Bounds3D(100,200))])\n",
    "})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7f4b834b78b645cdb822ed3c298c6e25",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VGridWidget(vgrid_spec={'compressed': True, 'data': b'x\\x9c\\xcdTKO\\xc30\\x0c\\xfe+(g\\xd4\\xa6\\xdd\\xc6cG\\x8e\\\\\\xe1…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_helper([my_first_ism])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try it yourself below! Create some new `IntervalSetMapping` objects and visualize them on our videos.\n",
    "\n",
    "**NB: if you try to visualize an `IntervalSet` with only a single Interval in it, the timeline may not appear until you click on the video.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try it yourself!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checkpoint: check on your understanding\n",
    "\n",
    "At this point, you should be familiar with a few concepts:\n",
    "* We represent events in videos as *sets* of intervals - objects that are defined by a Bounds object and an optional payload\n",
    "* An `IntervalSet` represents all the occurrences of a particular event in a single video\n",
    "* An `IntervalSetMapping` organizes multiple `IntervalSet`s and associates each one with a different video\n",
    "\n",
    "Since we told you that `bboxes_cydet` and `cyclist_gt` are `IntervalSetMapping` objects, you may have also surmised that we have some functions for manipulating these sets of intervals - we've already looked at a few examples of the `filter` function. But now that we know more about the underlying data representation, we can take a closer look at `filter`.\n",
    "\n",
    "From the [documentation](https://rekallpy.readthedocs.io/en/latest/index.html#rekall.IntervalSet.filter):\n",
    "\n",
    "```Python\n",
    "def filter(self, predicate):\n",
    "    \"\"\"\n",
    "    Filter the set and keep intervals that pass the predicate.\n",
    "    \n",
    "    Args:\n",
    "        predicate: A function that takes an Interval and returns a bool.\n",
    "        \n",
    "    Returns:\n",
    "        A new IntervalSet which is the filtered set.\n",
    "    \"\"\"\n",
    "```\n",
    "\n",
    "So `filter` expects a function that will take an `Interval` and return `True` or `False`. It runs the predicate function on every interval, and only keeps the ones where the predicate returns `True`.\n",
    "\n",
    "Notice that the documentation says that `filter` returns a new `IntervalSet` - that's because it's actually a function on `IntervalSet`! `IntervalSetMapping` simply reflects all the functions in `IntervalSet` and applies the functions to every `IntervalSet`. So even though we wrote the function over `IntervalSet`s, we can run them on `IntervalSetMapping` objects like `bboxes_cydet` and `cyclist_gt`.\n",
    "\n",
    "## Checkpoint exercise: duplicate certain labels across the entirety of a video\n",
    "Let's go through a simple exercise to check your understanding of these concepts.\n",
    "\n",
    "We'll define a simple `IntervalSetMapping` with a few objects. We'll want you to duplicate the Intervals at time 0 of video 0 throughout videos 0 and 3, but only if the class in their payload is `car`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "objects_to_duplicate = IntervalSetMapping({\n",
    "    0: IntervalSet([\n",
    "        Interval(Bounds3D(0, 10, 0.1, 0.3, 0.1, 0.9), payload={ 'class': 'car' }),\n",
    "        Interval(Bounds3D(0, 10, 0.4, 0.6, 0.1, 0.9), payload={ 'class': 'car' }),\n",
    "        Interval(Bounds3D(0, 10, 0.7, 0.9, 0.1, 0.9), payload={ 'class': 'godzilla' }),\n",
    "        Interval(Bounds3D(10, 20, 0.1, 0.3, 0.1, 0.9), payload={ 'class': 'car' }),\n",
    "        Interval(Bounds3D(10, 20, 0.4, 0.6, 0.1, 0.9), payload={ 'class': 'car' }),\n",
    "        Interval(Bounds3D(10, 20, 0.7, 0.9, 0.1, 0.9), payload={ 'class': 'godzilla' })\n",
    "    ]),\n",
    "    3: IntervalSet([\n",
    "        Interval(Bounds3D(0, 10, 0.2, 0.4, 0.1, 0.9), payload={ 'class': 'car' }),\n",
    "        Interval(Bounds3D(0, 10, 0.5, 0.7, 0.1, 0.9), payload={ 'class': 'car' }),\n",
    "        Interval(Bounds3D(0, 10, 0.75, 0.95, 0.1, 0.9), payload={ 'class': 'godzilla' }),\n",
    "        Interval(Bounds3D(10, 20, 0.2, 0.4, 0.1, 0.9), payload={ 'class': 'car' }),\n",
    "        Interval(Bounds3D(10, 20, 0.5, 0.7, 0.1, 0.9), payload={ 'class': 'car' }),\n",
    "        Interval(Bounds3D(10, 20, 0.75, 0.95, 0.1, 0.9), payload={ 'class': 'godzilla' })\n",
    "    ])\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In particular, we'll want the resulting Intervals to have the following properties:\n",
    "* You should have Intervals with time extents of (0, 10), (10, 20), (20, 30), etc.\n",
    "* The Intervals should have the same X/Y extent and payload as the Intervals at *time 0* of video 0 that have payload 'car'.\n",
    "* The Intervals should cover the entire video.\n",
    "\n",
    "A few hints to get you started:\n",
    "* You can access `IntervalSet 0` of `objects_to_duplicate` like this: `objects_to_duplicate[0]`.\n",
    "* You can loop through all the video keys of `objects_to_duplicate` using an iterator: `[k for k in objects_to_duplicate]`.\n",
    "* You can filter down to Intervals that start at time 0 like this: `objects_to_duplicate.filter(lambda interval: interval['t0'] == 0)`\n",
    "* You can access all the Intervals in an `IntervalSet` like this: `objects_to_duplicate[0].get_intervals()`. This will return a list of Intervals sorted by time.\n",
    "* For example, to get the t2 value of the last interval in `bboxes_cydet`: `bboxes_cydet[0].get_intervals()[-1]['t2']`\n",
    "* You can access the bounds of an interval like so: `interval['bounds']`\n",
    "* You can copy a bounds like so: `interval['bounds'].copy()`\n",
    "* Remember you can use Python ranges to get a value every ten seconds: `range(0, end, 10)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Construct an IntervalSetMapping with the answer here!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise solution:\n",
    "\n",
    "Example solution to this exercise - don't look if you haven't tried it!\n",
    "\n",
    "```Python\n",
    "objects_at_start = objects_to_duplicate[0].filter(\n",
    "    lambda interval: interval['t1'] == 0 and interval['payload']['class'] == 'car'\n",
    ").get_intervals()\n",
    "\n",
    "video_lengths = {\n",
    "    key: bboxes_cydet[key].get_intervals()[-1]['t2']\n",
    "    for key in bboxes_cydet\n",
    "}\n",
    "\n",
    "objects_duplicated = IntervalSetMapping({\n",
    "    key: IntervalSet([\n",
    "        Interval(Bounds3D(\n",
    "            t1 = t,\n",
    "            t2 = t + 10,\n",
    "            x1 = interval['x1'],\n",
    "            x2 = interval['x2'],\n",
    "            y1 = interval['y1'],\n",
    "            y2 = interval['y2']\n",
    "        ))\n",
    "        for t in range(0, int(video_lengths[key]), 10)\n",
    "        for interval in objects_at_start\n",
    "    ])\n",
    "    for key in bboxes_cydet\n",
    "})\n",
    "\n",
    "visualize_cydet([objects_duplicated])\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Defining New Events through Composition and Manipulation Functions for Cyclist Detection\n",
    "\n",
    "Now that we have that basic understanding of Rekall's data representation, let's look at some more complex manipulation functions to try to make a cyclist detector.\n",
    "\n",
    "First, let's look at the [`join`](https://rekallpy.readthedocs.io/en/latest/index.html#rekall.IntervalSet.join) function. This function computes the cross product of two `IntervalSet`s, filters the resulting pairs by a predicate, and then merges the resulting pairs back into a single `Interval` using a merge operation:\n",
    "\n",
    "![simple_join.png](https://olimar.stanford.edu/hdd/rekall_tutorials/simple_join.png)\n",
    "\n",
    "Here's an example of using a `join` operation to create an `IntervalSetMapping` object containing instances of a `person` bounding box overlapping with a `bicycle` bounding box.\n",
    "```Python\n",
    "person_intersect_bike = person.join(\n",
    "    bikes,\n",
    "    predicate = and_pred(\n",
    "        Bounds3D.T(equal()),\n",
    "        Bounds3D.X(overlaps()),\n",
    "        Bounds3D.Y(overlaps())\n",
    "    ),\n",
    "    merge_op = lambda interval1, interval2: Interval(\n",
    "        interval1['bounds'].span(interval2['bounds'])\n",
    "    ),\n",
    "    window = 0.0,\n",
    "    progress_bar = True\n",
    ")\n",
    "```\n",
    "\n",
    "This function joins `person` and `bikes`. The predicate expects a function of the following format:\n",
    "```Python\n",
    "def predicate(interval1, interval2):\n",
    "    # return True or False\n",
    "```\n",
    "\n",
    "We provide a number of spatial and temporal predicates in Rekall, outlined [here](https://rekallpy.readthedocs.io/en/latest/source/rekall.predicates.html). In this case, we're only keeping pairs of (person detection, bike detection) if they have the same time bounds (`Bounds3D.T(equal())`), and the `X` and `Y` bounds overlap (`Bounds3D.X(overlaps())`, `Bounds3D.Y(overlaps())`). The `and_pred` wrapper takes in an arbitrary number of predicates and makes sure that all of them pass.\n",
    "\n",
    "The `merge_op` expects a function of this form:\n",
    "```Python\n",
    "def merge_op(interval1, interval2):\n",
    "    # return a new Interval\n",
    "```\n",
    "\n",
    "In this case, we are returning a new Interval whose bounds span both the Intervals in the pair - basically, the minimum bounding box that covers both of them.\n",
    "\n",
    "We pass in a `window` of `0.0` - this is an optimization that limits the pairs in the cross product to only those Intervals whose time bounds are apart from each other by `window` or less time. A `window` value of `0` limits the pairs to only those that overlap. Since we're already filtering by that in the time dimension, we know that this optimization won't change our results.\n",
    "\n",
    "Finally, we pass `progress_bar=True` just to visualize a progress bar while we wait for this computation to complete (there are a lot of detections to process).\n",
    "\n",
    "Let's run this function and visualize the results below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 2/2 [00:08<00:00,  4.12s/it]\n"
     ]
    }
   ],
   "source": [
    "# Use the join above to construct bounding boxes where a person and bike overlap!\n",
    "person_intersect_bike = person.join(\n",
    "    bikes,\n",
    "    predicate = and_pred(\n",
    "        Bounds3D.T(equal()),\n",
    "        Bounds3D.X(overlaps()),\n",
    "        Bounds3D.Y(overlaps())\n",
    "    ),\n",
    "    merge_op = lambda interval1, interval2: Interval(\n",
    "        interval1['bounds'].span(interval2['bounds'])\n",
    "    ),\n",
    "    window = 0.0,\n",
    "    progress_bar = True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9405d44839704964b48726030ebcf069",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VGridWidget(vgrid_spec={'compressed': True, 'data': b'x\\x9c\\xd4\\x9d\\xcb\\xae\\xa4\\xc9u\\x9d_\\xc5\\xe0\\xd8 \\xe2~\\xf…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "visualize_helper([person_intersect_bike, bikes, person])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Congratulations!\n",
    "\n",
    "You've now used a simple Rekall query to detect bicyclists by composing person detections with bicycle detections.\n",
    "\n",
    "Next, check out the parking space detection tutorial to take a deeper dive into some of Rekall's functions and detect empty parking spaces using nothing more than the outputs of an off-the-shelf object detector!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}