notebooks/03-process.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Interactively run workflow calcium imaging\n",
    "\n",
    "+ This notebook walks you through the steps in detail to run the `workflow-calcium-imaging`.  \n",
    "\n",
    "+ The workflow requires the calcium imaging acquired data from ScanImage or Scanbox and processed data from Suite2p or CaImAn.\n",
    "\n",
    "+ If you haven't configured the paths, refer to [01-configure](01-configure.ipynb).\n",
    "\n",
    "+ To overview the schema structures, refer to [02-workflow-structure](02-workflow-structure.ipynb).\n",
    "\n",
    "+ If you need a more automatic approach to run the workflow, refer to [03-automate](03-automate-optional.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's change the directory to the package root directory to load the local configuration (`dj_local_conf.json`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "if os.path.basename(os.getcwd()) == \"notebooks\": os.chdir(\"..\")\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `Pipeline.py`\n",
    "\n",
    "+ This script `activates` the DataJoint `elements` and declares other required tables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Connecting user1@127.0.0.1:3306\n"
     ]
    }
   ],
   "source": [
    "import datajoint as dj\n",
    "from workflow_calcium_imaging.pipeline import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Schema diagrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<svg height=\"318pt\" viewBox=\"0.00 0.00 484.50 318.00\" width=\"485pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 314)\">\n<polygon fill=\"white\" points=\"-4,4 -4,-314 480.5,-314 480.5,4 -4,4\" stroke=\"transparent\"/>\n<!-- scan.Scan -->\n<g class=\"node\" id=\"node1\">\n<title>scan.Scan</title>\n<g id=\"a_node1\"><a xlink:title=\"→ Session\r\nscan_id              \r\n------------------------------\r\n→ Equipment\r\n→ scan.AcquisitionSoftware\r\nscan_notes=&quot;&quot;        \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"288.5,-168 215.5,-168 215.5,-133 288.5,-133 288.5,-168\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"223.5\" y=\"-148.4\">scan.Scan</text>\n</a>\n</g>\n</g>\n<!-- scan.ScanInfo -->\n<g class=\"node\" id=\"node8\">\n<title>scan.ScanInfo</title>\n<g id=\"a_node8\"><a xlink:title=\"→ scan.Scan\r\n------------------------------\r\nnfields              \r\nnchannels            \r\nndepths              \r\nnframes              \r\nnrois                \r\nx                    \r\ny                    \r\nz                    \r\nfps                  \r\nbidirectional        \r\nusecs_per_line=null  \r\nfill_fraction=null   \r\n\">\n<ellipse cx=\"117\" cy=\"-79.5\" fill=\"#00007f\" fill-opacity=\"0.250980\" rx=\"57.5\" ry=\"17.5\" stroke=\"#00007f\" stroke-opacity=\"0.250980\"/>\n<text fill=\"#00007f\" fill-opacity=\"0.627451\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"117\" y=\"-76.4\">scan.ScanInfo</text>\n</a>\n</g>\n</g>\n<!-- scan.Scan&#45;&gt;scan.ScanInfo -->\n<g class=\"edge\" id=\"edge1\">\n<title>scan.Scan-&gt;scan.ScanInfo</title>\n<path d=\"M219.66,-132.97C197.04,-121.41 167.11,-106.11 145.41,-95.02\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"2\"/>\n</g>\n<!-- scan.ScanLocation -->\n<g class=\"node\" id=\"node10\">\n<title>scan.ScanLocation</title>\n<g id=\"a_node10\"><a xlink:title=\"→ scan.Scan\r\n------------------------------\r\n→ Location\r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"311.5,-97 192.5,-97 192.5,-62 311.5,-62 311.5,-97\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"252\" y=\"-76.4\">scan.ScanLocation</text>\n</a>\n</g>\n</g>\n<!-- scan.Scan&#45;&gt;scan.ScanLocation -->\n<g class=\"edge\" id=\"edge2\">\n<title>scan.Scan-&gt;scan.ScanLocation</title>\n<path d=\"M252,-132.8C252,-121.95 252,-107.87 252,-97.05\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"2\"/>\n</g>\n<!-- imaging.ProcessingTask -->\n<g class=\"node\" id=\"node11\">\n<title>imaging.ProcessingTask</title>\n<g id=\"a_node11\"><a xlink:title=\"→ scan.Scan\r\n→ imaging.ProcessingParamSet\r\n------------------------------\r\nprocessing_output_dir \r\ntask_mode=&quot;load&quot;     \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"476.5,-97 329.5,-97 329.5,-62 476.5,-62 476.5,-97\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"403\" y=\"-76.4\">imaging.ProcessingTask</text>\n</a>\n</g>\n</g>\n<!-- scan.Scan&#45;&gt;imaging.ProcessingTask -->\n<g class=\"edge\" id=\"edge3\">\n<title>scan.Scan-&gt;imaging.ProcessingTask</title>\n<path d=\"M288.17,-132.97C311.99,-122.09 343.05,-107.89 366.87,-97.01\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- scan.ScanInfo.Field -->\n<g class=\"node\" id=\"node2\">\n<title>scan.ScanInfo.Field</title>\n<g id=\"a_node2\"><a xlink:title=\"→ scan.ScanInfo\r\nfield_idx            \r\n------------------------------\r\npx_height            \r\npx_width             \r\num_height=null       \r\num_width=null        \r\nfield_x              \r\nfield_y              \r\nfield_z              \r\ndelay_image=null     \r\nroi=null             \r\n\">\n<polygon fill=\"transparent\" points=\"104,-22.5 0,-22.5 0,-3.5 104,-3.5 104,-22.5\" stroke=\"transparent\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"8\" y=\"-11.5\">scan.ScanInfo.Field</text>\n</a>\n</g>\n</g>\n<!-- Subject -->\n<g class=\"node\" id=\"node3\">\n<title>Subject</title>\n<g id=\"a_node3\"><a xlink:title=\"subject              \r\n------------------------------\r\nsex                  \r\nsubject_birth_date   \r\nsubject_description=&quot;&quot; \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"224.5,-310 167.5,-310 167.5,-275 224.5,-275 224.5,-310\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"175.5\" y=\"-290.4\">Subject</text>\n</a>\n</g>\n</g>\n<!-- Session -->\n<g class=\"node\" id=\"node4\">\n<title>Session</title>\n<g id=\"a_node4\"><a xlink:title=\"→ Subject\r\nsession_datetime     \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"226,-239 166,-239 166,-204 226,-204 226,-239\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"darkgreen\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"174\" y=\"-219.4\">Session</text>\n</a>\n</g>\n</g>\n<!-- Subject&#45;&gt;Session -->\n<g class=\"edge\" id=\"edge4\">\n<title>Subject-&gt;Session</title>\n<path d=\"M196,-274.8C196,-263.95 196,-249.87 196,-239.05\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- Session&#45;&gt;scan.Scan -->\n<g class=\"edge\" id=\"edge5\">\n<title>Session-&gt;scan.Scan</title>\n<path d=\"M209.56,-203.8C218.36,-192.95 229.79,-178.87 238.57,-168.05\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- scan.Channel -->\n<g class=\"node\" id=\"node5\">\n<title>scan.Channel</title>\n<g id=\"a_node5\"><a xlink:title=\"channel              \r\n\">\n<polygon fill=\"#000000\" fill-opacity=\"0.125490\" points=\"319.5,-310 242.5,-310 242.5,-275 319.5,-275 319.5,-310\" stroke=\"transparent\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"250.5\" y=\"-291\">scan.Channel</text>\n</a>\n</g>\n</g>\n<!-- scan.AcquisitionSoftware -->\n<g class=\"node\" id=\"node6\">\n<title>scan.AcquisitionSoftware</title>\n<g id=\"a_node6\"><a xlink:title=\"acq_software         \r\n\">\n<polygon fill=\"#000000\" fill-opacity=\"0.125490\" points=\"372,-239 244,-239 244,-204 372,-204 372,-239\" stroke=\"transparent\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"252\" y=\"-220\">scan.AcquisitionSoftware</text>\n</a>\n</g>\n</g>\n<!-- scan.AcquisitionSoftware&#45;&gt;scan.Scan -->\n<g class=\"edge\" id=\"edge6\">\n<title>scan.AcquisitionSoftware-&gt;scan.Scan</title>\n<path d=\"M294.44,-203.8C285.64,-192.95 274.21,-178.87 265.43,-168.05\" fill=\"none\" stroke=\"#000000\" stroke-dasharray=\"5,2\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- scan.ScanInfo.ScanFile -->\n<g class=\"node\" id=\"node7\">\n<title>scan.ScanInfo.ScanFile</title>\n<g id=\"a_node7\"><a xlink:title=\"→ scan.ScanInfo\r\nfile_path            \r\n\">\n<polygon fill=\"transparent\" points=\"243.5,-22.5 122.5,-22.5 122.5,-3.5 243.5,-3.5 243.5,-22.5\" stroke=\"transparent\"/>\n<text font-family=\"arial\" font-size=\"10.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"130.5\" y=\"-11.5\">scan.ScanInfo.ScanFile</text>\n</a>\n</g>\n</g>\n<!-- scan.ScanInfo&#45;&gt;scan.ScanInfo.Field -->\n<g class=\"edge\" id=\"edge7\">\n<title>scan.ScanInfo-&gt;scan.ScanInfo.Field</title>\n<path d=\"M100.93,-62.56C88.3,-50.02 71.17,-33.03 60.89,-22.82\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- scan.ScanInfo&#45;&gt;scan.ScanInfo.ScanFile -->\n<g class=\"edge\" id=\"edge8\">\n<title>scan.ScanInfo-&gt;scan.ScanInfo.ScanFile</title>\n<path d=\"M133.31,-62.56C146.14,-50.02 163.53,-33.03 173.97,-22.82\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"0.75\"/>\n</g>\n<!-- imaging.Processing -->\n<g class=\"node\" id=\"node9\">\n<title>imaging.Processing</title>\n<g id=\"a_node9\"><a xlink:title=\"→ imaging.ProcessingTask\r\n------------------------------\r\nprocessing_time      \r\npackage_version=&quot;&quot;   \r\n\">\n<ellipse cx=\"403\" cy=\"-13\" fill=\"#ff0000\" fill-opacity=\"0.125490\" rx=\"13\" ry=\"13\" stroke=\"#ff0000\" stroke-opacity=\"0.125490\"/>\n<text fill=\"#7f0000\" fill-opacity=\"0.627451\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"middle\" x=\"403\" y=\"-9.9\">imaging.Processing</text>\n</a>\n</g>\n</g>\n<!-- imaging.ProcessingTask&#45;&gt;imaging.Processing -->\n<g class=\"edge\" id=\"edge9\">\n<title>imaging.ProcessingTask-&gt;imaging.Processing</title>\n<path d=\"M403,-61.89C403,-50.69 403,-36.17 403,-26.07\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\"2\"/>\n</g>\n</g>\n</svg>",
      "text/plain": [
       "<datajoint.diagram.Diagram at 0x7fd3f8d40490>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dj.Diagram(subject.Subject) + dj.Diagram(session.Session) + dj.Diagram(scan) + dj.Diagram(imaging.Processing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert an entry into `subject.Subject`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# Animal Subject\n",
       "subject              : varchar(32)                  # \n",
       "---\n",
       "sex                  : enum('M','F','U')            # \n",
       "subject_birth_date   : date                         # \n",
       "subject_description=\"\" : varchar(1024)                # "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subject.Subject.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "subject.Subject.insert1(dict(subject='subject3', \n",
    "                             sex='F', \n",
    "                             subject_birth_date='2020-01-01', \n",
    "                             subject_description='Scanbox acquisition. Suite2p processing.'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert an entry into `lab.Equipment`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "Equipment.insert1(dict(scanner='Scanbox'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert an entry into `session.Session`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> Subject\n",
      "session_datetime     : datetime(3)                  \n",
      "\n"
     ]
    }
   ],
   "source": [
    "session.Session.describe();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \n",
       "subject              : varchar(32)                  # \n",
       "session_datetime     : datetime(3)                  # "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.Session.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b></b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject3</td>\n",
       "<td>2021-04-30 12:22:15.032000</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet\n",
       "+----------+ +------------+\n",
       "subject3     2021-04-30 12:\n",
       " (Total: 1)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session_key = dict(subject='subject3', session_datetime='2021-04-30 12:22:15.032')\n",
    "\n",
    "session.Session.insert1(session_key)\n",
    "\n",
    "session.Session()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert an entry into `session.SessionDirectory`\n",
    "\n",
    "+ The `session_dir` is the relative path to the `imaging_root_data_dir` for the given session, in POSIX format with `/`.\n",
    "\n",
    "+ Instead of a relative path, `session_dir` could be an absolute path but it is not recommended as the absolute path would have to match the `imaging_root_data_dir` in `dj_local_conf.json`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-> Session\n",
      "---\n",
      "session_dir          : varchar(256)                 # Path to the data directory for a particular session\n",
      "\n"
     ]
    }
   ],
   "source": [
    "session.SessionDirectory.describe();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \n",
       "subject              : varchar(32)                  # \n",
       "session_datetime     : datetime(3)                  # \n",
       "---\n",
       "session_dir          : varchar(256)                 # Path to the data directory for a particular session"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.SessionDirectory.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b></b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">session_dir</p>\n",
       "                            <span class=\"djtooltiptext\">Path to the data directory for a particular session</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject3</td>\n",
       "<td>2021-04-30 12:22:15.032000</td>\n",
       "<td>subject3/210107_run00_orientation_8dir</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet session_dir   \n",
       "+----------+ +------------+ +------------+\n",
       "subject3     2021-04-30 12: subject3/21010\n",
       " (Total: 1)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.SessionDirectory.insert1(dict(subject='subject3', \n",
    "                                      session_datetime='2021-04-30 12:22:15.032', \n",
    "                                      session_dir='subject3/210107_run00_orientation_8dir'))\n",
    "\n",
    "session.SessionDirectory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert an entry into `scan.Scan`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# \n",
       "subject              : varchar(32)                  # \n",
       "session_datetime     : datetime(3)                  # \n",
       "scan_id              : int                          # \n",
       "---\n",
       "scanner              : varchar(32)                  # \n",
       "acq_software         : varchar(24)                  # \n",
       "scan_notes=\"\"        : varchar(4095)                # free-notes"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scan.Scan.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b></b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">scan_id</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">scanner</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">acq_software</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">scan_notes</p>\n",
       "                            <span class=\"djtooltiptext\">free-notes</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject3</td>\n",
       "<td>2021-04-30 12:22:15.032000</td>\n",
       "<td>0</td>\n",
       "<td>Scanbox</td>\n",
       "<td>Scanbox</td>\n",
       "<td></td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *scan_id    scanner     acq_software   scan_notes    \n",
       "+----------+ +------------+ +---------+ +---------+ +------------+ +------------+\n",
       "subject3     2021-04-30 12: 0           Scanbox     Scanbox                      \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scan.Scan.insert1(dict(subject='subject3', \n",
    "                       session_datetime='2021-04-30 12:22:15.032', \n",
    "                       scan_id=0, \n",
    "                       scanner='Scanbox', \n",
    "                       acq_software='Scanbox',\n",
    "                       scan_notes=''))\n",
    "scan.Scan()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate `scan.ScanInfo`\n",
    "\n",
    "+ This imported table stores information about the acquired image (e.g. image dimensions, file paths, etc.).\n",
    "+ `populate` automatically calls `make` for every key for which the auto-populated table is missing data.\n",
    "+ `populate_settings` passes arguments to the `populate` method.\n",
    "+ `display_progress=True` reports the progress bar"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# general data about the reso/meso scans, from ScanImage header\n",
      "-> scan.Scan\n",
      "---\n",
      "nfields              : tinyint                      # number of fields\n",
      "nchannels            : tinyint                      # number of channels\n",
      "ndepths              : int                          # Number of scanning depths (planes)\n",
      "nframes              : int                          # number of recorded frames\n",
      "nrois                : tinyint                      # number of ROIs (see scanimage's multi ROI imaging)\n",
      "x                    : float                        # (um) ScanImage's 0 point in the motor coordinate system\n",
      "y                    : float                        # (um) ScanImage's 0 point in the motor coordinate system\n",
      "z                    : float                        # (um) ScanImage's 0 point in the motor coordinate system\n",
      "fps                  : float                        # (Hz) frames per second - Volumetric Scan Rate\n",
      "bidirectional        : tinyint                      # true = bidirectional scanning\n",
      "usecs_per_line=null  : float                        # microseconds per scan line\n",
      "fill_fraction=null   : float                        # raster scan temporal fill fraction (see scanimage)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "scan.ScanInfo.describe();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "# general data about the reso/meso scans, from ScanImage header\n",
       "subject              : varchar(32)                  # \n",
       "session_datetime     : datetime(3)                  # \n",
       "scan_id              : int                          # \n",
       "---\n",
       "nfields              : tinyint                      # number of fields\n",
       "nchannels            : tinyint                      # number of channels\n",
       "ndepths              : int                          # Number of scanning depths (planes)\n",
       "nframes              : int                          # number of recorded frames\n",
       "nrois                : tinyint                      # number of ROIs (see scanimage's multi ROI imaging)\n",
       "x                    : float                        # (um) ScanImage's 0 point in the motor coordinate system\n",
       "y                    : float                        # (um) ScanImage's 0 point in the motor coordinate system\n",
       "z                    : float                        # (um) ScanImage's 0 point in the motor coordinate system\n",
       "fps                  : float                        # (Hz) frames per second - Volumetric Scan Rate\n",
       "bidirectional        : tinyint                      # true = bidirectional scanning\n",
       "usecs_per_line=null  : float                        # microseconds per scan line\n",
       "fill_fraction=null   : float                        # raster scan temporal fill fraction (see scanimage)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scan.ScanInfo.heading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "populate_settings = {'display_progress': True}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ScanInfo: 100%|██████████| 1/1 [00:00<00:00, 38.28it/s]{'scanning_mode': 'bidirectional', 'frame_rate': 7.8125, 'num_frames': 7530, 'num_channels': 1, 'num_planes': 4, 'frame_size': array([512, 796], dtype=uint16), 'num_target_frames': 0, 'num_stored_frames': 30123, 'stage_pos': [0, 0, -311.71], 'stage_angle': 9.65, 'etl_pos': [203, 255, 314, 379], 'filename': 'run00_orientation_8dir_000_000.sbx', 'resonant_freq': 8000, 'scanbox_version': 3, 'records_per_buffer': 256, 'magnification': 1.7, 'um_per_pixel_x': nan, 'um_per_pixel_y': nan, 'objective': 'Nikon_16x_dlr', 'messages': array([], dtype=object), 'event_id': array([], dtype=uint8), 'usernotes': array([], dtype='<U1'), 'ballmotion': array([], dtype='<U1')}\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    \n",
       "    <style type=\"text/css\">\n",
       "        .Relation{\n",
       "            border-collapse:collapse;\n",
       "        }\n",
       "        .Relation th{\n",
       "            background: #A0A0A0; color: #ffffff; padding:4px; border:#f0e0e0 1px solid;\n",
       "            font-weight: normal; font-family: monospace; font-size: 100%;\n",
       "        }\n",
       "        .Relation td{\n",
       "            padding:4px; border:#f0e0e0 1px solid; font-size:100%;\n",
       "        }\n",
       "        .Relation tr:nth-child(odd){\n",
       "            background: #ffffff;\n",
       "        }\n",
       "        .Relation tr:nth-child(even){\n",
       "            background: #f3f1ff;\n",
       "        }\n",
       "        /* Tooltip container */\n",
       "        .djtooltip {\n",
       "        }\n",
       "        /* Tooltip text */\n",
       "        .djtooltip .djtooltiptext {\n",
       "            visibility: hidden;\n",
       "            width: 120px;\n",
       "            background-color: black;\n",
       "            color: #fff;\n",
       "            text-align: center;\n",
       "            padding: 5px 0;\n",
       "            border-radius: 6px;\n",
       "            /* Position the tooltip text - see examples below! */\n",
       "            position: absolute;\n",
       "            z-index: 1;\n",
       "        }\n",
       "        #primary {\n",
       "            font-weight: bold;\n",
       "            color: black;\n",
       "        }\n",
       "        #nonprimary {\n",
       "            font-weight: normal;\n",
       "            color: white;\n",
       "        }\n",
       "\n",
       "        /* Show the tooltip text when you mouse over the tooltip container */\n",
       "        .djtooltip:hover .djtooltiptext {\n",
       "            visibility: visible;\n",
       "        }\n",
       "    </style>\n",
       "    \n",
       "    <b>general data about the reso/meso scans, from ScanImage header</b>\n",
       "        <div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "        <table border=\"1\" class=\"Relation\">\n",
       "            <thead> <tr style=\"text-align: right;\"> <th> <div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">subject</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">session_datetime</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"primary\">scan_id</p>\n",
       "                            <span class=\"djtooltiptext\"></span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">nfields</p>\n",
       "                            <span class=\"djtooltiptext\">number of fields</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">nchannels</p>\n",
       "                            <span class=\"djtooltiptext\">number of channels</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">ndepths</p>\n",
       "                            <span class=\"djtooltiptext\">Number of scanning depths (planes)</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">nframes</p>\n",
       "                            <span class=\"djtooltiptext\">number of recorded frames</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">nrois</p>\n",
       "                            <span class=\"djtooltiptext\">number of ROIs (see scanimage's multi ROI imaging)</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">x</p>\n",
       "                            <span class=\"djtooltiptext\">(um) ScanImage's 0 point in the motor coordinate system</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">y</p>\n",
       "                            <span class=\"djtooltiptext\">(um) ScanImage's 0 point in the motor coordinate system</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">z</p>\n",
       "                            <span class=\"djtooltiptext\">(um) ScanImage's 0 point in the motor coordinate system</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">fps</p>\n",
       "                            <span class=\"djtooltiptext\">(Hz) frames per second - Volumetric Scan Rate</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">bidirectional</p>\n",
       "                            <span class=\"djtooltiptext\">true = bidirectional scanning</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">usecs_per_line</p>\n",
       "                            <span class=\"djtooltiptext\">microseconds per scan line</span>\n",
       "                        </div></th><th><div class=\"djtooltip\">\n",
       "                            <p id=\"nonprimary\">fill_fraction</p>\n",
       "                            <span class=\"djtooltiptext\">raster scan temporal fill fraction (see scanimage)</span>\n",
       "                        </div> </th> </tr> </thead>\n",
       "            <tbody> <tr> <td>subject3</td>\n",
       "<td>2021-04-30 12:22:15.032000</td>\n",
       "<td>0</td>\n",
       "<td>4</td>\n",
       "<td>1</td>\n",
       "<td>4</td>\n",
       "<td>7530</td>\n",
       "<td>0</td>\n",
       "<td>0.0</td>\n",
       "<td>0.0</td>\n",
       "<td>-311.71</td>\n",
       "<td>7.8125</td>\n",
       "<td>0</td>\n",
       "<td>nan</td>\n",
       "<td>nan</td> </tr> </tbody>\n",
       "        </table>\n",
       "        \n",
       "        <p>Total: 1</p></div>\n",
       "        "
      ],
      "text/plain": [
       "*subject     *session_datet *scan_id    nfields     nchannels     ndepths     nframes     nrois     x       y       z           fps        bidirectional  usecs_per_line fill_fraction \n",
       "+----------+ +------------+ +---------+ +---------+ +-----------+ +---------+ +---------+ +-------+ +-----+ +-----+ +---------+ +--------+ +------------+ +------------+ +------------+\n",
       "subject3     2021-04-30 12: 0           4           1             4           7530        0         0.0     0.0     -311.71     7.8125     0              nan            nan           \n",
       " (Total: 1)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scan.ScanInfo.populate(**populate_settings)\n",
    "scan.ScanInfo()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Insert a new entry into `imaging.ProcessingParamSet` for Suite2p or CaImAn\n",
    "\n",
    "+ Define and insert the parameters that will be used for the Suite2p or CaImAn processing.\n",
    "\n",
    "+ This step is not needed if you are using an existing ProcessingParamSet.\n",
    "\n",
    "### Define Suite2p parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "params_suite2p = {'look_one_level_down': 0.0,\n",
    "                  'fast_disk': [],\n",
    "                  'delete_bin': False,\n",
    "                  'mesoscan': False,\n",
    "                  'h5py': [],\n",
    "                  'h5py_key': 'data',\n",
    "                  'save_path0': [],\n",
    "                  'subfolders': [],\n",
    "                  'nplanes': 1,\n",
    "                  'nchannels': 1,\n",
    "                  'functional_chan': 1,\n",
    "                  'tau': 1.0,\n",
    "                  'fs': 10.0,\n",
    "                  'force_sktiff': False,\n",
    "                  'preclassify': 0.0,\n",
    "                  'save_mat': False,\n",
    "                  'combined': True,\n",
    "                  'aspect': 1.0,\n",
    "                  'do_bidiphase': False,\n",
    "                  'bidiphase': 0.0,\n",
    "                  'do_registration': True,\n",
    "                  'keep_movie_raw': False,\n",
    "                  'nimg_init': 300,\n",
    "                  'batch_size': 500,\n",
    "                  'maxregshift': 0.1,\n",
    "                  'align_by_chan': 1,\n",
    "                  'reg_tif': False,\n",
    "                  'reg_tif_chan2': False,\n",
    "                  'subpixel': 10,\n",
    "                  'smooth_sigma': 1.15,\n",
    "                  'th_badframes': 1.0,\n",
    "                  'pad_fft': False,\n",
    "                  'nonrigid': True,\n",
    "                  'block_size': [128, 128],\n",
    "                  'snr_thresh': 1.2,\n",
    "                  'maxregshiftNR': 5.0,\n",
    "                  '1Preg': False,\n",
    "                  'spatial_hp': 50.0,\n",
    "                  'pre_smooth': 2.0,\n",
    "                  'spatial_taper': 50.0,\n",
    "                  'roidetect': True,\n",
    "                  'sparse_mode': False,\n",
    "                  'diameter': 12,\n",
    "                  'spatial_scale': 0,\n",
    "                  'connected': True,\n",
    "                  'nbinned': 5000,\n",
    "                  'max_iterations': 20,\n",
    "                  'threshold_scaling': 1.0,\n",
    "                  'max_overlap': 0.75,\n",
    "                  'high_pass': 100.0,\n",
    "                  'inner_neuropil_radius': 2,\n",
    "                  'min_neuropil_pixels': 350,\n",
    "                  'allow_overlap': False,\n",
    "                  'chan2_thres': 0.65,\n",
    "                  'baseline': 'maximin',\n",
    "                  'win_baseline': 60.0,\n",
    "                  'sig_baseline': 10.0,\n",
    "                  'prctile_baseline': 8.0,\n",
    "                  'neucoeff': 0.7,\n",
    "                  'xrange': np.array([0, 0]),\n",
    "                  'yrange': np.array([0, 0])}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Insert Suite2p parameters\n",
    "\n",
    "+ A method of the class `ProcessingParamset` called `insert_new_params` is a helper function to insert the Suite2p or CaIman parameters and ensures that the parameter set inserted is not duplicated."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "imaging.ProcessingParamSet.insert_new_params(\n",
    "    processing_method='suite2p', \n",
    "    paramset_idx=0, \n",
    "    params=params_suite2p,\n",
    "    paramset_desc='Calcium imaging analysis with Suite2p using default Suite2p parameters')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert new ProcessingTask to trigger ingestion of motion correction and segmentation results\n",
    "\n",
    "+ Motion correction and segmentation are performed for each scan in Suite2p or CaImAn.\n",
    "\n",
    "+ An entry in `ProcessingTask` indicates a set of motion correction and segmentation results (generated from Suite2p or CaImAn outside of `workflow-calcium-imaging`) are ready to be ingested.  In a future release, an entry in `ProcessingTask` can also indicate a new processing job (using Suite2p or CaImAn) is to be triggered.\n",
    "\n",
    "+ Two pieces of information need to be specified:\n",
    "\n",
    "    + The `paramset_idx` is the parameter set stored in `imaging.ProcessingParamSet` that is used for the Suite2p or CaImAn processing job.\n",
    "    \n",
    "    + The `processing_output_dir` stores the directory of the processing results (relative to the imaging root data directory)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "imaging.ProcessingTask.insert1(dict(subject='subject3', \n",
    "                                    session_datetime='2021-04-30 12:22:15.032', \n",
    "                                    scan_id=0,\n",
    "                                    paramset_idx=0,\n",
    "                                    processing_output_dir='subject3/210107_run00_orientation_8dir/suite2p'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate `imaging.Processing`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Processing: 100%|██████████| 1/1 [00:00<00:00, 68.04it/s]\n"
     ]
    }
   ],
   "source": [
    "imaging.Processing.populate(**populate_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Insert new Curation following the ProcessingTask\n",
    "\n",
    "+ The next step in the pipeline is the curation of motion corection and segmentation results.\n",
    "\n",
    "+ If a manual curation was implemented, an entry needs to be manually inserted into the table `imaging.Curation`, which specifies the directory to the curated results in `curation_output_dir`. \n",
    "\n",
    "+ If we would like to use the processed outcome directly, an entry is also needed in `imaging.Curation`. A method `create1_from_processing_task` was provided to help this insertion. It copies the `processing_output_dir` in `imaging.ProcessingTask` to the field `curation_output_dir` in the table `imaging.Curation` with a new `curation_id`.\n",
    "\n",
    "    + In this example, we create/insert one `imaging.Curation` for each `imaging.ProcessingTask`, specifying the same output directory.\n",
    "\n",
    "    + To this end, we could also make use of a convenient function `imaging.Curation().create1_from_processing_task()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "imaging.Curation.insert1(dict(subject='subject3', \n",
    "                              session_datetime='2021-04-30 12:22:15.032', \n",
    "                              scan_id=0,\n",
    "                              paramset_idx=0,\n",
    "                              curation_id=0,\n",
    "                              curation_time='2021-04-30 12:22:15.032', \n",
    "                              curation_output_dir='subject3/210107_run00_orientation_8dir/suite2p',\n",
    "                              manual_curation=False,\n",
    "                              curation_note=''))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate `imaging.MotionCorrection`\n",
    "\n",
    "+ This table contains the rigid or non-rigid motion correction data including the shifts and summary images.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "MotionCorrection: 100%|██████████| 1/1 [00:06<00:00,  6.50s/it]\n"
     ]
    }
   ],
   "source": [
    "imaging.MotionCorrection.populate(**populate_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate `imaging.Segmentation`\n",
    "\n",
    "+ This table contains the mask coordinates, weights, and centers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Segmentation: 100%|██████████| 1/1 [00:03<00:00,  3.28s/it]\n"
     ]
    }
   ],
   "source": [
    "imaging.Segmentation.populate(**populate_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add another set of results from a new round of curation\n",
    "\n",
    "If you performed curation on an existing processed results (i.e. motion correction or segmentation) then:\n",
    "    \n",
    "+ Add an entry into `imaging.Curation` with the directory of the curated results and a new `curation_id`.\n",
    "\n",
    "+ Populate the `imaging.MotionCorrection` and `imaging.Segmentation` tables again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate `imaging.MaskClassification`\n",
    "\n",
    "+ This table contains the classification of the segmented masks and the confidence of classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "MaskClassification: 100%|██████████| 1/1 [00:00<00:00, 385.65it/s]\n"
     ]
    }
   ],
   "source": [
    "imaging.MaskClassification.populate(**populate_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate `imaging.Fluorescence`\n",
    "\n",
    "+ This table contains the fluorescence traces prior to filtering and spike extraction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Fluorescence: 100%|██████████| 1/1 [00:49<00:00, 49.05s/it]\n"
     ]
    }
   ],
   "source": [
    "imaging.Fluorescence.populate(**populate_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Populate `imaging.Activity`\n",
    "+ This table contains the inferred neural activity from the fluorescence traces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Activity: 100%|██████████| 1/1 [00:07<00:00,  7.89s/it]\n"
     ]
    }
   ],
   "source": [
    "imaging.Activity.populate(**populate_settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "\n",
    "+ Proceed to the [05-explore](05-explore.ipynb) to learn how to  query, fetch, and visualize the imaging data."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.12 ('elementsPractice')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "28657e2c4c8fc87d9c51bab987cae00550904d22e795ba7dc84df7e29d09653b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}