Update tutorial, README, license

kushalbakshi · Nov 28, 2023 · ce9a185 · ce9a185
1 parent 9f321ee
commit ce9a185
Show file tree

Hide file tree

Showing 3 changed files with 110 additions and 49 deletions.
diff --git a/LICENSE b/LICENSE
@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2022 DataJoint
+Copyright (c) 2023 DataJoint
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

diff --git a/README.md b/README.md
@@ -7,7 +7,8 @@ system, and analyzed with [CaImAn](https://github.com/flatironinstitute/CaImAn).
 DataJoint Elements collectively standardize and automate data collection and analysis 
 for neuroscience experiments. Each Element is a modular pipeline for data storage and 
 processing with corresponding database tables that can be combined with other Elements 
-to assemble a fully functional pipeline.
+to assemble a fully functional pipeline. This repository also provides a tutorial
+environment and notebooks to learn the pipeline.
 
 ## Experiment Flowchart
 
@@ -19,16 +20,63 @@ to assemble a fully functional pipeline.
 
 ## Getting Started
 
-+ Install from PyPI
++ Please fork this repository.
+
++ Clone the repository to your computer.
+
+```bash
+     git clone https://github.com/<enter_github_username>/element-miniscope.git
+```
+
++ Install with `pip`:
 
      ```bash
-     pip install element-miniscope
+     pip install -e .
      ```
-
-+ [Interactive tutorial on GitHub Codespaces](https://github.com/datajoint/workflow-miniscope#interactive-tutorial)
+
++ [Interactive tutorial on GitHub
+  Codespaces](https://github.com/datajoint/element-miniscope#interactive-tutorial)
 
 + [Documentation](https://datajoint.com/docs/elements/element-miniscope)
 
 ## Support
 
-+ If you need help getting started or run into any errors, please contact our team by email at [email protected].
++ If you need help getting started or run into any errors, please contact our team by
+  email at [email protected].
+
+## Interactive Tutorial
+
++ The easiest way to learn about DataJoint Elements is to use the tutorial notebooks within the included interactive environment configured using [Dev Container](https://containers.dev/).
+
+### Launch Environment
+
+Here are some options that provide a great experience:
+
+- (*recommended*) Cloud-based Environment
+  - Launch using [GitHub Codespaces](https://github.com/features/codespaces) using the `+` option which will `Create codespace on main` in the codebase repository on your fork with default options. For more control, see the `...` where you may create `New with options...`.
+  - Build time for a codespace is a few minutes. This is done infrequently and cached for convenience.
+  - Start time for a codespace is less than 1 minute. This will pull the built codespace from cache when you need it.
+  - *Tip*: Each month, GitHub renews a [free-tier](https://docs.github.com/en/billing/managing-billing-for-github-codespaces/about-billing-for-github-codespaces#monthly-included-storage-and-core-hours-for-personal-accounts) quota of compute and storage. Typically we run into the storage limits before anything else since Codespaces consume storage while stopped. It is best to delete Codespaces when not actively in use and recreate when needed. We'll soon be creating prebuilds to avoid larger build times. Once any portion of your quota is reached, you will need to wait for it to be reset at the end of your cycle or add billing info to your GitHub account to handle overages.
+  - *Tip*: GitHub auto names the codespace but you can rename the codespace so that it is easier to identify later.
+
+- Local Environment
+  > *Note: Access to example data is currently limited to MacOS and Linux due to the s3fs utility. Windows users are recommended to use the above environment.*
+  - Install [Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+  - Install [Docker](https://docs.docker.com/get-docker/)
+  - Install [VSCode](https://code.visualstudio.com/)
+  - Install the VSCode [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
+  - `git clone` the codebase repository and open it in VSCode
+  - Use the `Dev Containers extension` to `Reopen in Container` (More info is in the `Getting started` included with the extension.)
+
+You will know your environment has finished loading once you either see a terminal open related to `Running postStartCommand` with a final message of `Done` or the `README.md` is opened in `Preview`.
+
+Once the environment has launched, please run the following command in the terminal:
+```
+MYSQL_VER=8.0 docker compose -f docker-compose-db.yaml up --build -d
+```
+
+### Instructions
+
+1. We recommend you start by navigating to the `notebooks` directory on the left panel and go through the `tutorial.ipynb` Jupyter notebook. Execute the cells in the notebook to begin your walk through of the tutorial.
+
+2. Once you are done, see the options available to you in the menu in the bottom-left corner. For example, in Codespace you will have an option to `Stop Current Codespace` but when running Dev Container on your own machine the equivalent option is `Reopen folder locally`. By default, GitHub will also automatically stop the Codespace after 30 minutes of inactivity.  Once the Codespace is no longer being used, we recommend deleting the Codespace.
diff --git a/notebooks/tutorial.ipynb b/notebooks/tutorial.ipynb
@@ -13,43 +13,44 @@
     "tutorial aims to provide a comprehensive understanding of the open-source data pipeline\n",
     "created using `element-miniscope`. \n",
     "\n",
-    "The package is designed to seamlessly ingest and track scan-level metadata, image\n",
-    "processing, and curation results across subjects and sessions. By the end of this tutorial you will have a clear grasp on setting up and integrating\n",
-    "`element-miniscope` into your specific research projects and your lab.\n",
+    "The package is designed to seamlessly process, ingest, and track miniscope data\n",
+    "including scan-level metadata, image processing, and curation results across subjects\n",
+    "and sessions. By the end of this tutorial you will have a clear grasp on setting up and\n",
+    "integrating `element-miniscope` into your specific research projects and your lab.\n",
     "\n",
     "\n",
     "![flowchart](https://raw.githubusercontent.com/datajoint/element-miniscope/main/images/flowchart.svg)\n",
     "\n",
     "### Prerequisites\n",
     "\n",
-    "1. This tutorial assumes you already understand different table types in\n",
-    "   `datajoint-python`. \n",
-    "2. This tutorial assumes you have a basic understanding of the functionality of the\n",
-    "   `.populate()` method used in `datajoint-python` to automatically insert data into\n",
-    "   `Imported` and `Computed` tables.  \n",
-    "\n",
     "Please see the [datajoint tutorials GitHub\n",
-    "repository](https://github.com/datajoint/datajoint-tutorials/tree/main) before proceeding.\n",
-    "\n",
-    "**Key Components and Objectives**\n",
-    "\n",
-    "+ **Setup**\n",
-    "+ **Activate the DataJoint pipeline.**\n",
-    "1. **Insert session-level metadata.**\n",
-    "2. **Extract scan-level (recording-level) metadata.**\n",
-    "3. **Perform image processing using\n",
-    "   [`CaImAn`](https://github.com/flatironinstitute/CaImAn).**\n",
-    "4. **Curate the results (optional).**\n",
-    "5. **Visualize the results.**\n",
+    "repository](https://github.com/datajoint/datajoint-tutorials/tree/main) before\n",
+    "proceeding.\n",
+    "\n",
+    "A basic understanding of the following DataJoint concepts will be beneficial to your\n",
+    "understanding of this tutorial: \n",
+    "1. The `Imported` and `Computed` tables types in `datajoint-python`.\n",
+    "2. The functionality of the `.populate()` method. \n",
+    "\n",
+    "#### **Tutorial Overview**\n",
+    "\n",
+    "+ Setup\n",
+    "+ *Activate* the DataJoint pipeline.\n",
+    "+ *Insert* subject and session metadata.\n",
+    "+ *Populate* scan-level (recording-level) metadata.\n",
+    "+ Run image processing using\n",
+    "   [`CaImAn`](https://github.com/flatironinstitute/CaImAn).\n",
+    "+ Curate the results (optional).\n",
+    "+ Visualize the results.\n",
     "\n",
-    "#### Setup\n",
+    "### **Setup**\n",
     "\n",
     "This tutorial examines data from a UCLA miniscope recording acquired using the Miniscope\n",
     "DAQ V4. The goal is to extract fluorescence and deconvolved activity traces and use\n",
     "these traces to study neuronal calcium dynamics and neural activity during\n",
     "specific behaviors or tasks.\n",
     "\n",
-    "The results of this element can be combined with **other modalities** to create a\n",
+    "The results of this Element can be combined with **other modalities** to create a\n",
     "complete, customizable data pipeline for your specific lab or study. For instance, you\n",
     "can combine `element-miniscope` with `element-calcium-imaging` or `element-deeplabcut`\n",
     "to characterize neural activity along with markerless pose-estimation during behavior.\n",
@@ -73,8 +74,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "If the tutorial is run in Codespaces, there is a local database private to you for\n",
-    "experimentation. Let's connect to the database server."
+    "If the tutorial is run in Codespaces, a private, local database server is created and\n",
+    "made available for you. This is where we will insert and store our processed results.\n",
+    "Let's connect to the database server."
    ]
   },
   {
@@ -90,7 +92,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### **Activate the DataJoint Pipeline**\n",
+    "### **Activate the DataJoint Pipeline**\n",
     "\n",
     "This tutorial activates the `miniscope.py` module from `element-miniscope`, along with\n",
     "upstream depedencies from `element-animal` and `element-session`. Please refer to the\n",
@@ -131,11 +133,11 @@
     "recording-level (scan-level) metadata, image processing results, and optional curation\n",
     "of image processing. A few tables such as `subject.Subject` or `session.Session` are\n",
     "inherited from `element-animal` and `element-session`, respectively, and while important\n",
-    "for a complete pipeline, fall outside the scope of `element-miniscope` and will\n",
-    "therefore, not be explored extensively here. The primary focus of this tutorial is on\n",
-    "the `miniscope` schema.\n",
+    "for a complete pipeline, fall outside the scope of the `element-miniscope` tutorial and\n",
+    "will therefore, not be explored extensively here. The primary focus of this tutorial is\n",
+    "on the `miniscope` schema.\n",
     "\n",
-    "##### 1. **Insert session-level metadata.**\n",
+    "### **Insert subject and session metadata**\n",
     "\n",
     "Let's start with the first table in the schema diagram (i.e. `subject.Subject` table).\n",
     "\n",
@@ -258,8 +260,6 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##### 2. **Extract Scan-level Metadata.**\n",
-    "\n",
     "Every experimental session produces a set of data files. The `SessionDirectory` table's\n",
     "purpose is to locate these files. It references a directory path relative to a root\n",
     "directory, defined in `dj.config[\"custom\"]`. More\n",
@@ -348,6 +348,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "### **Populate scan-level (recording-level) metadata**\n",
+    "\n",
     "In the upcoming cells, we'll make use of the `populate()` method to fill the `miniscope.RecordingInfo` table and its part table."
    ]
   },
@@ -409,7 +411,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##### 3. **Perform Image Processing using CaImAn.**\n",
+    "### **Run Image Processing using CaImAn**\n",
     "\n",
     "We're almost ready to perform image processing with `CaImAn`. An important step before\n",
     "processing is managing the parameters which will be used in that step. To do so, we will\n",
@@ -613,7 +615,7 @@
     "invoke `element-event` or define your custom DataJoint tables to extend the\n",
     "pipeline.\n",
     "\n",
-    "##### 5. **Visualize the results.**\n",
+    "### **Visualize the results.**\n",
     "\n",
     "In this tutorial, we will do some exploratory analysis by fetching the data from the database and creating a few plots."
    ]
@@ -725,18 +727,29 @@
     "\n",
     "+ Covered the essential functionality of `element-miniscope`.\n",
     "+ Learned how to manually insert data into tables.\n",
-    "+ Execute image processing with `CaImAn`.\n",
-    "+ Visualize the results.\n",
+    "+ Executed and ingested results of image processing with `CaImAn`.\n",
+    "+ Visualized the results.\n",
+    "\n",
+    "#### Documentation and DataJoint Tutorials\n",
+    "\n",
+    "+ [Detailed documentation on\n",
+    "  `element-miniscope`.](https://datajoint.com/docs/elements/element-miniscope/)\n",
+    "+ [General `datajoint-python`\n",
+    "  tutorials.](https://github.com/datajoint/datajoint-tutorials) covering fundamentals,\n",
+    "  such as table tiers, query operations, fetch operations, automated computations with the\n",
+    "  make function, and more.\n",
+    "+ [Documentation for\n",
+    "  `datajoint-python`.](https://datajoint.com/docs/core/datajoint-python/)\n",
     "\n",
-    "##### Run this tutorial on your own data.\n",
+    "##### Run this tutorial on your own data\n",
     "\n",
     "To run this tutorial notebook on your own data, please use the following steps:\n",
-    "- Download the [mysql-docker image for\n",
+    "+ Download the [mysql-docker image for\n",
     "  DataJoint](https://github.com/datajoint/mysql-docker) and run the container according\n",
     "  to the instructions provide in the repository.\n",
-    "- Create a fork of this repository to your GitHub account.\n",
-    "- Clone the repository and open the files using your IDE.\n",
-    "- Add a code cell immediately after the first code cell in the notebook - we will setup\n",
+    "+ Create a fork of this repository to your GitHub account.\n",
+    "+ Clone the repository and open the files using your IDE.\n",
+    "+ Add a code cell immediately after the first code cell in the notebook - we will setup\n",
     "  the local connection using this cell. In this cell, type in the following code. \n",
     "\n",
     "```python\n",
@@ -750,7 +763,7 @@
     "dj.conn()\n",
     "```\n",
     "\n",
-    "- Run this code block above and proceed with the rest of the notebook."
+    "+ Run the code block above and proceed with the rest of the notebook."
    ]
   },
   {