Merge pull request #168 from neurodata/develop

Merge v0.2.1

.gitattributes

@@ -0,0 +1 @@
+*.ipynb  linguist-vendored=true

brainlit/__init__.py

@@ -13,4 +13,4 @@
 warnings.simplefilter("always", category=UserWarning)
 
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"

brainlit/archive/upload_skeleton.py

@@ -3,7 +3,7 @@
 import argparse
 import numpy as np
 from cloudvolume import CloudVolume, Skeleton, storage
-from .swc import swc2skeleton
+from brainlit.utils.swc import swc2skeleton
 import pandas as pd
 from pathlib import Path
 import tifffile as tf

docs/README.rst

@@ -0,0 +1,209 @@
+.. role:: raw-html-m2r(raw)
+   :format: html
+
+
+Brainlit
+========
+
+
+.. image:: https://travis-ci.com/neurodata/brainlit.svg?branch=master
+   :target: https://travis-ci.com/neurodata/brainlit
+   :alt: Build Status
+
+
+.. image:: https://badge.fury.io/py/brainlit.svg
+   :target: https://badge.fury.io/py/brainlit
+   :alt: PyPI version
+
+
+.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
+   :target: https://github.com/psf/black
+   :alt: Code style: black
+
+
+.. image:: https://codecov.io/gh/neurodata/brainlit/branch/master/graph/badge.svg
+   :target: https://codecov.io/gh/neurodata/brainlit
+   :alt: codecov
+
+
+.. image:: https://img.shields.io/docker/cloud/build/bvarjavand/brainlit
+   :target: https://img.shields.io/docker/cloud/build/bvarjavand/brainlit
+   :alt: Docker Cloud Build Status
+
+
+.. image:: https://img.shields.io/docker/image-size/bvarjavand/brainlit
+   :target: https://img.shields.io/docker/image-size/bvarjavand/brainlit
+   :alt: Docker Image Size (latest by date)
+
+
+.. image:: https://img.shields.io/badge/License-Apache%202.0-blue.svg
+   :target: https://opensource.org/licenses/Apache-2.0
+   :alt: License
+
+
+This repository is a container of methods that Neurodata usees to expose their open-source code while it is in the process of being merged with larger scientific libraries such as scipy, scikit-image, or scikit-learn. Additionally, methods for computational neuroscience on brains too specific for a general scientific library can be found here, such as image registration software tuned specifically for large brain volumes.
+
+
+.. image:: https://i.postimg.cc/QtG9Xs68/Brainlit.png
+   :target: https://i.postimg.cc/QtG9Xs68/Brainlit.png
+   :alt: Brainlight Features
+
+
+.. toctree::
+   :numbered:
+
+
+Motivation
+----------
+
+The repository originated as the project of a team in Joshua Vogelstein's class **Neurodata** at Johns Hopkins University. This project was focused on data science towards the `mouselight data <https://www.hhmi.org/news/mouselight-project-maps-1000-neurons-and-counting-in-the-mouse-brain>`_. It becme apparent that the tools developed for the class would be useful for other groups doing data science on large data volumes.
+The repository can now be considered a "holding bay" for code developed by Neurodata for collaborators and researchers to use.
+
+Installation
+------------
+
+Environment
+^^^^^^^^^^^
+
+(optional, any python >= 3.8 environment will suffice)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+
+* `get conda <https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html>`_
+* create a virtual environment: ``conda create --name brainlit python=3.8``
+* activate the environment: ``conda activate brainlit``
+
+Install from pypi
+^^^^^^^^^^^^^^^^^
+
+
+* install brainlit: ``pip install brainlit``
+
+Install from source
+^^^^^^^^^^^^^^^^^^^
+
+
+* clone the repo: ``git clone https://github.com/neurodata/brainlit.git``
+* cd into the repo: ``cd brainlit``
+* install brainlit: ``pip install -e .``
+
+How to use Brainlit
+-------------------
+
+Data setup
+^^^^^^^^^^
+
+The ``source`` data directory should have an octree data structure
+
+.. code-block::
+
+    data/
+   ├── default.0.tif
+   ├── transform.txt
+   ├── 1/
+   │   ├── 1/, ..., 8/
+   │   └── default.0.tif
+   ├── 2/ ... 8/
+   └── consensus-swcs (optional)
+       ├── G-001.swc
+       ├── G-002.swc
+       └── default.0.tif
+
+If your team wants to interact with cloud data, each member will need account credentials specified in ``~/.cloudvolume/secrets/x-secret.json``\ , where ``x`` is one of ``[aws, gc, azure]`` which contains your id and secret key for your cloud platform.
+We provide a template for ``aws`` in the repo for convenience.
+
+Create a session
+^^^^^^^^^^^^^^^^
+
+Each user will start their scripts with approximately the same lines:
+
+.. code-block::
+
+   from brainlit.utils.ngl import NeuroglancerSession
+
+   session = NeuroglancerSession(url='file:///abc123xyz')
+
+From here, any number of tools can be run such as the visualization or annotation tools. `Viz demo <https://github.com/neurodata/brainlit/blob/master/docs/notebooks/visualization/visualization.ipynb>`_.
+
+Features
+--------
+
+Registration
+^^^^^^^^^^^^
+
+The registration subpackage is a facsimile of ARDENT, a pip-installable (pip install ardent) package for nonlinear image registration wrapped in an object-oriented framework for ease of use. This is an implementation of the LDDMM algorithm with modifications, written by Devin Crowley and based on "Diffeomorphic registration with intensity transformation and missing data: Application to 3D digital pathology of Alzheimer's disease." This paper extends on an older LDDMM paper, "Computing large deformation metric mappings via geodesic flows of diffeomorphisms."
+
+This is the more recent paper:
+
+Tward, Daniel, et al. "Diffeomorphic registration with intensity transformation and missing data: Application to 3D digital pathology of Alzheimer's disease." Frontiers in neuroscience 14 (2020).
+
+https://doi.org/10.3389/fnins.2020.00052
+
+This is the original LDDMM paper:
+
+Beg, M. Faisal, et al. "Computing large deformation metric mappings via geodesic flows of diffeomorphisms." International journal of computer vision 61.2 (2005): 139-157.
+
+https://doi.org/10.1023/B:VISI.0000043755.93987.aa
+
+A tutorial is available in docs/notebooks/registration_demo.ipynb.
+
+Core
+----
+
+The core brain-lit package can be described by the diagram at the top of the readme:
+
+(Push and Pull Data)
+^^^^^^^^^^^^^^^^^^^^
+
+Brainlit uses the Seung Lab's `Cloudvolume <https://github.com/seung-lab/cloud-volume>`_ package to push and pull data through the cloud or a local machine in an efficient and parallelized fashion. `Uploading demo <https://github.com/neurodata/brainlit/blob/master/docs/notebooks/utils/uploading_brains.ipynb>`_.\ :raw-html-m2r:`<br>`
+The only requirement is to have an account on a cloud service on s3, azure, or google cloud.
+
+Loading data via local filepath of an octree structure is also supported. `Octree demo <https://github.com/neurodata/brainlit/blob/master/docs/notebooks/utils/upload_brains.ipynb>`_.
+
+Visualize
+^^^^^^^^^
+
+Brainlit supports many methods to visualize large data. Visualizing the entire data can be done via Google's `Neuroglancer <https://github.com/google/neuroglancer>`_\ , which provides a web link as shown below.
+
+screenshot
+
+Brainlit also has tools to visualize chunks of data as 2d slices or as a 3d model. `Visualization demo <https://github.com/neurodata/brainlit/blob/master/docs/notebooks/visualization/visualization.ipynb>`_.
+
+screenshot
+
+Manually Segment
+^^^^^^^^^^^^^^^^
+
+Brainlit includes a lightweight manual segmentation pipeline. This allows collaborators of a projec to pull data from the cloud, create annotations, and push their annotations back up as a separate channel. `Auto demo <https://github.com/neurodata/brainlit/blob/master/docs/notebooks/pipelines/manual_segementation.ipynb>`_.
+
+Automatically and Semi-automatically Segment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Similar to the above pipeline, segmentations can be automatically or semi-automatically generated and pushed to a separate channel for viewing. `Semi-auto demo <https://github.com/neurodata/brainlit/blob/master/docs/notebooks/pipelines/seg_pipeline_demo.ipynb>`_.
+
+API Reference
+-------------
+
+
+.. image:: https://readthedocs.org/projects/brainlight/badge/?version=latest
+   :target: https://brainlight.readthedocs.io/en/latest/?badge=latest
+   :alt: Documentation Status
+
+The documentation can be found at `https://brainlight.readthedocs.io/en/latest/ <https://brainlight.readthedocs.io/en/latest/>`_.
+
+Tests
+-----
+
+Running tests can easily be done by moving to the root directory of the brainlit package ant typing ``pytest tests`` or ``python -m pytest tests``.\ :raw-html-m2r:`<br>`
+Running a specific test, such as ``test_upload.py`` can be done simply by ``ptest tests/test_upload.py``.
+
+Contributing
+------------
+
+Contribution guidelines can be found via `CONTRIBUTING.md <https://github.com/neurodata/brainlit/blob/master/CONTRIBUTING.md>`_
+
+Credits
+-------
+
+Thanks to the neurodata team and the group in the neurodata class which started the project.
+This project is currently managed by Tommy Athey and Bijan Varjavand.

docs/conf.py

@@ -43,6 +43,9 @@
     "sphinx.ext.intersphinx",
 ]
 
+nbsphinx_kernel_name = "docs"
+nbsphinx_allow_errors = True
+
 autoapi_dirs = ["../brainlit"]
 autoapi_add_toctree_entry = False
 autoapi_generate_api_docs = False

docs/index.rst

@@ -5,15 +5,10 @@
 Overview of Brainlit 
 ====================
 
-Brainlit is a Python package for reading and analyzing brain data.
-The brain data is assumed to consist of image files in an octree structure to handle mutliple resolutions.
-Optionally, the package is able to handle skeletonized axon data stored in a `.swc` file format.
-
-Brainlit is able to handle this data, visualizing and running analysis with morphological and statistical methods.
-A diagram demonstrating the capabilities of the package is shown.
+.. toctree::
+    :maxdepth: 1
 
-.. image:: images/figure.png
-   :width: 600
+    README
 
 Documentation
 =============

docs/notebooks/utils/downloading_brains.ipynb

@@ -132,4 +132,4 @@
    "source": []
   }
  ]
-}
+}

docs/notebooks/utils/utils.ipynb

@@ -33,12 +33,18 @@
     "## Converting image data to neuroglancer precomputed format\n",
     "\n",
     "Image data will be assumed to be stored locally in octree format and at multiple resolutions, such as\n",
-    "```default.0.tif 0/default.0.tif 1/default.0.tif 2/default.0.tif 3/default.0.tif 4/default.0.tif 5/default.0.tif 6/default.0.tif 7/default.0.tif 8/default.0.tif```.\n",
+    "```default.0.tif 0/default.0.tif 1/default.0.tif ... 8/default.0.tif```.\n",
     "\n",
-    "A user only needs to specity a path to the octree top level, and specify the number of resolutions to use.\n",
-    "### The octree path can be modified to generated files from different data"
+    "A user only needs to specity a path to the octree top level, and specify the number of resolutions to use."
    ]
   },
+  {
+   "source": [
+    "### The octree path can be modified to generated files from different data"
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
   {
    "cell_type": "code",
    "execution_count": 2,
@@ -82,7 +88,6 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The image layer is then defined\n",
     "### The URL-formated layer location parameter can be modified to send generated files to\n",
     " - any location on a local machine by using the \"file://\" prefix and then filepath\n",
     " - a google storage account by using the \"gs://\" prefix and then url\n",
@@ -132,7 +137,7 @@
     "SWC data will be assumed to be stored locally in `.swc` format, such as\n",
     "```default.0.swc```.\n",
     "\n",
-    "As before, this tutorial simply shows how to generate the requisite files, and the user can provide any destination for said files. The dafault is a local folder, but this can be modified to any url."
+    "As before, this tutorial simply shows how to generate the requisite files, and the user can provide any destination for said files. The dafault is a local folder, but this can be modified to any url.\n"
    ]
   },
   {
@@ -148,8 +153,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "We use `get_volume_info` from the `upload_skeleton` module instead, defining a volume in the same way.\n",
-    "### The octree path can be modified to generated files from different data"
+    "### The octree path can be modified to generated files from different data\n",
+    "\n",
+    "We use `get_volume_info` from the `upload_skeleton` module instead, defining a volume in the same way."
    ]
   },
   {