Zebrafish pipeline plan

Zebrafish pipeline plan

Short term (Spring 2018)

Create a fully-functioning visualization prototype sufficient to show feasibility for an NIH grant. The prototype should

1. Pull images from the NI-DAQ card (can start by mocking an image server: just draw files from disk)
2. Run CaImAn's preprocessing pipeline on these images.
   1. Dockerize CaImAn
   2. Assess how well CaImAn functions on zebrafish (as opposed to mouse cortical) data
3. Send preprocessed data back to server:
   1. Locations of cells
   2. Deconvolved spike trains
4. Build a visualization server like Jupyter that will let a user navigate to a particular port on localhost and see the data streaming in.
   1. This can be either a plugin to the server, or the server itself.
   2. We probably want to use WebGL for performance.
   3. The server in this case is part web server (it needs to send the js and html to the client) but the client should also be able to get images from the server via, e.g., REST calls.
   4. Might also be nice to paint the cells found by CaImAn as an example of how effective the algorithm is.

Questions to be resolved:

1. Should the web server be part of the data server or separate? In Jupyter, it's all the same, because it's just middleware handling a kernel. For our case, a separate web server could be small and hook into the data server via the C API, but this may be too complicated (or unnecessary) for a first pass.
2. Communication between pipeline pieces: gRPC? http?
   1. Apparently, gRPC has some performance issues: link, link, link