Scripts and dependencies updates for Python 3.8 - 3.10 (#2009)

* multichannel replaced with channel_axis in scikit-image ver 0.19

* in scikit-image 0.20 watershed was moved from morphology module to segmentation

* in scikit-image 0.20 selem was changed to footprint

selsm was replaced with footprint in scikit-image 0.20

* updates due to scikit-image 0.21.0 prefix GeometricTransform with _

* updating napari version and removing anisotropy in spot sizes

* fix(requirements): edit Makefile and update REQUIREMENTS.txt to generate all req files

updating requirements files to support sphinx's make html

* numpy.float is deprecated in ver 1.20

* append is deprecated in pandas 2.0 use concat instead. This update is backward compatible.

* Linting corrections following automatic checks in PR

* updating yaml.load() to support version 5.1+

* updating pip version

* linting corrections to crop.py and check_all_funcs.py

* mypy assignment error: replaced None with empty string

* mypy assignment error: replaced None with empty dict

* mypy assignment error: add Optional to args defaults with None

added: from typing import Optional

* mypy assignment error: silencing variable type IntensityTable

* check_all_func.py: added import typing

* round and ch_values from_spot_data of IntensityTable expected Sequence but got ndarray instead: replaced np.arange with range

* silencing mypy errors with type: ignore, to pass PR automatic checks

* check_all_funcs.py: F401 'typing' imported but unused: silencing linter

* fixing 6 files with linting errors E126, E128, E251

* fixing linting errors in 2 files: I101 and I201

* fixing conflict due to identical  in 2 json schema files

* correcting spots detected from 53 to 54

* update expected values for registered images and two other minor counts

* small correction to expected values when testing registration on ISS test data

* correcting channel_ and round_must_be_present tests and corresponding field_of_view schema

* fixiing wrong underline length in docstring

* defining 'en' language and adding xarray to autodoc_mock_imports

* removing py 3.7 and adding 3.10 to starfish-prod-ci.yml

* commenting out napari gui and moving to the end of script

* removed extension 'sphinx_autodoc_typehints' due to failed guarded type import with xarray

* fix: update docker image tp python 3.8 instead of 3.7

* docs(packaging): update python version compatible with starfish

docs(packaging): minor syntax correction

* docs: updating README and installation file with python versions and how to fix broken napari or jupyter

.github/workflows/starfish-prod-ci.yml

@@ -41,7 +41,7 @@ jobs:
     strategy:
       matrix:
         os: ["windows-latest", "ubuntu-latest"]
-        python-version: ["3.7", "3.8", "3.9"]
+        python-version: ["3.8", "3.9", "3.10"]
       fail-fast: false
     runs-on: ${{ matrix.os }}
     steps:
@@ -71,7 +71,7 @@ jobs:
     strategy:
       matrix:
         os: ["windows-latest", "ubuntu-latest"]
-        python-version: ["3.7", "3.8", "3.9"]
+        python-version: ["3.8", "3.9", "3.10"]
       fail-fast: false
     runs-on: ${{ matrix.os }}
     steps:

Makefile

@@ -12,7 +12,7 @@ DOCKER_BUILD?=1
 
 VERSION=$(shell sh -c "git describe --exact --dirty 2> /dev/null")
 # if you update this, you will need to update the version pin for the "Install Napari & Test napari (pinned)" test in .travis.yml
-PIP_VERSION=21.2.4
+PIP_VERSION=24.0
 
 define print_help
     @printf "    %-28s   $(2)\n" $(1)
@@ -80,8 +80,8 @@ help-docs:
 
 ### REQUIREMENTS #############################################
 #
-GENERATED_REQUIREMENT_FILES=starfish/REQUIREMENTS-STRICT.txt requirements/REQUIREMENTS-CI.txt requirements/REQUIREMENTS-NAPARI-CI.txt
-SOURCE_REQUIREMENT_FILES=REQUIREMENTS.txt requirements/REQUIREMENTS-CI.txt.in requirements/REQUIREMENTS-NAPARI-CI.txt.in
+GENERATED_REQUIREMENT_FILES=starfish/REQUIREMENTS-STRICT.txt requirements/REQUIREMENTS-CI.txt requirements/REQUIREMENTS-NAPARI-CI.txt requirements/REQUIREMENTS-JUPYTER.txt
+SOURCE_REQUIREMENT_FILES=REQUIREMENTS.txt requirements/REQUIREMENTS-CI.txt.in requirements/REQUIREMENTS-NAPARI-CI.txt.in requirements/REQUIREMENTS-JUPYTER.txt.in
 
 # This rule pins the requirements with the minimal set of changes required to satisfy the
 # requirements.  This is typically run when a new requirement is added, and we want to
@@ -103,7 +103,6 @@ starfish/REQUIREMENTS-STRICT.txt : REQUIREMENTS.txt
 	[ ! -e .$<-env ] || exit 1
 	$(call create_venv, .$<-env)
 	.$<-env/bin/pip install --upgrade pip==$(PIP_VERSION)
-	.$<-env/bin/pip install -r $@
 	.$<-env/bin/pip install -r $<
 	echo "# You should not edit this file directly.  Instead, you should edit one of the following files ($^) and run make $@" >| $@
 	.$<-env/bin/pip freeze --all | grep -v "pip==$(PIP_VERSION)" >> $@
@@ -113,7 +112,6 @@ requirements/REQUIREMENTS-%.txt : requirements/REQUIREMENTS-%.txt.in REQUIREMENT
 	[ ! -e .$<-env ] || exit 1
 	$(call create_venv, .$<-env)
 	.$<-env/bin/pip install --upgrade pip==$(PIP_VERSION)
-	.$<-env/bin/pip install -r $@
 	for src in $^; do \
 		.$<-env/bin/pip install -r $$src; \
 	done

README.rst

@@ -47,19 +47,19 @@ See `spacetx-starfish.readthedocs.io <https://spacetx-starfish.readthedocs.io/en
 Installation
 ------------
 
-starfish supports python 3.7 and above and can easily be installed from PyPI:
+starfish supports python 3.8-11 and can easily be installed from PyPI:
 
 .. code-block:: bash
 
     $ pip install starfish[napari]
 
-`For more detailed installation instructions, see here <https://spacetx-starfish.readthedocs.io/en/latest/installation/>`_.
+`Some operating systems might need different dependencies for napari. For more detailed installation instructions, see here <https://spacetx-starfish.readthedocs.io/en/latest/installation/>`_.
 
 Python Version Notice
 ---------------------
 
-starfish will be dropping support for python 3.6 in the next release due to
-minimum python=3.7 version requirements in upstream dependencies.
+starfish will be dropping support for python 3.8 in the next release due to
+minimum python=3.9 version requirements in upstream dependencies.
 
 Contributing
 ------------

REQUIREMENTS.txt

@@ -1,30 +1,27 @@
-# The following requirement is here because networkx restricts the version
-# of decorator.  Since pip is not a full dependency solver, it has already
-# installed decorator before networkx is installed, and bombs after the
-# fact.
-decorator < 5.0
 click
-dataclasses==0.6
+# docutils introduces an AttributeError: module 'docutils.nodes' has no attribute 'reprunicode'
+docutils<0.21
 h5py
-jsonschema
-matplotlib
-numpy != 1.13.0, >= 1.20.0
-pandas >= 0.23.4
+# jsonschema 4.18.0 made RefResolver deprecated
+jsonschema<4.18
+# matplotlib 3.8 changes colormap module
+matplotlib<3.8
+# pinning mistune for m2r2 in CI requirements
+mistune==0.8.4
+# numpy 1.25 slightly changes values in three tests causing them to fail
+numpy<1.25
 read_roi
 regional
-semantic_version
-# 0.16.[012] are excluded because https://github.com/scikit-image/scikit-image/pull/3984 introduced
-# a bug into max peak finder.  0.16.3 presumably will have the fix from
-# https://github.com/scikit-image/scikit-image/pull/4263.
-scikit-image >= 0.14.0, != 0.16.0.*, != 0.16.1.*, != 0.16.2.*, != 0.17.1.*, != 0.17.2.*, < 0.19.0
+# scikit-image is temporarily pinned as newer versions require updating module imports
+scikit-image==0.21
 scikit-learn
-scipy
-showit >= 1.1.4
-slicedimage==4.1.1
-sympy ~= 1.5.0
-urllib3 <1.27, >=1.25.4
+seaborn
+semantic_version
+showit
+slicedimage
+sympy
 tqdm
 trackpy
 validators
-xarray >= 0.14.1
-ipywidgets
+# xarray 2023.09 causses a ValueError: new dimensions ('y', 'x') must be a superset of existing dimensions ('dim_0', 'dim_1')
+xarray<2023.09

docker/Dockerfile

@@ -36,7 +36,7 @@
 ##
 ##   $ docker run -e TEST_ISS_KEEP_DATA=true --entrypoint=pytest spacetx/starfish -vsxk TestWithIssData
 ##
-FROM python:3.7-slim-stretch
+FROM python:3.8-slim-buster
 
 RUN useradd -m starfish
 USER starfish

docs/source/conf.py

@@ -52,7 +52,6 @@
     'sphinx.ext.coverage',
     'sphinx.ext.mathjax',
     'sphinx.ext.viewcode',
-    'sphinx_autodoc_typehints',
     'sphinxcontrib.programoutput',
     'sphinx_gallery.gen_gallery',
     'sphinx.ext.intersphinx',
@@ -91,7 +90,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = 'en'
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -130,6 +129,8 @@
     'download_all_examples': False,
     'default_thumb_file': f'{dir_}/_static/design/logo-solo.png',
     'min_reported_time': 10,
+    # Rerun stale examples even if their MD5 hash shows that the example did not change.
+    'run_stale_examples': True,
 
 }
 
@@ -276,9 +277,6 @@ def setup(app):
 napoleon_use_param = True
 napoleon_use_rtype = True
 
-# See https://stackoverflow.com/a/45565445/56887
-autodoc_mock_imports = ['_tkinter']
-
 rst_epilog = """
 .. include:: <s5defs.txt>
 """

docs/source/installation/index.rst

@@ -3,7 +3,7 @@
 Installation
 ============
 
-Starfish supports python 3.6 and above (python 3.7 recommended). To install the starfish package,
+Starfish supports python 3.8-11. To install the starfish package,
 first verify that your python version is compatible. You can check this by running :code:`python
 --version`.
 
@@ -12,13 +12,13 @@ The output should look similar to this:
 .. code-block:: bash
 
    $ python --version
-   Python 3.7.7
+   Python 3.8.18
 
 .. warning::
-    While starfish itself has no known issues with python 3.8, scikit-image is not fully
-    compatible with python 3.8. As such, installation of scikit-image, as part of starfish
-    installation, may unexpectedly fail. The workaround is to install numpy first before
-    installing starfish or scikit-image.
+    While starfish itself should work on any operating system, some napari dependencies might not be
+    compatible with Apple Silicon or Windows. As such, installation of napari, as part of starfish[napari]
+    installation, may unexpectedly fail. The workaround is to install napari first before
+    installing starfish or updating the dependencies via cloning the project and working in development mode.
 
 
 Using virtual environments
@@ -44,7 +44,7 @@ Conda_ users can set one up like so:
 
 .. code-block:: bash
 
-    $ conda create -n starfish "python=3.7"
+    $ conda create -n starfish "python=3.8"
     $ conda activate starfish
 
 Installing *starfish*
@@ -57,7 +57,7 @@ Starfish can easily be installed using pip:
     $ pip install starfish
 
 .. note::
-    If using python 3.8, first install numpy using pip before installing starfish.
+    If using Windows or Apple Silicon (M1+), first install napari using pip before installing starfish.
 
 To use napari for interactive image visualization via :py:func:`.display` you must also
 install napari:
@@ -72,8 +72,8 @@ Interactive visualization with napari also requires using Qt (e.g. by running th
 Installing *starfish* on Windows
 --------------------------------
 
-Windows users can install starfish in the same way. Again, we recommend using a conda or virtual
-environment with python 3.7. Here is how you would install starfish in a virtual environment
+Windows (cmd.exe) users can install starfish in the same way. Again, we recommend using a conda or virtual
+environment with python 3.8+. Here is how you would install starfish in a virtual environment
 created with python's ``venv`` module:
 
 .. code-block:: bat
@@ -82,12 +82,11 @@ created with python's ``venv`` module:
     > cd starfish
     > python -m venv .venv
     > .venv\Scripts\activate.bat
+    > pip install napari[all]
     > pip install starfish
-    > pip install starfish[napari]
 
 .. note::
-    Python 3.8 has trouble installing scikit-image v0.15.0 and the ``pip install numpy``
-    workaround does not solve this issue on Windows.
+    If you encounter issues, you need to update the dependencies via cloning the project and working in development mode.
 
 Jupyter notebook
 ----------------
@@ -97,7 +96,54 @@ the virtualenv kernel to jupyter by activating your virtual environment and then
 
 .. code-block:: bash
 
+    $ python -m pip install jupyter
     $ python -m ipykernel install --user --name=<venv_name>
 
 Now you should be able to select ``venv_name`` as the kernel in a jupyter notebook to have access
 to the starfish library.
+
+Installing *starfish* in development mode
+-----------------------------------------
+
+If you need to resolve dependency issues with napari and jupyter or want to tinker with the
+starfish package, it is best to work in development mode. 
+If you are on a mac, make sure you have the `XCode CommandLine Tools`_
+installed.
+
+.. _`XCode CommandLine Tools`: https://developer.apple.com/library/archive/technotes/tn2339/_index.html
+
+Check out the code for starfish:
+
+.. code-block:: bash
+
+    $ git clone https://github.com/spacetx/starfish.git
+    $ cd starfish
+
+Set up a `virtual environment`_:
+
+.. _`virtual environment`: #using-virtual-environments
+
+.. code-block:: bash
+
+    $ python -m venv .venv
+    $ source .venv/bin/activate
+
+Install starfish:
+
+.. code-block:: bash
+
+    $ make install-dev
+
+Update dependencies for napari and jupyter:
+
+.. code-block:: bash
+
+    $ make -B requirements/REQUIREMENTS-NAPARI-CI.txt
+    $ make -B requirements/REQUIREMENTS-JUPYTER.txt
+
+Install napari and jupyter:
+
+.. code-block:: bash
+
+    $ pip install -r requirements/REQUIREMENTS-NAPARI-CI.txt
+    $ pip install -r requirements/REQUIREMENTS-JUPYTER.txt

examples/data_formatting_examples/format_osmfish.py

@@ -132,7 +132,7 @@ def __init__(self, input_dir: str, metadata_yaml) -> None:
         import yaml
 
         with open(metadata_yaml, "r") as f:
-            self.osmfish_metadata = yaml.load(f)
+            self.osmfish_metadata = yaml.load(f, Loader=yaml.FullLoader)
         self.num_z = self.osmfish_metadata['ImageProperties']['HybImageSize']['zcount']
         self.input_dir = input_dir
 

notebooks/BaristaSeq.ipynb

@@ -289,7 +289,7 @@
     "from functools import partial\n",
     "\n",
     "# calculate the background\n",
-    "opening = partial(opening, selem=disk(5))\n",
+    "opening = partial(opening, footprint=disk(5))\n",
     "\n",
     "background = bleed_corrected.apply(\n",
     "    opening,\n",
@@ -479,4 +479,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 2
-}
+}

notebooks/Starfish_simulation.ipynb

@@ -40,7 +40,7 @@
    "outputs": [],
    "source": [
     "from numpy.random import permutation, rand, normal\n",
-    "from numpy import ones, zeros, concatenate, array, float\n",
+    "from numpy import ones, zeros, concatenate, array\n",
     "from numpy.random import poisson\n",
     "from pandas import DataFrame, concat\n",
     "from skimage.filters import gaussian\n",
@@ -144,4 +144,4 @@
  },
  "nbformat": 4,
  "nbformat_minor": 2
-}
+}

notebooks/py/BaristaSeq.py

@@ -212,7 +212,7 @@
 from functools import partial
 
 # calculate the background
-opening = partial(opening, selem=disk(5))
+opening = partial(opening, footprint=disk(5))
 
 background = bleed_corrected.apply(
     opening,

notebooks/py/Starfish_simulation.py

@@ -28,7 +28,7 @@ def code_sum(codeword):
 
 # EPY: START code
 from numpy.random import permutation, rand, normal
-from numpy import ones, zeros, concatenate, array, float
+from numpy import ones, zeros, concatenate, array
 from numpy.random import poisson
 from pandas import DataFrame, concat
 from skimage.filters import gaussian

notebooks/py/smFISH.py

@@ -32,10 +32,6 @@
 import starfish.data
 from starfish import FieldOfView, DecodedIntensityTable
 from starfish.types import TraceBuildingStrategies
-
-# equivalent to %gui qt
-ipython = get_ipython()
-ipython.magic("gui qt5")
 # EPY: END code
 
 # EPY: START markdown
@@ -171,5 +167,6 @@ def processing_pipeline(
 image, intensities = processing_pipeline(experiment, fov_name='fov_001')
 
 # uncomment the below line to visualize the output with the spot calls.
+# %gui qt
 # viewer = starfish.display(image, intensities)
 # EPY: END code