62 review edits harmonize pensacolarmd

.github/workflows/test_r.yaml

@@ -21,48 +21,59 @@ jobs:
 
     strategy:
       matrix:
-        os: [ubuntu-latest, macos-latest]
+        os: [windows-latest, macos-latest]
         python-version: ['3.8', '3.9', '3.10', '3.11']
-        include:
-          - os: windows-latest
-            python-version: "3.9"
     steps:
-      - uses: actions/checkout@v3
-         
-      - name: Set up Python ${{ matrix.python-version }}
-        uses: actions/setup-python@v2
+      - uses: actions/checkout@v4
+
+      - name: setup python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
 
       - name: Set up R
         uses: r-lib/actions/setup-r@v2
         with:
           r-version: 'release'
-
-      - uses: r-lib/actions/setup-pandoc@v2
 
-      - name: Update pip and install testing pkgs
-        run: |
-          python -VV
-          python -m pip install --upgrade pip
-          pip install pytest
+      #- uses: r-lib/actions/setup-renv@v2
 
-      # fiona doesn't have wheels for windows
-      - if: matrix.os == 'windows-latest'
-        run: |
-          pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.7.16/GDAL-3.7.1-cp39-cp39-win_amd64.whl
-          pip install https://github.com/cgohlke/geospatial-wheels/releases/download/v2023.7.16/Fiona-1.9.4.post1-cp39-cp39-win_amd64.whl
-
-      - name: Install package and dependencies
-        run: |
-          python -m pip install --no-deps .
-          pip install -r requirements.txt
+      - uses: r-lib/actions/setup-pandoc@v2
 
-      - name: Run pip env using R reticulate
-        run: |
-          install.packages("reticulate")
-          reticulate::import("harmonize_wq")
+      - name: R depends
         shell: Rscript {0}
+        run: |
+          install.packages(c("knitr", "rmarkdown", "reticulate"))
 
-      - name: Run pytest
-        run: pytest -v harmonize_wq
+      - name: setup r-reticulate venv & render rmd
+        shell: Rscript {0}
+        run: |
+          library(reticulate)
+
+          packages = c(
+              "pytest", "numpy<2.0", "pandas<2.0", "geopandas>=0.10.2, <0.13", "pint>=0.18",
+              "dataretrieval>=1.0, <1.0.5", "requests"
+              )
+
+          reticulate::install_miniconda()
+          reticulate::conda_create("wq_harmonize", python_version = "${{ matrix.python-version }}")
+          reticulate::conda_install("wq_harmonize", packages)
+
+          #path_to_venv <- virtualenv_create(
+          #  envname = "wq_harmonize",
+          #  python = Sys.which("python"), # placed on PATH by the setup-python action
+          #  packages
+          #)
+          #use_virtualenv("wq_harmonize")
+
+          reticulate::py_install("git+https://github.com/USEPA/harmonize-wq.git", pip = TRUE, envname = "wq_harmonize")
+
+          rmarkdown::render(input = "demos/Harmonize_Pensacola.Rmd")
+
+      - name: Upload artifact
+        if: ${{ (matrix.os == 'windows-latest') && (matrix.python-version == 3.11) }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: demos-artifact
+          # Upload entire demos folder
+          path: './demos'

demos/Harmonize_Pensacola.Rmd

@@ -55,6 +55,13 @@ data comparability and any thresholds for acceptance or rejection.
 
 ## Installation & Setup
 
+Using R/reticulate requires an installation of Python to bind to, and EPA's harmonize-wq package must be run from a python environment with the packages it depends on.
+
+For environment management, reticulate requires either conda (recommended) or virtualenv.
+
+There are multiple installers available for [Conda](https://conda.io/projects/conda/en/latest/user-guide/install/index.html).
+The examples use miniforge3, one of several verions of [miniforge](https://github.com/conda-forge/miniforge). In R, [miniconda](https://docs.anaconda.com/miniconda/) can be installed using reticulate::install_miniconda().
+
 #### Option 1: Install the harmonize-wq Package Using the Command Line
 
 To install and set up the harmonize-wq package using the command line:
@@ -64,82 +71,86 @@ To install and set up the harmonize-wq package using the command line:
     miniforge is installed. Go to your start menu and open the Miniforge
     Prompt.
 2.  At the Miniforge Prompt, run:
+    -   to update conda:
+
+        > conda update -n base -c conda-forge conda
+
     -   conda create --name wq_harmonize
-    -   activate wq_harmonize
-    -   conda install geopandas pip dataretrieval pint
-    -   may need to update conda
-        -   conda update -n base -c conda-forge conda
-    -   pip install harmonize-wq
-    -   pip install git+<https://github.com/USEPA/harmonize-wq.git> (dev
-        version)
+
+    -   conda activate wq_harmonize
+
+    -   conda install dependencies (from requirements.txt):
+
+        > conda install "numpy<2.0" "pandas<2.0" "geopandas>=0.10.2, <0.13" "pint>=0.18" "dataretrieval>=1.0, <1.0.5" "pip"
+
+    -   pip install harmonize-wq (dev-version shown):
+
+        > pip install git+<https://github.com/USEPA/harmonize-wq.git>
 
 #### Option 2: Install the harmonize-wq Package Using R
 
 **Alternatively**, you may be able to set up your environment and import
 the required Python packages using R.
 
-First, run the chunk below to install the reticulate package to use Python in R.
+First, run the chunk below to install the reticulate package to use
+Python in R.
 
 ```{r, results = 'hide'}
-install.packages("reticulate")
+install.packages("reticulate", repos = "http://cran.us.r-project.org")
 library(reticulate)
 ```
 
-Conda is required to use EPA's harmonize-wq package. 
-
-There are multiple installers available for Conda 
-(see: <https://conda.io/projects/conda/en/latest/user-guide/install/index.html>).
-
-One example installer is
-[miniforge](https://github.com/conda-forge/miniforge). We use miniforge3 in this 
-example.
-
-Once miniforge3 (or another installer of your choice) is installed, the
-reticulate package will automatically look for the installation of Conda (conda.exe) 
-on your computer. 
+Once miniforge3 (or another Conda installer of your choice) is
+installed, the reticulate package will automatically look for the
+installation of Conda (conda.exe) on your computer.
 
 ```{r, results = 'hide'}
 # options(reticulate.conda_binary = 'dir')
 ```
 
-However, you may still need to specify the location. If needed, update the code chuck below to specify the location of conda.exe on your computer.
+However, you may still need to specify the conda.exe location. To do so,
+update the last line of code below to specify your conda.exe location
 
 ```{r, results = 'hide'}
 # update the 'dir' in this chuck to specify the location of conda.exe on your computer
-# Note that the environment name may need to include the full path (e.g. "C:/Users/USERNAME/AppData/Local/miniforge3/Scripts/conda.exe")
-options(reticulate.conda_binary = "~/AppData/Local/miniforge3/Scripts/conda.exe")
+# Note: that the environment name may need to include the full path (e.g. "C:/Users/<USERNAME>/AppData/Local/miniforge3/Scripts/conda.exe")
+# options(reticulate.conda_binary = "~/AppData/Local/miniforge3/Scripts/conda.exe")
 ```
 
-Next, update the code chunk below to create a new Python environment in the envs
-folder on your computer called "wq_harmonize".
+Next, update the code chunk below to create a new Python environment in
+the envs folder called "wq_harmonize". Note that the environment name
+may need to include the full path (e.g.
+"C:/Users/<USERNAME>/AppData/Local/miniforge3/envs/wq_harmonize")
 
 ```{r, results = 'hide'}
-# Note that the environment name may need to include the full path (e.g. "C:/Users/USERNAME/AppData/Local/miniforge3/envs/wq_harmonize")
-reticulate::conda_create("~/AppData/Local/miniforge3/envs/wq_harmonize")
+# reticulate::conda_create("~/AppData/Local/miniforge3/envs/wq_harmonize")
 ```
 
-Install the following python and R packages to the newly created
-Python environment called "wq_harmonize".
+Install the following python packages to the newly created Python
+environment called "wq_harmonize".
 
 ```{r, results = 'hide'}
-reticulate::conda_install("wq_harmonize", "geopandas") # Python package
-reticulate::conda_install("wq_harmonize", "pint") # Python package
-reticulate::conda_install("wq_harmonize", "dataretrieval") # R package
+# packages = c(
+#              "numpy<2.0", "pandas<2.0", "geopandas>=0.10.2, <0.13", "pint>=0.18",
+#              "dataretrieval>=1.0, <1.0.5", "pip"
+#              )
+# reticulate::conda_install("wq_harmonize", packages)
 ```
 
-Install EPA's harmonize-wq package.
+Uncomment to install EPA's harmonize-wq package most recent release or
+development version.
 
 ```{r, results = 'hide'}
 # Install the most recent release of the harmonize-wq package
 # This only works with py_install() (pip = TRUE), which defaults to use virtualenvs
-reticulate::py_install("harmonize-wq", pip = TRUE, envname = "wq_harmonize")
+# reticulate::py_install("harmonize-wq", pip = TRUE, envname = "wq_harmonize")
 
-# Uncomment below to install the development version of harmonize-wq from GitHub instead (optional)
-# py_install("git+https://github.com/USEPA/harmonize-wq.git@new_release_0-3-8", pip = TRUE, envname = "wq_harmonize")
+# Install the development version of harmonize-wq from GitHub (optional)
+#py_install("git+https://github.com/USEPA/harmonize-wq.git", pip = TRUE, envname = "wq_harmonize")
 ```
 
-Specify the Python environment to be used, "wq_harmonize", and test that your Python
-environment is set up correctly.
+Specify the Python environment to be used, "wq_harmonize", and test that
+your Python environment is set up correctly.
 
 ```{r}
 # Specify environment to be used
@@ -167,6 +178,7 @@ import geopandas
 import dataretrieval.wqp as wqp
 import pint
 import mapclassify
+import matplotlib.pyplot as plt
 from harmonize_wq import harmonize
 from harmonize_wq import convert
 from harmonize_wq import wrangle
@@ -190,8 +202,8 @@ reticulate::repl_python()
 ```
 
 First, determine an area of interest (AOI), build a query, and retrieve
-water temperature and Secchi disk depth data from the Water Quality Portal (WQP)
-for the AOI using the dataretrieval package:
+water temperature and Secchi disk depth data from the Water Quality
+Portal (WQP) for the AOI using the dataretrieval package:
 
 ```{python, error = F}
 # File for area of interest (Pensacola and Perdido Bays, FL)
@@ -211,11 +223,11 @@ res_narrow, md_narrow = wqp.get_results(**query)
 res_narrow
 ```
 
-Next, harmonize and clean all results using the harmonize.harmonize_all, 
+Next, harmonize and clean all results using the harmonize.harmonize_all,
 clean.datetime, and clean.harmonize_depth functions.
 
-Enter a ? followed by the function name, for example ?harmonize.harmonize_all, 
-into the console for more details.
+Enter a ? followed by the function name, for example
+?harmonize.harmonize_all, into the console for more details.
 
 ```{python, error = F}
 df_harmonized = harmonize.harmonize_all(res_narrow, errors = 'raise')
@@ -257,5 +269,7 @@ stations_gdf, stations, site_md = location.get_harmonized_stations(query, aoi=ao
 
 # Map average temperature results at each station
 gdf_temperature = visualize.map_measure(df_wide, stations_gdf, 'Temperature')
+plt.figure()
 gdf_temperature.plot(column = 'mean', cmap = 'OrRd', legend = True)
+plt.show()
 ```