Prepare for Public Release with DOI

[dksasaki]

Description:
We are starting a to-do list with the objective of releasing this tool in a publication. The following tasks are needed before preparing for the publication:

To-Do List:

1. Configuration Files:

   • Reorganize configuration files so that each directory has its own configuration file, instead of one single general config file. Implement argparse.
     • boundary files
     • grid files (prototype/make_grid_.py)
     • grid smoothing
     • climatology and nudging
     • forcing files
     • rivers
     • tides

2. Bathymetry Smoothing:

   • Verify if the bathymetry smoothing can be run without compiling pyroms packages.

3. Meteorological Remapping:

   • Check if the remapping function is open source.

4. implement pyproject.toml for this package

   • Create pyproject.toml and auxilliary files and include dependencies
   • wrapping fortran functionalities into python via f2py
   • making the scripts available through the command line
   • push installable package to the rep.

5. Create Unit Testing:

   • Write unit tests for existing and new functionalities to ensure code reliability and correctness.

6. Include Docker Installation for Legacy:

   • Provide a Docker installation option to support legacy systems and ensure compatibility.

7. Include Docker Installation with ROMS:

   • Create a Docker installation that includes the ROMS (Regional Ocean Modeling System) for easier deployment and usage.

8. Implement Argparse:

   • Use argparse to handle command-line arguments for better user interaction and script flexibility.

9. Cleaning package (pylint might help)

   • Checking duplicated functions
   • Checking for variables or imported packages not used
   • Preparing docstring for better documentation

Note: This issue will be updated in the future

Update:

• we should use this branch of pyroms, since I modified it to make bathymetry smoothing easier.
• The bathymetry smoothing does not depend on the compilation of specific pyroms fortran wrappers. Some modifications were made in our pyroms branch to avoid dealing with basemap dependencies and fortran dependencies.

[nilodna]

Great @dksasaki. I will start with item 2, but before starting these tasks, I think we need to update the main and dev, so we can use the gitflow once again. What do you think?

I believe both dev and main are outdated and the 42_feature_including-nudging is the most stable and updated version of all. Since you was the last one to code something here, can you confirm that?

[dksasaki]

@nilodna you are right, 42_feature_including-nudging is our stable version as of now. I'm opening (and accepting) two pull requests - #45, #44 .

[dksasaki]

@nilodna, it might be better to modify the directory structure following item 1, before proceeding to other changes. This way we can avoid conflicts in pull requests in the future. Before I proceed, below is a scratch of the directory structure.

I created a new branch: https://github.com/lhico/pyroms_tools/tree/43-prepare-for-publication-release .

Let me know what are your ideas.

pyroms_tools/
├── pyroms
└── tools
    ├── atmos
    │   ├── atmos_forcing.py
    │   └── config.yaml
    ├── boundary
    │   ├── boundary_create.py
    │   ├── boundary_template.py
    │   └── config.yaml
    ├── grid
    │   ├── config.yaml
    │   └── grid_create.py
    ├── initial
    │   ├── config.yaml
    │   ├── initial_create.py
    │   └── initial_template.py
    ├── nudging
    │   ├── config.yaml
    │   └── nudging_create.py
    └── rivers
        ├── config.yaml
        └── rivers_create.py

[dksasaki]

The branch of pyroms we should use is: https://github.com/lhico/pyroms

I am updating a few things in this package so we can use it with a more recent version of python (3.9).

In case I succeed, I'll create a new repo with our own modified version of pyroms.

UPDATE:

I reconfigured our pyroms version. The bathymetry smoothing function does not depend on any fortran script in pyroms. I am using python 3.9, which was installed as follows (I hope I haven't missed anything):

# package installation
micromamba install -c conda-forge python=3.9

micromamba install -c conda-forge dask netCDF4 ipython
micromamba install -c conda-forge esmpy xarray numpy shapely cf_xarray sparse numba
micromamba install -c conda-forge xesmf

# pyroms installation 
git clone https://github.com/lhico/pyroms

micromamba install -c conda-forge scikit-build-core cmake ninja
pip install build
micromamba install -c conda-forge lpsolve55

cd pyroms/pyroms
python setup.py install

cd ../pyroms_toolbox
python setup.py install

cd ../bathy_smoother
python setup.py install

[nilodna]

Awesome @dksasaki. Do you want to work in a specific item? I was thinking work over item 1, but since I coded the meteorological extrapolation functions, I can work on this item too.

Let me know your preferences.

[dksasaki]

@nilodna, I've already started working on item 1 and 7 - I believe that it would be great if you start in the meteorological files and perhaps unit tests - I mean create some idealized files that represents a bathymetry, glorys and atmospheric forcing based on the era5, glorys and gebco files? We can further discuss this part.

BTW, creating a grid, smoothing the topography, initial conditions and boundary conditions were already updated. There are more details i'll need to work on, but it is a start.

[nilodna]

@nilodna, I've already started working on item 1 and 7 - I believe that it would be great if you start in the meteorological files and perhaps unit tests - I mean create some idealized files that represents a bathymetry, glorys and atmospheric forcing based on the era5, glorys and gebco files? We can further discuss this part.

Ok, I'll start on the "unit tests", but I don't think we can call them as unit tests because they are not suppose to run automatically during the installation of the package. Instead, they are more a testcases, so the user can run these experiments and compare with what is expected.

I have a few ideas. Give me some days and we can discuss.

[dksasaki]

One idea is to actually have unit tests while the package is installed - we can create some xarray datasets on the fly and test the scripts to evaluate if there is any issue with them.

[dksasaki]

95d53f7 I've included a pyproject.toml and a install.sh

• install.sh is a scratch that contain instructions on the installation of dependencies of pyroms_tools.
• pyproject.toml configures the installation of the package via python -m pip install . (or pip install .)

executables should be made available through the terminal as the following is stated in the pyproject.toml

[project.scripts]
make_grid        = "pyroms_tools.grid.make_grid_:main"
make_grid_smooth = "pyroms_tools.grid.make_grid_smooth:main"
make_grid_prototype = "pyroms_tools.grid.make_grid_prototype:main"
make_ic_template    = "pyroms_tools.initial.ic_file_template:main"
make_ic_build       = "pyroms_tools.initial.ic_file_build:main"
make_boundary_template    = "pyroms_tools.boundary.boundary_file_template:main"
make_boundary_build       = "pyroms_tools.boundary.boundary_file_build:main"

b43e91f In order to use one of the command above, type, for instance (an example of the config.yaml was also included in the root of the package)

make_grid --config config.yaml
make_grid_smooth --config config.yaml

b7581af I also included a minimum working installation of a fortran wrapping . This package will be moved to a different section of the code, but it is placed at this position for simplicity for now.

[dksasaki]

95d53f7 an example tool to edit mapped grid values has been added pyroms_tools/rivers/edit_mask.py. In the future we can use this tools to edit masks

[nilodna]

One idea is to actually have unit tests while the package is installed - we can create some xarray datasets on the fly and test the scripts to evaluate if there is any issue with them.

I have been thinking on this comment and I believe we can do something similar to what is done in xarray. They have files containing the expected output and during the installation they create datasets and compare with those files for assertion.

We can do something like that. We can create a domain, initial and boundary conditions beforehand (small files) and share them with the package on the tests folder (tests/expected). During the installation we can test the creation of these datasets comparing them with the files containing what is expected.

What do you think of that?