diff --git a/README.md b/README.md index b9a59e70..c6ed488c 100644 --- a/README.md +++ b/README.md @@ -30,6 +30,8 @@ + + VirtualShipParcels is a command line simulator allowing students to plan and conduct a virtual research expedition, receiving measurements as if they were coming from actual oceanographic instruments including: diff --git a/docs/user-guide/documentation/copernicus_products.md b/docs/user-guide/documentation/copernicus_products.md index 78361984..fec42d44 100644 --- a/docs/user-guide/documentation/copernicus_products.md +++ b/docs/user-guide/documentation/copernicus_products.md @@ -2,7 +2,7 @@ VirtualShip supports running experiments anywhere in the global ocean from 1993 through to the present day (and approximately two weeks into the future), using the suite of products available from the [Copernicus Marine Data Store](https://data.marine.copernicus.eu/products). -The data sourcing task is handled by the `virtualship fetch` command. The three products relied on by `fetch` to source data for all [VirtualShip instruments](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) (both physical and biogeochemical) are: +The data sourcing task is handled by the `virtualship run` command, which in turn relies on the [copernicusmarine toolbox](https://github.com/mercator-ocean/copernicus-marine-toolbox?tab=readme-ov-file) for 'streaming' data from the Copernicus Marine Data Store. The three products relied on in `run` to source data for all [VirtualShip instruments](https://virtualship.readthedocs.io/en/latest/user-guide/assignments/Research_proposal_intro.html#Measurement-Options) (both physical and biogeochemical) are: 1. **Reanalysis** (or "hindcast" for biogeochemistry). 2. **Renalysis interim** (or "hindcast interim" for biogeochemistry). @@ -15,7 +15,7 @@ The Copernicus Marine Service describe the differences between the three product As a general rule of thumb the three different products span different periods across the historical period to present and are intended to allow for continuity across the previous ~ 30 years. ```{note} -The ethos for automated dataset selection in `virtualship fetch` is to prioritise the Reanalysis/Hindcast products where possible (the 'work horse'), then _interim products where possible for continuity, and finally filling the very near-present (and near-future) temporal range with the Analysis & Forecast products. +The ethos for automated dataset selection in `virtualship run` is to prioritise the Reanalysis/Hindcast products where possible (the 'work horse'), then _interim products where possible for continuity, and finally filling the very near-present (and near-future) temporal range with the Analysis & Forecast products. ``` ```{warning} @@ -24,13 +24,13 @@ In the rare situation where the start and end times of an expedition schedule sp ### Data availability -The following tables summarise which Copernicus product is selected by `virtualship fetch` per combination of time period and variable (see legend below). +The following tables summarise which Copernicus product is selected by `virtualship run` per combination of time period and variable (see legend below). For biogeochemical variables `ph` and `phyc`, monthly products are required for hindcast and hindcast interim periods. For all other variables, daily products are available. #### Physical products -| Period | Product ID | Temporal Resolution | Typical Years Covered | Variables | +| Period | Dataset ID | Temporal Resolution | Typical Years Covered | Variables | | :------------------ | :--------------------------------------- | :------------------ | :---------------------------------- | :------------------------- | | Reanalysis | `cmems_mod_glo_phy_my_0.083deg_P1D-m` | Daily | ~30 years ago to ~5 years ago | `uo`, `vo`, `so`, `thetao` | | Reanalysis Interim | `cmems_mod_glo_phy_myint_0.083deg_P1D-m` | Daily | ~5 years ago to ~2 months ago | `uo`, `vo`, `so`, `thetao` | @@ -40,7 +40,7 @@ For biogeochemical variables `ph` and `phyc`, monthly products are required for #### Biogeochemical products -| Period | Product ID | Temporal Resolution | Typical Years Covered | Variables | Notes | +| Period | Dataset ID | Temporal Resolution | Typical Years Covered | Variables | Notes | | :---------------------------- | :----------------------------------------- | :------------------ | :---------------------------------- | :-------------------------------- | :------------------------------------- | | Hindcast | `cmems_mod_glo_bgc_my_0.25deg_P1D-m` | Daily | ~30 years ago to ~5 years ago | `o2`, `chl`, `no3`, `po4`, `nppv` | Most BGC variables except `ph`, `phyc` | | Hindcast (monthly) | `cmems_mod_glo_bgc_my_0.25deg_P1M-m` | Monthly | ~30 years ago to ~5 years ago | `ph`, `phyc` | Only `ph`, `phyc` (monthly only) | diff --git a/docs/user-guide/documentation/example_copernicus_download.ipynb b/docs/user-guide/documentation/example_copernicus_download.ipynb new file mode 100644 index 00000000..1e630520 --- /dev/null +++ b/docs/user-guide/documentation/example_copernicus_download.ipynb @@ -0,0 +1,193 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "a48322c9", + "metadata": {}, + "source": [ + "# Example Copernicus data download \n", + "\n", + "This notebook provides a rough, non-optimised example of how to download Copernicus Marine data using the `copernicusmarine` Python package.\n", + "\n", + "This will download:\n", + "- Global bathymetry data (static)\n", + "- Global biogeochemical monthly data (0.25 degree hindcast)\n", + "- Global physical daily data (0.25 degree reanalysis)\n", + "\n", + "For a singular year (2023) and two months (June and July).\n", + "\n", + "This notebook is intended as a basic example only. Modifications will be needed to adapt this to your own use case." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "7f5a7cc7", + "metadata": {}, + "outputs": [], + "source": [ + "import copernicusmarine\n", + "import os\n", + "from datetime import datetime" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e7279d5a", + "metadata": {}, + "outputs": [], + "source": [ + "YEAR = \"2023\"\n", + "MONTHS = [\"06\", \"07\"]\n", + "DAYS = [\n", + " \"01\",\n", + " \"02\",\n", + " \"03\",\n", + " \"04\",\n", + " \"05\",\n", + " \"06\",\n", + " \"07\",\n", + " \"08\",\n", + " \"09\",\n", + " \"10\",\n", + " \"11\",\n", + " \"12\",\n", + " \"13\",\n", + " \"14\",\n", + " \"15\",\n", + " \"16\",\n", + " \"17\",\n", + " \"18\",\n", + " \"19\",\n", + " \"20\",\n", + " \"21\",\n", + " \"22\",\n", + " \"23\",\n", + " \"24\",\n", + " \"25\",\n", + " \"26\",\n", + " \"27\",\n", + " \"28\",\n", + " \"29\",\n", + " \"30\",\n", + " \"31\",\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "1a583dba", + "metadata": {}, + "outputs": [], + "source": [ + "### PHYSICAL DAILY FILES\n", + "\n", + "os.chdir(\"~/data/phys/\")\n", + "DATASET_ID = \"cmems_mod_glo_phy-all_my_0.25deg_P1D-m\"\n", + "\n", + "for month in MONTHS:\n", + " for day in DAYS:\n", + " # check is valid date\n", + " try:\n", + " datetime(year=int(YEAR), month=int(month), day=int(day), hour=0)\n", + " except ValueError:\n", + " continue\n", + "\n", + " filename = f\"{DATASET_ID}_global_fulldepth_{YEAR}_{month}_{day}.nc\"\n", + "\n", + " if os.path.exists(filename):\n", + " print(f\"File {filename} already exists, skipping...\")\n", + " continue\n", + "\n", + " copernicusmarine.subset(\n", + " dataset_id=DATASET_ID,\n", + " variables=[\"uo_glor\", \"vo_glor\", \"thetao_glor\", \"so_glor\"],\n", + " minimum_longitude=-180,\n", + " maximum_longitude=179.75,\n", + " minimum_latitude=-80,\n", + " maximum_latitude=90,\n", + " start_datetime=f\"{YEAR}-{month}-{day}T00:00:00\",\n", + " end_datetime=f\"{YEAR}-{month}-{day}T00:00:00\",\n", + " minimum_depth=0.5057600140571594,\n", + " maximum_depth=5902.0576171875,\n", + " output_filename=filename,\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "89921772", + "metadata": {}, + "outputs": [], + "source": [ + "### BIOGEOCHEMICAL MONTHLY FILES\n", + "\n", + "os.chdir(\"~/data/bgc/\")\n", + "DATASET_ID = \"cmems_mod_glo_bgc_my_0.25deg_P1M-m\"\n", + "DAY = \"01\"\n", + "\n", + "for month in MONTHS:\n", + " try:\n", + " datetime(year=int(YEAR), month=int(month), day=int(DAY), hour=0)\n", + " except ValueError:\n", + " continue\n", + "\n", + " filename = f\"{DATASET_ID}_global_fulldepth_{YEAR}_{month}_{DAY}.nc\"\n", + "\n", + " if os.path.exists(filename):\n", + " print(f\"File {filename} already exists, skipping...\")\n", + " continue\n", + "\n", + " copernicusmarine.subset(\n", + " dataset_id=\"cmems_mod_glo_bgc_my_0.25deg_P1M-m\",\n", + " variables=[\"chl\", \"no3\", \"nppv\", \"o2\", \"ph\", \"phyc\", \"po4\"],\n", + " minimum_longitude=-180,\n", + " maximum_longitude=179.75,\n", + " minimum_latitude=-80,\n", + " maximum_latitude=90,\n", + " start_datetime=f\"{YEAR}-{month}-{DAY}T00:00:00\",\n", + " end_datetime=f\"{YEAR}-{month}-{DAY}T00:00:00\",\n", + " minimum_depth=0.5057600140571594,\n", + " maximum_depth=5902.05810546875,\n", + " output_filename=filename,\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8b5495c6", + "metadata": {}, + "outputs": [], + "source": [ + "### BATHYMETRY FILE\n", + "os.chdir(\"~/data/bathymetry/\")\n", + "DATASET_ID = \"cmems_mod_glo_phy_anfc_0.083deg_static\"\n", + "filename = \"cmems_mod_glo_phy_anfc_0.083deg_static_bathymetry.nc\"\n", + "\n", + "copernicusmarine.subset(\n", + " dataset_id=DATASET_ID,\n", + " dataset_part=\"bathy\",\n", + " variables=[\"deptho\"],\n", + " minimum_longitude=-180,\n", + " maximum_longitude=179.91668701171875,\n", + " minimum_latitude=-80,\n", + " maximum_latitude=90,\n", + " minimum_depth=0.49402499198913574,\n", + " maximum_depth=0.49402499198913574,\n", + " output_filename=filename,\n", + ")" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/user-guide/documentation/pre_download_data.md b/docs/user-guide/documentation/pre_download_data.md new file mode 100644 index 00000000..fc55f339 --- /dev/null +++ b/docs/user-guide/documentation/pre_download_data.md @@ -0,0 +1,87 @@ +# Pre-downloading data + +By default, VirtualShip will automatically 'stream' data from the Copernicus Marine Service via the [copernicusmarine toolbox](https://github.com/mercator-ocean/copernicus-marine-toolbox?tab=readme-ov-file). However, for users who wish to manage data locally, it is possible to pre-download the required datasets and feed them into VirtualShip simulations. + + + +As outlined in the [Quickstart Guide](https://virtualship.readthedocs.io/en/latest/user-guide/quickstart.html), the `virtualship run` command supports an optional `--from-data` argument, which allows users to specify a local directory containing the necessary data files. + +```{tip} +See the [for example...](#for-example) section for an example data download workflow. +``` + +### Data requirements + +For pre-downloaded data, VirtualShip only supports daily and monthly resolution physical and biogeochemical data, along with a static bathymetry file. + +In addition, all pre-downloaded data must be split into separate files per timestep (i.e. one .nc file per day or month). + +```{note} +**Monthly data**: when using monthly data, ensure that your final .nc file download is for the month *after* your expedition schedule end date. This is to ensure that a Parcels FieldSet can be generated under-the-hood which fully covers the expedition period. For example, if your expedition runs from 1st May to 15th May, your final monthly data file should be in June. Daily data files only need to cover the expedition period exactly. +``` + +Further, VirtualShip expects pre-downloaded data to be organised in a specific directory & filename structure within the specified local data directory. The expected structure is as outlined in the subsequent sections. + +#### Directory structure + +Assuming the local data directory (as supplied in the `--from-data` argument) is named `data/`, the expected subdirectory structure is: + +```bash +. +└── data + ├── bathymetry # containing the singular bathymetry .nc file + ├── bgc # containing biogeochemical .nc files + └── phys # containing physical .nc files +``` + +#### Filename conventions + +Within these subdirectories, the expected filename conventions are: + +- Physical data files (in `data/phys/`) should be named as follows: + - `_.nc` + - e.g. `cmems_mod_glo_phy-all_my_0.25deg_P1D-m_1998_05_01.nc` and so on for each timestep. +- Biogeochemical data files (in `data/bgc/`) should be named as follows: + - `_.nc` + - e.g. `cmems_mod_glo_bgc_my_0.25deg_P1M-m_1998_05_01.nc` and so on for each timestep. +- Bathymetry data file (in `data/bathymetry/`) should be named as follows: + - `cmems_mod_glo_phy_anfc_0.083deg_static_bathymetry.nc` + +```{tip} +Take care to use an underscore (`_`) as the separator between date components in the filenames (i.e. `YYYY_MM_DD`). +``` + +```{note} +Using the `` in the filenames is vital in order to correctly identify the temporal resolution of the data (daily or monthly). The `P1D` in the example above indicates daily data, whereas `P1M` would indicate monthly data. + +See [here](https://help.marine.copernicus.eu/en/articles/6820094-how-is-the-nomenclature-of-copernicus-marine-data-defined#h_34a5a6f21d) for more information on Copernicus dataset nomenclature. + +See also our own [documentation](https://virtualship.readthedocs.io/en/latest/user-guide/documentation/copernicus_products.html#data-availability) on the Copernicus datasets used natively by VirtualShip when 'streaming' data if you wish to use the same datasets for pre-download. +``` + +```{note} +**Monthly data**: the `DD` component of the date in the filename for monthly .nc files should always be `01`, representing the first day of the month. This ensures that a Parcels FieldSet can be generated under-the-hood which fully covers the expedition period from the start. +``` + +#### Further assumptions + +The following assumptions are also made about the data: + +1. All pre-downloaded data files must be in NetCDF format (`.nc`). +2. Physical data files must contain the following variables: `uo`, `vo`, `so`, `thetao` + - Or these strings must appear as substrings within the variable names (e.g. `uo_glor` is acceptable for `uo`). +3. If using BGC instruments (e.g. `CTD_BGC`), the relevant biogeochemical data files must contain the following variables: `o2`, `chl`, `no3`, `po4`, `nppv`, `ph`, `phyc`. + - Or these strings must appear as substrings within the variable names (e.g. `o2_glor` is acceptable for `o2`). +4. Bathymetry data files must contain a variable named `deptho`. + +#### Also of note + +1. Whilst not mandatory to use data downloaded only from the Copernicus Marine Service (any existing data you may hold can be re-organised accordingly), the assumptions made by VirtualShip regarding directory structure and filename conventions are motivated by alignment with the Copernicus Marine Service's practices. + - If you want to use pre-existing data with VirtualShip, which you may have accessed from a different source, it is possible to do so by restructuring and/or renaming your data files as necessary. +2. The whole VirtualShip pre-downloaded data workflow should support global data or subsets thereof, provided the data files contain the necessary variables and are structured as outlined above. + +#### For example... + +Example Python code for automating the data download from Copernicus Marine can be found in [Example Copernicus Download](example_copernicus_download.ipynb). + + diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md index 2578fbea..fac1c26c 100644 --- a/docs/user-guide/index.md +++ b/docs/user-guide/index.md @@ -15,4 +15,6 @@ assignments/index :maxdepth: 1 documentation/copernicus_products.md +documentation/pre_download_data.md +documentation/example_copernicus_download.ipynb ``` diff --git a/docs/user-guide/quickstart.md b/docs/user-guide/quickstart.md index 2f1d4f63..2d648723 100644 --- a/docs/user-guide/quickstart.md +++ b/docs/user-guide/quickstart.md @@ -1,5 +1,11 @@ # VirtualShip Quickstart Guide 🚢 +```{warning} +This quickstart guide is designed for use with VirtualShip v0.2.2 and currently out of date with the latest version of VirtualShip (v1.0.0). It will be updated soon. + +In particular, the `virtualship fetch` command is no longer supported. Instead, data fetching is now integrated into the `virtualship run` command. See [#226](https://github.com/Parcels-code/virtualship/pull/226) for details in the meantime. +``` + Welcome to this Quickstart to using VirtualShip. In this guide we will conduct a virtual expedition in the North Sea. Note, however, that you can plan your own expedition anywhere in the global ocean and conduct whatever set of measurements you wish! This Quickstart is available as an instructional video below, or you can continue with the step-by-step guide. @@ -46,10 +52,10 @@ virtualship init EXPEDITION_NAME --from-mfp CoordinatesExport.xlsx The `CoordinatesExport.xlsx` in the `virtualship init` command refers to the .xlsx file exported from MFP. Replace the filename with the name of your exported .xlsx file (and make sure to move it from the Downloads to the folder/directory in which you are running the expedition). ``` -This will create a folder/directory called `EXPEDITION_NAME` with two files: `schedule.yaml` and `ship_config.yaml` based on the sampling site coordinates that you specified in your MFP export. The `--from-mfp` flag indictates that the exported coordinates will be used. +This will create a folder/directory called `EXPEDITION_NAME` with a single file: `expedition.yaml` containing details on the ship and instrument configurations, as well as the expedition schedule based on the sampling site coordinates that you specified in your MFP export. The `--from-mfp` flag indicates that the exported coordinates will be used. ```{note} -For advanced users: it is also possible to run the expedition initialisation step without an MFP .xlsx export file. In this case you should simply run `virtualship init EXPEDITION_NAME` in the CLI. This will write example `schedule.yaml` and `ship_config.yaml` files in the `EXPEDITION_NAME` folder/directory. These files contain example waypoints, timings and instrument selections, but can be edited or propagated through the rest of the workflow unedited to run a sample expedition. +For advanced users: it is also possible to run the expedition initialisation step without an MFP .xlsx export file. In this case you should simply run `virtualship init EXPEDITION_NAME` in the CLI. This will write an example `expedition.yaml` file in the `EXPEDITION_NAME` folder/directory. This file contains example waypoints, timings, instrument selections, and ship configuration, but can be edited or propagated through the rest of the workflow unedited to run a sample expedition. ``` ## Expedition scheduling & ship configuration @@ -61,7 +67,7 @@ virtualship plan EXPEDITION_NAME ``` ```{tip} -Using the `virtualship plan` tool is optional. Advanced users can also edit the `schedule.yaml` and `ship_config.yaml` files directly if preferred. +Using the `virtualship plan` tool is optional. Advanced users can also edit the `expedition.yaml` file directly if preferred. ``` The planning tool should look something like this and offers an intuitive way to make your selections: @@ -115,7 +121,7 @@ For advanced users: you can also make further customisations to behaviours of al When you are happy with your ship configuration and schedule plan, press _Save Changes_. ```{note} -On pressing _Save Changes_ the tool will check the selections are valid (for example that the ship will be able to reach each waypoint in time). If they are, the changes will be saved to the `ship_config.yaml` and `schedule.yaml` files, ready for the next steps. If your selections are invalid you should be provided with information on how to fix them. +On pressing _Save Changes_ the tool will check the selections are valid (for example that the ship will be able to reach each waypoint in time). If they are, the changes will be saved to the `expedition.yaml` file, ready for the next steps. If your selections are invalid you should be provided with information on how to fix them. ``` ## Fetch the data @@ -151,5 +157,3 @@ It might take up to an hour to simulate the measurements depending on your choic Upon successfully completing the simulation, results from the expedition will be stored in the `EXPEDITION_NAME/results` directory, written as [Zarr](https://zarr.dev/) files. From here you can carry on your analysis (offline). We encourage you to explore and analyse these data using [Xarray](https://docs.xarray.dev/en/stable/). We also provide various further [VirtualShip tutorials](https://virtualship.readthedocs.io/en/latest/user-guide/tutorials/index.html) which provide examples of how to visualise data recorded by the VirtualShip instruments. - - diff --git a/docs/user-guide/tutorials/Argo_data_tutorial.ipynb b/docs/user-guide/tutorials/Argo_data_tutorial.ipynb index 30cee460..e8235315 100644 --- a/docs/user-guide/tutorials/Argo_data_tutorial.ipynb +++ b/docs/user-guide/tutorials/Argo_data_tutorial.ipynb @@ -14,7 +14,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": null, "metadata": {}, "outputs": [], "source": [ @@ -28,25 +28,26 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We have downloaded the data from Copernicus Marine Service, using `virtualship fetch` and the information in following `schedule.yaml` file:\n", + "We have downloaded the data from Copernicus Marine Service, using `virtualship fetch` and the information in following `schedule` section of the `expedition.yaml` file:\n", "```yaml\n", - "space_time_region:\n", - " spatial_range:\n", - " minimum_longitude: -5\n", - " maximum_longitude: 5\n", - " minimum_latitude: -5\n", - " maximum_latitude: 5\n", - " minimum_depth: 0\n", - " maximum_depth: 2000\n", - " time_range:\n", - " start_time: 2023-01-01 00:00:00\n", - " end_time: 2023-02-01 00:00:00\n", - "waypoints:\n", - " - instrument: ARGO_FLOAT\n", - " location:\n", - " latitude: 0.02\n", - " longitude: 0.02\n", - " time: 2023-01-01 02:00:00\n", + "schedule:\n", + " space_time_region:\n", + " spatial_range:\n", + " minimum_longitude: -5\n", + " maximum_longitude: 5\n", + " minimum_latitude: -5\n", + " maximum_latitude: 5\n", + " minimum_depth: 0\n", + " maximum_depth: 2000\n", + " time_range:\n", + " start_time: 2023-01-01 00:00:00\n", + " end_time: 2023-02-01 00:00:00\n", + " waypoints:\n", + " - instrument: ARGO_FLOAT\n", + " location:\n", + " latitude: 0.02\n", + " longitude: 0.02\n", + " time: 2023-01-01 02:00:00\n", "```\n", "\n", "After running `virtualship run`, we have a `results/argo_floats.zarr` file with the data from the float." @@ -54,7 +55,7 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -79,7 +80,7 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -111,7 +112,7 @@ ], "metadata": { "kernelspec": { - "display_name": "parcels", + "display_name": "ship", "language": "python", "name": "python3" }, @@ -125,7 +126,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.12.3" + "version": "3.12.9" } }, "nbformat": 4, diff --git a/src/virtualship/cli/_creds.py b/src/virtualship/cli/_creds.py deleted file mode 100644 index 9f1d2435..00000000 --- a/src/virtualship/cli/_creds.py +++ /dev/null @@ -1,103 +0,0 @@ -from __future__ import annotations - -from pathlib import Path - -import click -import pydantic -import yaml - -from virtualship.errors import CredentialFileError - -CREDENTIALS_FILE = "credentials.yaml" - - -class Credentials(pydantic.BaseModel): - """Credentials to be used in `virtualship fetch` command.""" - - COPERNICUS_USERNAME: str - COPERNICUS_PASSWORD: str - - @classmethod - def from_yaml(cls, path: str | Path) -> Credentials: - """ - Load credentials from a yaml file. - - :param path: Path to the file to load from. - :returns Credentials: The credentials. - """ - with open(path) as file: - data = yaml.safe_load(file) - - if not isinstance(data, dict): - raise CredentialFileError("Credential file is of an invalid format.") - - return cls(**data) - - def dump(self) -> str: - """ - Dump credentials to a yaml string. - - :param creds: The credentials to dump. - :returns str: The yaml string. - """ - return yaml.safe_dump(self.model_dump()) - - def to_yaml(self, path: str | Path) -> None: - """ - Write credentials to a yaml file. - - :param path: Path to the file to write to. - """ - with open(path, "w") as file: - file.write(self.dump()) - - -def get_dummy_credentials_yaml() -> str: - return ( - Credentials( - COPERNICUS_USERNAME="my_username", COPERNICUS_PASSWORD="my_password" - ) - .dump() - .strip() - ) - - -def get_credentials_flow( - username: str | None, password: str | None, creds_path: Path -) -> tuple[str, str]: - """ - Execute flow of getting credentials for use in the `fetch` command. - - - If username and password are provided via CLI, use them (ignore the credentials file if exists). - - If username and password are not provided, try to load them from the credentials file. - - If no credentials are provided, print a message on how to make credentials file and prompt for credentials. - - :param username: The username provided via CLI. - :param password: The password provided via CLI. - :param creds_path: The path to the credentials file. - """ - if username and password: - if creds_path.exists(): - click.echo( - f"Credentials file exists at '{creds_path}', but username and password are already provided. Ignoring credentials file." - ) - return username, password - - try: - creds = Credentials.from_yaml(creds_path) - click.echo(f"Loaded credentials from '{creds_path}'.") - return creds.COPERNICUS_USERNAME, creds.COPERNICUS_PASSWORD - except FileNotFoundError: - msg = f"""Credentials not provided. Credentials can be obtained from https://data.marine.copernicus.eu/register. Either pass in via `--username` and `--password` arguments, or via config file at '{creds_path}'. Config file should be YAML along following format: -### {creds_path} - -{get_dummy_credentials_yaml().strip()} - -### - -Prompting for credentials instead... -""" - click.echo(msg) - username = click.prompt("username") - password = click.prompt("password", hide_input=True) - return username, password diff --git a/src/virtualship/cli/_fetch.py b/src/virtualship/cli/_fetch.py deleted file mode 100644 index 4e816f91..00000000 --- a/src/virtualship/cli/_fetch.py +++ /dev/null @@ -1,563 +0,0 @@ -from __future__ import annotations - -import hashlib -import shutil -from datetime import datetime, timedelta -from pathlib import Path -from typing import TYPE_CHECKING - -import numpy as np -from pydantic import BaseModel - -from virtualship.errors import CopernicusCatalogueError, IncompleteDownloadError -from virtualship.utils import ( - _dump_yaml, - _generic_load_yaml, - _get_schedule, - _get_ship_config, -) - -if TYPE_CHECKING: - from virtualship.models import SpaceTimeRegion - -import click -import copernicusmarine -from copernicusmarine.core_functions.credentials_utils import InvalidUsernameOrPassword - -import virtualship.cli._creds as creds -from virtualship.utils import SCHEDULE - -DOWNLOAD_METADATA = "download_metadata.yaml" - - -def _fetch(path: str | Path, username: str | None, password: str | None) -> None: - """ - Download input data for an expedition. - - Entrypoint for the tool to download data based on space-time region provided in the - schedule file. Data is downloaded from Copernicus Marine, credentials for which can be - obtained via registration: https://data.marine.copernicus.eu/register . Credentials can - be provided on prompt, via command line arguments, or via a YAML config file. Run - `virtualship fetch` on an expedition for more info. - """ - from virtualship.models import InstrumentType - - if sum([username is None, password is None]) == 1: - raise ValueError("Both username and password must be provided when using CLI.") - - path = Path(path) - - data_folder = path / "data" - data_folder.mkdir(exist_ok=True) - - schedule = _get_schedule(path) - ship_config = _get_ship_config(path) - - schedule.verify( - ship_config.ship_speed_knots, - input_data=None, - check_space_time_region=True, - ignore_missing_fieldsets=True, - ) - - space_time_region_hash = get_space_time_region_hash(schedule.space_time_region) - - existing_download = get_existing_download(data_folder, space_time_region_hash) - if existing_download is not None: - click.echo( - f"Data download for space-time region already completed ('{existing_download}')." - ) - return - - creds_path = path / creds.CREDENTIALS_FILE - username, password = creds.get_credentials_flow(username, password, creds_path) - - # Extract space_time_region details from the schedule - spatial_range = schedule.space_time_region.spatial_range - time_range = schedule.space_time_region.time_range - start_datetime = time_range.start_time - end_datetime = time_range.end_time - instruments_in_schedule = schedule.get_instruments() - - # Create download folder and set download metadata - download_folder = data_folder / hash_to_filename(space_time_region_hash) - download_folder.mkdir() - DownloadMetadata(download_complete=False).to_yaml( - download_folder / DOWNLOAD_METADATA - ) - shutil.copyfile(path / SCHEDULE, download_folder / SCHEDULE) - - # data download - if ( - ( - {"XBT", "CTD", "CDT_BGC", "SHIP_UNDERWATER_ST"} - & set(instrument.name for instrument in instruments_in_schedule) - ) - or ship_config.ship_underwater_st_config is not None - or ship_config.adcp_config is not None - ): - print("Ship data will be downloaded. Please wait...") - - phys_product_id = select_product_id( - physical=True, - schedule_start=start_datetime, - schedule_end=end_datetime, - username=username, - password=password, - ) - - # Define all ship datasets to download, including bathymetry - download_dict = { - "Bathymetry": { - "dataset_id": "cmems_mod_glo_phy_my_0.083deg_static", - "variables": ["deptho"], - "output_filename": "bathymetry.nc", - }, - "UVdata": { - "dataset_id": phys_product_id, - "variables": ["uo", "vo"], - "output_filename": "ship_uv.nc", - }, - "Sdata": { - "dataset_id": phys_product_id, - "variables": ["so"], - "output_filename": "ship_s.nc", - }, - "Tdata": { - "dataset_id": phys_product_id, - "variables": ["thetao"], - "output_filename": "ship_t.nc", - }, - } - - # Iterate over all datasets and download each based on space_time_region - try: - for dataset in download_dict.values(): - copernicusmarine.subset( - dataset_id=dataset["dataset_id"], - variables=dataset["variables"], - minimum_longitude=spatial_range.minimum_longitude, - maximum_longitude=spatial_range.maximum_longitude, - minimum_latitude=spatial_range.minimum_latitude, - maximum_latitude=spatial_range.maximum_latitude, - start_datetime=start_datetime, - end_datetime=end_datetime, - minimum_depth=abs(spatial_range.minimum_depth), - maximum_depth=abs(spatial_range.maximum_depth), - output_filename=dataset["output_filename"], - output_directory=download_folder, - username=username, - password=password, - overwrite=True, - coordinates_selection_method="outside", - ) - except InvalidUsernameOrPassword as e: - shutil.rmtree(download_folder) - raise e - - click.echo("Ship data download based on space-time region completed.") - - if InstrumentType.DRIFTER in instruments_in_schedule: - print("Drifter data will be downloaded. Please wait...") - drifter_download_dict = { - "UVdata": { - "dataset_id": phys_product_id, - "variables": ["uo", "vo"], - "output_filename": "drifter_uv.nc", - }, - "Tdata": { - "dataset_id": phys_product_id, - "variables": ["thetao"], - "output_filename": "drifter_t.nc", - }, - } - - # Iterate over all datasets and download each based on space_time_region - try: - for dataset in drifter_download_dict.values(): - copernicusmarine.subset( - dataset_id=dataset["dataset_id"], - variables=dataset["variables"], - minimum_longitude=spatial_range.minimum_longitude - 3.0, - maximum_longitude=spatial_range.maximum_longitude + 3.0, - minimum_latitude=spatial_range.minimum_latitude - 3.0, - maximum_latitude=spatial_range.maximum_latitude + 3.0, - start_datetime=start_datetime, - end_datetime=end_datetime + timedelta(days=21), - minimum_depth=abs(1), - maximum_depth=abs(1), - output_filename=dataset["output_filename"], - output_directory=download_folder, - username=username, - password=password, - overwrite=True, - coordinates_selection_method="outside", - ) - except InvalidUsernameOrPassword as e: - shutil.rmtree(download_folder) - raise e - - click.echo("Drifter data download based on space-time region completed.") - - if InstrumentType.ARGO_FLOAT in instruments_in_schedule: - print("Argo float data will be downloaded. Please wait...") - argo_download_dict = { - "UVdata": { - "dataset_id": phys_product_id, - "variables": ["uo", "vo"], - "output_filename": "argo_float_uv.nc", - }, - "Sdata": { - "dataset_id": phys_product_id, - "variables": ["so"], - "output_filename": "argo_float_s.nc", - }, - "Tdata": { - "dataset_id": phys_product_id, - "variables": ["thetao"], - "output_filename": "argo_float_t.nc", - }, - } - - # Iterate over all datasets and download each based on space_time_region - try: - for dataset in argo_download_dict.values(): - copernicusmarine.subset( - dataset_id=dataset["dataset_id"], - variables=dataset["variables"], - minimum_longitude=spatial_range.minimum_longitude - 3.0, - maximum_longitude=spatial_range.maximum_longitude + 3.0, - minimum_latitude=spatial_range.minimum_latitude - 3.0, - maximum_latitude=spatial_range.maximum_latitude + 3.0, - start_datetime=start_datetime, - end_datetime=end_datetime + timedelta(days=21), - minimum_depth=abs(1), - maximum_depth=abs(spatial_range.maximum_depth), - output_filename=dataset["output_filename"], - output_directory=download_folder, - username=username, - password=password, - overwrite=True, - coordinates_selection_method="outside", - ) - except InvalidUsernameOrPassword as e: - shutil.rmtree(download_folder) - raise e - - click.echo("Argo_float data download based on space-time region completed.") - - if InstrumentType.CTD_BGC in instruments_in_schedule: - print("CTD_BGC data will be downloaded. Please wait...") - - bgc_args = { - "physical": False, - "schedule_start": start_datetime, - "schedule_end": end_datetime, - "username": username, - "password": password, - } - - ctd_bgc_download_dict = { - "o2data": { - "dataset_id": select_product_id(**{**bgc_args, "variable": "o2"}), - "variables": ["o2"], - "output_filename": "ctd_bgc_o2.nc", - }, - "chlorodata": { - "dataset_id": select_product_id(**{**bgc_args, "variable": "chl"}), - "variables": ["chl"], - "output_filename": "ctd_bgc_chl.nc", - }, - "nitratedata": { - "dataset_id": select_product_id(**{**bgc_args, "variable": "no3"}), - "variables": ["no3"], - "output_filename": "ctd_bgc_no3.nc", - }, - "phosphatedata": { - "dataset_id": select_product_id(**{**bgc_args, "variable": "po4"}), - "variables": ["po4"], - "output_filename": "ctd_bgc_po4.nc", - }, - "phdata": { - "dataset_id": select_product_id( - **{**bgc_args, "variable": "ph"} - ), # this will be monthly resolution if reanalysis(_interim) period - "variables": ["ph"], - "output_filename": "ctd_bgc_ph.nc", - }, - "phytoplanktondata": { - "dataset_id": select_product_id( - **{**bgc_args, "variable": "phyc"} - ), # this will be monthly resolution if reanalysis(_interim) period, - "variables": ["phyc"], - "output_filename": "ctd_bgc_phyc.nc", - }, - "primaryproductiondata": { - "dataset_id": select_product_id(**{**bgc_args, "variable": "nppv"}), - "variables": ["nppv"], - "output_filename": "ctd_bgc_nppv.nc", - }, - } - - # Iterate over all datasets and download each based on space_time_region - try: - for dataset in ctd_bgc_download_dict.values(): - copernicusmarine.subset( - dataset_id=dataset["dataset_id"], - variables=dataset["variables"], - minimum_longitude=spatial_range.minimum_longitude - 3.0, - maximum_longitude=spatial_range.maximum_longitude + 3.0, - minimum_latitude=spatial_range.minimum_latitude - 3.0, - maximum_latitude=spatial_range.maximum_latitude + 3.0, - start_datetime=start_datetime, - end_datetime=end_datetime + timedelta(days=21), - minimum_depth=abs(1), - maximum_depth=abs(spatial_range.maximum_depth), - output_filename=dataset["output_filename"], - output_directory=download_folder, - username=username, - password=password, - overwrite=True, - coordinates_selection_method="outside", - ) - except InvalidUsernameOrPassword as e: - shutil.rmtree(download_folder) - raise e - - click.echo("CTD_BGC data download based on space-time region completed.") - - complete_download(download_folder) - - -def _hash(s: str, *, length: int) -> str: - """Create a hash of a string.""" - assert length % 2 == 0, "Length must be even." - half_length = length // 2 - - return hashlib.shake_128(s.encode("utf-8")).hexdigest(half_length) - - -def create_hash(s: str) -> str: - """Create an 8 digit hash of a string.""" - return _hash(s, length=8) - - -def hash_model(model: BaseModel, salt: int = 0) -> str: - """ - Hash a Pydantic model. - - :param region: The region to hash. - :param salt: Salt to add to the hash. - :returns: The hash. - """ - return create_hash(model.model_dump_json() + str(salt)) - - -def get_space_time_region_hash(space_time_region: SpaceTimeRegion) -> str: - # Increment salt in the event of breaking data fetching changes with prior versions - # of virtualship where you want to force new hashes (i.e., new data downloads) - salt = 0 - return hash_model(space_time_region, salt=salt) - - -def filename_to_hash(filename: str) -> str: - """Extract hash from filename of the format YYYYMMDD_HHMMSS_{hash}.""" - parts = filename.split("_") - if len(parts) != 3: - raise ValueError( - f"Filename '{filename}' must have 3 parts delimited with underscores." - ) - return parts[-1] - - -def hash_to_filename(hash: str) -> str: - """Return a filename of the format YYYYMMDD_HHMMSS_{hash}.""" - if "_" in hash: - raise ValueError("Hash cannot contain underscores.") - return f"{datetime.now().strftime('%Y%m%d_%H%M%S')}_{hash}" - - -class DownloadMetadata(BaseModel): - """Metadata for a data download.""" - - download_complete: bool - download_date: datetime | None = None - - def to_yaml(self, file_path: str | Path) -> None: - with open(file_path, "w") as file: - _dump_yaml(self, file) - - @classmethod - def from_yaml(cls, file_path: str | Path) -> DownloadMetadata: - return _generic_load_yaml(file_path, cls) - - -def get_existing_download( - data_folder: Path, space_time_region_hash: str -) -> Path | None: - """Check if a download has already been completed. If so, return the path for existing download.""" - for download_path in data_folder.rglob("*"): - try: - hash = filename_to_hash(download_path.name) - except ValueError: - continue - - if hash == space_time_region_hash: - assert_complete_download(download_path) - return download_path - - return None - - -def assert_complete_download(download_path: Path) -> None: - download_metadata = download_path / DOWNLOAD_METADATA - try: - with open(download_metadata) as file: - assert DownloadMetadata.from_yaml(file).download_complete - except (FileNotFoundError, AssertionError) as e: - raise IncompleteDownloadError( - f"Download at {download_path} was found, but looks to be incomplete " - f"(likely due to interupting it mid-download). Please delete this folder and retry." - ) from e - return - - -def complete_download(download_path: Path) -> None: - """Mark a download as complete.""" - download_metadata = download_path / DOWNLOAD_METADATA - metadata = DownloadMetadata(download_complete=True, download_date=datetime.now()) - metadata.to_yaml(download_metadata) - return - - -def select_product_id( - physical: bool, - schedule_start: datetime, - schedule_end: datetime, - username: str, - password: str, - variable: str | None = None, # only needed for BGC datasets -) -> str: - """ - Determine which copernicus product id should be selected (reanalysis, reanalysis-interim, analysis & forecast), for prescribed schedule and physical vs. BGC. - - BGC is more complicated than physical products. Often (re)analysis period and variable dependent, hence more custom logic here. - """ - product_ids = { - "phys": { - "reanalysis": "cmems_mod_glo_phy_my_0.083deg_P1D-m", - "reanalysis_interim": "cmems_mod_glo_phy_myint_0.083deg_P1D-m", - "analysis": "cmems_mod_glo_phy_anfc_0.083deg_P1D-m", - }, - "bgc": { - "reanalysis": "cmems_mod_glo_bgc_my_0.25deg_P1D-m", - "reanalysis_interim": "cmems_mod_glo_bgc_myint_0.25deg_P1D-m", - "analysis": None, # will be set per variable - }, - } - - bgc_analysis_ids = { - "o2": "cmems_mod_glo_bgc-bio_anfc_0.25deg_P1D-m", - "chl": "cmems_mod_glo_bgc-pft_anfc_0.25deg_P1D-m", - "no3": "cmems_mod_glo_bgc-nut_anfc_0.25deg_P1D-m", - "po4": "cmems_mod_glo_bgc-nut_anfc_0.25deg_P1D-m", - "ph": "cmems_mod_glo_bgc-car_anfc_0.25deg_P1D-m", - "phyc": "cmems_mod_glo_bgc-pft_anfc_0.25deg_P1D-m", - "nppv": "cmems_mod_glo_bgc-bio_anfc_0.25deg_P1D-m", - } - - # pH and phytoplankton variables are available as *monthly* products only in renalysis(_interim) period - monthly_bgc_reanalysis_ids = { - "ph": "cmems_mod_glo_bgc_my_0.25deg_P1M-m", - "phyc": "cmems_mod_glo_bgc_my_0.25deg_P1M-m", - } - monthly_bgc_reanalysis_interim_ids = { - "ph": "cmems_mod_glo_bgc_myint_0.25deg_P1M-m", - "phyc": "cmems_mod_glo_bgc_myint_0.25deg_P1M-m", - } - - key = "phys" if physical else "bgc" - selected_id = None - - for period, pid in product_ids[key].items(): - # for BGC analysis, set pid per variable - if key == "bgc" and period == "analysis": - if variable is None or variable not in bgc_analysis_ids: - continue - pid = bgc_analysis_ids[variable] - # for BGC reanalysis, check if requires monthly product - if ( - key == "bgc" - and period == "reanalysis" - and variable in monthly_bgc_reanalysis_ids - ): - monthly_pid = monthly_bgc_reanalysis_ids[variable] - ds_monthly = copernicusmarine.open_dataset( - monthly_pid, - username=username, - password=password, - ) - time_end_monthly = ds_monthly["time"][-1].values - if np.datetime64(schedule_end) <= time_end_monthly: - pid = monthly_pid - # for BGC reanalysis_interim, check if requires monthly product - if ( - key == "bgc" - and period == "reanalysis_interim" - and variable in monthly_bgc_reanalysis_interim_ids - ): - monthly_pid = monthly_bgc_reanalysis_interim_ids[variable] - ds_monthly = copernicusmarine.open_dataset( - monthly_pid, username=username, password=password - ) - time_end_monthly = ds_monthly["time"][-1].values - if np.datetime64(schedule_end) <= time_end_monthly: - pid = monthly_pid - if pid is None: - continue - ds = copernicusmarine.open_dataset(pid, username=username, password=password) - time_end = ds["time"][-1].values - if np.datetime64(schedule_end) <= time_end: - selected_id = pid - break - - if selected_id is None: - raise CopernicusCatalogueError( - "No suitable product found in the Copernicus Marine Catalogue for the scheduled time and variable." - ) - - # handle the rare situation where start time and end time span different products, which is possible for reanalysis and reanalysis_interim - # in this case, return the analysis product which spans far back enough - if start_end_in_product_timerange( - selected_id, schedule_start, schedule_end, username, password - ): - return selected_id - - else: - return ( - product_ids["phys"]["analysis"] if physical else bgc_analysis_ids[variable] - ) - - -def start_end_in_product_timerange( - selected_id: str, - schedule_start: datetime, - schedule_end: datetime, - username: str, - password: str, -) -> bool: - """Check schedule_start and schedule_end are both within a selected Copernicus product's time range.""" - ds_selected = copernicusmarine.open_dataset( - selected_id, username=username, password=password - ) - time_values = ds_selected["time"].values - time_min, time_max = np.min(time_values), np.max(time_values) - - if ( - np.datetime64(schedule_start) >= time_min - and np.datetime64(schedule_end) <= time_max - ): - return True - - else: - return False diff --git a/src/virtualship/cli/_plan.py b/src/virtualship/cli/_plan.py index c426dade..8b2adb15 100644 --- a/src/virtualship/cli/_plan.py +++ b/src/virtualship/cli/_plan.py @@ -1,7 +1,6 @@ import datetime import os import traceback -from typing import ClassVar from textual import on from textual.app import App, ComposeResult @@ -30,23 +29,23 @@ type_to_textual, ) from virtualship.errors import UnexpectedError, UserError -from virtualship.models.location import Location -from virtualship.models.schedule import Schedule, Waypoint -from virtualship.models.ship_config import ( +from virtualship.instruments.types import InstrumentType +from virtualship.models import ( ADCPConfig, ArgoFloatConfig, CTD_BGCConfig, CTDConfig, DrifterConfig, - InstrumentType, + Expedition, + Location, ShipConfig, ShipUnderwaterSTConfig, - XBTConfig, -) -from virtualship.models.space_time_region import ( SpatialRange, TimeRange, + Waypoint, + XBTConfig, ) +from virtualship.utils import EXPEDITION UNEXPECTED_MSG_ONSAVE = ( "Please ensure that:\n" @@ -81,227 +80,236 @@ def log_exception_to_file( f.write("\n") -class WaypointWidget(Static): - def __init__(self, waypoint: Waypoint, index: int): +DEFAULT_TS_CONFIG = {"period_minutes": 5.0} + +DEFAULT_ADCP_CONFIG = { + "num_bins": 40, + "period_minutes": 5.0, +} + +INSTRUMENT_FIELDS = { + "adcp_config": { + "class": ADCPConfig, + "title": "Onboard ADCP", + "attributes": [ + {"name": "num_bins"}, + {"name": "period", "minutes": True}, + ], + }, + "ship_underwater_st_config": { + "class": ShipUnderwaterSTConfig, + "title": "Onboard Temperature/Salinity", + "attributes": [ + {"name": "period", "minutes": True}, + ], + }, + "ctd_config": { + "class": CTDConfig, + "title": "CTD", + "attributes": [ + {"name": "max_depth_meter"}, + {"name": "min_depth_meter"}, + {"name": "stationkeeping_time", "minutes": True}, + ], + }, + "ctd_bgc_config": { + "class": CTD_BGCConfig, + "title": "CTD-BGC", + "attributes": [ + {"name": "max_depth_meter"}, + {"name": "min_depth_meter"}, + {"name": "stationkeeping_time", "minutes": True}, + ], + }, + "xbt_config": { + "class": XBTConfig, + "title": "XBT", + "attributes": [ + {"name": "min_depth_meter"}, + {"name": "max_depth_meter"}, + {"name": "fall_speed_meter_per_second"}, + {"name": "deceleration_coefficient"}, + ], + }, + "argo_float_config": { + "class": ArgoFloatConfig, + "title": "Argo Float", + "attributes": [ + {"name": "min_depth_meter"}, + {"name": "max_depth_meter"}, + {"name": "drift_depth_meter"}, + {"name": "vertical_speed_meter_per_second"}, + {"name": "cycle_days"}, + {"name": "drift_days"}, + ], + }, + "drifter_config": { + "class": DrifterConfig, + "title": "Drifter", + "attributes": [ + {"name": "depth_meter"}, + {"name": "lifetime", "minutes": True}, + ], + }, +} + + +class ExpeditionEditor(Static): + def __init__(self, path: str): super().__init__() - self.waypoint = waypoint - self.index = index + self.path = path + self.expedition = None def compose(self) -> ComposeResult: try: - with Collapsible( - title=f"[b]Waypoint {self.index + 1}[/b]", - collapsed=True, - id=f"wp{self.index + 1}", - ): - if self.index > 0: - yield Button( - "Copy Time & Instruments from Previous", - id=f"wp{self.index}_copy", - variant="warning", - ) - yield Label("Location:") - yield Label(" Latitude:") - yield Input( - id=f"wp{self.index}_lat", - value=str(self.waypoint.location.lat) - if self.waypoint.location.lat - is not None # is not None to handle if lat is 0.0 - else "", - validators=[ - Function( - is_valid_lat, - f"INVALID: value must be {is_valid_lat.__doc__.lower()}", - ) - ], - type="number", - placeholder="°N", - classes="latitude-input", - ) - yield Label( - "", - id=f"validation-failure-label-wp{self.index}_lat", - classes="-hidden validation-failure", - ) + self.expedition = Expedition.from_yaml(self.path.joinpath(EXPEDITION)) + except Exception as e: + raise UserError( + f"There is an issue in {self.path.joinpath(EXPEDITION)}:\n\n{e}" + ) from None - yield Label(" Longitude:") - yield Input( - id=f"wp{self.index}_lon", - value=str(self.waypoint.location.lon) - if self.waypoint.location.lon - is not None # is not None to handle if lon is 0.0 - else "", - validators=[ - Function( - is_valid_lon, - f"INVALID: value must be {is_valid_lon.__doc__.lower()}", - ) - ], - type="number", - placeholder="°E", - classes="longitude-input", - ) - yield Label( - "", - id=f"validation-failure-label-wp{self.index}_lon", - classes="-hidden validation-failure", - ) + try: + ## 1) SHIP SPEED & INSTRUMENTS CONFIG EDITOR - yield Label("Time:") - with Horizontal(): - yield Label("Year:") - yield Select( - [ - (str(year), year) - # TODO: change from hard coding? ...flexibility for different datasets... - for year in range( - 1993, - datetime.datetime.now().year + 1, - ) - ], - id=f"wp{self.index}_year", - value=int(self.waypoint.time.year) - if self.waypoint.time - else Select.BLANK, - prompt="YYYY", - classes="year-select", - ) - yield Label("Month:") - yield Select( - [(f"{m:02d}", m) for m in range(1, 13)], - id=f"wp{self.index}_month", - value=int(self.waypoint.time.month) - if self.waypoint.time - else Select.BLANK, - prompt="MM", - classes="month-select", - ) - yield Label("Day:") - yield Select( - [(f"{d:02d}", d) for d in range(1, 32)], - id=f"wp{self.index}_day", - value=int(self.waypoint.time.day) - if self.waypoint.time - else Select.BLANK, - prompt="DD", - classes="day-select", - ) - yield Label("Hour:") - yield Select( - [(f"{h:02d}", h) for h in range(24)], - id=f"wp{self.index}_hour", - value=int(self.waypoint.time.hour) - if self.waypoint.time - else Select.BLANK, - prompt="hh", - classes="hour-select", - ) - yield Label("Min:") - yield Select( - [(f"{m:02d}", m) for m in range(0, 60, 5)], - id=f"wp{self.index}_minute", - value=int(self.waypoint.time.minute) - if self.waypoint.time - else Select.BLANK, - prompt="mm", - classes="minute-select", - ) + yield Label( + "[b]Ship & Instruments Config Editor[/b]", + id="title_ship_instruments_config", + markup=True, + ) + yield Rule(line_style="heavy") - yield Label("Instruments:") - for instrument in InstrumentType: - is_selected = instrument in (self.waypoint.instrument or []) - with Horizontal(): - yield Label(instrument.value) - yield Switch( - value=is_selected, id=f"wp{self.index}_{instrument.value}" - ) + # SECTION: "Ship Speed & Onboard Measurements" - if instrument.value == "DRIFTER": - yield Label("Count") - yield Input( - id=f"wp{self.index}_drifter_count", - value=str( - self.get_drifter_count() if is_selected else "" - ), - type="integer", - placeholder="# of drifters", - validators=Integer( - minimum=1, - failure_description="INVALID: value must be > 0", - ), - classes="drifter-count-input", - ) - yield Label( - "", - id=f"validation-failure-label-wp{self.index}_drifter_count", - classes="-hidden validation-failure", + with Collapsible( + title="[b]Ship Speed & Onboard Measurements[/b]", + id="speed_collapsible", + collapsed=False, + ): + attr = "ship_speed_knots" + validators = group_validators(ShipConfig, attr) + with Horizontal(classes="ship_speed"): + yield Label("[b]Ship Speed (knots):[/b]") + yield Input( + id="speed", + type=type_to_textual(get_field_type(ShipConfig, attr)), + validators=[ + Function( + validator, + f"INVALID: value must be {validator.__doc__.lower()}", ) + for validator in validators + ], + classes="ship_speed_input", + placeholder="knots", + value=str( + self.expedition.ship_config.ship_speed_knots + if self.expedition.ship_config.ship_speed_knots + else "" + ), + ) + yield Label("", id="validation-failure-label-speed", classes="-hidden") - except Exception as e: - raise UnexpectedError(unexpected_msg_compose(e)) from None - - def get_drifter_count(self) -> int: - return sum( - 1 for inst in self.waypoint.instrument if inst == InstrumentType.DRIFTER - ) - - def copy_from_previous(self) -> None: - """Copy inputs from previous waypoint widget (time and instruments only, not lat/lon).""" - try: - if self.index > 0: - schedule_editor = self.parent - if schedule_editor: - time_components = ["year", "month", "day", "hour", "minute"] - for comp in time_components: - prev = schedule_editor.query_one(f"#wp{self.index - 1}_{comp}") - curr = self.query_one(f"#wp{self.index}_{comp}") - if prev and curr: - curr.value = prev.value - - for instrument in InstrumentType: - prev_switch = schedule_editor.query_one( - f"#wp{self.index - 1}_{instrument.value}" - ) - curr_switch = self.query_one( - f"#wp{self.index}_{instrument.value}" - ) - if prev_switch and curr_switch: - curr_switch.value = prev_switch.value - except Exception as e: - raise UnexpectedError(unexpected_msg_compose(e)) from None + with Horizontal(classes="ts-section"): + yield Label("[b]Onboard Temperature/Salinity:[/b]") + yield Switch( + value=bool( + self.expedition.instruments_config.ship_underwater_st_config + ), + id="has_onboard_ts", + ) - @on(Button.Pressed, "Button") - def button_pressed(self, event: Button.Pressed) -> None: - if event.button.id == f"wp{self.index}_copy": - self.copy_from_previous() + with Horizontal(classes="adcp-section"): + yield Label("[b]Onboard ADCP:[/b]") + yield Switch( + value=bool(self.expedition.instruments_config.adcp_config), + id="has_adcp", + ) - @on(Switch.Changed) - def on_switch_changed(self, event: Switch.Changed) -> None: - if event.switch.id == f"wp{self.index}_DRIFTER": - drifter_count_input = self.query_one( - f"#wp{self.index}_drifter_count", Input - ) - if not event.value: - drifter_count_input.value = "" - else: - if not drifter_count_input.value: - drifter_count_input.value = "1" + # adcp type selection + with Horizontal(id="adcp_type_container", classes="-hidden"): + is_deep = ( + self.expedition.instruments_config.adcp_config + and self.expedition.instruments_config.adcp_config.max_depth_meter + == -1000.0 + ) + yield Label(" OceanObserver:") + yield Switch(value=is_deep, id="adcp_deep") + yield Label(" SeaSeven:") + yield Switch(value=not is_deep, id="adcp_shallow") + yield Button("?", id="info_button", variant="warning") + ## SECTION: "Instrument Configurations"" -class ScheduleEditor(Static): - def __init__(self, path: str): - super().__init__() - self.path = path - self.schedule = None + with Collapsible( + title="[b]Instrument Configurations[/b] (advanced users only)", + collapsed=True, + ): + for instrument_name, info in INSTRUMENT_FIELDS.items(): + config_class = info["class"] + attributes = info["attributes"] + # instrument-specific configs now live under instruments_config + config_instance = getattr( + self.expedition.instruments_config, instrument_name, None + ) + title = info.get("title", instrument_name.replace("_", " ").title()) + with Collapsible( + title=f"[b]{title}[/b]", + collapsed=True, + ): + if instrument_name in ( + "adcp_config", + "ship_underwater_st_config", + ): + yield Label( + f"NOTE: entries will be ignored here if {info['title']} is OFF in Ship Speed & Onboard Measurements." + ) + with Container(classes="instrument-config"): + for attr_meta in attributes: + attr = attr_meta["name"] + is_minutes = attr_meta.get("minutes", False) + validators = group_validators(config_class, attr) + if config_instance: + raw_value = getattr(config_instance, attr, "") + if is_minutes and raw_value != "": + try: + value = str( + raw_value.total_seconds() / 60.0 + ) + except AttributeError: + value = str(raw_value) + else: + value = str(raw_value) + else: + value = "" + label = f"{attr.replace('_', ' ').title()}:" + yield Label( + label + if not is_minutes + else label.replace(":", " Minutes:") + ) + yield Input( + id=f"{instrument_name}_{attr}", + type=type_to_textual( + get_field_type(config_class, attr) + ), + validators=[ + Function( + validator, + f"INVALID: value must be {validator.__doc__.lower()}", + ) + for validator in validators + ], + value=value, + ) + yield Label( + "", + id=f"validation-failure-label-{instrument_name}_{attr}", + classes="-hidden validation-failure", + ) - def compose(self) -> ComposeResult: - try: - self.schedule = Schedule.from_yaml(f"{self.path}/schedule.yaml") - except Exception as e: - raise UserError(f"There is an issue in schedule.yaml:\n\n{e}") from None + ## 2) SCHEDULE EDITOR - try: - yield Label("[b]Schedule Editor[/b]", id="title", markup=True) + yield Label("[b]Schedule Editor[/b]", id="title_schedule", markup=True) yield Rule(line_style="heavy") # SECTION: "Waypoints & Instrument Selection" @@ -327,8 +335,8 @@ def compose(self) -> ComposeResult: title="[b]Space-Time Region[/b] (advanced users only)", collapsed=True, ): - if self.schedule.space_time_region: - str_data = self.schedule.space_time_region + if self.expedition.schedule.space_time_region: + str_data = self.expedition.schedule.space_time_region yield Label("Minimum Latitude:") yield Input( @@ -501,13 +509,180 @@ def compose(self) -> ComposeResult: def on_mount(self) -> None: self.refresh_waypoint_widgets() + adcp_present = ( + getattr(self.expedition.instruments_config, "adcp_config", None) + if self.expedition.instruments_config + else False + ) + self.show_hide_adcp_type(bool(adcp_present)) def refresh_waypoint_widgets(self): waypoint_list = self.query_one("#waypoint_list", VerticalScroll) waypoint_list.remove_children() - for i, waypoint in enumerate(self.schedule.waypoints): + for i, waypoint in enumerate(self.expedition.schedule.waypoints): waypoint_list.mount(WaypointWidget(waypoint, i)) + def save_changes(self) -> bool: + """Save changes to expedition.yaml.""" + try: + self._update_ship_speed() + self._update_instrument_configs() + self._update_schedule() + self.expedition.to_yaml(self.path.joinpath(EXPEDITION)) + return True + except Exception as e: + log_exception_to_file( + e, + self.path, + context_message=f"Error saving {self.path.joinpath(EXPEDITION)}:", + ) + raise UnexpectedError( + UNEXPECTED_MSG_ONSAVE + + f"\n\nTraceback will be logged in {self.path}/virtualship_error.txt. Please attach this/copy the contents to any issue submitted." + ) from None + + def _update_ship_speed(self): + attr = "ship_speed_knots" + field_type = get_field_type(type(self.expedition.ship_config), attr) + value = field_type(self.query_one("#speed").value) + ShipConfig.model_validate( + {**self.expedition.ship_config.model_dump(), attr: value} + ) + self.expedition.ship_config.ship_speed_knots = value + + def _update_instrument_configs(self): + for instrument_name, info in INSTRUMENT_FIELDS.items(): + config_class = info["class"] + attributes = info["attributes"] + kwargs = {} + # special handling for onboard ADCP and T/S + if instrument_name == "adcp_config": + has_adcp = self.query_one("#has_adcp", Switch).value + if not has_adcp: + setattr(self.expedition.instruments_config, instrument_name, None) + continue + if instrument_name == "ship_underwater_st_config": + has_ts = self.query_one("#has_onboard_ts", Switch).value + if not has_ts: + setattr(self.expedition.instruments_config, instrument_name, None) + continue + for attr_meta in attributes: + attr = attr_meta["name"] + is_minutes = attr_meta.get("minutes", False) + input_id = f"{instrument_name}_{attr}" + value = self.query_one(f"#{input_id}").value + field_type = get_field_type(config_class, attr) + if is_minutes and field_type is datetime.timedelta: + value = datetime.timedelta(minutes=float(value)) + else: + value = field_type(value) + kwargs[attr] = value + # ADCP max_depth_meter based on deep/shallow switch + if instrument_name == "adcp_config": + if self.query_one("#adcp_deep", Switch).value: + kwargs["max_depth_meter"] = -1000.0 + else: + kwargs["max_depth_meter"] = -150.0 + setattr( + self.expedition.instruments_config, + instrument_name, + config_class(**kwargs), + ) + + def _update_schedule(self): + start_time_input = self.query_one("#start_time").value + end_time_input = self.query_one("#end_time").value + waypoint_times = [ + wp.time + for wp in self.expedition.schedule.waypoints + if hasattr(wp, "time") and wp.time + ] + if not start_time_input and waypoint_times: + start_time = min(waypoint_times) + else: + start_time = start_time_input + if not end_time_input and waypoint_times: + end_time = max(waypoint_times) + datetime.timedelta(minutes=60480.0) + else: + end_time = end_time_input + time_range = TimeRange(start_time=start_time, end_time=end_time) + self.expedition.schedule.space_time_region.time_range = time_range + + for i, wp in enumerate(self.expedition.schedule.waypoints): + wp.location = Location( + latitude=float(self.query_one(f"#wp{i}_lat").value), + longitude=float(self.query_one(f"#wp{i}_lon").value), + ) + wp.time = datetime.datetime( + int(self.query_one(f"#wp{i}_year").value), + int(self.query_one(f"#wp{i}_month").value), + int(self.query_one(f"#wp{i}_day").value), + int(self.query_one(f"#wp{i}_hour").value), + int(self.query_one(f"#wp{i}_minute").value), + 0, + ) + wp.instrument = [] + for instrument in [inst for inst in InstrumentType if not inst.is_underway]: + switch_on = self.query_one(f"#wp{i}_{instrument.value}").value + if instrument.value == "DRIFTER" and switch_on: + count_str = self.query_one(f"#wp{i}_drifter_count").value + count = int(count_str) + assert count > 0 + wp.instrument.extend([InstrumentType.DRIFTER] * count) + elif switch_on: + wp.instrument.append(instrument) + + # take min/max lat/lon to be most extreme values of waypoints or space_time_region inputs (so as to cover possibility of user edits in either place) + # also prevents situation where e.g. user defines a space time region inconsistent with waypoint locations and vice versa (warning also provided) + waypoint_lats = [ + wp.location.latitude for wp in self.expedition.schedule.waypoints + ] + waypoint_lons = [ + wp.location.longitude for wp in self.expedition.schedule.waypoints + ] + wp_min_lat, wp_max_lat = ( + min(waypoint_lats) if waypoint_lats else -90.0, + max(waypoint_lats) if waypoint_lats else 90.0, + ) + wp_min_lon, wp_max_lon = ( + min(waypoint_lons) if waypoint_lons else -180.0, + max(waypoint_lons) if waypoint_lons else 180.0, + ) + + st_reg_min_lat = float(self.query_one("#min_lat").value) + st_reg_max_lat = float(self.query_one("#max_lat").value) + st_reg_min_lon = float(self.query_one("#min_lon").value) + st_reg_max_lon = float(self.query_one("#max_lon").value) + + min_lat = min(wp_min_lat, st_reg_min_lat) + max_lat = max(wp_max_lat, st_reg_max_lat) + min_lon = min(wp_min_lon, st_reg_min_lon) + max_lon = max(wp_max_lon, st_reg_max_lon) + + spatial_range = SpatialRange( + minimum_longitude=min_lon, + maximum_longitude=max_lon, + minimum_latitude=min_lat, + maximum_latitude=max_lat, + minimum_depth=self.query_one("#min_depth").value, + maximum_depth=self.query_one("#max_depth").value, + ) + self.expedition.schedule.space_time_region.spatial_range = spatial_range + + # provide warning if user defines a space time region inconsistent with waypoint locations + if ( + (wp_min_lat < st_reg_min_lat) + or (wp_max_lat > st_reg_max_lat) + or (wp_min_lon < st_reg_min_lon) + or (wp_max_lon > st_reg_max_lon) + ): + self.notify( + "[b]WARNING[/b]. One or more waypoint locations lie outside the defined space-time region. Take care if manually adjusting the space-time region." + "\n\nThe space-time region will be automatically adjusted on saving to include all waypoint locations.", + severity="warning", + timeout=10, + ) + @on(Input.Changed) def show_invalid_reasons(self, event: Input.Changed) -> None: input_id = event.input.id @@ -547,331 +722,38 @@ def show_invalid_reasons(self, event: Input.Changed) -> None: def add_waypoint(self) -> None: """Add a new waypoint to the schedule. Copies time from last waypoint if possible (Lat/lon and instruments blank).""" try: - if self.schedule.waypoints: - last_wp = self.schedule.waypoints[-1] + if self.expedition.schedule.waypoints: + last_wp = self.expedition.schedule.waypoints[-1] new_time = last_wp.time if last_wp.time else None new_wp = Waypoint( location=Location( latitude=0.0, longitude=0.0, ), - time=new_time, - instrument=[], - ) - else: - new_wp = Waypoint( - location=Location(latitude=0.0, longitude=0.0), - time=None, - instrument=[], - ) - self.schedule.waypoints.append(new_wp) - self.refresh_waypoint_widgets() - - except Exception as e: - raise UnexpectedError(unexpected_msg_compose(e)) from None - - @on(Button.Pressed, "#remove_waypoint") - def remove_waypoint(self) -> None: - """Remove the last waypoint from the schedule.""" - try: - if self.schedule.waypoints: - self.schedule.waypoints.pop() - self.refresh_waypoint_widgets() - else: - self.notify("No waypoints to remove.", severity="error", timeout=5) - - except Exception as e: - raise UnexpectedError(unexpected_msg_compose(e)) from None - - def save_changes(self) -> bool: - """Save changes to schedule.yaml.""" - try: - ## spacetime region - spatial_range = SpatialRange( - minimum_longitude=self.query_one("#min_lon").value, - maximum_longitude=self.query_one("#max_lon").value, - minimum_latitude=self.query_one("#min_lat").value, - maximum_latitude=self.query_one("#max_lat").value, - minimum_depth=self.query_one("#min_depth").value, - maximum_depth=self.query_one("#max_depth").value, - ) - - # auto fill start and end times if input is blank - start_time_input = self.query_one("#start_time").value - end_time_input = self.query_one("#end_time").value - waypoint_times = [ - wp.time - for wp in self.schedule.waypoints - if hasattr(wp, "time") and wp.time - ] - - if not start_time_input and waypoint_times: - start_time = min(waypoint_times) - else: - start_time = start_time_input - - if not end_time_input and waypoint_times: - end_time = max(waypoint_times) + datetime.timedelta( - minutes=60480.0 - ) # with buffer (corresponds to default drifter lifetime) - else: - end_time = end_time_input - - time_range = TimeRange( - start_time=start_time, - end_time=end_time, - ) - - self.schedule.space_time_region.spatial_range = spatial_range - self.schedule.space_time_region.time_range = time_range - - ## waypoints - for i, wp in enumerate(self.schedule.waypoints): - wp.location = Location( - latitude=float(self.query_one(f"#wp{i}_lat").value), - longitude=float(self.query_one(f"#wp{i}_lon").value), - ) - wp.time = datetime.datetime( - int(self.query_one(f"#wp{i}_year").value), - int(self.query_one(f"#wp{i}_month").value), - int(self.query_one(f"#wp{i}_day").value), - int(self.query_one(f"#wp{i}_hour").value), - int(self.query_one(f"#wp{i}_minute").value), - 0, - ) - - wp.instrument = [] - for instrument in InstrumentType: - switch_on = self.query_one(f"#wp{i}_{instrument.value}").value - if instrument.value == "DRIFTER" and switch_on: - count_str = self.query_one(f"#wp{i}_drifter_count").value - count = int(count_str) - assert count > 0 - wp.instrument.extend([InstrumentType.DRIFTER] * count) - elif switch_on: - wp.instrument.append(instrument) - - # save - self.schedule.to_yaml(f"{self.path}/schedule.yaml") - return True - - except Exception as e: - log_exception_to_file( - e, self.path, context_message="Error saving schedule:" - ) - - raise UnexpectedError( - UNEXPECTED_MSG_ONSAVE - + f"\n\nTraceback will be logged in {self.path}/virtualship_error.txt. Please attach this/copy the contents to any issue submitted." - ) from None - - -class ConfigEditor(Container): - DEFAULT_ADCP_CONFIG: ClassVar[dict[str, float]] = { - "num_bins": 40, - "period_minutes": 5.0, - } - - DEFAULT_TS_CONFIG: ClassVar[dict[str, float]] = {"period_minutes": 5.0} - - INSTRUMENT_FIELDS: ClassVar[dict[str, dict]] = { - "adcp_config": { - "class": ADCPConfig, - "title": "Onboard ADCP", - "attributes": [ - {"name": "num_bins"}, - {"name": "period", "minutes": True}, - ], - }, - "ship_underwater_st_config": { - "class": ShipUnderwaterSTConfig, - "title": "Onboard Temperature/Salinity", - "attributes": [ - {"name": "period", "minutes": True}, - ], - }, - "ctd_config": { - "class": CTDConfig, - "title": "CTD", - "attributes": [ - {"name": "max_depth_meter"}, - {"name": "min_depth_meter"}, - {"name": "stationkeeping_time", "minutes": True}, - ], - }, - "ctd_bgc_config": { - "class": CTD_BGCConfig, - "title": "CTD-BGC", - "attributes": [ - {"name": "max_depth_meter"}, - {"name": "min_depth_meter"}, - {"name": "stationkeeping_time", "minutes": True}, - ], - }, - "xbt_config": { - "class": XBTConfig, - "title": "XBT", - "attributes": [ - {"name": "min_depth_meter"}, - {"name": "max_depth_meter"}, - {"name": "fall_speed_meter_per_second"}, - {"name": "deceleration_coefficient"}, - ], - }, - "argo_float_config": { - "class": ArgoFloatConfig, - "title": "Argo Float", - "attributes": [ - {"name": "min_depth_meter"}, - {"name": "max_depth_meter"}, - {"name": "drift_depth_meter"}, - {"name": "vertical_speed_meter_per_second"}, - {"name": "cycle_days"}, - {"name": "drift_days"}, - ], - }, - "drifter_config": { - "class": DrifterConfig, - "title": "Drifter", - "attributes": [ - {"name": "depth_meter"}, - {"name": "lifetime", "minutes": True}, - ], - }, - } - - def __init__(self, path: str): - super().__init__() - self.path = path - self.config = None - - def compose(self) -> ComposeResult: - try: - self.config = ShipConfig.from_yaml(f"{self.path}/ship_config.yaml") - except Exception as e: - raise UserError(f"There is an issue in ship_config.yaml:\n\n{e}") from None - - try: - ## SECTION: "Ship Speed & Onboard Measurements" - - yield Label("[b]Ship Config Editor[/b]", id="title", markup=True) - yield Rule(line_style="heavy") - - with Collapsible( - title="[b]Ship Speed & Onboard Measurements[/b]", id="speed_collapsible" - ): - attr = "ship_speed_knots" - validators = group_validators(ShipConfig, attr) - with Horizontal(classes="ship_speed"): - yield Label("[b]Ship Speed (knots):[/b]") - yield Input( - id="speed", - type=type_to_textual(get_field_type(ShipConfig, attr)), - validators=[ - Function( - validator, - f"INVALID: value must be {validator.__doc__.lower()}", - ) - for validator in validators - ], - classes="ship_speed_input", - placeholder="knots", - value=str( - self.config.ship_speed_knots - if self.config.ship_speed_knots - else "" - ), - ) - yield Label("", id="validation-failure-label-speed", classes="-hidden") - - with Horizontal(classes="ts-section"): - yield Label("[b]Onboard Temperature/Salinity:[/b]") - yield Switch( - value=bool(self.config.ship_underwater_st_config), - id="has_onboard_ts", - ) - - with Horizontal(classes="adcp-section"): - yield Label("[b]Onboard ADCP:[/b]") - yield Switch(value=bool(self.config.adcp_config), id="has_adcp") - - # adcp type selection - with Horizontal(id="adcp_type_container", classes="-hidden"): - is_deep = ( - self.config.adcp_config - and self.config.adcp_config.max_depth_meter == -1000.0 - ) - yield Label(" OceanObserver:") - yield Switch(value=is_deep, id="adcp_deep") - yield Label(" SeaSeven:") - yield Switch(value=not is_deep, id="adcp_shallow") - yield Button("?", id="info_button", variant="warning") - - ## SECTION: "Instrument Configurations"" - - with Collapsible( - title="[b]Instrument Configurations[/b] (advanced users only)", - collapsed=True, - ): - for instrument_name, info in self.INSTRUMENT_FIELDS.items(): - config_class = info["class"] - attributes = info["attributes"] - config_instance = getattr(self.config, instrument_name, None) - title = info.get("title", instrument_name.replace("_", " ").title()) - with Collapsible( - title=f"[b]{title}[/b]", - collapsed=True, - ): - if instrument_name in ( - "adcp_config", - "ship_underwater_st_config", - ): - yield Label( - f"NOTE: entries will be ignored here if {info['title']} is OFF in Ship Speed & Onboard Measurements." - ) - with Container(classes="instrument-config"): - for attr_meta in attributes: - attr = attr_meta["name"] - is_minutes = attr_meta.get("minutes", False) - validators = group_validators(config_class, attr) - if config_instance: - raw_value = getattr(config_instance, attr, "") - if is_minutes and raw_value != "": - try: - value = str( - raw_value.total_seconds() / 60.0 - ) - except AttributeError: - value = str(raw_value) - else: - value = str(raw_value) - else: - value = "" - label = f"{attr.replace('_', ' ').title()}:" - yield Label( - label - if not is_minutes - else label.replace(":", " Minutes:") - ) - yield Input( - id=f"{instrument_name}_{attr}", - type=type_to_textual( - get_field_type(config_class, attr) - ), - validators=[ - Function( - validator, - f"INVALID: value must be {validator.__doc__.lower()}", - ) - for validator in validators - ], - value=value, - ) - yield Label( - "", - id=f"validation-failure-label-{instrument_name}_{attr}", - classes="-hidden validation-failure", - ) + time=new_time, + instrument=[], + ) + else: + new_wp = Waypoint( + location=Location(latitude=0.0, longitude=0.0), + time=None, + instrument=[], + ) + self.expedition.schedule.waypoints.append(new_wp) + self.refresh_waypoint_widgets() + + except Exception as e: + raise UnexpectedError(unexpected_msg_compose(e)) from None + + @on(Button.Pressed, "#remove_waypoint") + def remove_waypoint(self) -> None: + """Remove the last waypoint from the schedule.""" + try: + if self.expedition.schedule.waypoints: + self.expedition.schedule.waypoints.pop() + self.refresh_waypoint_widgets() + else: + self.notify("No waypoints to remove.", severity="error", timeout=5) except Exception as e: raise UnexpectedError(unexpected_msg_compose(e)) from None @@ -885,31 +767,6 @@ def info_pressed(self) -> None: timeout=20, ) - @on(Input.Changed) - def show_invalid_reasons(self, event: Input.Changed) -> None: - input_id = event.input.id - label_id = f"validation-failure-label-{input_id}" - label = self.query_one(f"#{label_id}", Label) - if not event.validation_result.is_valid: - message = ( - "\n".join(event.validation_result.failure_descriptions) - if isinstance(event.validation_result.failure_descriptions, list) - else str(event.validation_result.failure_descriptions) - ) - label.update(message) - label.remove_class("-hidden") - label.add_class("validation-failure") - else: - label.update("") - label.add_class("-hidden") - label.remove_class("validation-failure") - - def on_mount(self) -> None: - adcp_present = ( - getattr(self.config, "adcp_config", None) if self.config else False - ) - self.show_hide_adcp_type(bool(adcp_present)) - def show_hide_adcp_type(self, show: bool) -> None: container = self.query_one("#adcp_type_container") if show: @@ -919,29 +776,32 @@ def show_hide_adcp_type(self, show: bool) -> None: def _set_adcp_default_values(self): self.query_one("#adcp_config_num_bins").value = str( - self.DEFAULT_ADCP_CONFIG["num_bins"] + DEFAULT_ADCP_CONFIG["num_bins"] ) self.query_one("#adcp_config_period").value = str( - self.DEFAULT_ADCP_CONFIG["period_minutes"] + DEFAULT_ADCP_CONFIG["period_minutes"] ) self.query_one("#adcp_shallow").value = False self.query_one("#adcp_deep").value = True def _set_ts_default_values(self): self.query_one("#ship_underwater_st_config_period").value = str( - self.DEFAULT_TS_CONFIG["period_minutes"] + DEFAULT_TS_CONFIG["period_minutes"] ) @on(Switch.Changed, "#has_adcp") def on_adcp_toggle(self, event: Switch.Changed) -> None: self.show_hide_adcp_type(event.value) - if event.value and not self.config.adcp_config: + if event.value and not self.expedition.instruments_config.adcp_config: # ADCP was turned on and was previously null self._set_adcp_default_values() @on(Switch.Changed, "#has_onboard_ts") def on_ts_toggle(self, event: Switch.Changed) -> None: - if event.value and not self.config.ship_underwater_st_config: + if ( + event.value + and not self.expedition.instruments_config.ship_underwater_st_config + ): # T/S was turned on and was previously null self._set_ts_default_values() @@ -957,68 +817,213 @@ def shallow_changed(self, event: Switch.Changed) -> None: deep = self.query_one("#adcp_deep", Switch) deep.value = False - def save_changes(self) -> bool: - """Save changes to ship_config.yaml.""" + +class WaypointWidget(Static): + def __init__(self, waypoint: Waypoint, index: int): + super().__init__() + self.waypoint = waypoint + self.index = index + + def compose(self) -> ComposeResult: try: - # ship speed - attr = "ship_speed_knots" - field_type = get_field_type(type(self.config), attr) - value = field_type(self.query_one("#speed").value) - ShipConfig.model_validate( - {**self.config.model_dump(), attr: value} - ) # validate using a temporary model (raises if invalid) - self.config.ship_speed_knots = value - - # individual instrument configurations - for instrument_name, info in self.INSTRUMENT_FIELDS.items(): - config_class = info["class"] - attributes = info["attributes"] - kwargs = {} - - # special handling for onboard ADCP and T/S - # will skip to next instrument if toggle is off - if instrument_name == "adcp_config": - has_adcp = self.query_one("#has_adcp", Switch).value - if not has_adcp: - setattr(self.config, instrument_name, None) - continue - if instrument_name == "ship_underwater_st_config": - has_ts = self.query_one("#has_onboard_ts", Switch).value - if not has_ts: - setattr(self.config, instrument_name, None) - continue - - for attr_meta in attributes: - attr = attr_meta["name"] - is_minutes = attr_meta.get("minutes", False) - input_id = f"{instrument_name}_{attr}" - value = self.query_one(f"#{input_id}").value - field_type = get_field_type(config_class, attr) - if is_minutes and field_type is datetime.timedelta: - value = datetime.timedelta(minutes=float(value)) - else: - value = field_type(value) - kwargs[attr] = value - - # ADCP max_depth_meter based on deep/shallow switch - if instrument_name == "adcp_config": - if self.query_one("#adcp_deep", Switch).value: - kwargs["max_depth_meter"] = -1000.0 - else: - kwargs["max_depth_meter"] = -150.0 - - setattr(self.config, instrument_name, config_class(**kwargs)) - - # save - self.config.to_yaml(f"{self.path}/ship_config.yaml") - return True + with Collapsible( + title=f"[b]Waypoint {self.index + 1}[/b]", + collapsed=True, + id=f"wp{self.index + 1}", + ): + if self.index > 0: + yield Button( + "Copy Time & Instruments from Previous", + id=f"wp{self.index}_copy", + variant="warning", + ) + yield Label("Location:") + yield Label(" Latitude:") + yield Input( + id=f"wp{self.index}_lat", + value=str(self.waypoint.location.lat) + if self.waypoint.location.lat + is not None # is not None to handle if lat is 0.0 + else "", + validators=[ + Function( + is_valid_lat, + f"INVALID: value must be {is_valid_lat.__doc__.lower()}", + ) + ], + type="number", + placeholder="°N", + classes="latitude-input", + ) + yield Label( + "", + id=f"validation-failure-label-wp{self.index}_lat", + classes="-hidden validation-failure", + ) + + yield Label(" Longitude:") + yield Input( + id=f"wp{self.index}_lon", + value=str(self.waypoint.location.lon) + if self.waypoint.location.lon + is not None # is not None to handle if lon is 0.0 + else "", + validators=[ + Function( + is_valid_lon, + f"INVALID: value must be {is_valid_lon.__doc__.lower()}", + ) + ], + type="number", + placeholder="°E", + classes="longitude-input", + ) + yield Label( + "", + id=f"validation-failure-label-wp{self.index}_lon", + classes="-hidden validation-failure", + ) + + yield Label("Time:") + with Horizontal(): + yield Label("Year:") + yield Select( + [ + (str(year), year) + for year in range( + 1993, + datetime.datetime.now().year + 1, + ) + ], + id=f"wp{self.index}_year", + value=int(self.waypoint.time.year) + if self.waypoint.time + else Select.BLANK, + prompt="YYYY", + classes="year-select", + ) + yield Label("Month:") + yield Select( + [(f"{m:02d}", m) for m in range(1, 13)], + id=f"wp{self.index}_month", + value=int(self.waypoint.time.month) + if self.waypoint.time + else Select.BLANK, + prompt="MM", + classes="month-select", + ) + yield Label("Day:") + yield Select( + [(f"{d:02d}", d) for d in range(1, 32)], + id=f"wp{self.index}_day", + value=int(self.waypoint.time.day) + if self.waypoint.time + else Select.BLANK, + prompt="DD", + classes="day-select", + ) + yield Label("Hour:") + yield Select( + [(f"{h:02d}", h) for h in range(24)], + id=f"wp{self.index}_hour", + value=int(self.waypoint.time.hour) + if self.waypoint.time + else Select.BLANK, + prompt="hh", + classes="hour-select", + ) + yield Label("Min:") + yield Select( + [(f"{m:02d}", m) for m in range(0, 60, 5)], + id=f"wp{self.index}_minute", + value=int(self.waypoint.time.minute) + if self.waypoint.time + else Select.BLANK, + prompt="mm", + classes="minute-select", + ) + + yield Label("Instruments:") + for instrument in [i for i in InstrumentType if not i.is_underway]: + is_selected = instrument in (self.waypoint.instrument or []) + with Horizontal(): + yield Label(instrument.value) + yield Switch( + value=is_selected, id=f"wp{self.index}_{instrument.value}" + ) + + if instrument.value == "DRIFTER": + yield Label("Count") + yield Input( + id=f"wp{self.index}_drifter_count", + value=str( + self.get_drifter_count() if is_selected else "" + ), + type="integer", + placeholder="# of drifters", + validators=Integer( + minimum=1, + failure_description="INVALID: value must be > 0", + ), + classes="drifter-count-input", + ) + yield Label( + "", + id=f"validation-failure-label-wp{self.index}_drifter_count", + classes="-hidden validation-failure", + ) except Exception as e: - log_exception_to_file( - e, self.path, context_message="Error saving ship config:" - ) + raise UnexpectedError(unexpected_msg_compose(e)) from None + + def get_drifter_count(self) -> int: + return sum( + 1 for inst in self.waypoint.instrument if inst == InstrumentType.DRIFTER + ) + + def copy_from_previous(self) -> None: + """Copy inputs from previous waypoint widget (time and instruments only, not lat/lon).""" + try: + if self.index > 0: + schedule_editor = self.parent + if schedule_editor: + time_components = ["year", "month", "day", "hour", "minute"] + for comp in time_components: + prev = schedule_editor.query_one(f"#wp{self.index - 1}_{comp}") + curr = self.query_one(f"#wp{self.index}_{comp}") + if prev and curr: + curr.value = prev.value + + for instrument in [ + inst for inst in InstrumentType if not inst.is_underway + ]: + prev_switch = schedule_editor.query_one( + f"#wp{self.index - 1}_{instrument.value}" + ) + curr_switch = self.query_one( + f"#wp{self.index}_{instrument.value}" + ) + if prev_switch and curr_switch: + curr_switch.value = prev_switch.value + except Exception as e: + raise UnexpectedError(unexpected_msg_compose(e)) from None + + @on(Button.Pressed, "Button") + def button_pressed(self, event: Button.Pressed) -> None: + if event.button.id == f"wp{self.index}_copy": + self.copy_from_previous() - raise UnexpectedError(UNEXPECTED_MSG_ONSAVE) from None + @on(Switch.Changed) + def on_switch_changed(self, event: Switch.Changed) -> None: + if event.switch.id == f"wp{self.index}_DRIFTER": + drifter_count_input = self.query_one( + f"#wp{self.index}_drifter_count", Input + ) + if not event.value: + drifter_count_input.value = "" + else: + if not drifter_count_input.value: + drifter_count_input.value = "1" class PlanScreen(Screen): @@ -1029,8 +1034,7 @@ def __init__(self, path: str): def compose(self) -> ComposeResult: try: with VerticalScroll(): - yield ConfigEditor(self.path) - yield ScheduleEditor(self.path) + yield ExpeditionEditor(self.path) with Horizontal(): yield Button("Save Changes", id="save_button", variant="success") yield Button("Exit", id="exit_button", variant="error") @@ -1039,20 +1043,20 @@ def compose(self) -> ComposeResult: def sync_ui_waypoints(self): """Update the waypoints models with current UI values (spacetime only) from the live UI inputs.""" - schedule_editor = self.query_one(ScheduleEditor) + expedition_editor = self.query_one(ExpeditionEditor) errors = [] - for i, wp in enumerate(schedule_editor.schedule.waypoints): + for i, wp in enumerate(expedition_editor.expedition.schedule.waypoints): try: wp.location = Location( - latitude=float(schedule_editor.query_one(f"#wp{i}_lat").value), - longitude=float(schedule_editor.query_one(f"#wp{i}_lon").value), + latitude=float(expedition_editor.query_one(f"#wp{i}_lat").value), + longitude=float(expedition_editor.query_one(f"#wp{i}_lon").value), ) wp.time = datetime.datetime( - int(schedule_editor.query_one(f"#wp{i}_year").value), - int(schedule_editor.query_one(f"#wp{i}_month").value), - int(schedule_editor.query_one(f"#wp{i}_day").value), - int(schedule_editor.query_one(f"#wp{i}_hour").value), - int(schedule_editor.query_one(f"#wp{i}_minute").value), + int(expedition_editor.query_one(f"#wp{i}_year").value), + int(expedition_editor.query_one(f"#wp{i}_month").value), + int(expedition_editor.query_one(f"#wp{i}_day").value), + int(expedition_editor.query_one(f"#wp{i}_hour").value), + int(expedition_editor.query_one(f"#wp{i}_minute").value), 0, ) except Exception as e: @@ -1075,26 +1079,23 @@ def exit_pressed(self) -> None: @on(Button.Pressed, "#save_button") def save_pressed(self) -> None: """Save button press.""" - config_editor = self.query_one(ConfigEditor) - schedule_editor = self.query_one(ScheduleEditor) + expedition_editor = self.query_one(ExpeditionEditor) try: - ship_speed_value = self.get_ship_speed(config_editor) + ship_speed_value = self.get_ship_speed(expedition_editor) self.sync_ui_waypoints() # call to ensure waypoint inputs are synced # verify schedule - schedule_editor.schedule.verify( + expedition_editor.expedition.schedule.verify( ship_speed_value, - input_data=None, check_space_time_region=True, - ignore_missing_fieldsets=True, + ignore_land_test=True, ) - config_saved = config_editor.save_changes() - schedule_saved = schedule_editor.save_changes() + expedition_saved = expedition_editor.save_changes() - if config_saved and schedule_saved: + if expedition_saved: self.notify( "Changes saved successfully", severity="information", @@ -1109,9 +1110,9 @@ def save_pressed(self) -> None: ) return False - def get_ship_speed(self, config_editor): + def get_ship_speed(self, expedition_editor): try: - ship_speed = float(config_editor.query_one("#speed").value) + ship_speed = float(expedition_editor.query_one("#speed").value) assert ship_speed > 0 except Exception as e: log_exception_to_file( @@ -1130,12 +1131,6 @@ class PlanApp(App): align: center middle; } - ConfigEditor { - padding: 1; - margin-bottom: 1; - height: auto; - } - VerticalScroll { width: 100%; height: 100%; @@ -1210,7 +1205,12 @@ class PlanApp(App): margin: 0 1; } - #title { + #title_ship_instruments_config { + text-style: bold; + padding: 1; + } + + #title_schedule { text-style: bold; padding: 1; } diff --git a/src/virtualship/cli/_run.py b/src/virtualship/cli/_run.py new file mode 100644 index 00000000..f07fbab2 --- /dev/null +++ b/src/virtualship/cli/_run.py @@ -0,0 +1,191 @@ +"""do_expedition function.""" + +import logging +import os +import shutil +import time +from pathlib import Path + +import copernicusmarine +import pyproj + +from virtualship.expedition.simulate_schedule import ( + MeasurementsToSimulate, + ScheduleProblem, + simulate_schedule, +) +from virtualship.models import Schedule +from virtualship.models.checkpoint import Checkpoint +from virtualship.utils import ( + CHECKPOINT, + _get_expedition, + expedition_cost, + get_instrument_class, +) + +# projection used to sail between waypoints +projection = pyproj.Geod(ellps="WGS84") + + +# parcels logger (suppress INFO messages to prevent log being flooded) +external_logger = logging.getLogger("parcels.tools.loggers") +external_logger.setLevel(logging.WARNING) + +# copernicusmarine logger (suppress INFO messages to prevent log being flooded) +logging.getLogger("copernicusmarine").setLevel("ERROR") + + +def _run(expedition_dir: str | Path, from_data: Path | None = None) -> None: + """ + Perform an expedition, providing terminal feedback and file output. + + :param expedition_dir: The base directory for the expedition. + """ + # start timing + start_time = time.time() + print("[TIMER] Expedition started...") + + print("\n╔═════════════════════════════════════════════════╗") + print("║ VIRTUALSHIP EXPEDITION STATUS ║") + print("╚═════════════════════════════════════════════════╝") + + if from_data is None: + # TODO: caution, if collaborative environments, will this mean everyone uses the same credentials file? + # TODO: need to think about how to deal with this for when using collaborative environments AND streaming data via copernicusmarine + COPERNICUS_CREDS_FILE = os.path.expandvars( + "$HOME/.copernicusmarine/.copernicusmarine-credentials" + ) + + if ( + os.path.isfile(COPERNICUS_CREDS_FILE) + and os.path.getsize(COPERNICUS_CREDS_FILE) > 0 + ): + pass + else: + print( + "\nPlease enter your log in details for the Copernicus Marine Service (only necessary the first time you run VirtualShip). \n\nIf you have not registered yet, please do so at https://marine.copernicus.eu/.\n\n" + "If you did not expect to see this message, and intended to use pre-downloaded data instead of streaming via Copernicus Marine, please use the '--from-data' option to specify the path to the data.\n" + ) + copernicusmarine.login() + + if isinstance(expedition_dir, str): + expedition_dir = Path(expedition_dir) + + expedition = _get_expedition(expedition_dir) + + # Verify instruments_config file is consistent with schedule + expedition.instruments_config.verify(expedition) + + # load last checkpoint + checkpoint = _load_checkpoint(expedition_dir) + if checkpoint is None: + checkpoint = Checkpoint(past_schedule=Schedule(waypoints=[])) + + # verify that schedule and checkpoint match + checkpoint.verify(expedition.schedule) + + print("\n---- WAYPOINT VERIFICATION ----") + + expedition.schedule.verify( + expedition.ship_config.ship_speed_knots, + from_data=Path(from_data) if from_data else None, + ) + + # simulate the schedule + schedule_results = simulate_schedule( + projection=projection, + expedition=expedition, + ) + if isinstance(schedule_results, ScheduleProblem): + print( + f"SIMULATION PAUSED: update your schedule (`virtualship plan`) and continue the expedition by executing the `virtualship run` command again.\nCheckpoint has been saved to {expedition_dir.joinpath(CHECKPOINT)}." + ) + _save_checkpoint( + Checkpoint( + past_schedule=Schedule( + waypoints=expedition.schedule.waypoints[ + : schedule_results.failed_waypoint_i + ] + ) + ), + expedition_dir, + ) + return + + # delete and create results directory + if os.path.exists(expedition_dir.joinpath("results")): + shutil.rmtree(expedition_dir.joinpath("results")) + os.makedirs(expedition_dir.joinpath("results")) + + print("\n----- EXPEDITION SUMMARY ------") + + # expedition cost in US$ + _write_expedition_cost(expedition, schedule_results, expedition_dir) + + print("\n--- MEASUREMENT SIMULATIONS ---") + + # simulate measurements + print("\nSimulating measurements. This may take a while...\n") + + instruments_in_expedition = expedition.get_instruments() + + for itype in instruments_in_expedition: + # get instrument class + instrument_class = get_instrument_class(itype) + if instrument_class is None: + raise RuntimeError(f"No instrument class found for type {itype}.") + + # get measurements to simulate + attr = MeasurementsToSimulate.get_attr_for_instrumenttype(itype) + measurements = getattr(schedule_results.measurements_to_simulate, attr) + + # initialise instrument + instrument = instrument_class( + expedition=expedition, + from_data=Path(from_data) if from_data is not None else None, + ) + + # execute simulation + instrument.execute( + measurements=measurements, + out_path=expedition_dir.joinpath("results", f"{itype.name.lower()}.zarr"), + ) + + print("\nAll measurement simulations are complete.") + + print("\n----- EXPEDITION RESULTS ------") + print("\nYour expedition has concluded successfully!") + print( + f"Your measurements can be found in the '{expedition_dir}/results' directory." + ) + print("\n------------- END -------------\n") + + # end timing + end_time = time.time() + elapsed = end_time - start_time + print(f"[TIMER] Expedition completed in {elapsed / 60.0:.2f} minutes.") + + +def _load_checkpoint(expedition_dir: Path) -> Checkpoint | None: + file_path = expedition_dir.joinpath(CHECKPOINT) + try: + return Checkpoint.from_yaml(file_path) + except FileNotFoundError: + return None + + +def _save_checkpoint(checkpoint: Checkpoint, expedition_dir: Path) -> None: + file_path = expedition_dir.joinpath(CHECKPOINT) + checkpoint.to_yaml(file_path) + + +def _write_expedition_cost(expedition, schedule_results, expedition_dir): + """Calculate the expedition cost, write it to a file, and print summary.""" + assert expedition.schedule.waypoints[0].time is not None, ( + "First waypoint has no time. This should not be possible as it should have been verified before." + ) + time_past = schedule_results.time - expedition.schedule.waypoints[0].time + cost = expedition_cost(schedule_results, time_past) + with open(expedition_dir.joinpath("results", "cost.txt"), "w") as file: + file.writelines(f"cost: {cost} US$") + print(f"\nExpedition duration: {time_past}\nExpedition cost: US$ {cost:,.0f}.") diff --git a/src/virtualship/cli/commands.py b/src/virtualship/cli/commands.py index 72d37866..b5840f17 100644 --- a/src/virtualship/cli/commands.py +++ b/src/virtualship/cli/commands.py @@ -2,13 +2,13 @@ import click -from virtualship import utils -from virtualship.cli._fetch import _fetch from virtualship.cli._plan import _plan -from virtualship.expedition.do_expedition import do_expedition +from virtualship.cli._run import _run from virtualship.utils import ( - SCHEDULE, - SHIP_CONFIG, + COPERNICUSMARINE_BGC_VARIABLES, + COPERNICUSMARINE_PHYS_VARIABLES, + EXPEDITION, + get_example_expedition, mfp_to_yaml, ) @@ -28,47 +28,39 @@ ) def init(path, from_mfp): """ - Initialize a directory for a new expedition, with an example schedule and ship config files. + Initialize a directory for a new expedition, with an expedition.yaml file. - If --mfp-file is provided, it will generate the schedule from the MPF file instead. + If --mfp-file is provided, it will generate the expedition.yaml from the MPF file instead. """ path = Path(path) path.mkdir(exist_ok=True) - config = path / SHIP_CONFIG - schedule = path / SCHEDULE + expedition = path / EXPEDITION - if config.exists(): + if expedition.exists(): raise FileExistsError( - f"File '{config}' already exist. Please remove it or choose another directory." + f"File '{expedition}' already exist. Please remove it or choose another directory." ) - if schedule.exists(): - raise FileExistsError( - f"File '{schedule}' already exist. Please remove it or choose another directory." - ) - - config.write_text(utils.get_example_config()) if from_mfp: mfp_file = Path(from_mfp) - # Generate schedule.yaml from the MPF file + # Generate expedition.yaml from the MPF file click.echo(f"Generating schedule from {mfp_file}...") - mfp_to_yaml(mfp_file, schedule) + mfp_to_yaml(mfp_file, expedition) click.echo( "\n⚠️ The generated schedule does not contain TIME values or INSTRUMENT selections. ⚠️" "\n\nNow please either use the `\033[4mvirtualship plan\033[0m` app to complete the schedule configuration, " - "\nOR edit 'schedule.yaml' and manually add the necessary time values and instrument selections." - "\n\nIf editing 'schedule.yaml' manually:" + "\nOR edit 'expedition.yaml' and manually add the necessary time values and instrument selections under the 'schedule' heading." + "\n\nIf editing 'expedition.yaml' manually:" "\n\n🕒 Expected time format: 'YYYY-MM-DD HH:MM:SS' (e.g., '2023-10-20 01:00:00')." "\n\n🌡️ Expected instrument(s) format: one line per instrument e.g." f"\n\n{' ' * 15}waypoints:\n{' ' * 15}- instrument:\n{' ' * 19}- CTD\n{' ' * 19}- ARGO_FLOAT\n" ) else: - # Create a default example schedule - # schedule_body = utils.get_example_schedule() - schedule.write_text(utils.get_example_schedule()) + # Create a default example expedition YAML + expedition.write_text(get_example_expedition()) - click.echo(f"Created '{config.name}' and '{schedule.name}' at {path}.") + click.echo(f"Created '{expedition.name}' at {path}.") @click.command() @@ -78,7 +70,7 @@ def init(path, from_mfp): ) def plan(path): """ - Launch UI to help build schedule and ship config files. + Launch UI to help build expedition configuration (YAML) file. Should you encounter any issues with using this tool, please report an issue describing the problem to the VirtualShip issue tracker at: https://github.com/OceanParcels/virtualship/issues" """ @@ -91,35 +83,15 @@ def plan(path): type=click.Path(exists=True, file_okay=False, dir_okay=True, readable=True), ) @click.option( - "--username", + "--from-data", type=str, default=None, - help="Copernicus Marine username.", -) -@click.option( - "--password", - type=str, - default=None, - help="Copernicus Marine password.", -) -def fetch(path: str | Path, username: str | None, password: str | None) -> None: - """ - Download input data for an expedition. - - Entrypoint for the tool to download data based on space-time region provided in the - schedule file. Data is downloaded from Copernicus Marine, credentials for which can be - obtained via registration: https://data.marine.copernicus.eu/register . Credentials can - be provided on prompt, via command line arguments, or via a YAML config file. Run - `virtualship fetch` on a expedition for more info. - """ - _fetch(path, username, password) - - -@click.command() -@click.argument( - "path", - type=click.Path(exists=True, file_okay=False, dir_okay=True, readable=True), + help="Use pre-downloaded data, saved to disk, for expedition, instead of streaming directly via Copernicus Marine" + "Assumes all data is stored in prescribed directory, and all variables (as listed below) are present." + f"Required variables are: {set(COPERNICUSMARINE_PHYS_VARIABLES + COPERNICUSMARINE_BGC_VARIABLES)}" + "Assumes that variable names at least contain the standard Copernicus Marine variable name as a substring." + "Will also take the first file found containing the variable name substring. CAUTION if multiple files contain the same variable name substring.", ) -def run(path): - """Run the expedition.""" - do_expedition(Path(path)) +def run(path, from_data): + """Execute the expedition simulations.""" + _run(Path(path), from_data) diff --git a/src/virtualship/cli/main.py b/src/virtualship/cli/main.py index 6ee12eff..a02a5ffb 100644 --- a/src/virtualship/cli/main.py +++ b/src/virtualship/cli/main.py @@ -11,7 +11,6 @@ def cli(): cli.add_command(commands.init) cli.add_command(commands.plan) -cli.add_command(commands.fetch) cli.add_command(commands.run) if __name__ == "__main__": diff --git a/src/virtualship/cli/validator_utils.py b/src/virtualship/cli/validator_utils.py index 83239ac8..402e48b1 100644 --- a/src/virtualship/cli/validator_utils.py +++ b/src/virtualship/cli/validator_utils.py @@ -123,7 +123,6 @@ def make_validator(condition, reference, value_type): Therefore, reference values for the conditions cannot be fed in dynamically and necessitates 'hard-coding' the condition and reference value combination. At present, Pydantic models in VirtualShip only require gt/ge/lt/le relative to **0.0** so the 'reference' value is always checked as being == 0.0 Additional custom conditions can be 'hard-coded' as new condition and reference combinations if Pydantic model specifications change in the future and/or new instruments are added to VirtualShip etc. - TODO: Perhaps there's scope here though for a more flexible implementation in a future PR... """ diff --git a/src/virtualship/errors.py b/src/virtualship/errors.py index 6026936e..ac1aa8a1 100644 --- a/src/virtualship/errors.py +++ b/src/virtualship/errors.py @@ -22,8 +22,8 @@ class ScheduleError(RuntimeError): pass -class ConfigError(RuntimeError): - """An error in the config.""" +class InstrumentsConfigError(RuntimeError): + """An error in the InstrumentsConfig.""" pass @@ -40,6 +40,12 @@ class UnexpectedError(Exception): pass +class UnderwayConfigsError(Exception): + """Error raised when underway instrument configurations (ADCP or underwater ST) are missing.""" + + pass + + class CopernicusCatalogueError(Exception): """Error raised when a relevant product is not found in the Copernicus Catalogue.""" diff --git a/src/virtualship/expedition/__init__.py b/src/virtualship/expedition/__init__.py index 43d24844..7f072bbf 100644 --- a/src/virtualship/expedition/__init__.py +++ b/src/virtualship/expedition/__init__.py @@ -1,9 +1 @@ -"""Everything for simulating an expedition.""" - -from .do_expedition import do_expedition -from .input_data import InputData - -__all__ = [ - "InputData", - "do_expedition", -] +"""Simulating an expedition.""" diff --git a/src/virtualship/expedition/do_expedition.py b/src/virtualship/expedition/do_expedition.py deleted file mode 100644 index 56ee79fa..00000000 --- a/src/virtualship/expedition/do_expedition.py +++ /dev/null @@ -1,173 +0,0 @@ -"""do_expedition function.""" - -import os -import shutil -from pathlib import Path - -import pyproj - -from virtualship.cli._fetch import get_existing_download, get_space_time_region_hash -from virtualship.models import Schedule, ShipConfig -from virtualship.utils import ( - CHECKPOINT, - _get_schedule, - _get_ship_config, -) - -from .checkpoint import Checkpoint -from .expedition_cost import expedition_cost -from .input_data import InputData -from .simulate_measurements import simulate_measurements -from .simulate_schedule import ScheduleProblem, simulate_schedule - -# projection used to sail between waypoints -projection = pyproj.Geod(ellps="WGS84") - - -def do_expedition(expedition_dir: str | Path, input_data: Path | None = None) -> None: - """ - Perform an expedition, providing terminal feedback and file output. - - :param expedition_dir: The base directory for the expedition. - :param input_data: Input data folder (override used for testing). - """ - print("\n╔═════════════════════════════════════════════════╗") - print("║ VIRTUALSHIP EXPEDITION STATUS ║") - print("╚═════════════════════════════════════════════════╝") - - if isinstance(expedition_dir, str): - expedition_dir = Path(expedition_dir) - - ship_config = _get_ship_config(expedition_dir) - schedule = _get_schedule(expedition_dir) - - # Verify ship_config file is consistent with schedule - ship_config.verify(schedule) - - # load last checkpoint - checkpoint = _load_checkpoint(expedition_dir) - if checkpoint is None: - checkpoint = Checkpoint(past_schedule=Schedule(waypoints=[])) - - # verify that schedule and checkpoint match - checkpoint.verify(schedule) - - # load fieldsets - loaded_input_data = _load_input_data( - expedition_dir=expedition_dir, - schedule=schedule, - ship_config=ship_config, - input_data=input_data, - ) - - print("\n---- WAYPOINT VERIFICATION ----") - - # verify schedule is valid - schedule.verify(ship_config.ship_speed_knots, loaded_input_data) - - # simulate the schedule - schedule_results = simulate_schedule( - projection=projection, ship_config=ship_config, schedule=schedule - ) - if isinstance(schedule_results, ScheduleProblem): - print( - "Update your schedule and continue the expedition by running the tool again." - ) - _save_checkpoint( - Checkpoint( - past_schedule=Schedule( - waypoints=schedule.waypoints[: schedule_results.failed_waypoint_i] - ) - ), - expedition_dir, - ) - return - - # delete and create results directory - if os.path.exists(expedition_dir.joinpath("results")): - shutil.rmtree(expedition_dir.joinpath("results")) - os.makedirs(expedition_dir.joinpath("results")) - - print("\n----- EXPEDITION SUMMARY ------") - - # calculate expedition cost in US$ - assert schedule.waypoints[0].time is not None, ( - "First waypoint has no time. This should not be possible as it should have been verified before." - ) - time_past = schedule_results.time - schedule.waypoints[0].time - cost = expedition_cost(schedule_results, time_past) - with open(expedition_dir.joinpath("results", "cost.txt"), "w") as file: - file.writelines(f"cost: {cost} US$") - print(f"\nExpedition duration: {time_past}\nExpedition cost: US$ {cost:,.0f}.") - - print("\n--- MEASUREMENT SIMULATIONS ---") - - # simulate measurements - print("\nSimulating measurements. This may take a while...\n") - simulate_measurements( - expedition_dir, - ship_config, - loaded_input_data, - schedule_results.measurements_to_simulate, - ) - print("\nAll measurement simulations are complete.") - - print("\n----- EXPEDITION RESULTS ------") - print("\nYour expedition has concluded successfully!") - print( - f"Your measurements can be found in the '{expedition_dir}/results' directory." - ) - print("\n------------- END -------------\n") - - -def _load_input_data( - expedition_dir: Path, - schedule: Schedule, - ship_config: ShipConfig, - input_data: Path | None, -) -> InputData: - """ - Load the input data. - - :param expedition_dir: Directory of the expedition. - :type expedition_dir: Path - :param schedule: Schedule object. - :type schedule: Schedule - :param ship_config: Ship configuration. - :type ship_config: ShipConfig - :param input_data: Folder containing input data. - :type input_data: Path | None - :return: InputData object. - :rtype: InputData - """ - if input_data is None: - space_time_region_hash = get_space_time_region_hash(schedule.space_time_region) - input_data = get_existing_download(expedition_dir, space_time_region_hash) - - assert input_data is not None, ( - "Input data hasn't been found. Have you run the `virtualship fetch` command?" - ) - - return InputData.load( - directory=input_data, - load_adcp=ship_config.adcp_config is not None, - load_argo_float=ship_config.argo_float_config is not None, - load_ctd=ship_config.ctd_config is not None, - load_ctd_bgc=ship_config.ctd_bgc_config is not None, - load_drifter=ship_config.drifter_config is not None, - load_xbt=ship_config.xbt_config is not None, - load_ship_underwater_st=ship_config.ship_underwater_st_config is not None, - ) - - -def _load_checkpoint(expedition_dir: Path) -> Checkpoint | None: - file_path = expedition_dir.joinpath(CHECKPOINT) - try: - return Checkpoint.from_yaml(file_path) - except FileNotFoundError: - return None - - -def _save_checkpoint(checkpoint: Checkpoint, expedition_dir: Path) -> None: - file_path = expedition_dir.joinpath(CHECKPOINT) - checkpoint.to_yaml(file_path) diff --git a/src/virtualship/expedition/expedition_cost.py b/src/virtualship/expedition/expedition_cost.py deleted file mode 100644 index cab6ab7d..00000000 --- a/src/virtualship/expedition/expedition_cost.py +++ /dev/null @@ -1,27 +0,0 @@ -"""expedition_cost function.""" - -from datetime import timedelta - -from .simulate_schedule import ScheduleOk - - -def expedition_cost(schedule_results: ScheduleOk, time_past: timedelta) -> float: - """ - Calculate the cost of the expedition in US$. - - :param schedule_results: Results from schedule simulation. - :param time_past: Time the expedition took. - :returns: The calculated cost of the expedition in US$. - """ - SHIP_COST_PER_DAY = 30000 - DRIFTER_DEPLOY_COST = 2500 - ARGO_DEPLOY_COST = 15000 - - ship_cost = SHIP_COST_PER_DAY / 24 * time_past.total_seconds() // 3600 - num_argos = len(schedule_results.measurements_to_simulate.argo_floats) - argo_cost = num_argos * ARGO_DEPLOY_COST - num_drifters = len(schedule_results.measurements_to_simulate.drifters) - drifter_cost = num_drifters * DRIFTER_DEPLOY_COST - - cost = ship_cost + argo_cost + drifter_cost - return cost diff --git a/src/virtualship/expedition/input_data.py b/src/virtualship/expedition/input_data.py deleted file mode 100644 index fa48e0a7..00000000 --- a/src/virtualship/expedition/input_data.py +++ /dev/null @@ -1,255 +0,0 @@ -"""InputData class.""" - -from __future__ import annotations - -from dataclasses import dataclass -from pathlib import Path - -from parcels import Field, FieldSet - - -@dataclass -class InputData: - """A collection of fieldsets that function as input data for simulation.""" - - adcp_fieldset: FieldSet | None - argo_float_fieldset: FieldSet | None - ctd_fieldset: FieldSet | None - ctd_bgc_fieldset: FieldSet | None - drifter_fieldset: FieldSet | None - xbt_fieldset: FieldSet | None - ship_underwater_st_fieldset: FieldSet | None - - @classmethod - def load( - cls, - directory: str | Path, - load_adcp: bool, - load_argo_float: bool, - load_ctd: bool, - load_ctd_bgc: bool, - load_drifter: bool, - load_xbt: bool, - load_ship_underwater_st: bool, - ) -> InputData: - """ - Create an instance of this class from netCDF files. - - For now this function makes a lot of assumption about file location and contents. - - :param directory: Input data directory. - :param load_adcp: Whether to load the ADCP fieldset. - :param load_argo_float: Whether to load the argo float fieldset. - :param load_ctd: Whether to load the CTD fieldset. - :param load_ctd_bgc: Whether to load the CTD BGC fieldset. - :param load_drifter: Whether to load the drifter fieldset. - :param load_ship_underwater_st: Whether to load the ship underwater ST fieldset. - :returns: An instance of this class with loaded fieldsets. - """ - directory = Path(directory) - if load_drifter: - drifter_fieldset = cls._load_drifter_fieldset(directory) - else: - drifter_fieldset = None - if load_argo_float: - argo_float_fieldset = cls._load_argo_float_fieldset(directory) - else: - argo_float_fieldset = None - if load_ctd_bgc: - ctd_bgc_fieldset = cls._load_ctd_bgc_fieldset(directory) - else: - ctd_bgc_fieldset = None - if load_adcp or load_ctd or load_ship_underwater_st or load_xbt: - ship_fieldset = cls._load_ship_fieldset(directory) - if load_adcp: - adcp_fieldset = ship_fieldset - else: - adcp_fieldset = None - if load_ctd: - ctd_fieldset = ship_fieldset - else: - ctd_fieldset = None - if load_ship_underwater_st: - ship_underwater_st_fieldset = ship_fieldset - else: - ship_underwater_st_fieldset = None - if load_xbt: - xbt_fieldset = ship_fieldset - else: - xbt_fieldset = None - - return InputData( - adcp_fieldset=adcp_fieldset, - argo_float_fieldset=argo_float_fieldset, - ctd_fieldset=ctd_fieldset, - ctd_bgc_fieldset=ctd_bgc_fieldset, - drifter_fieldset=drifter_fieldset, - xbt_fieldset=xbt_fieldset, - ship_underwater_st_fieldset=ship_underwater_st_fieldset, - ) - - @classmethod - def _load_ship_fieldset(cls, directory: Path) -> FieldSet: - filenames = { - "U": directory.joinpath("ship_uv.nc"), - "V": directory.joinpath("ship_uv.nc"), - "S": directory.joinpath("ship_s.nc"), - "T": directory.joinpath("ship_t.nc"), - } - variables = {"U": "uo", "V": "vo", "S": "so", "T": "thetao"} - dimensions = { - "lon": "longitude", - "lat": "latitude", - "time": "time", - "depth": "depth", - } - - # create the fieldset and set interpolation methods - fieldset = FieldSet.from_netcdf( - filenames, variables, dimensions, allow_time_extrapolation=True - ) - fieldset.T.interp_method = "linear_invdist_land_tracer" - fieldset.S.interp_method = "linear_invdist_land_tracer" - - # make depth negative - for g in fieldset.gridset.grids: - g.negate_depth() - - # add bathymetry data - bathymetry_file = directory.joinpath("bathymetry.nc") - bathymetry_variables = ("bathymetry", "deptho") - bathymetry_dimensions = {"lon": "longitude", "lat": "latitude"} - bathymetry_field = Field.from_netcdf( - bathymetry_file, bathymetry_variables, bathymetry_dimensions - ) - # make depth negative - bathymetry_field.data = -bathymetry_field.data - fieldset.add_field(bathymetry_field) - - # read in data already - fieldset.computeTimeChunk(0, 1) - - return fieldset - - @classmethod - def _load_ctd_bgc_fieldset(cls, directory: Path) -> FieldSet: - filenames = { - "U": directory.joinpath("ship_uv.nc"), - "V": directory.joinpath("ship_uv.nc"), - "o2": directory.joinpath("ctd_bgc_o2.nc"), - "chl": directory.joinpath("ctd_bgc_chl.nc"), - "no3": directory.joinpath("ctd_bgc_no3.nc"), - "po4": directory.joinpath("ctd_bgc_po4.nc"), - "ph": directory.joinpath("ctd_bgc_ph.nc"), - "phyc": directory.joinpath("ctd_bgc_phyc.nc"), - "nppv": directory.joinpath("ctd_bgc_nppv.nc"), - } - variables = { - "U": "uo", - "V": "vo", - "o2": "o2", - "chl": "chl", - "no3": "no3", - "po4": "po4", - "ph": "ph", - "phyc": "phyc", - "nppv": "nppv", - } - dimensions = { - "lon": "longitude", - "lat": "latitude", - "time": "time", - "depth": "depth", - } - - fieldset = FieldSet.from_netcdf( - filenames, variables, dimensions, allow_time_extrapolation=True - ) - fieldset.o2.interp_method = "linear_invdist_land_tracer" - fieldset.chl.interp_method = "linear_invdist_land_tracer" - fieldset.no3.interp_method = "linear_invdist_land_tracer" - fieldset.po4.interp_method = "linear_invdist_land_tracer" - fieldset.ph.interp_method = "linear_invdist_land_tracer" - fieldset.phyc.interp_method = "linear_invdist_land_tracer" - fieldset.nppv.interp_method = "linear_invdist_land_tracer" - - # make depth negative - for g in fieldset.gridset.grids: - g.negate_depth() - - # add bathymetry data - bathymetry_file = directory.joinpath("bathymetry.nc") - bathymetry_variables = ("bathymetry", "deptho") - bathymetry_dimensions = {"lon": "longitude", "lat": "latitude"} - bathymetry_field = Field.from_netcdf( - bathymetry_file, bathymetry_variables, bathymetry_dimensions - ) - # make depth negative - bathymetry_field.data = -bathymetry_field.data - fieldset.add_field(bathymetry_field) - - # read in data already - fieldset.computeTimeChunk(0, 1) - - return fieldset - - @classmethod - def _load_drifter_fieldset(cls, directory: Path) -> FieldSet: - filenames = { - "U": directory.joinpath("drifter_uv.nc"), - "V": directory.joinpath("drifter_uv.nc"), - "T": directory.joinpath("drifter_t.nc"), - } - variables = {"U": "uo", "V": "vo", "T": "thetao"} - dimensions = { - "lon": "longitude", - "lat": "latitude", - "time": "time", - "depth": "depth", - } - - fieldset = FieldSet.from_netcdf( - filenames, variables, dimensions, allow_time_extrapolation=False - ) - fieldset.T.interp_method = "linear_invdist_land_tracer" - - # make depth negative - for g in fieldset.gridset.grids: - g.negate_depth() - - # read in data already - fieldset.computeTimeChunk(0, 1) - - return fieldset - - @classmethod - def _load_argo_float_fieldset(cls, directory: Path) -> FieldSet: - filenames = { - "U": directory.joinpath("argo_float_uv.nc"), - "V": directory.joinpath("argo_float_uv.nc"), - "S": directory.joinpath("argo_float_s.nc"), - "T": directory.joinpath("argo_float_t.nc"), - } - variables = {"U": "uo", "V": "vo", "S": "so", "T": "thetao"} - dimensions = { - "lon": "longitude", - "lat": "latitude", - "time": "time", - "depth": "depth", - } - - fieldset = FieldSet.from_netcdf( - filenames, variables, dimensions, allow_time_extrapolation=False - ) - fieldset.T.interp_method = "linear_invdist_land_tracer" - fieldset.S.interp_method = "linear_invdist_land_tracer" - - # make depth negative - for g in fieldset.gridset.grids: - if max(g.depth) > 0: - g.negate_depth() - - # read in data already - fieldset.computeTimeChunk(0, 1) - - return fieldset diff --git a/src/virtualship/expedition/simulate_measurements.py b/src/virtualship/expedition/simulate_measurements.py deleted file mode 100644 index 20ba2cdb..00000000 --- a/src/virtualship/expedition/simulate_measurements.py +++ /dev/null @@ -1,163 +0,0 @@ -"""simulate_measurements function.""" - -from __future__ import annotations - -import logging -from datetime import timedelta -from pathlib import Path -from typing import TYPE_CHECKING - -from yaspin import yaspin - -from virtualship.instruments.adcp import simulate_adcp -from virtualship.instruments.argo_float import simulate_argo_floats -from virtualship.instruments.ctd import simulate_ctd -from virtualship.instruments.ctd_bgc import simulate_ctd_bgc -from virtualship.instruments.drifter import simulate_drifters -from virtualship.instruments.ship_underwater_st import simulate_ship_underwater_st -from virtualship.instruments.xbt import simulate_xbt -from virtualship.models import ShipConfig -from virtualship.utils import ship_spinner - -from .simulate_schedule import MeasurementsToSimulate - -if TYPE_CHECKING: - from .input_data import InputData - -# parcels logger (suppress INFO messages to prevent log being flooded) -external_logger = logging.getLogger("parcels.tools.loggers") -external_logger.setLevel(logging.WARNING) - - -def simulate_measurements( - expedition_dir: str | Path, - ship_config: ShipConfig, - input_data: InputData, - measurements: MeasurementsToSimulate, -) -> None: - """ - Simulate measurements using Parcels. - - Saves everything in expedition_dir/results. - - :param expedition_dir: Base directory of the expedition. - :param ship_config: Ship configuration. - :param input_data: Input data for simulation. - :param measurements: The measurements to simulate. - :raises RuntimeError: In case fieldsets of configuration is not provided. Make sure to check this before calling this function. - """ - if isinstance(expedition_dir, str): - expedition_dir = Path(expedition_dir) - - if len(measurements.ship_underwater_sts) > 0: - if ship_config.ship_underwater_st_config is None: - raise RuntimeError("No configuration for ship underwater ST provided.") - if input_data.ship_underwater_st_fieldset is None: - raise RuntimeError("No fieldset for ship underwater ST provided.") - with yaspin( - text="Simulating onboard temperature and salinity measurements... ", - side="right", - spinner=ship_spinner, - ) as spinner: - simulate_ship_underwater_st( - fieldset=input_data.ship_underwater_st_fieldset, - out_path=expedition_dir.joinpath("results", "ship_underwater_st.zarr"), - depth=-2, - sample_points=measurements.ship_underwater_sts, - ) - spinner.ok("✅") - - if len(measurements.adcps) > 0: - if ship_config.adcp_config is None: - raise RuntimeError("No configuration for ADCP provided.") - if input_data.adcp_fieldset is None: - raise RuntimeError("No fieldset for ADCP provided.") - with yaspin( - text="Simulating onboard ADCP... ", side="right", spinner=ship_spinner - ) as spinner: - simulate_adcp( - fieldset=input_data.adcp_fieldset, - out_path=expedition_dir.joinpath("results", "adcp.zarr"), - max_depth=ship_config.adcp_config.max_depth_meter, - min_depth=-5, - num_bins=ship_config.adcp_config.num_bins, - sample_points=measurements.adcps, - ) - spinner.ok("✅") - - if len(measurements.ctds) > 0: - if ship_config.ctd_config is None: - raise RuntimeError("No configuration for CTD provided.") - if input_data.ctd_fieldset is None: - raise RuntimeError("No fieldset for CTD provided.") - with yaspin( - text="Simulating CTD casts... ", side="right", spinner=ship_spinner - ) as spinner: - simulate_ctd( - out_path=expedition_dir.joinpath("results", "ctd.zarr"), - fieldset=input_data.ctd_fieldset, - ctds=measurements.ctds, - outputdt=timedelta(seconds=10), - ) - spinner.ok("✅") - - if len(measurements.ctd_bgcs) > 0: - if ship_config.ctd_bgc_config is None: - raise RuntimeError("No configuration for CTD_BGC provided.") - if input_data.ctd_bgc_fieldset is None: - raise RuntimeError("No fieldset for CTD_BGC provided.") - with yaspin( - text="Simulating BGC CTD casts... ", side="right", spinner=ship_spinner - ) as spinner: - simulate_ctd_bgc( - out_path=expedition_dir.joinpath("results", "ctd_bgc.zarr"), - fieldset=input_data.ctd_bgc_fieldset, - ctd_bgcs=measurements.ctd_bgcs, - outputdt=timedelta(seconds=10), - ) - spinner.ok("✅") - - if len(measurements.xbts) > 0: - if ship_config.xbt_config is None: - raise RuntimeError("No configuration for XBTs provided.") - if input_data.xbt_fieldset is None: - raise RuntimeError("No fieldset for XBTs provided.") - with yaspin( - text="Simulating XBTs... ", side="right", spinner=ship_spinner - ) as spinner: - simulate_xbt( - out_path=expedition_dir.joinpath("results", "xbts.zarr"), - fieldset=input_data.xbt_fieldset, - xbts=measurements.xbts, - outputdt=timedelta(seconds=1), - ) - spinner.ok("✅") - - if len(measurements.drifters) > 0: - print("Simulating drifters... ") - if ship_config.drifter_config is None: - raise RuntimeError("No configuration for drifters provided.") - if input_data.drifter_fieldset is None: - raise RuntimeError("No fieldset for drifters provided.") - simulate_drifters( - out_path=expedition_dir.joinpath("results", "drifters.zarr"), - fieldset=input_data.drifter_fieldset, - drifters=measurements.drifters, - outputdt=timedelta(hours=5), - dt=timedelta(minutes=5), - endtime=None, - ) - - if len(measurements.argo_floats) > 0: - print("Simulating argo floats... ") - if ship_config.argo_float_config is None: - raise RuntimeError("No configuration for argo floats provided.") - if input_data.argo_float_fieldset is None: - raise RuntimeError("No fieldset for argo floats provided.") - simulate_argo_floats( - out_path=expedition_dir.joinpath("results", "argo_floats.zarr"), - argo_floats=measurements.argo_floats, - fieldset=input_data.argo_float_fieldset, - outputdt=timedelta(minutes=5), - endtime=None, - ) diff --git a/src/virtualship/expedition/simulate_schedule.py b/src/virtualship/expedition/simulate_schedule.py index 95fa2f5f..0a567b1c 100644 --- a/src/virtualship/expedition/simulate_schedule.py +++ b/src/virtualship/expedition/simulate_schedule.py @@ -4,6 +4,7 @@ from dataclasses import dataclass, field from datetime import datetime, timedelta +from typing import ClassVar import pyproj @@ -11,12 +12,11 @@ from virtualship.instruments.ctd import CTD from virtualship.instruments.ctd_bgc import CTD_BGC from virtualship.instruments.drifter import Drifter +from virtualship.instruments.types import InstrumentType from virtualship.instruments.xbt import XBT from virtualship.models import ( - InstrumentType, + Expedition, Location, - Schedule, - ShipConfig, Spacetime, Waypoint, ) @@ -40,7 +40,26 @@ class ScheduleProblem: @dataclass class MeasurementsToSimulate: - """The measurements to simulate, as concluded from schedule simulation.""" + """ + The measurements to simulate, as concluded from schedule simulation. + + Provides a mapping from InstrumentType to the correct attribute name for robust access. + """ + + _instrumenttype_to_attr: ClassVar[dict] = { + InstrumentType.ADCP: "adcps", + InstrumentType.UNDERWATER_ST: "ship_underwater_sts", + InstrumentType.ARGO_FLOAT: "argo_floats", + InstrumentType.DRIFTER: "drifters", + InstrumentType.CTD: "ctds", + InstrumentType.CTD_BGC: "ctd_bgcs", + InstrumentType.XBT: "xbts", + } + + @classmethod + def get_attr_for_instrumenttype(cls, instrument_type): + """Return the attribute name for a given InstrumentType.""" + return cls._instrumenttype_to_attr[instrument_type] adcps: list[Spacetime] = field(default_factory=list, init=False) ship_underwater_sts: list[Spacetime] = field(default_factory=list, init=False) @@ -52,23 +71,21 @@ class MeasurementsToSimulate: def simulate_schedule( - projection: pyproj.Geod, ship_config: ShipConfig, schedule: Schedule + projection: pyproj.Geod, expedition: Expedition ) -> ScheduleOk | ScheduleProblem: """ Simulate a schedule. :param projection: The projection to use for sailing. - :param ship_config: Ship configuration. - :param schedule: The schedule to simulate. + :param expedition: Expedition object containing the schedule to simulate. :returns: Either the results of a successfully simulated schedule, or information on where the schedule became infeasible. """ - return _ScheduleSimulator(projection, ship_config, schedule).simulate() + return _ScheduleSimulator(projection, expedition).simulate() class _ScheduleSimulator: _projection: pyproj.Geod - _ship_config: ShipConfig - _schedule: Schedule + _expedition: Expedition _time: datetime """Current time.""" @@ -82,18 +99,15 @@ class _ScheduleSimulator: _next_ship_underwater_st_time: datetime """Next moment ship underwater ST measurement will be done.""" - def __init__( - self, projection: pyproj.Geod, ship_config: ShipConfig, schedule: Schedule - ) -> None: + def __init__(self, projection: pyproj.Geod, expedition: Expedition) -> None: self._projection = projection - self._ship_config = ship_config - self._schedule = schedule + self._expedition = expedition - assert self._schedule.waypoints[0].time is not None, ( + assert self._expedition.schedule.waypoints[0].time is not None, ( "First waypoint must have a time. This should have been verified before calling this function." ) - self._time = schedule.waypoints[0].time - self._location = schedule.waypoints[0].location + self._time = expedition.schedule.waypoints[0].time + self._location = expedition.schedule.waypoints[0].location self._measurements_to_simulate = MeasurementsToSimulate() @@ -101,15 +115,16 @@ def __init__( self._next_ship_underwater_st_time = self._time def simulate(self) -> ScheduleOk | ScheduleProblem: - for wp_i, waypoint in enumerate(self._schedule.waypoints): + for wp_i, waypoint in enumerate(self._expedition.schedule.waypoints): # sail towards waypoint self._progress_time_traveling_towards(waypoint.location) # check if waypoint was reached in time if waypoint.time is not None and self._time > waypoint.time: print( - # TODO: I think this should be wp_i + 1, not wp_i; otherwise it will be off by one - f"Waypoint {wp_i} could not be reached in time. Current time: {self._time}. Waypoint time: {waypoint.time}." + f"Waypoint {wp_i + 1} could not be reached in time. Current time: {self._time}. Waypoint time: {waypoint.time}." + "\n\nHave you ensured that your schedule includes sufficient time for taking measurements, e.g. CTD casts (in addition to the time it takes to sail between waypoints)?\n" + "**Note**, the `virtualship plan` tool will not account for measurement times when verifying the schedule, only the time it takes to sail between waypoints.\n" ) return ScheduleProblem(self._time, wp_i) else: @@ -131,7 +146,9 @@ def _progress_time_traveling_towards(self, location: Location) -> None: lons2=location.lon, lats2=location.lat, ) - ship_speed_meter_per_second = self._ship_config.ship_speed_knots * 1852 / 3600 + ship_speed_meter_per_second = ( + self._expedition.ship_config.ship_speed_knots * 1852 / 3600 + ) azimuth1 = geodinv[0] distance_to_next_waypoint = geodinv[2] time_to_reach = timedelta( @@ -140,7 +157,7 @@ def _progress_time_traveling_towards(self, location: Location) -> None: end_time = self._time + time_to_reach # note all ADCP measurements - if self._ship_config.adcp_config is not None: + if self._expedition.instruments_config.adcp_config is not None: location = self._location time = self._time while self._next_adcp_time <= end_time: @@ -162,11 +179,12 @@ def _progress_time_traveling_towards(self, location: Location) -> None: ) self._next_adcp_time = ( - self._next_adcp_time + self._ship_config.adcp_config.period + self._next_adcp_time + + self._expedition.instruments_config.adcp_config.period ) # note all ship underwater ST measurements - if self._ship_config.ship_underwater_st_config is not None: + if self._expedition.instruments_config.ship_underwater_st_config is not None: location = self._location time = self._time while self._next_ship_underwater_st_time <= end_time: @@ -189,7 +207,7 @@ def _progress_time_traveling_towards(self, location: Location) -> None: self._next_ship_underwater_st_time = ( self._next_ship_underwater_st_time - + self._ship_config.ship_underwater_st_config.period + + self._expedition.instruments_config.ship_underwater_st_config.period ) self._time = end_time @@ -199,24 +217,25 @@ def _progress_time_stationary(self, time_passed: timedelta) -> None: end_time = self._time + time_passed # note all ADCP measurements - if self._ship_config.adcp_config is not None: + if self._expedition.instruments_config.adcp_config is not None: while self._next_adcp_time <= end_time: self._measurements_to_simulate.adcps.append( Spacetime(self._location, self._next_adcp_time) ) self._next_adcp_time = ( - self._next_adcp_time + self._ship_config.adcp_config.period + self._next_adcp_time + + self._expedition.instruments_config.adcp_config.period ) # note all ship underwater ST measurements - if self._ship_config.ship_underwater_st_config is not None: + if self._expedition.instruments_config.ship_underwater_st_config is not None: while self._next_ship_underwater_st_time <= end_time: self._measurements_to_simulate.ship_underwater_sts.append( Spacetime(self._location, self._next_ship_underwater_st_time) ) self._next_ship_underwater_st_time = ( self._next_ship_underwater_st_time - + self._ship_config.ship_underwater_st_config.period + + self._expedition.instruments_config.ship_underwater_st_config.period ) self._time = end_time @@ -241,48 +260,52 @@ def _make_measurements(self, waypoint: Waypoint) -> timedelta: self._measurements_to_simulate.argo_floats.append( ArgoFloat( spacetime=Spacetime(self._location, self._time), - min_depth=self._ship_config.argo_float_config.min_depth_meter, - max_depth=self._ship_config.argo_float_config.max_depth_meter, - drift_depth=self._ship_config.argo_float_config.drift_depth_meter, - vertical_speed=self._ship_config.argo_float_config.vertical_speed_meter_per_second, - cycle_days=self._ship_config.argo_float_config.cycle_days, - drift_days=self._ship_config.argo_float_config.drift_days, + min_depth=self._expedition.instruments_config.argo_float_config.min_depth_meter, + max_depth=self._expedition.instruments_config.argo_float_config.max_depth_meter, + drift_depth=self._expedition.instruments_config.argo_float_config.drift_depth_meter, + vertical_speed=self._expedition.instruments_config.argo_float_config.vertical_speed_meter_per_second, + cycle_days=self._expedition.instruments_config.argo_float_config.cycle_days, + drift_days=self._expedition.instruments_config.argo_float_config.drift_days, ) ) elif instrument is InstrumentType.CTD: self._measurements_to_simulate.ctds.append( CTD( spacetime=Spacetime(self._location, self._time), - min_depth=self._ship_config.ctd_config.min_depth_meter, - max_depth=self._ship_config.ctd_config.max_depth_meter, + min_depth=self._expedition.instruments_config.ctd_config.min_depth_meter, + max_depth=self._expedition.instruments_config.ctd_config.max_depth_meter, ) ) - time_costs.append(self._ship_config.ctd_config.stationkeeping_time) + time_costs.append( + self._expedition.instruments_config.ctd_config.stationkeeping_time + ) elif instrument is InstrumentType.CTD_BGC: self._measurements_to_simulate.ctd_bgcs.append( CTD_BGC( spacetime=Spacetime(self._location, self._time), - min_depth=self._ship_config.ctd_bgc_config.min_depth_meter, - max_depth=self._ship_config.ctd_bgc_config.max_depth_meter, + min_depth=self._expedition.instruments_config.ctd_bgc_config.min_depth_meter, + max_depth=self._expedition.instruments_config.ctd_bgc_config.max_depth_meter, ) ) - time_costs.append(self._ship_config.ctd_bgc_config.stationkeeping_time) + time_costs.append( + self._expedition.instruments_config.ctd_bgc_config.stationkeeping_time + ) elif instrument is InstrumentType.DRIFTER: self._measurements_to_simulate.drifters.append( Drifter( spacetime=Spacetime(self._location, self._time), - depth=self._ship_config.drifter_config.depth_meter, - lifetime=self._ship_config.drifter_config.lifetime, + depth=self._expedition.instruments_config.drifter_config.depth_meter, + lifetime=self._expedition.instruments_config.drifter_config.lifetime, ) ) elif instrument is InstrumentType.XBT: self._measurements_to_simulate.xbts.append( XBT( spacetime=Spacetime(self._location, self._time), - min_depth=self._ship_config.xbt_config.min_depth_meter, - max_depth=self._ship_config.xbt_config.max_depth_meter, - fall_speed=self._ship_config.xbt_config.fall_speed_meter_per_second, - deceleration_coefficient=self._ship_config.xbt_config.deceleration_coefficient, + min_depth=self._expedition.instruments_config.xbt_config.min_depth_meter, + max_depth=self._expedition.instruments_config.xbt_config.max_depth_meter, + fall_speed=self._expedition.instruments_config.xbt_config.fall_speed_meter_per_second, + deceleration_coefficient=self._expedition.instruments_config.xbt_config.deceleration_coefficient, ) ) else: diff --git a/src/virtualship/instruments/__init__.py b/src/virtualship/instruments/__init__.py index 6a6ffbca..b593ed38 100644 --- a/src/virtualship/instruments/__init__.py +++ b/src/virtualship/instruments/__init__.py @@ -1,6 +1,14 @@ -"""Measurement instrument that can be used with Parcels.""" +"""Instruments in VirtualShip.""" -from . import adcp, argo_float, ctd, ctd_bgc, drifter, ship_underwater_st, xbt +from . import ( + adcp, + argo_float, + ctd, + ctd_bgc, + drifter, + ship_underwater_st, + xbt, +) __all__ = [ "adcp", diff --git a/src/virtualship/instruments/adcp.py b/src/virtualship/instruments/adcp.py index af2c285e..2a761e14 100644 --- a/src/virtualship/instruments/adcp.py +++ b/src/virtualship/instruments/adcp.py @@ -1,14 +1,32 @@ -"""ADCP instrument.""" - -from pathlib import Path +from dataclasses import dataclass +from typing import ClassVar import numpy as np -from parcels import FieldSet, ParticleSet, ScipyParticle, Variable +from parcels import ParticleSet, ScipyParticle, Variable + +from virtualship.instruments.base import Instrument +from virtualship.instruments.types import InstrumentType +from virtualship.utils import ( + register_instrument, +) + +# ===================================================== +# SECTION: Dataclass +# ===================================================== + + +@dataclass +class ADCP: + """ADCP configuration.""" + + name: ClassVar[str] = "ADCP" + + +# ===================================================== +# SECTION: Particle Class +# ===================================================== -from virtualship.models import Spacetime -# we specifically use ScipyParticle because we have many small calls to execute -# there is some overhead with JITParticle and this ends up being significantly faster _ADCPParticle = ScipyParticle.add_variables( [ Variable("U", dtype=np.float32, initial=np.nan), @@ -16,6 +34,10 @@ ] ) +# ===================================================== +# SECTION: Kernels +# ===================================================== + def _sample_velocity(particle, fieldset, time): particle.U, particle.V = fieldset.UV.eval( @@ -23,56 +45,77 @@ def _sample_velocity(particle, fieldset, time): ) -def simulate_adcp( - fieldset: FieldSet, - out_path: str | Path, - max_depth: float, - min_depth: float, - num_bins: int, - sample_points: list[Spacetime], -) -> None: - """ - Use Parcels to simulate an ADCP in a fieldset. - - :param fieldset: The fieldset to simulate the ADCP in. - :param out_path: The path to write the results to. - :param max_depth: Maximum depth the ADCP can measure. - :param min_depth: Minimum depth the ADCP can measure. - :param num_bins: How many samples to take in the complete range between max_depth and min_depth. - :param sample_points: The places and times to sample at. - """ - sample_points.sort(key=lambda p: p.time) - - bins = np.linspace(max_depth, min_depth, num_bins) - num_particles = len(bins) - particleset = ParticleSet.from_list( - fieldset=fieldset, - pclass=_ADCPParticle, - lon=np.full( - num_particles, 0.0 - ), # initial lat/lon are irrelevant and will be overruled later. - lat=np.full(num_particles, 0.0), - depth=bins, - time=0, # same for time - ) +# ===================================================== +# SECTION: Instrument Class +# ===================================================== - # define output file for the simulation - # outputdt set to infinite as we just want to write at the end of every call to 'execute' - out_file = particleset.ParticleFile(name=out_path, outputdt=np.inf) - for point in sample_points: - particleset.lon_nextloop[:] = point.location.lon - particleset.lat_nextloop[:] = point.location.lat - particleset.time_nextloop[:] = fieldset.time_origin.reltime( - np.datetime64(point.time) - ) +@register_instrument(InstrumentType.ADCP) +class ADCPInstrument(Instrument): + """ADCP instrument class.""" - # perform one step using the particleset - # dt and runtime are set so exactly one step is made. - particleset.execute( - [_sample_velocity], - dt=1, - runtime=1, + def __init__(self, expedition, from_data): + """Initialize ADCPInstrument.""" + variables = {"U": "uo", "V": "vo"} + + super().__init__( + expedition, + variables, + add_bathymetry=False, + allow_time_extrapolation=True, verbose_progress=False, - output_file=out_file, + spacetime_buffer_size=None, + limit_spec=None, + from_data=from_data, + ) + + def simulate(self, measurements, out_path) -> None: + """Simulate ADCP measurements.""" + config_max_depth = ( + self.expedition.instruments_config.adcp_config.max_depth_meter + ) + + if config_max_depth < -1600.0: + print( + f"\n\n⚠️ Warning: The configured ADCP max depth of {abs(config_max_depth)} m exceeds the 1600 m limit for the technology (e.g. https://www.geomar.de/en/research/fb1/fb1-po/observing-systems/adcp)." + "\n\n This expedition will continue using the prescribed configuration. However, note, the results will not necessarily represent authentic ADCP instrument readings and could also lead to slower simulations ." + "\n\n If this was unintented, consider re-adjusting your ADCP configuration in your expedition.yaml or via `virtualship plan`.\n\n" + ) + + MAX_DEPTH = config_max_depth + MIN_DEPTH = -5.0 + NUM_BINS = self.expedition.instruments_config.adcp_config.num_bins + + measurements.sort(key=lambda p: p.time) + + fieldset = self.load_input_data() + + bins = np.linspace(MAX_DEPTH, MIN_DEPTH, NUM_BINS) + num_particles = len(bins) + particleset = ParticleSet.from_list( + fieldset=fieldset, + pclass=_ADCPParticle, + lon=np.full( + num_particles, 0.0 + ), # initial lat/lon are irrelevant and will be overruled later.s + lat=np.full(num_particles, 0.0), + depth=bins, + time=0, ) + + out_file = particleset.ParticleFile(name=out_path, outputdt=np.inf) + + for point in measurements: + particleset.lon_nextloop[:] = point.location.lon + particleset.lat_nextloop[:] = point.location.lat + particleset.time_nextloop[:] = fieldset.time_origin.reltime( + np.datetime64(point.time) + ) + + particleset.execute( + [_sample_velocity], + dt=1, + runtime=1, + verbose_progress=self.verbose_progress, + output_file=out_file, + ) diff --git a/src/virtualship/instruments/argo_float.py b/src/virtualship/instruments/argo_float.py index d0976367..204f0b3d 100644 --- a/src/virtualship/instruments/argo_float.py +++ b/src/virtualship/instruments/argo_float.py @@ -1,27 +1,32 @@ -"""Argo float instrument.""" - import math from dataclasses import dataclass -from datetime import datetime, timedelta -from pathlib import Path +from datetime import timedelta +from typing import ClassVar import numpy as np from parcels import ( AdvectionRK4, - FieldSet, JITParticle, ParticleSet, StatusCode, Variable, ) -from virtualship.models import Spacetime +from virtualship.instruments.base import Instrument +from virtualship.instruments.types import InstrumentType +from virtualship.models.spacetime import Spacetime +from virtualship.utils import register_instrument + +# ===================================================== +# SECTION: Dataclass +# ===================================================== @dataclass class ArgoFloat: - """Configuration for a single Argo float.""" + """Argo float configuration.""" + name: ClassVar[str] = "ArgoFloat" spacetime: Spacetime min_depth: float max_depth: float @@ -31,6 +36,10 @@ class ArgoFloat: drift_days: float +# ===================================================== +# SECTION: Particle Class +# ===================================================== + _ArgoParticle = JITParticle.add_variables( [ Variable("cycle_phase", dtype=np.int32, initial=0.0), @@ -47,6 +56,10 @@ class ArgoFloat: ] ) +# ===================================================== +# SECTION: Kernels +# ===================================================== + def _argo_float_vertical_movement(particle, fieldset, time): if particle.cycle_phase == 0: @@ -115,72 +128,92 @@ def _check_error(particle, fieldset, time): particle.delete() -def simulate_argo_floats( - fieldset: FieldSet, - out_path: str | Path, - argo_floats: list[ArgoFloat], - outputdt: timedelta, - endtime: datetime | None, -) -> None: - """ - Use Parcels to simulate a set of Argo floats in a fieldset. - - :param fieldset: The fieldset to simulate the Argo floats in. - :param out_path: The path to write the results to. - :param argo_floats: A list of Argo floats to simulate. - :param outputdt: Interval which dictates the update frequency of file output during simulation - :param endtime: Stop at this time, or if None, continue until the end of the fieldset. - """ - DT = 10.0 # dt of Argo float simulation integrator - - if len(argo_floats) == 0: - print( - "No Argo floats provided. Parcels currently crashes when providing an empty particle set, so no argo floats simulation will be done and no files will be created." +# ===================================================== +# SECTION: Instrument Class +# ===================================================== + + +@register_instrument(InstrumentType.ARGO_FLOAT) +class ArgoFloatInstrument(Instrument): + """ArgoFloat instrument class.""" + + def __init__(self, expedition, from_data): + """Initialize ArgoFloatInstrument.""" + variables = {"U": "uo", "V": "vo", "S": "so", "T": "thetao"} + spacetime_buffer_size = { + "latlon": 3.0, # [degrees] + "time": 21.0, # [days] + } + + super().__init__( + expedition, + variables, + add_bathymetry=False, + allow_time_extrapolation=False, + verbose_progress=True, + spacetime_buffer_size=spacetime_buffer_size, + limit_spec=None, + from_data=from_data, + ) + + def simulate(self, measurements, out_path) -> None: + """Simulate Argo float measurements.""" + DT = 10.0 # dt of Argo float simulation integrator + OUTPUT_DT = timedelta(minutes=5) + ENDTIME = None + + if len(measurements) == 0: + print( + "No Argo floats provided. Parcels currently crashes when providing an empty particle set, so no argo floats simulation will be done and no files will be created." + ) + # TODO when Parcels supports it this check can be removed. + return + + fieldset = self.load_input_data() + + # define parcel particles + argo_float_particleset = ParticleSet( + fieldset=fieldset, + pclass=_ArgoParticle, + lat=[argo.spacetime.location.lat for argo in measurements], + lon=[argo.spacetime.location.lon for argo in measurements], + depth=[argo.min_depth for argo in measurements], + time=[argo.spacetime.time for argo in measurements], + min_depth=[argo.min_depth for argo in measurements], + max_depth=[argo.max_depth for argo in measurements], + drift_depth=[argo.drift_depth for argo in measurements], + vertical_speed=[argo.vertical_speed for argo in measurements], + cycle_days=[argo.cycle_days for argo in measurements], + drift_days=[argo.drift_days for argo in measurements], + ) + + # define output file for the simulation + out_file = argo_float_particleset.ParticleFile( + name=out_path, + outputdt=OUTPUT_DT, + chunks=[len(argo_float_particleset), 100], + ) + + # get earliest between fieldset end time and provide end time + fieldset_endtime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[-1]) + if ENDTIME is None: + actual_endtime = fieldset_endtime + elif ENDTIME > fieldset_endtime: + print("WARN: Requested end time later than fieldset end time.") + actual_endtime = fieldset_endtime + else: + actual_endtime = np.timedelta64(ENDTIME) + + # execute simulation + argo_float_particleset.execute( + [ + _argo_float_vertical_movement, + AdvectionRK4, + _keep_at_surface, + _check_error, + ], + endtime=actual_endtime, + dt=DT, + output_file=out_file, + verbose_progress=self.verbose_progress, ) - # TODO when Parcels supports it this check can be removed. - return - - # define parcel particles - argo_float_particleset = ParticleSet( - fieldset=fieldset, - pclass=_ArgoParticle, - lat=[argo.spacetime.location.lat for argo in argo_floats], - lon=[argo.spacetime.location.lon for argo in argo_floats], - depth=[argo.min_depth for argo in argo_floats], - time=[argo.spacetime.time for argo in argo_floats], - min_depth=[argo.min_depth for argo in argo_floats], - max_depth=[argo.max_depth for argo in argo_floats], - drift_depth=[argo.drift_depth for argo in argo_floats], - vertical_speed=[argo.vertical_speed for argo in argo_floats], - cycle_days=[argo.cycle_days for argo in argo_floats], - drift_days=[argo.drift_days for argo in argo_floats], - ) - - # define output file for the simulation - out_file = argo_float_particleset.ParticleFile( - name=out_path, outputdt=outputdt, chunks=[len(argo_float_particleset), 100] - ) - - # get earliest between fieldset end time and provide end time - fieldset_endtime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[-1]) - if endtime is None: - actual_endtime = fieldset_endtime - elif endtime > fieldset_endtime: - print("WARN: Requested end time later than fieldset end time.") - actual_endtime = fieldset_endtime - else: - actual_endtime = np.timedelta64(endtime) - - # execute simulation - argo_float_particleset.execute( - [ - _argo_float_vertical_movement, - AdvectionRK4, - _keep_at_surface, - _check_error, - ], - endtime=actual_endtime, - dt=DT, - output_file=out_file, - verbose_progress=True, - ) diff --git a/src/virtualship/instruments/base.py b/src/virtualship/instruments/base.py new file mode 100644 index 00000000..22b0b54a --- /dev/null +++ b/src/virtualship/instruments/base.py @@ -0,0 +1,215 @@ +from __future__ import annotations + +import abc +from collections import OrderedDict +from datetime import timedelta +from pathlib import Path +from typing import TYPE_CHECKING + +import copernicusmarine +import xarray as xr +from parcels import FieldSet +from yaspin import yaspin + +from virtualship.errors import CopernicusCatalogueError +from virtualship.utils import ( + COPERNICUSMARINE_PHYS_VARIABLES, + _find_files_in_timerange, + _find_nc_file_with_variable, + _get_bathy_data, + _select_product_id, + ship_spinner, +) + +if TYPE_CHECKING: + from virtualship.models import Expedition + + +class Instrument(abc.ABC): + """Base class for instruments and their simulation.""" + + def __init__( + self, + expedition: Expedition, + variables: dict, + add_bathymetry: bool, + allow_time_extrapolation: bool, + verbose_progress: bool, + from_data: Path | None, + spacetime_buffer_size: dict | None = None, + limit_spec: dict | None = None, + ): + """Initialise instrument.""" + self.expedition = expedition + self.from_data = from_data + + self.variables = OrderedDict(variables) + self.dimensions = { + "lon": "longitude", + "lat": "latitude", + "time": "time", + "depth": "depth", + } # same dimensions for all instruments + self.add_bathymetry = add_bathymetry + self.allow_time_extrapolation = allow_time_extrapolation + self.verbose_progress = verbose_progress + self.spacetime_buffer_size = spacetime_buffer_size + self.limit_spec = limit_spec + + def load_input_data(self) -> FieldSet: + """Load and return the input data as a FieldSet for the instrument.""" + try: + fieldset = self._generate_fieldset() + except Exception as e: + raise CopernicusCatalogueError( + f"Failed to load input data directly from Copernicus Marine (or local data) for instrument '{self.__class__.__name__}'. Original error: {e}" + ) from e + + # interpolation methods + for var in (v for v in self.variables if v not in ("U", "V")): + getattr(fieldset, var).interp_method = "linear_invdist_land_tracer" + + # depth negative + for g in fieldset.gridset.grids: + g.negate_depth() + + # bathymetry data + if self.add_bathymetry: + bathymetry_field = _get_bathy_data( + self.expedition.schedule.space_time_region, + latlon_buffer=self.spacetime_buffer_size.get("latlon") + if self.spacetime_buffer_size + else None, + from_data=self.from_data, + ).bathymetry + bathymetry_field.data = -bathymetry_field.data + fieldset.add_field(bathymetry_field) + + return fieldset + + @abc.abstractmethod + def simulate( + self, + data_dir: Path, + measurements: list, + out_path: str | Path, + ) -> None: + """Simulate instrument measurements.""" + + def execute(self, measurements: list, out_path: str | Path) -> None: + """Run instrument simulation.""" + if not self.verbose_progress: + with yaspin( + text=f"Simulating {self.__class__.__name__.split('Instrument')[0]} measurements... ", + side="right", + spinner=ship_spinner, + ) as spinner: + self.simulate(measurements, out_path) + spinner.ok("✅\n") + else: + print( + f"Simulating {self.__class__.__name__.split('Instrument')[0]} measurements... " + ) + self.simulate(measurements, out_path) + print("\n") + + def _get_copernicus_ds( + self, + physical: bool, + var: str, + ) -> xr.Dataset: + """Get Copernicus Marine dataset for direct ingestion.""" + product_id = _select_product_id( + physical=physical, + schedule_start=self.expedition.schedule.space_time_region.time_range.start_time, + schedule_end=self.expedition.schedule.space_time_region.time_range.end_time, + variable=var if not physical else None, + ) + + latlon_buffer = self._get_spec_value("buffer", "latlon", 0.0) + time_buffer = self._get_spec_value("buffer", "time", 0.0) + depth_min = self._get_spec_value("limit", "depth_min", None) + depth_max = self._get_spec_value("limit", "depth_max", None) + + return copernicusmarine.open_dataset( + dataset_id=product_id, + minimum_longitude=self.expedition.schedule.space_time_region.spatial_range.minimum_longitude + - latlon_buffer, + maximum_longitude=self.expedition.schedule.space_time_region.spatial_range.maximum_longitude + + latlon_buffer, + minimum_latitude=self.expedition.schedule.space_time_region.spatial_range.minimum_latitude + - latlon_buffer, + maximum_latitude=self.expedition.schedule.space_time_region.spatial_range.maximum_latitude + + latlon_buffer, + variables=[var], + start_datetime=self.expedition.schedule.space_time_region.time_range.start_time, + end_datetime=self.expedition.schedule.space_time_region.time_range.end_time + + timedelta(days=time_buffer), + minimum_depth=depth_min, + maximum_depth=depth_max, + coordinates_selection_method="outside", + ) + + def _generate_fieldset(self) -> FieldSet: + """ + Create and combine FieldSets for each variable, supporting both local and Copernicus Marine data sources. + + Per variable avoids issues when using copernicusmarine and creating directly one FieldSet of ds's sourced from different Copernicus Marine product IDs, which is often the case for BGC variables. + """ + fieldsets_list = [] + keys = list(self.variables.keys()) + + for key in keys: + var = self.variables[key] + if self.from_data is not None: # load from local data + physical = var in COPERNICUSMARINE_PHYS_VARIABLES + if physical: + data_dir = self.from_data.joinpath("phys") + else: + data_dir = self.from_data.joinpath("bgc") + + schedule_start = ( + self.expedition.schedule.space_time_region.time_range.start_time + ) + schedule_end = ( + self.expedition.schedule.space_time_region.time_range.end_time + ) + + files = _find_files_in_timerange( + data_dir, + schedule_start, + schedule_end, + ) + + _, full_var_name = _find_nc_file_with_variable( + data_dir, var + ) # get full variable name from one of the files; var may only appear as substring in variable name in file + + ds = xr.open_mfdataset( + [data_dir.joinpath(f) for f in files] + ) # using: ds --> .from_xarray_dataset seems more robust than .from_netcdf for handling different temporal resolutions for different variables ... + + fs = FieldSet.from_xarray_dataset( + ds, + variables={key: full_var_name}, + dimensions=self.dimensions, + mesh="spherical", + ) + else: # stream via Copernicus Marine Service + physical = var in COPERNICUSMARINE_PHYS_VARIABLES + ds = self._get_copernicus_ds(physical=physical, var=var) + fs = FieldSet.from_xarray_dataset( + ds, {key: var}, self.dimensions, mesh="spherical" + ) + fieldsets_list.append(fs) + + base_fieldset = fieldsets_list[0] + for fs, key in zip(fieldsets_list[1:], keys[1:], strict=False): + base_fieldset.add_field(getattr(fs, key)) + + return base_fieldset + + def _get_spec_value(self, spec_type: str, key: str, default=None): + """Helper to extract a value from spacetime_buffer_size or limit_spec.""" + spec = self.spacetime_buffer_size if spec_type == "buffer" else self.limit_spec + return spec.get(key) if spec and spec.get(key) is not None else default diff --git a/src/virtualship/instruments/ctd.py b/src/virtualship/instruments/ctd.py index 41185007..73248cf9 100644 --- a/src/virtualship/instruments/ctd.py +++ b/src/virtualship/instruments/ctd.py @@ -1,24 +1,36 @@ -"""CTD instrument.""" - from dataclasses import dataclass from datetime import timedelta -from pathlib import Path +from typing import TYPE_CHECKING, ClassVar import numpy as np -from parcels import FieldSet, JITParticle, ParticleSet, Variable +from parcels import JITParticle, ParticleSet, Variable + +from virtualship.instruments.base import Instrument +from virtualship.instruments.types import InstrumentType + +if TYPE_CHECKING: + from virtualship.models.spacetime import Spacetime +from virtualship.utils import add_dummy_UV, register_instrument -from virtualship.models import Spacetime +# ===================================================== +# SECTION: Dataclass +# ===================================================== @dataclass class CTD: - """Configuration for a single CTD.""" + """CTD configuration.""" - spacetime: Spacetime + name: ClassVar[str] = "CTD" + spacetime: "Spacetime" min_depth: float max_depth: float +# ===================================================== +# SECTION: Particle Class +# ===================================================== + _CTDParticle = JITParticle.add_variables( [ Variable("salinity", dtype=np.float32, initial=np.nan), @@ -31,6 +43,11 @@ class CTD: ) +# ===================================================== +# SECTION: Kernels +# ===================================================== + + def _sample_temperature(particle, fieldset, time): particle.temperature = fieldset.T[time, particle.depth, particle.lat, particle.lon] @@ -53,85 +70,112 @@ def _ctd_cast(particle, fieldset, time): particle.delete() -def simulate_ctd( - fieldset: FieldSet, - out_path: str | Path, - ctds: list[CTD], - outputdt: timedelta, -) -> None: - """ - Use Parcels to simulate a set of CTDs in a fieldset. - - :param fieldset: The fieldset to simulate the CTDs in. - :param out_path: The path to write the results to. - :param ctds: A list of CTDs to simulate. - :param outputdt: Interval which dictates the update frequency of file output during simulation - :raises ValueError: Whenever provided CTDs, fieldset, are not compatible with this function. - """ - WINCH_SPEED = 1.0 # sink and rise speed in m/s - DT = 10.0 # dt of CTD simulation integrator - - if len(ctds) == 0: - print( - "No CTDs provided. Parcels currently crashes when providing an empty particle set, so no CTD simulation will be done and no files will be created." +# ===================================================== +# SECTION: Instrument Class +# ===================================================== + + +@register_instrument(InstrumentType.CTD) +class CTDInstrument(Instrument): + """CTD instrument class.""" + + def __init__(self, expedition, from_data): + """Initialize CTDInstrument.""" + variables = {"S": "so", "T": "thetao"} + + super().__init__( + expedition, + variables, + add_bathymetry=True, + allow_time_extrapolation=True, + verbose_progress=False, + spacetime_buffer_size=None, + limit_spec=None, + from_data=from_data, ) - # TODO when Parcels supports it this check can be removed. - return - - fieldset_starttime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[0]) - fieldset_endtime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[-1]) - - # deploy time for all ctds should be later than fieldset start time - if not all( - [np.datetime64(ctd.spacetime.time) >= fieldset_starttime for ctd in ctds] - ): - raise ValueError("CTD deployed before fieldset starts.") - - # depth the ctd will go to. shallowest between ctd max depth and bathymetry. - max_depths = [ - max( - ctd.max_depth, - fieldset.bathymetry.eval( - z=0, y=ctd.spacetime.location.lat, x=ctd.spacetime.location.lon, time=0 - ), + + def simulate(self, measurements, out_path) -> None: + """Simulate CTD measurements.""" + WINCH_SPEED = 1.0 # sink and rise speed in m/s + DT = 10.0 # dt of CTD simulation integrator + OUTPUT_DT = timedelta(seconds=10) # output dt for CTD simulation + + if len(measurements) == 0: + print( + "No CTDs provided. Parcels currently crashes when providing an empty particle set, so no CTD simulation will be done and no files will be created." + ) + # TODO when Parcels supports it this check can be removed. + return + + fieldset = self.load_input_data() + + # add dummy U + add_dummy_UV(fieldset) # TODO: parcels v3 bodge; remove when parcels v4 is used + + fieldset_starttime = fieldset.T.grid.time_origin.fulltime( + fieldset.T.grid.time_full[0] + ) + fieldset_endtime = fieldset.T.grid.time_origin.fulltime( + fieldset.T.grid.time_full[-1] ) - for ctd in ctds - ] - # CTD depth can not be too shallow, because kernel would break. - # This shallow is not useful anyway, no need to support. - if not all([max_depth <= -DT * WINCH_SPEED for max_depth in max_depths]): - raise ValueError( - f"CTD max_depth or bathymetry shallower than maximum {-DT * WINCH_SPEED}" + # deploy time for all ctds should be later than fieldset start time + if not all( + [ + np.datetime64(ctd.spacetime.time) >= fieldset_starttime + for ctd in measurements + ] + ): + raise ValueError("CTD deployed before fieldset starts.") + + # depth the ctd will go to. shallowest between ctd max depth and bathymetry. + max_depths = [ + max( + ctd.max_depth, + fieldset.bathymetry.eval( + z=0, + y=ctd.spacetime.location.lat, + x=ctd.spacetime.location.lon, + time=0, + ), + ) + for ctd in measurements + ] + + # CTD depth can not be too shallow, because kernel would break. + # This shallow is not useful anyway, no need to support. + if not all([max_depth <= -DT * WINCH_SPEED for max_depth in max_depths]): + raise ValueError( + f"CTD max_depth or bathymetry shallower than maximum {-DT * WINCH_SPEED}" + ) + + # define parcel particles + ctd_particleset = ParticleSet( + fieldset=fieldset, + pclass=_CTDParticle, + lon=[ctd.spacetime.location.lon for ctd in measurements], + lat=[ctd.spacetime.location.lat for ctd in measurements], + depth=[ctd.min_depth for ctd in measurements], + time=[ctd.spacetime.time for ctd in measurements], + max_depth=max_depths, + min_depth=[ctd.min_depth for ctd in measurements], + winch_speed=[WINCH_SPEED for _ in measurements], ) - # define parcel particles - ctd_particleset = ParticleSet( - fieldset=fieldset, - pclass=_CTDParticle, - lon=[ctd.spacetime.location.lon for ctd in ctds], - lat=[ctd.spacetime.location.lat for ctd in ctds], - depth=[ctd.min_depth for ctd in ctds], - time=[ctd.spacetime.time for ctd in ctds], - max_depth=max_depths, - min_depth=[ctd.min_depth for ctd in ctds], - winch_speed=[WINCH_SPEED for _ in ctds], - ) - - # define output file for the simulation - out_file = ctd_particleset.ParticleFile(name=out_path, outputdt=outputdt) - - # execute simulation - ctd_particleset.execute( - [_sample_salinity, _sample_temperature, _ctd_cast], - endtime=fieldset_endtime, - dt=DT, - verbose_progress=False, - output_file=out_file, - ) - - # there should be no particles left, as they delete themselves when they resurface - if len(ctd_particleset.particledata) != 0: - raise ValueError( - "Simulation ended before CTD resurfaced. This most likely means the field time dimension did not match the simulation time span." + # define output file for the simulation + out_file = ctd_particleset.ParticleFile(name=out_path, outputdt=OUTPUT_DT) + + # execute simulation + ctd_particleset.execute( + [_sample_salinity, _sample_temperature, _ctd_cast], + endtime=fieldset_endtime, + dt=DT, + verbose_progress=self.verbose_progress, + output_file=out_file, ) + + # there should be no particles left, as they delete themselves when they resurface + if len(ctd_particleset.particledata) != 0: + raise ValueError( + "Simulation ended before CTD resurfaced. This most likely means the field time dimension did not match the simulation time span." + ) diff --git a/src/virtualship/instruments/ctd_bgc.py b/src/virtualship/instruments/ctd_bgc.py index 0a34f61b..fab9e07b 100644 --- a/src/virtualship/instruments/ctd_bgc.py +++ b/src/virtualship/instruments/ctd_bgc.py @@ -1,24 +1,34 @@ -"""CTD_BGC instrument.""" - from dataclasses import dataclass from datetime import timedelta -from pathlib import Path +from typing import ClassVar import numpy as np -from parcels import FieldSet, JITParticle, ParticleSet, Variable +from parcels import JITParticle, ParticleSet, Variable + +from virtualship.instruments.base import Instrument +from virtualship.instruments.types import InstrumentType +from virtualship.models.spacetime import Spacetime +from virtualship.utils import add_dummy_UV, register_instrument -from virtualship.models import Spacetime +# ===================================================== +# SECTION: Dataclass +# ===================================================== @dataclass class CTD_BGC: - """Configuration for a single BGC CTD.""" + """CTD_BGC configuration.""" + name: ClassVar[str] = "CTD_BGC" spacetime: Spacetime min_depth: float max_depth: float +# ===================================================== +# SECTION: Particle Class +# ===================================================== + _CTD_BGCParticle = JITParticle.add_variables( [ Variable("o2", dtype=np.float32, initial=np.nan), @@ -35,6 +45,10 @@ class CTD_BGC: ] ) +# ===================================================== +# SECTION: Kernels +# ===================================================== + def _sample_o2(particle, fieldset, time): particle.o2 = fieldset.o2[time, particle.depth, particle.lat, particle.lon] @@ -78,100 +92,128 @@ def _ctd_bgc_cast(particle, fieldset, time): particle.delete() -def simulate_ctd_bgc( - fieldset: FieldSet, - out_path: str | Path, - ctd_bgcs: list[CTD_BGC], - outputdt: timedelta, -) -> None: - """ - Use Parcels to simulate a set of BGC CTDs in a fieldset. - - :param fieldset: The fieldset to simulate the BGC CTDs in. - :param out_path: The path to write the results to. - :param ctds: A list of BGC CTDs to simulate. - :param outputdt: Interval which dictates the update frequency of file output during simulation - :raises ValueError: Whenever provided BGC CTDs, fieldset, are not compatible with this function. - """ - WINCH_SPEED = 1.0 # sink and rise speed in m/s - DT = 10.0 # dt of CTD simulation integrator - - if len(ctd_bgcs) == 0: - print( - "No BGC CTDs provided. Parcels currently crashes when providing an empty particle set, so no BGC CTD simulation will be done and no files will be created." +# ===================================================== +# SECTION: Instrument Class +# ===================================================== + + +@register_instrument(InstrumentType.CTD_BGC) +class CTD_BGCInstrument(Instrument): + """CTD_BGC instrument class.""" + + def __init__(self, expedition, from_data): + """Initialize CTD_BGCInstrument.""" + variables = { + "o2": "o2", + "chl": "chl", + "no3": "no3", + "po4": "po4", + "ph": "ph", + "phyc": "phyc", + "nppv": "nppv", + } + super().__init__( + expedition, + variables, + add_bathymetry=True, + allow_time_extrapolation=True, + verbose_progress=False, + spacetime_buffer_size=None, + limit_spec=None, + from_data=from_data, ) - # TODO when Parcels supports it this check can be removed. - return - fieldset_starttime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[0]) - fieldset_endtime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[-1]) + def simulate(self, measurements, out_path) -> None: + """Simulate BGC CTD measurements using Parcels.""" + WINCH_SPEED = 1.0 # sink and rise speed in m/s + DT = 10.0 # dt of CTD_BGC simulation integrator + OUTPUT_DT = timedelta(seconds=10) # output dt for CTD_BGC simulation - # deploy time for all ctds should be later than fieldset start time - if not all( - [ - np.datetime64(ctd_bgc.spacetime.time) >= fieldset_starttime - for ctd_bgc in ctd_bgcs - ] - ): - raise ValueError("BGC CTD deployed before fieldset starts.") - - # depth the bgc ctd will go to. shallowest between bgc ctd max depth and bathymetry. - max_depths = [ - max( - ctd_bgc.max_depth, - fieldset.bathymetry.eval( - z=0, - y=ctd_bgc.spacetime.location.lat, - x=ctd_bgc.spacetime.location.lon, - time=0, - ), + if len(measurements) == 0: + print( + "No BGC CTDs provided. Parcels currently crashes when providing an empty particle set, so no BGC CTD simulation will be done and no files will be created." + ) + # TODO when Parcels supports it this check can be removed. + return + + fieldset = self.load_input_data() + + # add dummy U + add_dummy_UV(fieldset) # TODO: parcels v3 bodge; remove when parcels v4 is used + + fieldset_starttime = fieldset.o2.grid.time_origin.fulltime( + fieldset.o2.grid.time_full[0] + ) + fieldset_endtime = fieldset.o2.grid.time_origin.fulltime( + fieldset.o2.grid.time_full[-1] ) - for ctd_bgc in ctd_bgcs - ] - # CTD depth can not be too shallow, because kernel would break. - # This shallow is not useful anyway, no need to support. - if not all([max_depth <= -DT * WINCH_SPEED for max_depth in max_depths]): - raise ValueError( - f"BGC CTD max_depth or bathymetry shallower than maximum {-DT * WINCH_SPEED}" + # deploy time for all ctds should be later than fieldset start time + if not all( + [ + np.datetime64(ctd_bgc.spacetime.time) >= fieldset_starttime + for ctd_bgc in measurements + ] + ): + raise ValueError("BGC CTD deployed before fieldset starts.") + + # depth the bgc ctd will go to. shallowest between bgc ctd max depth and bathymetry. + max_depths = [ + max( + ctd_bgc.max_depth, + fieldset.bathymetry.eval( + z=0, + y=ctd_bgc.spacetime.location.lat, + x=ctd_bgc.spacetime.location.lon, + time=0, + ), + ) + for ctd_bgc in measurements + ] + + # CTD depth can not be too shallow, because kernel would break. + # This shallow is not useful anyway, no need to support. + if not all([max_depth <= -DT * WINCH_SPEED for max_depth in max_depths]): + raise ValueError( + f"BGC CTD max_depth or bathymetry shallower than maximum {-DT * WINCH_SPEED}" + ) + + # define parcel particles + ctd_bgc_particleset = ParticleSet( + fieldset=fieldset, + pclass=_CTD_BGCParticle, + lon=[ctd_bgc.spacetime.location.lon for ctd_bgc in measurements], + lat=[ctd_bgc.spacetime.location.lat for ctd_bgc in measurements], + depth=[ctd_bgc.min_depth for ctd_bgc in measurements], + time=[ctd_bgc.spacetime.time for ctd_bgc in measurements], + max_depth=max_depths, + min_depth=[ctd_bgc.min_depth for ctd_bgc in measurements], + winch_speed=[WINCH_SPEED for _ in measurements], ) - # define parcel particles - ctd_bgc_particleset = ParticleSet( - fieldset=fieldset, - pclass=_CTD_BGCParticle, - lon=[ctd_bgc.spacetime.location.lon for ctd_bgc in ctd_bgcs], - lat=[ctd_bgc.spacetime.location.lat for ctd_bgc in ctd_bgcs], - depth=[ctd_bgc.min_depth for ctd_bgc in ctd_bgcs], - time=[ctd_bgc.spacetime.time for ctd_bgc in ctd_bgcs], - max_depth=max_depths, - min_depth=[ctd_bgc.min_depth for ctd_bgc in ctd_bgcs], - winch_speed=[WINCH_SPEED for _ in ctd_bgcs], - ) - - # define output file for the simulation - out_file = ctd_bgc_particleset.ParticleFile(name=out_path, outputdt=outputdt) - - # execute simulation - ctd_bgc_particleset.execute( - [ - _sample_o2, - _sample_chlorophyll, - _sample_nitrate, - _sample_phosphate, - _sample_ph, - _sample_phytoplankton, - _sample_primary_production, - _ctd_bgc_cast, - ], - endtime=fieldset_endtime, - dt=DT, - verbose_progress=False, - output_file=out_file, - ) - - # there should be no particles left, as they delete themselves when they resurface - if len(ctd_bgc_particleset.particledata) != 0: - raise ValueError( - "Simulation ended before BGC CTD resurfaced. This most likely means the field time dimension did not match the simulation time span." + # define output file for the simulation + out_file = ctd_bgc_particleset.ParticleFile(name=out_path, outputdt=OUTPUT_DT) + + # execute simulation + ctd_bgc_particleset.execute( + [ + _sample_o2, + _sample_chlorophyll, + _sample_nitrate, + _sample_phosphate, + _sample_ph, + _sample_phytoplankton, + _sample_primary_production, + _ctd_bgc_cast, + ], + endtime=fieldset_endtime, + dt=DT, + verbose_progress=self.verbose_progress, + output_file=out_file, ) + + # there should be no particles left, as they delete themselves when they resurface + if len(ctd_bgc_particleset.particledata) != 0: + raise ValueError( + "Simulation ended before BGC CTD resurfaced. This most likely means the field time dimension did not match the simulation time span." + ) diff --git a/src/virtualship/instruments/drifter.py b/src/virtualship/instruments/drifter.py index 5aef240f..e962278d 100644 --- a/src/virtualship/instruments/drifter.py +++ b/src/virtualship/instruments/drifter.py @@ -1,24 +1,34 @@ -"""Drifter instrument.""" - from dataclasses import dataclass -from datetime import datetime, timedelta -from pathlib import Path +from datetime import timedelta +from typing import ClassVar import numpy as np -from parcels import AdvectionRK4, FieldSet, JITParticle, ParticleSet, Variable +from parcels import AdvectionRK4, JITParticle, ParticleSet, Variable + +from virtualship.instruments.base import Instrument +from virtualship.instruments.types import InstrumentType +from virtualship.models.spacetime import Spacetime +from virtualship.utils import register_instrument -from virtualship.models import Spacetime +# ===================================================== +# SECTION: Dataclass +# ===================================================== @dataclass class Drifter: - """Configuration for a single Drifter.""" + """Drifter configuration.""" + name: ClassVar[str] = "Drifter" spacetime: Spacetime depth: float # depth at which it floats and samples lifetime: timedelta | None # if none, lifetime is infinite +# ===================================================== +# SECTION: Particle Class +# ===================================================== + _DrifterParticle = JITParticle.add_variables( [ Variable("temperature", dtype=np.float32, initial=np.nan), @@ -28,6 +38,10 @@ class Drifter: ] ) +# ===================================================== +# SECTION: Kernels +# ===================================================== + def _sample_temperature(particle, fieldset, time): particle.temperature = fieldset.T[time, particle.depth, particle.lat, particle.lon] @@ -40,74 +54,100 @@ def _check_lifetime(particle, fieldset, time): particle.delete() -def simulate_drifters( - fieldset: FieldSet, - out_path: str | Path, - drifters: list[Drifter], - outputdt: timedelta, - dt: timedelta, - endtime: datetime | None = None, -) -> None: - """ - Use Parcels to simulate a set of drifters in a fieldset. - - :param fieldset: The fieldset to simulate the Drifters in. - :param out_path: The path to write the results to. - :param drifters: A list of drifters to simulate. - :param outputdt: Interval which dictates the update frequency of file output during simulation. - :param dt: Dt for integration. - :param endtime: Stop at this time, or if None, continue until the end of the fieldset or until all drifters ended. If this is earlier than the last drifter ended or later than the end of the fieldset, a warning will be printed. - """ - if len(drifters) == 0: - print( - "No drifters provided. Parcels currently crashes when providing an empty particle set, so no drifter simulation will be done and no files will be created." +# ===================================================== +# SECTION: Instrument Class +# ===================================================== + + +@register_instrument(InstrumentType.DRIFTER) +class DrifterInstrument(Instrument): + """Drifter instrument class.""" + + def __init__(self, expedition, from_data): + """Initialize DrifterInstrument.""" + variables = {"U": "uo", "V": "vo", "T": "thetao"} + spacetime_buffer_size = { + "latlon": 6.0, # [degrees] + "time": 21.0, # [days] + } + limit_spec = { + "depth_min": 1.0, # [meters] + "depth_max": 1.0, # [meters] + } + + super().__init__( + expedition, + variables, + add_bathymetry=False, + allow_time_extrapolation=False, + verbose_progress=True, + spacetime_buffer_size=spacetime_buffer_size, + limit_spec=limit_spec, + from_data=from_data, + ) + + def simulate(self, measurements, out_path) -> None: + """Simulate Drifter measurements.""" + OUTPUT_DT = timedelta(hours=5) + DT = timedelta(minutes=5) + ENDTIME = None + + if len(measurements) == 0: + print( + "No drifters provided. Parcels currently crashes when providing an empty particle set, so no drifter simulation will be done and no files will be created." + ) + # TODO when Parcels supports it this check can be removed. + return + + fieldset = self.load_input_data() + + # define parcel particles + drifter_particleset = ParticleSet( + fieldset=fieldset, + pclass=_DrifterParticle, + lat=[drifter.spacetime.location.lat for drifter in measurements], + lon=[drifter.spacetime.location.lon for drifter in measurements], + depth=[drifter.depth for drifter in measurements], + time=[drifter.spacetime.time for drifter in measurements], + has_lifetime=[ + 1 if drifter.lifetime is not None else 0 for drifter in measurements + ], + lifetime=[ + 0 if drifter.lifetime is None else drifter.lifetime.total_seconds() + for drifter in measurements + ], ) - # TODO when Parcels supports it this check can be removed. - return - - # define parcel particles - drifter_particleset = ParticleSet( - fieldset=fieldset, - pclass=_DrifterParticle, - lat=[drifter.spacetime.location.lat for drifter in drifters], - lon=[drifter.spacetime.location.lon for drifter in drifters], - depth=[drifter.depth for drifter in drifters], - time=[drifter.spacetime.time for drifter in drifters], - has_lifetime=[1 if drifter.lifetime is not None else 0 for drifter in drifters], - lifetime=[ - 0 if drifter.lifetime is None else drifter.lifetime.total_seconds() - for drifter in drifters - ], - ) - - # define output file for the simulation - out_file = drifter_particleset.ParticleFile( - name=out_path, outputdt=outputdt, chunks=[len(drifter_particleset), 100] - ) - - # get earliest between fieldset end time and provide end time - fieldset_endtime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[-1]) - if endtime is None: - actual_endtime = fieldset_endtime - elif endtime > fieldset_endtime: - print("WARN: Requested end time later than fieldset end time.") - actual_endtime = fieldset_endtime - else: - actual_endtime = np.timedelta64(endtime) - - # execute simulation - drifter_particleset.execute( - [AdvectionRK4, _sample_temperature, _check_lifetime], - endtime=actual_endtime, - dt=dt, - output_file=out_file, - verbose_progress=True, - ) - - # if there are more particles left than the number of drifters with an indefinite endtime, warn the user - if len(drifter_particleset.particledata) > len( - [d for d in drifters if d.lifetime is None] - ): - print( - "WARN: Some drifters had a life time beyond the end time of the fieldset or the requested end time." + + # define output file for the simulation + out_file = drifter_particleset.ParticleFile( + name=out_path, + outputdt=OUTPUT_DT, + chunks=[len(drifter_particleset), 100], ) + + # get earliest between fieldset end time and prescribed end time + fieldset_endtime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[-1]) + if ENDTIME is None: + actual_endtime = fieldset_endtime + elif ENDTIME > fieldset_endtime: + print("WARN: Requested end time later than fieldset end time.") + actual_endtime = fieldset_endtime + else: + actual_endtime = np.timedelta64(ENDTIME) + + # execute simulation + drifter_particleset.execute( + [AdvectionRK4, _sample_temperature, _check_lifetime], + endtime=actual_endtime, + dt=DT, + output_file=out_file, + verbose_progress=self.verbose_progress, + ) + + # if there are more particles left than the number of drifters with an indefinite endtime, warn the user + if len(drifter_particleset.particledata) > len( + [d for d in measurements if d.lifetime is None] + ): + print( + "WARN: Some drifters had a life time beyond the end time of the fieldset or the requested end time." + ) diff --git a/src/virtualship/instruments/ship_underwater_st.py b/src/virtualship/instruments/ship_underwater_st.py index 7b08ad4b..088a439f 100644 --- a/src/virtualship/instruments/ship_underwater_st.py +++ b/src/virtualship/instruments/ship_underwater_st.py @@ -1,14 +1,29 @@ -"""Ship salinity and temperature.""" - -from pathlib import Path +from dataclasses import dataclass +from typing import ClassVar import numpy as np -from parcels import FieldSet, ParticleSet, ScipyParticle, Variable +from parcels import ParticleSet, ScipyParticle, Variable + +from virtualship.instruments.base import Instrument +from virtualship.instruments.types import InstrumentType +from virtualship.utils import add_dummy_UV, register_instrument + +# ===================================================== +# SECTION: Dataclass +# ===================================================== + + +@dataclass +class Underwater_ST: + """Underwater_ST configuration.""" + + name: ClassVar[str] = "Underwater_ST" -from virtualship.models import Spacetime -# we specifically use ScipyParticle because we have many small calls to execute -# there is some overhead with JITParticle and this ends up being significantly faster +# ===================================================== +# SECTION: Particle Class +# ===================================================== + _ShipSTParticle = ScipyParticle.add_variables( [ Variable("S", dtype=np.float32, initial=np.nan), @@ -16,6 +31,10 @@ ] ) +# ===================================================== +# SECTION: Kernels +# ===================================================== + # define function sampling Salinity def _sample_salinity(particle, fieldset, time): @@ -27,50 +46,63 @@ def _sample_temperature(particle, fieldset, time): particle.T = fieldset.T[time, particle.depth, particle.lat, particle.lon] -def simulate_ship_underwater_st( - fieldset: FieldSet, - out_path: str | Path, - depth: float, - sample_points: list[Spacetime], -) -> None: - """ - Use Parcels to simulate underway data, measuring salinity and temperature at the given depth along the ship track in a fieldset. - - :param fieldset: The fieldset to simulate the sampling in. - :param out_path: The path to write the results to. - :param depth: The depth at which to measure. 0 is water surface, negative is into the water. - :param sample_points: The places and times to sample at. - """ - sample_points.sort(key=lambda p: p.time) - - particleset = ParticleSet.from_list( - fieldset=fieldset, - pclass=_ShipSTParticle, - lon=0.0, # initial lat/lon are irrelevant and will be overruled later - lat=0.0, - depth=depth, - time=0, # same for time - ) - - # define output file for the simulation - # outputdt set to infinie as we want to just want to write at the end of every call to 'execute' - out_file = particleset.ParticleFile(name=out_path, outputdt=np.inf) - - # iterate over each point, manually set lat lon time, then - # execute the particle set for one step, performing one set of measurement - for point in sample_points: - particleset.lon_nextloop[:] = point.location.lon - particleset.lat_nextloop[:] = point.location.lat - particleset.time_nextloop[:] = fieldset.time_origin.reltime( - np.datetime64(point.time) - ) +# ===================================================== +# SECTION: Instrument Class +# ===================================================== + + +@register_instrument(InstrumentType.UNDERWATER_ST) +class Underwater_STInstrument(Instrument): + """Underwater_ST instrument class.""" - # perform one step using the particleset - # dt and runtime are set so exactly one step is made. - particleset.execute( - [_sample_salinity, _sample_temperature], - dt=1, - runtime=1, + def __init__(self, expedition, from_data): + """Initialize Underwater_STInstrument.""" + variables = {"S": "so", "T": "thetao"} + + super().__init__( + expedition, + variables, + add_bathymetry=False, + allow_time_extrapolation=True, verbose_progress=False, - output_file=out_file, + spacetime_buffer_size=None, + limit_spec=None, + from_data=from_data, + ) + + def simulate(self, measurements, out_path) -> None: + """Simulate underway salinity and temperature measurements.""" + DEPTH = -2.0 + + measurements.sort(key=lambda p: p.time) + + fieldset = self.load_input_data() + + # add dummy U + add_dummy_UV(fieldset) # TODO: parcels v3 bodge; remove when parcels v4 is used + + particleset = ParticleSet.from_list( + fieldset=fieldset, + pclass=_ShipSTParticle, + lon=0.0, + lat=0.0, + depth=DEPTH, + time=0, ) + + out_file = particleset.ParticleFile(name=out_path, outputdt=np.inf) + + for point in measurements: + particleset.lon_nextloop[:] = point.location.lon + particleset.lat_nextloop[:] = point.location.lat + particleset.time_nextloop[:] = fieldset.time_origin.reltime( + np.datetime64(point.time) + ) + + particleset.execute( + [_sample_salinity, _sample_temperature], + dt=1, + runtime=1, + verbose_progress=self.verbose_progress, + output_file=out_file, + ) diff --git a/src/virtualship/instruments/types.py b/src/virtualship/instruments/types.py new file mode 100644 index 00000000..9ae221e9 --- /dev/null +++ b/src/virtualship/instruments/types.py @@ -0,0 +1,18 @@ +from enum import Enum + + +class InstrumentType(Enum): + """Types of the instruments.""" + + CTD = "CTD" + CTD_BGC = "CTD_BGC" + DRIFTER = "DRIFTER" + ARGO_FLOAT = "ARGO_FLOAT" + XBT = "XBT" + ADCP = "ADCP" + UNDERWATER_ST = "UNDERWATER_ST" + + @property + def is_underway(self) -> bool: + """Return True if instrument is an underway instrument (ADCP, UNDERWATER_ST).""" + return self in {InstrumentType.ADCP, InstrumentType.UNDERWATER_ST} diff --git a/src/virtualship/instruments/xbt.py b/src/virtualship/instruments/xbt.py index 6d75be8c..f0f5d130 100644 --- a/src/virtualship/instruments/xbt.py +++ b/src/virtualship/instruments/xbt.py @@ -1,19 +1,25 @@ -"""XBT instrument.""" - from dataclasses import dataclass from datetime import timedelta -from pathlib import Path +from typing import ClassVar import numpy as np -from parcels import FieldSet, JITParticle, ParticleSet, Variable +from parcels import JITParticle, ParticleSet, Variable + +from virtualship.instruments.base import Instrument +from virtualship.instruments.types import InstrumentType +from virtualship.models.spacetime import Spacetime +from virtualship.utils import add_dummy_UV, register_instrument -from virtualship.models import Spacetime +# ===================================================== +# SECTION: Dataclass +# ===================================================== @dataclass class XBT: - """Configuration for a single XBT.""" + """XBT configuration.""" + name: ClassVar[str] = "XBT" spacetime: Spacetime min_depth: float max_depth: float @@ -21,6 +27,10 @@ class XBT: deceleration_coefficient: float +# ===================================================== +# SECTION: Particle Class +# ===================================================== + _XBTParticle = JITParticle.add_variables( [ Variable("temperature", dtype=np.float32, initial=np.nan), @@ -31,6 +41,10 @@ class XBT: ] ) +# ===================================================== +# SECTION: Kernels +# ===================================================== + def _sample_temperature(particle, fieldset, time): particle.temperature = fieldset.T[time, particle.depth, particle.lat, particle.lon] @@ -54,88 +68,111 @@ def _xbt_cast(particle, fieldset, time): particle_ddepth = particle.max_depth - particle.depth -def simulate_xbt( - fieldset: FieldSet, - out_path: str | Path, - xbts: list[XBT], - outputdt: timedelta, -) -> None: - """ - Use Parcels to simulate a set of XBTs in a fieldset. - - :param fieldset: The fieldset to simulate the XBTs in. - :param out_path: The path to write the results to. - :param xbts: A list of XBTs to simulate. - :param outputdt: Interval which dictates the update frequency of file output during simulation - :raises ValueError: Whenever provided XBTs, fieldset, are not compatible with this function. - """ - DT = 10.0 # dt of XBT simulation integrator - - if len(xbts) == 0: - print( - "No XBTs provided. Parcels currently crashes when providing an empty particle set, so no XBT simulation will be done and no files will be created." - ) - # TODO when Parcels supports it this check can be removed. - return - - fieldset_starttime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[0]) - fieldset_endtime = fieldset.time_origin.fulltime(fieldset.U.grid.time_full[-1]) - - # deploy time for all xbts should be later than fieldset start time - if not all( - [np.datetime64(xbt.spacetime.time) >= fieldset_starttime for xbt in xbts] - ): - raise ValueError("XBT deployed before fieldset starts.") - - # depth the xbt will go to. shallowest between xbt max depth and bathymetry. - max_depths = [ - max( - xbt.max_depth, - fieldset.bathymetry.eval( - z=0, y=xbt.spacetime.location.lat, x=xbt.spacetime.location.lon, time=0 - ), +# ===================================================== +# SECTION: Instrument Class +# ===================================================== + + +@register_instrument(InstrumentType.XBT) +class XBTInstrument(Instrument): + """XBT instrument class.""" + + def __init__(self, expedition, from_data): + """Initialize XBTInstrument.""" + variables = {"T": "thetao"} + super().__init__( + expedition, + variables, + add_bathymetry=True, + allow_time_extrapolation=True, + verbose_progress=False, + spacetime_buffer_size=None, + limit_spec=None, + from_data=from_data, ) - for xbt in xbts - ] - # initial fall speeds - initial_fall_speeds = [xbt.fall_speed for xbt in xbts] + def simulate(self, measurements, out_path) -> None: + """Simulate XBT measurements.""" + DT = 10.0 # dt of XBT simulation integrator + OUTPUT_DT = timedelta(seconds=10) - # XBT depth can not be too shallow, because kernel would break. - # This shallow is not useful anyway, no need to support. - for max_depth, fall_speed in zip(max_depths, initial_fall_speeds, strict=False): - if not max_depth <= -DT * fall_speed: - raise ValueError( - f"XBT max_depth or bathymetry shallower than maximum {-DT * fall_speed}" + if len(measurements) == 0: + print( + "No XBTs provided. Parcels currently crashes when providing an empty particle set, so no XBT simulation will be done and no files will be created." ) + # TODO when Parcels supports it this check can be removed. + return - # define xbt particles - xbt_particleset = ParticleSet( - fieldset=fieldset, - pclass=_XBTParticle, - lon=[xbt.spacetime.location.lon for xbt in xbts], - lat=[xbt.spacetime.location.lat for xbt in xbts], - depth=[xbt.min_depth for xbt in xbts], - time=[xbt.spacetime.time for xbt in xbts], - max_depth=max_depths, - min_depth=[xbt.min_depth for xbt in xbts], - fall_speed=[xbt.fall_speed for xbt in xbts], - ) + fieldset = self.load_input_data() - # define output file for the simulation - out_file = xbt_particleset.ParticleFile(name=out_path, outputdt=outputdt) + # add dummy U + add_dummy_UV(fieldset) # TODO: parcels v3 bodge; remove when parcels v4 is used - # execute simulation - xbt_particleset.execute( - [_sample_temperature, _xbt_cast], - endtime=fieldset_endtime, - dt=DT, - verbose_progress=False, - output_file=out_file, - ) + fieldset_starttime = fieldset.T.grid.time_origin.fulltime( + fieldset.T.grid.time_full[0] + ) + fieldset_endtime = fieldset.T.grid.time_origin.fulltime( + fieldset.T.grid.time_full[-1] + ) + + # deploy time for all xbts should be later than fieldset start time + if not all( + [ + np.datetime64(xbt.spacetime.time) >= fieldset_starttime + for xbt in measurements + ] + ): + raise ValueError("XBT deployed before fieldset starts.") + + # depth the xbt will go to. shallowest between xbt max depth and bathymetry. + max_depths = [ + max( + xbt.max_depth, + fieldset.bathymetry.eval( + z=0, + y=xbt.spacetime.location.lat, + x=xbt.spacetime.location.lon, + time=0, + ), + ) + for xbt in measurements + ] + + # initial fall speeds + initial_fall_speeds = [xbt.fall_speed for xbt in measurements] + + # XBT depth can not be too shallow, because kernel would break. + for max_depth, fall_speed in zip(max_depths, initial_fall_speeds, strict=False): + if not max_depth <= -DT * fall_speed: + raise ValueError( + f"XBT max_depth or bathymetry shallower than minimum {-DT * fall_speed}. It is likely the XBT cannot be deployed in this area, which is too shallow." + ) + + # define xbt particles + xbt_particleset = ParticleSet( + fieldset=fieldset, + pclass=_XBTParticle, + lon=[xbt.spacetime.location.lon for xbt in measurements], + lat=[xbt.spacetime.location.lat for xbt in measurements], + depth=[xbt.min_depth for xbt in measurements], + time=[xbt.spacetime.time for xbt in measurements], + max_depth=max_depths, + min_depth=[xbt.min_depth for xbt in measurements], + fall_speed=[xbt.fall_speed for xbt in measurements], + ) + + out_file = xbt_particleset.ParticleFile(name=out_path, outputdt=OUTPUT_DT) - # there should be no particles left, as they delete themselves when they finish profiling - if len(xbt_particleset.particledata) != 0: - raise ValueError( - "Simulation ended before XBT finished profiling. This most likely means the field time dimension did not match the simulation time span." + xbt_particleset.execute( + [_sample_temperature, _xbt_cast], + endtime=fieldset_endtime, + dt=DT, + verbose_progress=self.verbose_progress, + output_file=out_file, ) + + # there should be no particles left, as they delete themselves when they finish profiling + if len(xbt_particleset.particledata) != 0: + raise ValueError( + "Simulation ended before XBT finished profiling. This most likely means the field time dimension did not match the simulation time span." + ) diff --git a/src/virtualship/models/__init__.py b/src/virtualship/models/__init__.py index 48106056..5eaabb85 100644 --- a/src/virtualship/models/__init__.py +++ b/src/virtualship/models/__init__.py @@ -1,18 +1,20 @@ """Pydantic models and data classes used to configure virtualship (i.e., in the configuration files or settings).""" -from .location import Location -from .schedule import Schedule, Waypoint -from .ship_config import ( +from .expedition import ( ADCPConfig, ArgoFloatConfig, CTD_BGCConfig, CTDConfig, DrifterConfig, - InstrumentType, + Expedition, + InstrumentsConfig, + Schedule, ShipConfig, ShipUnderwaterSTConfig, + Waypoint, XBTConfig, ) +from .location import Location from .space_time_region import ( SpaceTimeRegion, SpatialRange, @@ -25,8 +27,8 @@ __all__ = [ # noqa: RUF022 "Location", "Schedule", + "ShipConfig", "Waypoint", - "InstrumentType", "ArgoFloatConfig", "ADCPConfig", "CTDConfig", @@ -34,9 +36,10 @@ "ShipUnderwaterSTConfig", "DrifterConfig", "XBTConfig", - "ShipConfig", "SpatialRange", "TimeRange", "SpaceTimeRegion", "Spacetime", + "Expedition", + "InstrumentsConfig", ] diff --git a/src/virtualship/expedition/checkpoint.py b/src/virtualship/models/checkpoint.py similarity index 95% rename from src/virtualship/expedition/checkpoint.py rename to src/virtualship/models/checkpoint.py index 6daf1a9b..98fe1ae0 100644 --- a/src/virtualship/expedition/checkpoint.py +++ b/src/virtualship/models/checkpoint.py @@ -8,7 +8,8 @@ import yaml from virtualship.errors import CheckpointError -from virtualship.models import InstrumentType, Schedule +from virtualship.instruments.types import InstrumentType +from virtualship.models import Schedule class _YamlDumper(yaml.SafeDumper): diff --git a/src/virtualship/models/expedition.py b/src/virtualship/models/expedition.py new file mode 100644 index 00000000..4847b10c --- /dev/null +++ b/src/virtualship/models/expedition.py @@ -0,0 +1,423 @@ +from __future__ import annotations + +import itertools +from datetime import datetime, timedelta +from pathlib import Path + +import numpy as np +import pydantic +import pyproj +import yaml + +from virtualship.errors import InstrumentsConfigError, ScheduleError +from virtualship.instruments.types import InstrumentType +from virtualship.utils import _get_bathy_data, _validate_numeric_mins_to_timedelta + +from .location import Location +from .space_time_region import SpaceTimeRegion + +projection: pyproj.Geod = pyproj.Geod(ellps="WGS84") + + +class Expedition(pydantic.BaseModel): + """Expedition class, including schedule and ship config.""" + + schedule: Schedule + instruments_config: InstrumentsConfig + ship_config: ShipConfig + + model_config = pydantic.ConfigDict(extra="forbid") + + def to_yaml(self, file_path: str) -> None: + """Write exepedition object to yaml file.""" + with open(file_path, "w") as file: + yaml.dump(self.model_dump(by_alias=True), file) + + @classmethod + def from_yaml(cls, file_path: str) -> Expedition: + """Load config from yaml file.""" + with open(file_path) as file: + data = yaml.safe_load(file) + return Expedition(**data) + + def get_instruments(self) -> set[InstrumentType]: + """Return a set of unique InstrumentType enums used in the expedition.""" + instruments_in_expedition = [] + # from waypoints + for waypoint in self.schedule.waypoints: + if waypoint.instrument: + for instrument in waypoint.instrument: + if instrument: + instruments_in_expedition.append(instrument) + + # check for underway instruments and add if present in expeditions + try: + if self.instruments_config.adcp_config is not None: + instruments_in_expedition.append(InstrumentType.ADCP) + if self.instruments_config.ship_underwater_st_config is not None: + instruments_in_expedition.append(InstrumentType.UNDERWATER_ST) + return sorted(set(instruments_in_expedition), key=lambda x: x.name) + except Exception as e: + raise InstrumentsConfigError( + "Underway instrument config attribute(s) are missing from YAML. Must be Config object or None." + ) from e + + +class ShipConfig(pydantic.BaseModel): + """Configuration of the ship.""" + + ship_speed_knots: float = pydantic.Field(gt=0.0) + + # TODO: room here for adding more ship config options in future PRs (e.g. max_days_at_sea)... + + model_config = pydantic.ConfigDict(extra="forbid") + + +class Schedule(pydantic.BaseModel): + """Schedule of the virtual ship.""" + + waypoints: list[Waypoint] + space_time_region: SpaceTimeRegion | None = None + + model_config = pydantic.ConfigDict(extra="forbid") + + def verify( + self, + ship_speed: float, + ignore_land_test: bool = False, + *, + check_space_time_region: bool = False, + from_data: Path | None = None, + ) -> None: + """ + Verify the feasibility and correctness of the schedule's waypoints. + + This method checks various conditions to ensure the schedule is valid: + 1. At least one waypoint is provided. + 2. The first waypoint has a specified time. + 3. Waypoint times are in ascending order. + 4. All waypoints are in water (not on land). + 5. The ship can arrive on time at each waypoint given its speed. + """ + print("\nVerifying route... ") + + if check_space_time_region and self.space_time_region is None: + raise ScheduleError( + "space_time_region not found in schedule, please define it to proceed." + ) + + if len(self.waypoints) == 0: + raise ScheduleError("At least one waypoint must be provided.") + + # check first waypoint has a time + if self.waypoints[0].time is None: + raise ScheduleError("First waypoint must have a specified time.") + + # check waypoint times are in ascending order + timed_waypoints = [wp for wp in self.waypoints if wp.time is not None] + checks = [ + next.time >= cur.time for cur, next in itertools.pairwise(timed_waypoints) + ] + if not all(checks): + invalid_i = [i for i, c in enumerate(checks) if c] + raise ScheduleError( + f"Waypoint(s) {', '.join(f'#{i + 1}' for i in invalid_i)}: each waypoint should be timed after all previous waypoints", + ) + + # check if all waypoints are in water using bathymetry data + land_waypoints = [] + if not ignore_land_test: + try: + bathymetry_field = _get_bathy_data( + self.space_time_region, + latlon_buffer=None, + from_data=from_data, + ).bathymetry + except Exception as e: + raise ScheduleError( + f"Problem loading bathymetry data (used to verify waypoints are in water) directly via copernicusmarine. \n\n original message: {e}" + ) from e + + for wp_i, wp in enumerate(self.waypoints): + try: + value = bathymetry_field.eval( + 0, # time + 0, # depth (surface) + wp.location.lat, + wp.location.lon, + ) + if value == 0.0 or (isinstance(value, float) and np.isnan(value)): + land_waypoints.append((wp_i, wp)) + except Exception as e: + raise ScheduleError( + f"Waypoint #{wp_i + 1} at location {wp.location} could not be evaluated against bathymetry data. There may be a problem with the waypoint location being outside of the space_time_region or with the bathymetry data itself.\n\n Original error: {e}" + ) from e + + if len(land_waypoints) > 0: + raise ScheduleError( + f"The following waypoint(s) throw(s) error(s): {['#' + str(wp_i + 1) + ' ' + str(wp) for (wp_i, wp) in land_waypoints]}\n\nINFO: They are likely on land (bathymetry data cannot be interpolated to their location(s)).\n" + ) + + # check that ship will arrive on time at each waypoint (in case no unexpected event happen) + time = self.waypoints[0].time + for wp_i, (wp, wp_next) in enumerate( + zip(self.waypoints, self.waypoints[1:], strict=False) + ): + if wp.instrument is InstrumentType.CTD: + time += timedelta(minutes=20) + + geodinv: tuple[float, float, float] = projection.inv( + wp.location.lon, + wp.location.lat, + wp_next.location.lon, + wp_next.location.lat, + ) + distance = geodinv[2] + + time_to_reach = timedelta(seconds=distance / ship_speed * 3600 / 1852) + arrival_time = time + time_to_reach + + if wp_next.time is None: + time = arrival_time + elif arrival_time > wp_next.time: + raise ScheduleError( + f"Waypoint planning is not valid: would arrive too late at waypoint number {wp_i + 2}. " + f"location: {wp_next.location} time: {wp_next.time} instrument: {wp_next.instrument}" + ) + else: + time = wp_next.time + + print("... All good to go!") + + +class Waypoint(pydantic.BaseModel): + """A Waypoint to sail to with an optional time and an optional instrument.""" + + location: Location + time: datetime | None = None + instrument: InstrumentType | list[InstrumentType] | None = None + + @pydantic.field_serializer("instrument") + def serialize_instrument(self, instrument): + """Ensure InstrumentType is serialized as a string (or list of strings).""" + if isinstance(instrument, list): + return [inst.value for inst in instrument] + return instrument.value if instrument else None + + +class ArgoFloatConfig(pydantic.BaseModel): + """Configuration for argos floats.""" + + min_depth_meter: float = pydantic.Field(le=0.0) + max_depth_meter: float = pydantic.Field(le=0.0) + drift_depth_meter: float = pydantic.Field(le=0.0) + vertical_speed_meter_per_second: float = pydantic.Field(lt=0.0) + cycle_days: float = pydantic.Field(gt=0.0) + drift_days: float = pydantic.Field(gt=0.0) + + +class ADCPConfig(pydantic.BaseModel): + """Configuration for ADCP instrument.""" + + max_depth_meter: float = pydantic.Field(le=0.0) + num_bins: int = pydantic.Field(gt=0.0) + period: timedelta = pydantic.Field( + serialization_alias="period_minutes", + validation_alias="period_minutes", + gt=timedelta(), + ) + + model_config = pydantic.ConfigDict(populate_by_name=True) + + @pydantic.field_serializer("period") + def _serialize_period(self, value: timedelta, _info): + return value.total_seconds() / 60.0 + + @pydantic.field_validator("period", mode="before") + def _validate_period(cls, value: int | float | timedelta) -> timedelta: + return _validate_numeric_mins_to_timedelta(value) + + +class CTDConfig(pydantic.BaseModel): + """Configuration for CTD instrument.""" + + stationkeeping_time: timedelta = pydantic.Field( + serialization_alias="stationkeeping_time_minutes", + validation_alias="stationkeeping_time_minutes", + gt=timedelta(), + ) + min_depth_meter: float = pydantic.Field(le=0.0) + max_depth_meter: float = pydantic.Field(le=0.0) + + model_config = pydantic.ConfigDict(populate_by_name=True) + + @pydantic.field_serializer("stationkeeping_time") + def _serialize_stationkeeping_time(self, value: timedelta, _info): + return value.total_seconds() / 60.0 + + @pydantic.field_validator("stationkeeping_time", mode="before") + def _validate_stationkeeping_time(cls, value: int | float | timedelta) -> timedelta: + return _validate_numeric_mins_to_timedelta(value) + + +class CTD_BGCConfig(pydantic.BaseModel): + """Configuration for CTD_BGC instrument.""" + + stationkeeping_time: timedelta = pydantic.Field( + serialization_alias="stationkeeping_time_minutes", + validation_alias="stationkeeping_time_minutes", + gt=timedelta(), + ) + min_depth_meter: float = pydantic.Field(le=0.0) + max_depth_meter: float = pydantic.Field(le=0.0) + + model_config = pydantic.ConfigDict(populate_by_name=True) + + @pydantic.field_serializer("stationkeeping_time") + def _serialize_stationkeeping_time(self, value: timedelta, _info): + return value.total_seconds() / 60.0 + + @pydantic.field_validator("stationkeeping_time", mode="before") + def _validate_stationkeeping_time(cls, value: int | float | timedelta) -> timedelta: + return _validate_numeric_mins_to_timedelta(value) + + +class ShipUnderwaterSTConfig(pydantic.BaseModel): + """Configuration for underwater ST.""" + + period: timedelta = pydantic.Field( + serialization_alias="period_minutes", + validation_alias="period_minutes", + gt=timedelta(), + ) + + model_config = pydantic.ConfigDict(populate_by_name=True) + + @pydantic.field_serializer("period") + def _serialize_period(self, value: timedelta, _info): + return value.total_seconds() / 60.0 + + @pydantic.field_validator("period", mode="before") + def _validate_period(cls, value: int | float | timedelta) -> timedelta: + return _validate_numeric_mins_to_timedelta(value) + + +class DrifterConfig(pydantic.BaseModel): + """Configuration for drifters.""" + + depth_meter: float = pydantic.Field(le=0.0) + lifetime: timedelta = pydantic.Field( + serialization_alias="lifetime_minutes", + validation_alias="lifetime_minutes", + gt=timedelta(), + ) + + model_config = pydantic.ConfigDict(populate_by_name=True) + + @pydantic.field_serializer("lifetime") + def _serialize_lifetime(self, value: timedelta, _info): + return value.total_seconds() / 60.0 + + @pydantic.field_validator("lifetime", mode="before") + def _validate_lifetime(cls, value: int | float | timedelta) -> timedelta: + return _validate_numeric_mins_to_timedelta(value) + + +class XBTConfig(pydantic.BaseModel): + """Configuration for xbt instrument.""" + + min_depth_meter: float = pydantic.Field(le=0.0) + max_depth_meter: float = pydantic.Field(le=0.0) + fall_speed_meter_per_second: float = pydantic.Field(gt=0.0) + deceleration_coefficient: float = pydantic.Field(gt=0.0) + + +class InstrumentsConfig(pydantic.BaseModel): + """Configuration of instruments.""" + + argo_float_config: ArgoFloatConfig | None = None + """ + Argo float configuration. + + If None, no argo floats can be deployed. + """ + + adcp_config: ADCPConfig | None = None + """ + ADCP configuration. + + If None, no ADCP measurements will be performed. + """ + + ctd_config: CTDConfig | None = None + """ + CTD configuration. + + If None, no CTDs can be cast. + """ + + ctd_bgc_config: CTD_BGCConfig | None = None + """ + CTD_BGC configuration. + + If None, no BGC CTDs can be cast. + """ + + ship_underwater_st_config: ShipUnderwaterSTConfig | None = None + """ + Ship underwater salinity temperature measurementconfiguration. + + If None, no ST measurements will be performed. + """ + + drifter_config: DrifterConfig | None = None + """ + Drifter configuration. + + If None, no drifters can be deployed. + """ + + xbt_config: XBTConfig | None = None + """ + XBT configuration. + + If None, no XBTs can be cast. + """ + + model_config = pydantic.ConfigDict(extra="forbid") + + def verify(self, expedition: Expedition) -> None: + """ + Verify instrument configurations against the schedule. + + Removes instrument configs not present in the schedule and checks that all scheduled instruments are configured. + Raises ConfigError if any scheduled instrument is missing a config. + """ + instruments_in_expedition = expedition.get_instruments() + instrument_config_map = { + InstrumentType.ARGO_FLOAT: "argo_float_config", + InstrumentType.DRIFTER: "drifter_config", + InstrumentType.XBT: "xbt_config", + InstrumentType.CTD: "ctd_config", + InstrumentType.CTD_BGC: "ctd_bgc_config", + InstrumentType.ADCP: "adcp_config", + InstrumentType.UNDERWATER_ST: "ship_underwater_st_config", + } + # Remove configs for unused instruments + for inst_type, config_attr in instrument_config_map.items(): + if ( + hasattr(self, config_attr) + and inst_type not in instruments_in_expedition + ): + setattr(self, config_attr, None) + # Check all scheduled instruments are configured + for inst_type in instruments_in_expedition: + config_attr = instrument_config_map.get(inst_type) + if ( + not config_attr + or not hasattr(self, config_attr) + or getattr(self, config_attr) is None + ): + raise InstrumentsConfigError( + f"Expedition includes instrument '{inst_type.value}', but instruments_config does not provide configuration for it." + ) diff --git a/src/virtualship/models/schedule.py b/src/virtualship/models/schedule.py deleted file mode 100644 index 3de44f09..00000000 --- a/src/virtualship/models/schedule.py +++ /dev/null @@ -1,236 +0,0 @@ -"""Schedule class.""" - -from __future__ import annotations - -import itertools -from datetime import datetime, timedelta -from pathlib import Path -from typing import TYPE_CHECKING - -import pydantic -import pyproj -import yaml - -from virtualship.errors import ScheduleError - -from .location import Location -from .ship_config import InstrumentType -from .space_time_region import SpaceTimeRegion - -if TYPE_CHECKING: - from parcels import FieldSet - - from virtualship.expedition.input_data import InputData - -projection: pyproj.Geod = pyproj.Geod(ellps="WGS84") - - -class Waypoint(pydantic.BaseModel): - """A Waypoint to sail to with an optional time and an optional instrument.""" - - location: Location - time: datetime | None = None - instrument: InstrumentType | list[InstrumentType] | None = None - - @pydantic.field_serializer("instrument") - def serialize_instrument(self, instrument): - """Ensure InstrumentType is serialized as a string (or list of strings).""" - if isinstance(instrument, list): - return [inst.value for inst in instrument] - return instrument.value if instrument else None - - -class Schedule(pydantic.BaseModel): - """Schedule of the virtual ship.""" - - waypoints: list[Waypoint] - space_time_region: SpaceTimeRegion | None = None - - model_config = pydantic.ConfigDict(extra="forbid") - - def to_yaml(self, file_path: str | Path) -> None: - """ - Write schedule to yaml file. - - :param file_path: Path to the file to write to. - """ - with open(file_path, "w") as file: - yaml.dump( - self.model_dump( - by_alias=True, - ), - file, - ) - - @classmethod - def from_yaml(cls, file_path: str | Path) -> Schedule: - """ - Load schedule from yaml file. - - :param file_path: Path to the file to load from. - :returns: The schedule. - """ - with open(file_path) as file: - data = yaml.safe_load(file) - return Schedule(**data) - - def get_instruments(self) -> set[InstrumentType]: - """ - Retrieve a set of unique instruments used in the schedule. - - This method iterates through all waypoints in the schedule and collects - the instruments associated with each waypoint. It returns a set of unique - instruments, either as objects or as names. - - :raises CheckpointError: If the past waypoints in the given schedule - have been changed compared to the checkpoint. - :return: set: A set of unique instruments used in the schedule. - - """ - instruments_in_schedule = [] - for waypoint in self.waypoints: - if waypoint.instrument: - for instrument in waypoint.instrument: - if instrument: - instruments_in_schedule.append(instrument) - return set(instruments_in_schedule) - - def verify( - self, - ship_speed: float, - input_data: InputData | None, - *, - check_space_time_region: bool = False, - ignore_missing_fieldsets: bool = False, - ) -> None: - """ - Verify the feasibility and correctness of the schedule's waypoints. - - This method checks various conditions to ensure the schedule is valid: - 1. At least one waypoint is provided. - 2. The first waypoint has a specified time. - 3. Waypoint times are in ascending order. - 4. All waypoints are in water (not on land). - 5. The ship can arrive on time at each waypoint given its speed. - - :param ship_speed: The ship's speed in knots. - :param input_data: An InputData object containing fieldsets used to check if waypoints are on water. - :param check_space_time_region: whether to check for missing space_time_region. - :param ignore_missing_fieldsets: whether to ignore warning for missing field sets. - :raises PlanningError: If any of the verification checks fail, indicating infeasible or incorrect waypoints. - :raises NotImplementedError: If an instrument in the schedule is not implemented. - :return: None. The method doesn't return a value but raises exceptions if verification fails. - """ - print("\nVerifying route... ") - - if check_space_time_region and self.space_time_region is None: - raise ScheduleError( - "space_time_region not found in schedule, please define it to fetch the data." - ) - - if len(self.waypoints) == 0: - raise ScheduleError("At least one waypoint must be provided.") - - # check first waypoint has a time - if self.waypoints[0].time is None: - raise ScheduleError("First waypoint must have a specified time.") - - # check waypoint times are in ascending order - timed_waypoints = [wp for wp in self.waypoints if wp.time is not None] - checks = [ - next.time >= cur.time for cur, next in itertools.pairwise(timed_waypoints) - ] - if not all(checks): - invalid_i = [i for i, c in enumerate(checks) if c] - raise ScheduleError( - f"Waypoint(s) {', '.join(f'#{i + 1}' for i in invalid_i)}: each waypoint should be timed after all previous waypoints", - ) - - # check if all waypoints are in water - # this is done by picking an arbitrary provided fieldset and checking if UV is not zero - - # get all available fieldsets - available_fieldsets = [] - if input_data is not None: - fieldsets = [ - input_data.adcp_fieldset, - input_data.argo_float_fieldset, - input_data.ctd_fieldset, - input_data.drifter_fieldset, - input_data.ship_underwater_st_fieldset, - ] - for fs in fieldsets: - if fs is not None: - available_fieldsets.append(fs) - - # check if there are any fieldsets, else it's an error - if len(available_fieldsets) == 0: - if not ignore_missing_fieldsets: - print( - "Cannot verify because no fieldsets have been loaded. This is probably " - "because you are not using any instruments in your schedule. This is not a problem, " - "but carefully check your waypoint locations manually." - ) - - else: - # pick any - fieldset = available_fieldsets[0] - # get waypoints with 0 UV - land_waypoints = [ - (wp_i, wp) - for wp_i, wp in enumerate(self.waypoints) - if _is_on_land_zero_uv(fieldset, wp) - ] - # raise an error if there are any - if len(land_waypoints) > 0: - raise ScheduleError( - f"The following waypoints are on land: {['#' + str(wp_i) + ' ' + str(wp) for (wp_i, wp) in land_waypoints]}" - ) - - # check that ship will arrive on time at each waypoint (in case no unexpected event happen) - time = self.waypoints[0].time - for wp_i, (wp, wp_next) in enumerate( - zip(self.waypoints, self.waypoints[1:], strict=False) - ): - if wp.instrument is InstrumentType.CTD: - time += timedelta(minutes=20) - - geodinv: tuple[float, float, float] = projection.inv( - wp.location.lon, - wp.location.lat, - wp_next.location.lon, - wp_next.location.lat, - ) - distance = geodinv[2] - - time_to_reach = timedelta(seconds=distance / ship_speed * 3600 / 1852) - arrival_time = time + time_to_reach - - if wp_next.time is None: - time = arrival_time - elif arrival_time > wp_next.time: - raise ScheduleError( - f"Waypoint planning is not valid: would arrive too late at waypoint number {wp_i + 2}. " - f"location: {wp_next.location} time: {wp_next.time} instrument: {wp_next.instrument}" - ) - else: - time = wp_next.time - - print("... All good to go!") - - -def _is_on_land_zero_uv(fieldset: FieldSet, waypoint: Waypoint) -> bool: - """ - Check if waypoint is on land by assuming zero velocity means land. - - :param fieldset: The fieldset to sample the velocity from. - :param waypoint: The waypoint to check. - :returns: If the waypoint is on land. - """ - return fieldset.UV.eval( - 0, - fieldset.gridset.grids[0].depth[0], - waypoint.location.lat, - waypoint.location.lon, - applyConversion=False, - ) == (0.0, 0.0) diff --git a/src/virtualship/models/ship_config.py b/src/virtualship/models/ship_config.py deleted file mode 100644 index be3ee30d..00000000 --- a/src/virtualship/models/ship_config.py +++ /dev/null @@ -1,320 +0,0 @@ -"""ShipConfig and supporting classes.""" - -from __future__ import annotations - -from datetime import timedelta -from enum import Enum -from pathlib import Path -from typing import TYPE_CHECKING - -import pydantic -import yaml - -from virtualship.errors import ConfigError -from virtualship.utils import _validate_numeric_mins_to_timedelta - -if TYPE_CHECKING: - from .schedule import Schedule - - -class InstrumentType(Enum): - """Types of the instruments.""" - - CTD = "CTD" - CTD_BGC = "CTD_BGC" - DRIFTER = "DRIFTER" - ARGO_FLOAT = "ARGO_FLOAT" - XBT = "XBT" - - -class ArgoFloatConfig(pydantic.BaseModel): - """Configuration for argos floats.""" - - min_depth_meter: float = pydantic.Field(le=0.0) - max_depth_meter: float = pydantic.Field(le=0.0) - drift_depth_meter: float = pydantic.Field(le=0.0) - vertical_speed_meter_per_second: float = pydantic.Field(lt=0.0) - cycle_days: float = pydantic.Field(gt=0.0) - drift_days: float = pydantic.Field(gt=0.0) - - -class ADCPConfig(pydantic.BaseModel): - """Configuration for ADCP instrument.""" - - max_depth_meter: float = pydantic.Field(le=0.0) - num_bins: int = pydantic.Field(gt=0.0) - period: timedelta = pydantic.Field( - serialization_alias="period_minutes", - validation_alias="period_minutes", - gt=timedelta(), - ) - - model_config = pydantic.ConfigDict(populate_by_name=True) - - @pydantic.field_serializer("period") - def _serialize_period(self, value: timedelta, _info): - return value.total_seconds() / 60.0 - - @pydantic.field_validator("period", mode="before") - def _validate_period(cls, value: int | float | timedelta) -> timedelta: - return _validate_numeric_mins_to_timedelta(value) - - -class CTDConfig(pydantic.BaseModel): - """Configuration for CTD instrument.""" - - stationkeeping_time: timedelta = pydantic.Field( - serialization_alias="stationkeeping_time_minutes", - validation_alias="stationkeeping_time_minutes", - gt=timedelta(), - ) - min_depth_meter: float = pydantic.Field(le=0.0) - max_depth_meter: float = pydantic.Field(le=0.0) - - model_config = pydantic.ConfigDict(populate_by_name=True) - - @pydantic.field_serializer("stationkeeping_time") - def _serialize_stationkeeping_time(self, value: timedelta, _info): - return value.total_seconds() / 60.0 - - @pydantic.field_validator("stationkeeping_time", mode="before") - def _validate_stationkeeping_time(cls, value: int | float | timedelta) -> timedelta: - return _validate_numeric_mins_to_timedelta(value) - - -class CTD_BGCConfig(pydantic.BaseModel): - """Configuration for CTD_BGC instrument.""" - - stationkeeping_time: timedelta = pydantic.Field( - serialization_alias="stationkeeping_time_minutes", - validation_alias="stationkeeping_time_minutes", - gt=timedelta(), - ) - min_depth_meter: float = pydantic.Field(le=0.0) - max_depth_meter: float = pydantic.Field(le=0.0) - - model_config = pydantic.ConfigDict(populate_by_name=True) - - @pydantic.field_serializer("stationkeeping_time") - def _serialize_stationkeeping_time(self, value: timedelta, _info): - return value.total_seconds() / 60.0 - - @pydantic.field_validator("stationkeeping_time", mode="before") - def _validate_stationkeeping_time(cls, value: int | float | timedelta) -> timedelta: - return _validate_numeric_mins_to_timedelta(value) - - -class ShipUnderwaterSTConfig(pydantic.BaseModel): - """Configuration for underwater ST.""" - - period: timedelta = pydantic.Field( - serialization_alias="period_minutes", - validation_alias="period_minutes", - gt=timedelta(), - ) - - model_config = pydantic.ConfigDict(populate_by_name=True) - - @pydantic.field_serializer("period") - def _serialize_period(self, value: timedelta, _info): - return value.total_seconds() / 60.0 - - @pydantic.field_validator("period", mode="before") - def _validate_period(cls, value: int | float | timedelta) -> timedelta: - return _validate_numeric_mins_to_timedelta(value) - - -class DrifterConfig(pydantic.BaseModel): - """Configuration for drifters.""" - - depth_meter: float = pydantic.Field(le=0.0) - lifetime: timedelta = pydantic.Field( - serialization_alias="lifetime_minutes", - validation_alias="lifetime_minutes", - gt=timedelta(), - ) - - model_config = pydantic.ConfigDict(populate_by_name=True) - - @pydantic.field_serializer("lifetime") - def _serialize_lifetime(self, value: timedelta, _info): - return value.total_seconds() / 60.0 - - @pydantic.field_validator("lifetime", mode="before") - def _validate_lifetime(cls, value: int | float | timedelta) -> timedelta: - return _validate_numeric_mins_to_timedelta(value) - - -class XBTConfig(pydantic.BaseModel): - """Configuration for xbt instrument.""" - - min_depth_meter: float = pydantic.Field(le=0.0) - max_depth_meter: float = pydantic.Field(le=0.0) - fall_speed_meter_per_second: float = pydantic.Field(gt=0.0) - deceleration_coefficient: float = pydantic.Field(gt=0.0) - - -class ShipConfig(pydantic.BaseModel): - """Configuration of the virtual ship.""" - - ship_speed_knots: float = pydantic.Field(gt=0.0) - """ - Velocity of the ship in knots. - """ - - argo_float_config: ArgoFloatConfig | None = None - """ - Argo float configuration. - - If None, no argo floats can be deployed. - """ - - adcp_config: ADCPConfig | None = None - """ - ADCP configuration. - - If None, no ADCP measurements will be performed. - """ - - ctd_config: CTDConfig | None = None - """ - CTD configuration. - - If None, no CTDs can be cast. - """ - - ctd_bgc_config: CTD_BGCConfig | None = None - """ - CTD_BGC configuration. - - If None, no BGC CTDs can be cast. - """ - - ship_underwater_st_config: ShipUnderwaterSTConfig | None = None - """ - Ship underwater salinity temperature measurementconfiguration. - - If None, no ST measurements will be performed. - """ - - drifter_config: DrifterConfig | None = None - """ - Drifter configuration. - - If None, no drifters can be deployed. - """ - - xbt_config: XBTConfig | None = None - """ - XBT configuration. - - If None, no XBTs can be cast. - """ - - model_config = pydantic.ConfigDict(extra="forbid") - - def to_yaml(self, file_path: str | Path) -> None: - """ - Write config to yaml file. - - :param file_path: Path to the file to write to. - """ - with open(file_path, "w") as file: - yaml.dump(self.model_dump(by_alias=True), file) - - @classmethod - def from_yaml(cls, file_path: str | Path) -> ShipConfig: - """ - Load config from yaml file. - - :param file_path: Path to the file to load from. - :returns: The config. - """ - with open(file_path) as file: - data = yaml.safe_load(file) - return ShipConfig(**data) - - def verify(self, schedule: Schedule) -> None: - """ - Verify the ship configuration against the provided schedule. - - This function performs two main tasks: - 1. Removes instrument configurations that are not present in the schedule. - 2. Verifies that all instruments in the schedule have corresponding configurations. - - Parameters - ---------- - schedule : Schedule - The schedule object containing the planned instruments and waypoints. - - Returns - ------- - None - - Raises - ------ - ConfigError - If an instrument in the schedule does not have a corresponding configuration. - - Notes - ----- - - Prints a message if a configuration is provided for an instrument not in the schedule. - - Sets the configuration to None for instruments not in the schedule. - - Raises a ConfigError for each instrument in the schedule that lacks a configuration. - - """ - instruments_in_schedule = schedule.get_instruments() - - for instrument in [ - "ARGO_FLOAT", - "DRIFTER", - "XBT", - "CTD", - "CTD_BGC", - ]: # TODO make instrument names consistent capitals or lowercase throughout codebase - if hasattr(self, instrument.lower() + "_config") and not any( - instrument == schedule_instrument.name - for schedule_instrument in instruments_in_schedule - ): - print(f"{instrument} configuration provided but not in schedule.") - setattr(self, instrument.lower() + "_config", None) - - # verify instruments in schedule have configuration - # TODO: the ConfigError message could be improved to explain that the **schedule** file has X instrument but the **ship_config** file does not - for instrument in instruments_in_schedule: - try: - InstrumentType(instrument) - except ValueError as e: - raise NotImplementedError("Instrument not supported.") from e - - if instrument == InstrumentType.ARGO_FLOAT and ( - not hasattr(self, "argo_float_config") or self.argo_float_config is None - ): - raise ConfigError( - "Planning has a waypoint with Argo float instrument, but configuration does not configure Argo floats." - ) - if instrument == InstrumentType.CTD and ( - not hasattr(self, "ctd_config") or self.ctd_config is None - ): - raise ConfigError( - "Planning has a waypoint with CTD instrument, but configuration does not configure CTDs." - ) - if instrument == InstrumentType.CTD_BGC and ( - not hasattr(self, "ctd_bgc_config") or self.ctd_bgc_config is None - ): - raise ConfigError( - "Planning has a waypoint with CTD_BGC instrument, but configuration does not configure CTD_BGCs." - ) - if instrument == InstrumentType.DRIFTER and ( - not hasattr(self, "drifter_config") or self.drifter_config is None - ): - raise ConfigError( - "Planning has a waypoint with drifter instrument, but configuration does not configure drifters." - ) - - if instrument == InstrumentType.XBT and ( - not hasattr(self, "xbt_config") or self.xbt_config is None - ): - raise ConfigError( - "Planning has a waypoint with XBT instrument, but configuration does not configure XBT." - ) diff --git a/src/virtualship/static/expedition.yaml b/src/virtualship/static/expedition.yaml new file mode 100644 index 00000000..2b770735 --- /dev/null +++ b/src/virtualship/static/expedition.yaml @@ -0,0 +1,75 @@ +schedule: + space_time_region: + spatial_range: + minimum_longitude: -5 + maximum_longitude: 5 + minimum_latitude: -5 + maximum_latitude: 5 + minimum_depth: 0 + maximum_depth: 2000 + time_range: + start_time: 1998-01-01 00:00:00 + end_time: 1998-02-01 00:00:00 + waypoints: + - instrument: + - CTD + - CTD_BGC + location: + latitude: 0 + longitude: 0 + time: 1998-01-01 00:00:00 + - instrument: + - DRIFTER + - CTD + location: + latitude: 0.01 + longitude: 0.01 + time: 1998-01-01 01:00:00 + - instrument: + - ARGO_FLOAT + location: + latitude: 0.02 + longitude: 0.02 + time: 1998-01-01 02:00:00 + - instrument: + - XBT + location: + latitude: 0.03 + longitude: 0.03 + time: 1998-01-01 03:00:00 + - location: + latitude: 0.03 + longitude: 0.03 + time: 1998-01-01 03:00:00 +instruments_config: + adcp_config: + num_bins: 40 + max_depth_meter: -1000.0 + period_minutes: 5.0 + argo_float_config: + cycle_days: 10.0 + drift_days: 9.0 + drift_depth_meter: -1000.0 + max_depth_meter: -2000.0 + min_depth_meter: 0.0 + vertical_speed_meter_per_second: -0.1 + ctd_config: + max_depth_meter: -2000.0 + min_depth_meter: -11.0 + stationkeeping_time_minutes: 20.0 + ctd_bgc_config: + max_depth_meter: -2000.0 + min_depth_meter: -11.0 + stationkeeping_time_minutes: 20.0 + drifter_config: + depth_meter: -1.0 + lifetime_minutes: 60480.0 + xbt_config: + max_depth_meter: -285.0 + min_depth_meter: -2.0 + fall_speed_meter_per_second: 6.7 + deceleration_coefficient: 0.00225 + ship_underwater_st_config: + period_minutes: 5.0 +ship_config: + ship_speed_knots: 10.0 diff --git a/src/virtualship/static/schedule.yaml b/src/virtualship/static/schedule.yaml deleted file mode 100644 index 34c5c01a..00000000 --- a/src/virtualship/static/schedule.yaml +++ /dev/null @@ -1,42 +0,0 @@ -space_time_region: - spatial_range: - minimum_longitude: -5 - maximum_longitude: 5 - minimum_latitude: -5 - maximum_latitude: 5 - minimum_depth: 0 - maximum_depth: 2000 - time_range: - start_time: 1998-01-01 00:00:00 - end_time: 1998-02-01 00:00:00 -waypoints: - - instrument: - - CTD - - CTD_BGC - location: - latitude: 0 - longitude: 0 - time: 1998-01-01 00:00:00 - - instrument: - - DRIFTER - - CTD - location: - latitude: 0.01 - longitude: 0.01 - time: 1998-01-01 01:00:00 - - instrument: - - ARGO_FLOAT - location: - latitude: 0.02 - longitude: 0.02 - time: 1998-01-01 02:00:00 - - instrument: - - XBT - location: - latitude: 0.03 - longitude: 0.03 - time: 1998-01-01 03:00:00 - - location: - latitude: 0.03 - longitude: 0.03 - time: 1998-01-01 03:00:00 diff --git a/src/virtualship/static/ship_config.yaml b/src/virtualship/static/ship_config.yaml deleted file mode 100644 index 34d6c6ea..00000000 --- a/src/virtualship/static/ship_config.yaml +++ /dev/null @@ -1,30 +0,0 @@ -ship_speed_knots: 10.0 -adcp_config: - num_bins: 40 - max_depth_meter: -1000.0 - period_minutes: 5.0 -argo_float_config: - cycle_days: 10.0 - drift_days: 9.0 - drift_depth_meter: -1000.0 - max_depth_meter: -2000.0 - min_depth_meter: 0.0 - vertical_speed_meter_per_second: -0.1 -ctd_config: - max_depth_meter: -2000.0 - min_depth_meter: -11.0 - stationkeeping_time_minutes: 20.0 -ctd_bgc_config: - max_depth_meter: -2000.0 - min_depth_meter: -11.0 - stationkeeping_time_minutes: 20.0 -drifter_config: - depth_meter: 0.0 - lifetime_minutes: 60480.0 -xbt_config: - max_depth_meter: -285.0 - min_depth_meter: -2.0 - fall_speed_meter_per_second: 6.7 - deceleration_coefficient: 0.00225 -ship_underwater_st_config: - period_minutes: 5.0 diff --git a/src/virtualship/utils.py b/src/virtualship/utils.py index 1f334f06..7e37617b 100644 --- a/src/virtualship/utils.py +++ b/src/virtualship/utils.py @@ -1,24 +1,33 @@ from __future__ import annotations +import glob import os +import re import warnings -from datetime import timedelta +from datetime import datetime, timedelta from functools import lru_cache from importlib.resources import files from pathlib import Path from typing import TYPE_CHECKING, TextIO -from yaspin import Spinner +import copernicusmarine +import numpy as np +import xarray as xr +from parcels import FieldSet + +from virtualship.errors import CopernicusCatalogueError if TYPE_CHECKING: - from virtualship.models import Schedule, ShipConfig + from virtualship.expedition.simulate_schedule import ScheduleOk + from virtualship.models import Expedition + import pandas as pd import yaml from pydantic import BaseModel +from yaspin import Spinner -SCHEDULE = "schedule.yaml" -SHIP_CONFIG = "ship_config.yaml" +EXPEDITION = "expedition.yaml" CHECKPOINT = "checkpoint.yaml" @@ -28,15 +37,10 @@ def load_static_file(name: str) -> str: @lru_cache(None) -def get_example_config() -> str: - """Get the example configuration file.""" - return load_static_file(SHIP_CONFIG) - - @lru_cache(None) -def get_example_schedule() -> str: - """Get the example schedule file.""" - return load_static_file(SCHEDULE) +def get_example_expedition() -> str: + """Get the example unified expedition configuration file.""" + return load_static_file(EXPEDITION) def _dump_yaml(model: BaseModel, stream: TextIO) -> str | None: @@ -121,7 +125,7 @@ def validate_coordinates(coordinates_data): def mfp_to_yaml(coordinates_file_path: str, yaml_output_path: str): # noqa: D417 """ - Generates a YAML file with spatial and temporal information based on instrument data from MFP excel file. + Generates an expedition.yaml file with schedule information based on data from MFP excel file. The ship and instrument configurations entries in the YAML file are sourced from the static version. Parameters ---------- @@ -134,7 +138,10 @@ def mfp_to_yaml(coordinates_file_path: str, yaml_output_path: str): # noqa: D41 4. returns the yaml information. """ + # avoid circular imports from virtualship.models import ( + Expedition, + InstrumentsConfig, Location, Schedule, SpaceTimeRegion, @@ -188,8 +195,23 @@ def mfp_to_yaml(coordinates_file_path: str, yaml_output_path: str): # noqa: D41 space_time_region=space_time_region, ) + # extract instruments config from static + instruments_config = InstrumentsConfig.model_validate( + yaml.safe_load(get_example_expedition()).get("instruments_config") + ) + + # extract ship config from static + ship_config = yaml.safe_load(get_example_expedition()).get("ship_config") + + # combine to Expedition object + expedition = Expedition( + schedule=schedule, + instruments_config=instruments_config, + ship_config=ship_config, + ) + # Save to YAML file - schedule.to_yaml(yaml_output_path) + expedition.to_yaml(yaml_output_path) def _validate_numeric_mins_to_timedelta(value: int | float | timedelta) -> timedelta: @@ -199,26 +221,16 @@ def _validate_numeric_mins_to_timedelta(value: int | float | timedelta) -> timed return timedelta(minutes=value) -def _get_schedule(expedition_dir: Path) -> Schedule: - """Load Schedule object from yaml config file in `expedition_dir`.""" - from virtualship.models import Schedule +def _get_expedition(expedition_dir: Path) -> Expedition: + """Load Expedition object from yaml config file in `expedition_dir`.""" + from virtualship.models import Expedition - file_path = expedition_dir.joinpath(SCHEDULE) + file_path = expedition_dir.joinpath(EXPEDITION) try: - return Schedule.from_yaml(file_path) - except FileNotFoundError as e: - raise FileNotFoundError(f'Schedule not found. Save it to "{file_path}".') from e - - -def _get_ship_config(expedition_dir: Path) -> ShipConfig: - from virtualship.models import ShipConfig - - file_path = expedition_dir.joinpath(SHIP_CONFIG) - try: - return ShipConfig.from_yaml(file_path) + return Expedition.from_yaml(file_path) except FileNotFoundError as e: raise FileNotFoundError( - f'Ship config not found. Save it to "{file_path}".' + f'{EXPEDITION} not found. Save it to "{file_path}".' ) from e @@ -238,3 +250,313 @@ def _get_ship_config(expedition_dir: Path) -> ShipConfig: "🚢 ", ], ) + + +# InstrumentType -> Instrument registry and registration utilities. +INSTRUMENT_CLASS_MAP = {} + + +def register_instrument(instrument_type): + def decorator(cls): + INSTRUMENT_CLASS_MAP[instrument_type] = cls + return cls + + return decorator + + +def get_instrument_class(instrument_type): + return INSTRUMENT_CLASS_MAP.get(instrument_type) + + +def add_dummy_UV(fieldset: FieldSet): + """Add a dummy U and V field to a FieldSet to satisfy parcels FieldSet completeness checks.""" + if "U" not in fieldset.__dict__.keys(): + for uv_var in ["U", "V"]: + dummy_field = getattr( + FieldSet.from_data( + {"U": 0, "V": 0}, {"lon": 0, "lat": 0}, mesh="spherical" + ), + uv_var, + ) + fieldset.add_field(dummy_field) + try: + fieldset.time_origin = ( + fieldset.T.grid.time_origin + if "T" in fieldset.__dict__.keys() + else fieldset.o2.grid.time_origin + ) + except Exception: + raise ValueError( + "Cannot determine time_origin for dummy UV fields. Assert T or o2 exists in fieldset." + ) from None + + +# Copernicus Marine product IDs + +PRODUCT_IDS = { + "phys": { + "reanalysis": "cmems_mod_glo_phy_my_0.083deg_P1D-m", + "reanalysis_interim": "cmems_mod_glo_phy_myint_0.083deg_P1D-m", + "analysis": "cmems_mod_glo_phy_anfc_0.083deg_P1D-m", + }, + "bgc": { + "reanalysis": "cmems_mod_glo_bgc_my_0.25deg_P1D-m", + "reanalysis_interim": "cmems_mod_glo_bgc_myint_0.25deg_P1D-m", + "analysis": None, # will be set per variable + }, +} + +BGC_ANALYSIS_IDS = { + "o2": "cmems_mod_glo_bgc-bio_anfc_0.25deg_P1D-m", + "chl": "cmems_mod_glo_bgc-pft_anfc_0.25deg_P1D-m", + "no3": "cmems_mod_glo_bgc-nut_anfc_0.25deg_P1D-m", + "po4": "cmems_mod_glo_bgc-nut_anfc_0.25deg_P1D-m", + "ph": "cmems_mod_glo_bgc-car_anfc_0.25deg_P1D-m", + "phyc": "cmems_mod_glo_bgc-pft_anfc_0.25deg_P1D-m", + "nppv": "cmems_mod_glo_bgc-bio_anfc_0.25deg_P1D-m", +} + +MONTHLY_BGC_REANALYSIS_IDS = { + "ph": "cmems_mod_glo_bgc_my_0.25deg_P1M-m", + "phyc": "cmems_mod_glo_bgc_my_0.25deg_P1M-m", +} +MONTHLY_BGC_REANALYSIS_INTERIM_IDS = { + "ph": "cmems_mod_glo_bgc_myint_0.25deg_P1M-m", + "phyc": "cmems_mod_glo_bgc_myint_0.25deg_P1M-m", +} + +# variables used in VirtualShip which are physical or biogeochemical variables, respectively +COPERNICUSMARINE_PHYS_VARIABLES = ["uo", "vo", "so", "thetao"] +COPERNICUSMARINE_BGC_VARIABLES = ["o2", "chl", "no3", "po4", "ph", "phyc", "nppv"] + + +def _select_product_id( + physical: bool, + schedule_start, + schedule_end, + username: str | None = None, + password: str | None = None, + variable: str | None = None, +) -> str: + """Determine which copernicus product id should be selected (reanalysis, reanalysis-interim, analysis & forecast), for prescribed schedule and physical vs. BGC.""" + key = "phys" if physical else "bgc" + selected_id = None + + for period, pid in PRODUCT_IDS[key].items(): + # for BGC analysis, set pid per variable + if key == "bgc" and period == "analysis": + if variable is None or variable not in BGC_ANALYSIS_IDS: + continue + pid = BGC_ANALYSIS_IDS[variable] + # for BGC reanalysis, check if requires monthly product + if ( + key == "bgc" + and period == "reanalysis" + and variable in MONTHLY_BGC_REANALYSIS_IDS + ): + monthly_pid = MONTHLY_BGC_REANALYSIS_IDS[variable] + ds_monthly = copernicusmarine.open_dataset( + monthly_pid, + username=username, + password=password, + ) + time_end_monthly = ds_monthly["time"][-1].values + if np.datetime64(schedule_end) <= time_end_monthly: + pid = monthly_pid + # for BGC reanalysis_interim, check if requires monthly product + if ( + key == "bgc" + and period == "reanalysis_interim" + and variable in MONTHLY_BGC_REANALYSIS_INTERIM_IDS + ): + monthly_pid = MONTHLY_BGC_REANALYSIS_INTERIM_IDS[variable] + ds_monthly = copernicusmarine.open_dataset( + monthly_pid, username=username, password=password + ) + time_end_monthly = ds_monthly["time"][-1].values + if np.datetime64(schedule_end) <= time_end_monthly: + pid = monthly_pid + if pid is None: + continue + ds = copernicusmarine.open_dataset(pid, username=username, password=password) + time_end = ds["time"][-1].values + if np.datetime64(schedule_end) <= time_end: + selected_id = pid + break + + if selected_id is None: + raise CopernicusCatalogueError( + "No suitable product found in the Copernicus Marine Catalogue for the scheduled time and variable." + ) + + if _start_end_in_product_timerange( + selected_id, schedule_start, schedule_end, username, password + ): + return selected_id + else: + return ( + PRODUCT_IDS["phys"]["analysis"] if physical else BGC_ANALYSIS_IDS[variable] + ) + + +def _start_end_in_product_timerange( + selected_id, schedule_start, schedule_end, username, password +): + ds_selected = copernicusmarine.open_dataset( + selected_id, username=username, password=password + ) + time_values = ds_selected["time"].values + import numpy as np + + time_min, time_max = np.min(time_values), np.max(time_values) + return ( + np.datetime64(schedule_start) >= time_min + and np.datetime64(schedule_end) <= time_max + ) + + +def _get_bathy_data( + space_time_region, latlon_buffer: float | None = None, from_data: Path | None = None +) -> FieldSet: + """Bathymetry data from local or 'streamed' directly from Copernicus Marine.""" + if from_data is not None: # load from local data + var = "deptho" + bathy_dir = from_data.joinpath("bathymetry") + try: + filename, _ = _find_nc_file_with_variable(bathy_dir, var) + except Exception as e: + # TODO: link to documentation on expected data structure!! + raise RuntimeError( + f"\n\n❗️ Could not find bathymetry variable '{var}' in data directory '{from_data}/bathymetry/'.\n\n❗️ Is the pre-downloaded data directory structure compliant with VirtualShip expectations?\n\n❗️ See the docs for more information on expectations: https://virtualship.readthedocs.io/en/latest/user-guide/index.html#documentation\n" + ) from e + ds_bathymetry = xr.open_dataset(bathy_dir.joinpath(filename)) + bathymetry_variables = {"bathymetry": "deptho"} + bathymetry_dimensions = {"lon": "longitude", "lat": "latitude"} + return FieldSet.from_xarray_dataset( + ds_bathymetry, bathymetry_variables, bathymetry_dimensions + ) + + else: # stream via Copernicus Marine Service + ds_bathymetry = copernicusmarine.open_dataset( + dataset_id="cmems_mod_glo_phy_my_0.083deg_static", + minimum_longitude=space_time_region.spatial_range.minimum_longitude + - (latlon_buffer if latlon_buffer is not None else 0), + maximum_longitude=space_time_region.spatial_range.maximum_longitude + + (latlon_buffer if latlon_buffer is not None else 0), + minimum_latitude=space_time_region.spatial_range.minimum_latitude + - (latlon_buffer if latlon_buffer is not None else 0), + maximum_latitude=space_time_region.spatial_range.maximum_latitude + + (latlon_buffer if latlon_buffer is not None else 0), + variables=["deptho"], + start_datetime=space_time_region.time_range.start_time, + end_datetime=space_time_region.time_range.end_time, + coordinates_selection_method="outside", + ) + bathymetry_variables = {"bathymetry": "deptho"} + bathymetry_dimensions = {"lon": "longitude", "lat": "latitude"} + + return FieldSet.from_xarray_dataset( + ds_bathymetry, bathymetry_variables, bathymetry_dimensions + ) + + +def expedition_cost(schedule_results: ScheduleOk, time_past: timedelta) -> float: + """ + Calculate the cost of the expedition in US$. + + :param schedule_results: Results from schedule simulation. + :param time_past: Time the expedition took. + :returns: The calculated cost of the expedition in US$. + """ + SHIP_COST_PER_DAY = 30000 + DRIFTER_DEPLOY_COST = 2500 + ARGO_DEPLOY_COST = 15000 + + ship_cost = SHIP_COST_PER_DAY / 24 * time_past.total_seconds() // 3600 + num_argos = len(schedule_results.measurements_to_simulate.argo_floats) + argo_cost = num_argos * ARGO_DEPLOY_COST + num_drifters = len(schedule_results.measurements_to_simulate.drifters) + drifter_cost = num_drifters * DRIFTER_DEPLOY_COST + + cost = ship_cost + argo_cost + drifter_cost + return cost + + +def _find_nc_file_with_variable(data_dir: Path, var: str) -> str | None: + """Search for a .nc file in the given directory containing the specified variable.""" + for nc_file in data_dir.glob("*.nc"): + try: + with xr.open_dataset(nc_file, chunks={}) as ds: + matched_vars = [v for v in ds.variables if var in v] + if matched_vars: + return nc_file.name, matched_vars[0] + except Exception: + continue + return None + + +def _find_files_in_timerange( + data_dir: Path, + schedule_start, + schedule_end, + date_pattern=r"\d{4}_\d{2}_\d{2}", + date_fmt="%Y_%m_%d", +) -> list: + """Find all files in data_dir whose filenames contain a date within [schedule_start, schedule_end] (inclusive).""" + # TODO: scope to make this more flexible for different date patterns / formats ... ? + + all_files = glob.glob(str(data_dir.joinpath("*"))) + if not all_files: + raise ValueError( + f"No files found in data directory {data_dir}. Please ensure the directory contains files with 'P1D' or 'P1M' in their names as per Copernicus Marine Product ID naming conventions." + ) + + if all("P1D" in s for s in all_files): + t_resolution = "daily" + elif all("P1M" in s for s in all_files): + t_resolution = "monthly" + else: + raise ValueError( + f"Could not determine time resolution from filenames in data directory. Please ensure all filenames in {data_dir} contain either 'P1D' (daily) or 'P1M' (monthly), " + f"as per the Copernicus Marine Product ID naming conventions." + ) + + if t_resolution == "monthly": + t_min = schedule_start.date().replace( + day=1 + ) # first day of month of the schedule start date + t_max = ( + schedule_end.date() + + timedelta( + days=32 + ) # buffer to ensure fieldset end date is always longer than schedule end date for monthly data + ) + else: # daily + t_min = schedule_start.date() + t_max = schedule_end.date() + + files_with_dates = [] + for file in data_dir.iterdir(): + if file.is_file(): + match = re.search(date_pattern, file.name) + if match: + file_date = datetime.strptime( + match.group(), date_fmt + ).date() # normalise to date only for comparison (given start/end dates have hour/minute components which may exceed those in file_date) + if t_min <= file_date <= t_max: + files_with_dates.append((file_date, file.name)) + + files_with_dates.sort( + key=lambda x: x[0] + ) # sort by extracted date; more robust than relying on filesystem order + + # catch if not enough data coverage found for the requested time range + if files_with_dates[-1][0] < schedule_end.date(): + raise ValueError( + f"Not enough data coverage found in {data_dir} for the requested time range {schedule_start} to {schedule_end}. " + f"Latest available data is for date {files_with_dates[-1][0]}." + f"If using monthly data, please ensure that the last month downloaded covers the schedule end date + 1 month." + f"See the docs for more details: https://virtualship.readthedocs.io/en/latest/user-guide/index.html#documentation" + ) + + return [fname for _, fname in files_with_dates] diff --git a/tests/cli/test_cli.py b/tests/cli/test_cli.py index 5148176b..a0d90d77 100644 --- a/tests/cli/test_cli.py +++ b/tests/cli/test_cli.py @@ -5,8 +5,8 @@ import xarray as xr from click.testing import CliRunner -from virtualship.cli.commands import fetch, init -from virtualship.utils import SCHEDULE, SHIP_CONFIG +from virtualship.cli.commands import init +from virtualship.utils import EXPEDITION @pytest.fixture @@ -23,9 +23,9 @@ def fake_open_dataset(*args, **kwargs): "time": ( "time", [ - np.datetime64("1993-01-01"), np.datetime64("2022-01-01"), - ], # mock up rough renanalysis period + np.datetime64("2025-01-01"), + ], # mock up rough reanalysis period, covers test schedule ) } ) @@ -51,51 +51,17 @@ def test_init(): with runner.isolated_filesystem(): result = runner.invoke(init, ["."]) assert result.exit_code == 0 - config = Path(SHIP_CONFIG) - schedule = Path(SCHEDULE) + expedition = Path(EXPEDITION) - assert config.exists() - assert schedule.exists() + assert expedition.exists() -def test_init_existing_config(): +def test_init_existing_expedition(): runner = CliRunner() with runner.isolated_filesystem(): - config = Path(SHIP_CONFIG) - config.write_text("test") + expedition = Path(EXPEDITION) + expedition.write_text("test") with pytest.raises(FileExistsError): result = runner.invoke(init, ["."]) raise result.exception - - -def test_init_existing_schedule(): - runner = CliRunner() - with runner.isolated_filesystem(): - schedule = Path(SCHEDULE) - schedule.write_text("test") - - with pytest.raises(FileExistsError): - result = runner.invoke(init, ["."]) - raise result.exception - - -@pytest.mark.parametrize( - "fetch_args", - [ - [".", "--username", "test"], - [".", "--password", "test"], - ], -) -@pytest.mark.usefixtures("copernicus_no_download") -def test_fetch_both_creds_via_cli(runner, fetch_args): - result = runner.invoke(fetch, fetch_args) - assert result.exit_code == 1 - assert "Both username and password" in result.exc_info[1].args[0] - - -@pytest.mark.usefixtures("copernicus_no_download") -def test_fetch(runner): - """Test the fetch command, but mock the downloads (and metadata interrogation).""" - result = runner.invoke(fetch, [".", "--username", "test", "--password", "test"]) - assert result.exit_code == 0 diff --git a/tests/cli/test_creds.py b/tests/cli/test_creds.py deleted file mode 100644 index 17ef2023..00000000 --- a/tests/cli/test_creds.py +++ /dev/null @@ -1,66 +0,0 @@ -import pydantic -import pytest - -from virtualship.cli._creds import CredentialFileError, Credentials - - -def test_load_credentials(tmp_file): - tmp_file.write_text( - """ - COPERNICUS_USERNAME: test_user - COPERNICUS_PASSWORD: test_password - """ - ) - - creds = Credentials.from_yaml(tmp_file) - assert creds.COPERNICUS_USERNAME == "test_user" - assert creds.COPERNICUS_PASSWORD == "test_password" - - -# parameterize with the contents of the file -@pytest.mark.parametrize( - "contents", - [ - pytest.param( - """ - INVALID_KEY: some_value - """, - id="invalid-key", - ), - pytest.param( - """ - # number not allowed, should be string (or quoted number) - USERNAME: 123 - """, - id="number-not-allowed", - ), - ], -) -def test_invalid_credentials(tmp_file, contents): - tmp_file.write_text(contents) - - with pytest.raises(pydantic.ValidationError): - Credentials.from_yaml(tmp_file) - - -def test_credentials_invalid_format(tmp_file): - tmp_file.write_text( - """ - INVALID_FORMAT_BUT_VALID_YAML - """ - ) - - with pytest.raises(CredentialFileError): - Credentials.from_yaml(tmp_file) - - -def test_rt_credentials(tmp_file): - """Test round-trip for credentials using Credentials.from_yaml() and Credentials.dump().""" - creds = Credentials( - COPERNICUS_USERNAME="test_user", COPERNICUS_PASSWORD="test_password" - ) - - creds.to_yaml(tmp_file) - creds_loaded = Credentials.from_yaml(tmp_file) - - assert creds == creds_loaded diff --git a/tests/cli/test_fetch.py b/tests/cli/test_fetch.py deleted file mode 100644 index 611911ff..00000000 --- a/tests/cli/test_fetch.py +++ /dev/null @@ -1,178 +0,0 @@ -from pathlib import Path - -import numpy as np -import pytest -import xarray as xr -from pydantic import BaseModel - -from virtualship.cli._fetch import ( - DOWNLOAD_METADATA, - DownloadMetadata, - IncompleteDownloadError, - _fetch, - assert_complete_download, - complete_download, - create_hash, - filename_to_hash, - get_existing_download, - hash_model, - hash_to_filename, - select_product_id, - start_end_in_product_timerange, -) -from virtualship.models import Schedule, ShipConfig -from virtualship.utils import get_example_config, get_example_schedule - - -@pytest.fixture -def copernicus_no_download(monkeypatch): - """Mock the copernicusmarine `subset` and `open_dataset` functions, approximating the reanalysis products.""" - - # mock for copernicusmarine.subset - def fake_download(output_filename, output_directory, **_): - Path(output_directory).joinpath(output_filename).touch() - - def fake_open_dataset(*args, **kwargs): - return xr.Dataset( - coords={ - "time": ( - "time", - [ - np.datetime64("1993-01-01"), - np.datetime64("2022-01-01"), - ], # mock up rough renanalysis period - ) - } - ) - - monkeypatch.setattr("virtualship.cli._fetch.copernicusmarine.subset", fake_download) - monkeypatch.setattr( - "virtualship.cli._fetch.copernicusmarine.open_dataset", fake_open_dataset - ) - yield - - -@pytest.fixture -def schedule(tmpdir): - out_path = tmpdir.join("schedule.yaml") - - with open(out_path, "w") as file: - file.write(get_example_schedule()) - - schedule = Schedule.from_yaml(out_path) - - return schedule - - -@pytest.fixture -def ship_config(tmpdir): - out_path = tmpdir.join("ship_config.yaml") - - with open(out_path, "w") as file: - file.write(get_example_config()) - - ship_config = ShipConfig.from_yaml(out_path) - - return ship_config - - -@pytest.mark.usefixtures("copernicus_no_download") -def test_fetch(schedule, ship_config, tmpdir): - """Test the fetch command, but mock the download and dataset metadata interrogation.""" - _fetch(Path(tmpdir), "test", "test") - - -def test_create_hash(): - assert len(create_hash("correct-length")) == 8 - assert create_hash("same") == create_hash("same") - assert create_hash("unique1") != create_hash("unique2") - - -def test_hash_filename_roundtrip(): - hash_ = create_hash("test") - assert filename_to_hash(hash_to_filename(hash_)) == hash_ - - -def test_hash_model(): - class TestModel(BaseModel): - a: int - b: str - - hash_model(TestModel(a=0, b="b")) - - -def test_complete_download(tmp_path): - # Setup - DownloadMetadata(download_complete=False).to_yaml(tmp_path / DOWNLOAD_METADATA) - - complete_download(tmp_path) - - assert_complete_download(tmp_path) - - -@pytest.mark.usefixtures("copernicus_no_download") -def test_select_product_id(schedule): - """Should return the physical reanalysis product id via the timings prescribed in the static schedule.yaml file.""" - result = select_product_id( - physical=True, - schedule_start=schedule.space_time_region.time_range.start_time, - schedule_end=schedule.space_time_region.time_range.end_time, - username="test", - password="test", - ) - assert result == "cmems_mod_glo_phy_my_0.083deg_P1D-m" - - -@pytest.mark.usefixtures("copernicus_no_download") -def test_start_end_in_product_timerange(schedule): - """Should return True for valid range ass determined by the static schedule.yaml file.""" - assert start_end_in_product_timerange( - selected_id="cmems_mod_glo_phy_my_0.083deg_P1D-m", - schedule_start=schedule.space_time_region.time_range.start_time, - schedule_end=schedule.space_time_region.time_range.end_time, - username="test", - password="test", - ) - - -def test_assert_complete_download_complete(tmp_path): - # Setup - DownloadMetadata(download_complete=True).to_yaml(tmp_path / DOWNLOAD_METADATA) - - assert_complete_download(tmp_path) - - -def test_assert_complete_download_incomplete(tmp_path): - # Setup - DownloadMetadata(download_complete=False).to_yaml(tmp_path / DOWNLOAD_METADATA) - - with pytest.raises(IncompleteDownloadError): - assert_complete_download(tmp_path) - - -def test_assert_complete_download_missing(tmp_path): - with pytest.raises(IncompleteDownloadError): - assert_complete_download(tmp_path) - - -@pytest.fixture -def existing_data_folder(tmp_path, monkeypatch): - # Setup - folders = [ - "YYYYMMDD_HHMMSS_hash", - "YYYYMMDD_HHMMSS_hash2", - "some-invalid-data-folder", - "YYYYMMDD_HHMMSS_hash3", - ] - data_folder = tmp_path - monkeypatch.setattr( - "virtualship.cli._fetch.assert_complete_download", lambda x: None - ) - for f in folders: - (data_folder / f).mkdir() - yield data_folder - - -def test_get_existing_download(existing_data_folder): - assert isinstance(get_existing_download(existing_data_folder, "hash"), Path) - assert get_existing_download(existing_data_folder, "missing-hash") is None diff --git a/tests/cli/test_plan.py b/tests/cli/test_plan.py index 6fef90a1..421feba0 100644 --- a/tests/cli/test_plan.py +++ b/tests/cli/test_plan.py @@ -9,7 +9,8 @@ import yaml from textual.widgets import Button, Collapsible, Input -from virtualship.cli._plan import ConfigEditor, PlanApp, ScheduleEditor +from virtualship.cli._plan import ExpeditionEditor, PlanApp +from virtualship.utils import EXPEDITION NEW_SPEED = "8.0" NEW_LAT = "0.05" @@ -33,12 +34,8 @@ async def test_UI_changes(): tmpdir = Path(tempfile.mkdtemp()) shutil.copy( - files("virtualship.static").joinpath("ship_config.yaml"), - tmpdir / "ship_config.yaml", - ) - shutil.copy( - files("virtualship.static").joinpath("schedule.yaml"), - tmpdir / "schedule.yaml", + files("virtualship.static").joinpath(EXPEDITION), + tmpdir / EXPEDITION, ) app = PlanApp(path=tmpdir) @@ -47,22 +44,23 @@ async def test_UI_changes(): await pilot.pause(0.5) plan_screen = pilot.app.screen - config_editor = plan_screen.query_one(ConfigEditor) - schedule_editor = plan_screen.query_one(ScheduleEditor) + expedition_editor = plan_screen.query_one(ExpeditionEditor) # get mock of UI notify method plan_screen.notify = MagicMock() # change ship speed - speed_collapsible = config_editor.query_one("#speed_collapsible", Collapsible) + speed_collapsible = expedition_editor.query_one( + "#speed_collapsible", Collapsible + ) if speed_collapsible.collapsed: speed_collapsible.collapsed = False await pilot.pause() - ship_speed_input = config_editor.query_one("#speed", Input) + ship_speed_input = expedition_editor.query_one("#speed", Input) await simulate_input(pilot, ship_speed_input, NEW_SPEED) # change waypoint lat/lon (e.g. first waypoint) - waypoints_collapsible = schedule_editor.query_one("#waypoints", Collapsible) + waypoints_collapsible = expedition_editor.query_one("#waypoints", Collapsible) if waypoints_collapsible.collapsed: waypoints_collapsible.collapsed = False await pilot.pause() @@ -104,11 +102,11 @@ async def test_UI_changes(): ) # verify changes to speed, lat, lon in saved YAML - ship_config_path = os.path.join(tmpdir, "ship_config.yaml") - with open(ship_config_path) as f: - saved_config = yaml.safe_load(f) + expedition_path = os.path.join(tmpdir, EXPEDITION) + with open(expedition_path) as f: + saved_expedition = yaml.safe_load(f) - assert saved_config["ship_speed_knots"] == float(NEW_SPEED) + assert saved_expedition["ship_config"]["ship_speed_knots"] == float(NEW_SPEED) # check schedule.verify() methods are working by purposefully making invalid schedule (i.e. ship speed too slow to reach waypoints) invalid_speed = "0.0001" diff --git a/tests/cli/test_run.py b/tests/cli/test_run.py new file mode 100644 index 00000000..19044234 --- /dev/null +++ b/tests/cli/test_run.py @@ -0,0 +1,64 @@ +from datetime import datetime +from pathlib import Path + +from virtualship.cli._run import _run +from virtualship.expedition.simulate_schedule import ( + MeasurementsToSimulate, + ScheduleOk, +) +from virtualship.utils import EXPEDITION, get_example_expedition + + +def _simulate_schedule(projection, expedition): + """Return a trivial ScheduleOk with no measurements to simulate.""" + return ScheduleOk( + time=datetime.now(), measurements_to_simulate=MeasurementsToSimulate() + ) + + +class DummyInstrument: + """Dummy instrument class that just creates empty output directories.""" + + def __init__(self, expedition, from_data=None): + """Initialize DummyInstrument.""" + self.expedition = expedition + self.from_data = from_data + + def execute(self, measurements, out_path): + """Mock execute method.""" + out_path = Path(out_path) + out_path.parent.mkdir(parents=True, exist_ok=True) + out_path.mkdir(parents=True, exist_ok=True) + + +def test_run(tmp_path, monkeypatch): + """Testing as if using pre-downloaded, local data.""" + expedition_dir = tmp_path / "expedition_dir" + expedition_dir.mkdir() + (expedition_dir / EXPEDITION).write_text(get_example_expedition()) + + monkeypatch.setattr("virtualship.cli._run.simulate_schedule", _simulate_schedule) + + monkeypatch.setattr( + "virtualship.models.InstrumentsConfig.verify", lambda self, expedition: None + ) + monkeypatch.setattr( + "virtualship.models.Schedule.verify", lambda self, *args, **kwargs: None + ) + + monkeypatch.setattr( + "virtualship.cli._run.get_instrument_class", lambda itype: DummyInstrument + ) + + fake_data_dir = tmp_path / "fake_data" + fake_data_dir.mkdir() + + _run(expedition_dir, from_data=fake_data_dir) + + results_dir = expedition_dir / "results" + + assert results_dir.exists() and results_dir.is_dir() + cost_file = results_dir / "cost.txt" + assert cost_file.exists() + content = cost_file.read_text() + assert "cost:" in content diff --git a/tests/conftest.py b/tests/conftest.py index 1b7a1de0..5ceac033 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,4 +1,4 @@ -"""Test configuration that is ran for every test.""" +"""Test configuration that is run for every test.""" import pytest diff --git a/tests/expedition/expedition_dir/expedition.yaml b/tests/expedition/expedition_dir/expedition.yaml new file mode 100644 index 00000000..0ed9e5f4 --- /dev/null +++ b/tests/expedition/expedition_dir/expedition.yaml @@ -0,0 +1,46 @@ +schedule: + waypoints: + - instrument: + - CTD + location: + latitude: 0 + longitude: 0 + time: 2023-01-01 00:00:00 + - instrument: + - DRIFTER + - ARGO_FLOAT + location: + latitude: 0.01 + longitude: 0.01 + time: 2023-01-02 00:00:00 + - location: # empty waypoint + latitude: 0.02 + longitude: 0.01 + time: 2023-01-02 03:00:00 +instruments_config: + adcp_config: + num_bins: 40 + max_depth_meter: -1000.0 + period_minutes: 5.0 + argo_float_config: + cycle_days: 10.0 + drift_days: 9.0 + drift_depth_meter: -1000.0 + max_depth_meter: -2000.0 + min_depth_meter: 0.0 + vertical_speed_meter_per_second: -0.1 + ctd_config: + max_depth_meter: -2000.0 + min_depth_meter: -11.0 + stationkeeping_time_minutes: 20.0 + ctd_bgc_config: + max_depth_meter: -2000.0 + min_depth_meter: -11.0 + stationkeeping_time_minutes: 20.0 + drifter_config: + depth_meter: -1.0 + lifetime_minutes: 40320.0 + ship_underwater_st_config: + period_minutes: 5.0 +ship_config: + ship_speed_knots: 10.0 diff --git a/tests/expedition/expedition_dir/schedule.yaml b/tests/expedition/expedition_dir/schedule.yaml deleted file mode 100644 index 29c14ac9..00000000 --- a/tests/expedition/expedition_dir/schedule.yaml +++ /dev/null @@ -1,18 +0,0 @@ -waypoints: - - instrument: - - CTD - location: - latitude: 0 - longitude: 0 - time: 2023-01-01 00:00:00 - - instrument: - - DRIFTER - - ARGO_FLOAT - location: - latitude: 0.01 - longitude: 0.01 - time: 2023-01-02 00:00:00 - - location: # empty waypoint - latitude: 0.02 - longitude: 0.01 - time: 2023-01-02 03:00:00 diff --git a/tests/expedition/expedition_dir/ship_config.yaml b/tests/expedition/expedition_dir/ship_config.yaml deleted file mode 100644 index 1bae9d1d..00000000 --- a/tests/expedition/expedition_dir/ship_config.yaml +++ /dev/null @@ -1,25 +0,0 @@ -ship_speed_knots: 10.0 -adcp_config: - num_bins: 40 - max_depth_meter: -1000.0 - period_minutes: 5.0 -argo_float_config: - cycle_days: 10.0 - drift_days: 9.0 - drift_depth_meter: -1000.0 - max_depth_meter: -2000.0 - min_depth_meter: 0.0 - vertical_speed_meter_per_second: -0.1 -ctd_config: - max_depth_meter: -2000.0 - min_depth_meter: -11.0 - stationkeeping_time_minutes: 20.0 -ctd_bgc_config: - max_depth_meter: -2000.0 - min_depth_meter: -11.0 - stationkeeping_time_minutes: 20.0 -drifter_config: - depth_meter: 0.0 - lifetime_minutes: 40320.0 -ship_underwater_st_config: - period_minutes: 5.0 diff --git a/tests/expedition/test_do_expedition.py b/tests/expedition/test_do_expedition.py deleted file mode 100644 index 0dbcd99a..00000000 --- a/tests/expedition/test_do_expedition.py +++ /dev/null @@ -1,13 +0,0 @@ -from pathlib import Path - -from pytest import CaptureFixture - -from virtualship.expedition import do_expedition - - -def test_do_expedition(capfd: CaptureFixture) -> None: - do_expedition("expedition_dir", input_data=Path("expedition_dir/input_data")) - out, _ = capfd.readouterr() - assert "Your expedition has concluded successfully!" in out, ( - "Expedition did not complete successfully." - ) diff --git a/tests/expedition/test_expedition.py b/tests/expedition/test_expedition.py new file mode 100644 index 00000000..78fff2c2 --- /dev/null +++ b/tests/expedition/test_expedition.py @@ -0,0 +1,383 @@ +from datetime import datetime, timedelta +from pathlib import Path +from unittest.mock import patch + +import numpy as np +import pyproj +import pytest +import xarray as xr +from parcels import FieldSet + +from virtualship.errors import InstrumentsConfigError, ScheduleError +from virtualship.models import ( + Expedition, + Location, + Schedule, + Waypoint, +) +from virtualship.models.space_time_region import ( + SpaceTimeRegion, + SpatialRange, + TimeRange, +) +from virtualship.utils import EXPEDITION, _get_expedition, get_example_expedition + +projection = pyproj.Geod(ellps="WGS84") + +expedition_dir = Path("expedition_dir") + + +def test_import_export_expedition(tmpdir) -> None: + out_path = tmpdir.join(EXPEDITION) + + # arbitrary time for testing + base_time = datetime.strptime("1950-01-01", "%Y-%m-%d") + + schedule = Schedule( + waypoints=[ + Waypoint(location=Location(0, 0), time=base_time, instrument=None), + Waypoint( + location=Location(1, 1), + time=base_time + timedelta(hours=1), + instrument=None, + ), + ] + ) + get_expedition = _get_expedition(expedition_dir) + expedition = Expedition( + schedule=schedule, + instruments_config=get_expedition.instruments_config, + ship_config=get_expedition.ship_config, + ) + expedition.to_yaml(out_path) + + expedition2 = Expedition.from_yaml(out_path) + assert expedition == expedition2 + + +def test_verify_schedule() -> None: + schedule = Schedule( + waypoints=[ + Waypoint(location=Location(0, 0), time=datetime(2022, 1, 1, 1, 0, 0)), + Waypoint(location=Location(1, 0), time=datetime(2022, 1, 2, 1, 0, 0)), + ] + ) + + ship_speed_knots = _get_expedition(expedition_dir).ship_config.ship_speed_knots + + schedule.verify(ship_speed_knots, ignore_land_test=True) + + +def test_get_instruments() -> None: + get_expedition = _get_expedition(expedition_dir) + schedule = Schedule( + waypoints=[ + Waypoint(location=Location(0, 0), instrument=["CTD"]), + Waypoint(location=Location(1, 0), instrument=["XBT", "ARGO_FLOAT"]), + Waypoint(location=Location(1, 0), instrument=["CTD"]), + ] + ) + expedition = Expedition( + schedule=schedule, + instruments_config=get_expedition.instruments_config, + ship_config=get_expedition.ship_config, + ) + assert ( + set(instrument.name for instrument in expedition.get_instruments()) + == { + "CTD", + "UNDERWATER_ST", # not added above but underway instruments are auto present from instruments_config in expedition_dir/expedition.yaml + "ADCP", # as above + "ARGO_FLOAT", + "XBT", + } + ) + + +def test_verify_on_land(): + """Test that schedule verification raises error for waypoints on land (0.0 m bathymetry).""" + # bathymetry fieldset with NaNs at specific locations + latitude = np.array([0, 1.0, 2.0]) + longitude = np.array([0, 1.0, 2.0]) + bathymetry = np.array( + [ + [100, 0.0, 100], + [100, 100, 0.0], + [0.0, 100, 100], + ] + ) + + ds_bathymetry = xr.Dataset( + { + "deptho": (("latitude", "longitude"), bathymetry), + }, + coords={ + "latitude": latitude, + "longitude": longitude, + }, + ) + + bathymetry_variables = {"bathymetry": "deptho"} + bathymetry_dimensions = {"lon": "longitude", "lat": "latitude"} + bathymetry_fieldset = FieldSet.from_xarray_dataset( + ds_bathymetry, bathymetry_variables, bathymetry_dimensions + ) + + # waypoints placed in NaN bathy cells + waypoints = [ + Waypoint( + location=Location(0.0, 1.0), time=datetime(2022, 1, 1, 1, 0, 0) + ), # NaN cell + Waypoint( + location=Location(1.0, 2.0), time=datetime(2022, 1, 2, 1, 0, 0) + ), # NaN cell + Waypoint( + location=Location(2.0, 0.0), time=datetime(2022, 1, 3, 1, 0, 0) + ), # NaN cell + ] + + spatial_range = SpatialRange( + minimum_latitude=min(wp.location.lat for wp in waypoints), + maximum_latitude=max(wp.location.lat for wp in waypoints), + minimum_longitude=min(wp.location.lon for wp in waypoints), + maximum_longitude=max(wp.location.lon for wp in waypoints), + ) + time_range = TimeRange( + start_time=min(wp.time for wp in waypoints if wp.time is not None), + end_time=max(wp.time for wp in waypoints if wp.time is not None), + ) + space_time_region = SpaceTimeRegion( + spatial_range=spatial_range, time_range=time_range + ) + schedule = Schedule(waypoints=waypoints, space_time_region=space_time_region) + ship_speed_knots = _get_expedition(expedition_dir).ship_config.ship_speed_knots + + with patch( + "virtualship.models.expedition._get_bathy_data", + return_value=bathymetry_fieldset, + ): + with pytest.raises( + ScheduleError, + match=r"The following waypoint\(s\) throw\(s\) error\(s\):", + ): + schedule.verify( + ship_speed_knots, + ignore_land_test=False, + from_data=None, + ) + + +@pytest.mark.parametrize( + "schedule,check_space_time_region,error,match", + [ + pytest.param( + Schedule(waypoints=[]), + False, + ScheduleError, + "At least one waypoint must be provided.", + id="NoWaypoints", + ), + pytest.param( + Schedule( + waypoints=[ + Waypoint(location=Location(0, 0)), + Waypoint( + location=Location(1, 0), time=datetime(2022, 1, 1, 1, 0, 0) + ), + ] + ), + False, + ScheduleError, + "First waypoint must have a specified time.", + id="FirstWaypointHasTime", + ), + pytest.param( + Schedule( + waypoints=[ + Waypoint( + location=Location(0, 0), time=datetime(2022, 1, 2, 1, 0, 0) + ), + Waypoint(location=Location(0, 0)), + Waypoint( + location=Location(1, 0), time=datetime(2022, 1, 1, 1, 0, 0) + ), + ] + ), + False, + ScheduleError, + "Waypoint\\(s\\) : each waypoint should be timed after all previous waypoints", + id="SequentialWaypoints", + ), + pytest.param( + Schedule( + waypoints=[ + Waypoint( + location=Location(0, 0), time=datetime(2022, 1, 1, 1, 0, 0) + ), + Waypoint( + location=Location(1, 0), time=datetime(2022, 1, 1, 1, 1, 0) + ), + ] + ), + False, + ScheduleError, + "Waypoint planning is not valid: would arrive too late at waypoint number 2...", + id="NotEnoughTime", + ), + pytest.param( + Schedule( + waypoints=[ + Waypoint( + location=Location(0, 0), time=datetime(2022, 1, 1, 1, 0, 0) + ), + Waypoint( + location=Location(1, 0), time=datetime(2022, 1, 2, 1, 1, 0) + ), + ] + ), + True, + ScheduleError, + "space_time_region not found in schedule, please define it to proceed.", + id="NoSpaceTimeRegion", + ), + ], +) +def test_verify_schedule_errors( + schedule: Schedule, check_space_time_region: bool, error, match +) -> None: + expedition = _get_expedition(expedition_dir) + + with pytest.raises(error, match=match): + schedule.verify( + expedition.ship_config.ship_speed_knots, + ignore_land_test=True, + check_space_time_region=check_space_time_region, + ) + + +@pytest.fixture +def expedition(tmp_file): + with open(tmp_file, "w") as file: + file.write(get_example_expedition()) + return Expedition.from_yaml(tmp_file) + + +@pytest.fixture +def expedition_no_xbt(expedition): + for waypoint in expedition.schedule.waypoints: + if waypoint.instrument and any( + instrument.name == "XBT" for instrument in waypoint.instrument + ): + waypoint.instrument = [ + instrument + for instrument in waypoint.instrument + if instrument.name != "XBT" + ] + + return expedition + + +@pytest.fixture +def instruments_config_no_xbt(expedition): + delattr(expedition.instruments_config, "xbt_config") + return expedition.instruments_config + + +@pytest.fixture +def instruments_config_no_ctd(expedition): + delattr(expedition.instruments_config, "ctd_config") + return expedition.instruments_config + + +@pytest.fixture +def instruments_config_no_ctd_bgc(expedition): + delattr(expedition.instruments_config, "ctd_bgc_config") + return expedition.instruments_config + + +@pytest.fixture +def instruments_config_no_argo_float(expedition): + delattr(expedition.instruments_config, "argo_float_config") + return expedition.instruments_config + + +@pytest.fixture +def instruments_config_no_drifter(expedition): + delattr(expedition.instruments_config, "drifter_config") + return expedition.instruments_config + + +@pytest.fixture +def instruments_config_no_adcp(expedition): + delattr(expedition.instruments_config, "adcp_config") + return expedition.instruments_config + + +@pytest.fixture +def instruments_config_no_underwater_st(expedition): + delattr(expedition.instruments_config, "ship_underwater_st_config") + return expedition.instruments_config + + +def test_verify_instruments_config(expedition) -> None: + expedition.instruments_config.verify(expedition) + + +def test_verify_instruments_config_no_instrument(expedition, expedition_no_xbt) -> None: + expedition.instruments_config.verify(expedition_no_xbt) + + +@pytest.mark.parametrize( + "instruments_config_fixture,error,match", + [ + pytest.param( + "instruments_config_no_xbt", + InstrumentsConfigError, + "Expedition includes instrument 'XBT', but instruments_config does not provide configuration for it.", + id="InstrumentsConfigNoXBT", + ), + pytest.param( + "instruments_config_no_ctd", + InstrumentsConfigError, + "Expedition includes instrument 'CTD', but instruments_config does not provide configuration for it.", + id="InstrumentsConfigNoCTD", + ), + pytest.param( + "instruments_config_no_ctd_bgc", + InstrumentsConfigError, + "Expedition includes instrument 'CTD_BGC', but instruments_config does not provide configuration for it.", + id="InstrumentsConfigNoCTD_BGC", + ), + pytest.param( + "instruments_config_no_argo_float", + InstrumentsConfigError, + "Expedition includes instrument 'ARGO_FLOAT', but instruments_config does not provide configuration for it.", + id="InstrumentsConfigNoARGO_FLOAT", + ), + pytest.param( + "instruments_config_no_drifter", + InstrumentsConfigError, + "Expedition includes instrument 'DRIFTER', but instruments_config does not provide configuration for it.", + id="InstrumentsConfigNoDRIFTER", + ), + pytest.param( + "instruments_config_no_adcp", + InstrumentsConfigError, + r"Underway instrument config attribute\(s\) are missing from YAML\. Must be Config object or None\.", + id="InstrumentsConfigNoADCP", + ), + pytest.param( + "instruments_config_no_underwater_st", + InstrumentsConfigError, + r"Underway instrument config attribute\(s\) are missing from YAML\. Must be Config object or None\.", + id="InstrumentsConfigNoUNDERWATER_ST", + ), + ], +) +def test_verify_instruments_config_errors( + request, expedition, instruments_config_fixture, error, match +) -> None: + instruments_config = request.getfixturevalue(instruments_config_fixture) + + with pytest.raises(error, match=match): + instruments_config.verify(expedition) diff --git a/tests/expedition/test_schedule.py b/tests/expedition/test_schedule.py deleted file mode 100644 index f4a8532e..00000000 --- a/tests/expedition/test_schedule.py +++ /dev/null @@ -1,160 +0,0 @@ -from datetime import datetime, timedelta -from pathlib import Path - -import pyproj -import pytest - -from virtualship.errors import ScheduleError -from virtualship.expedition.do_expedition import _load_input_data -from virtualship.models import Location, Schedule, Waypoint -from virtualship.utils import _get_ship_config - -projection = pyproj.Geod(ellps="WGS84") - -expedition_dir = Path("expedition_dir") - - -def test_import_export_schedule(tmpdir) -> None: - out_path = tmpdir.join("schedule.yaml") - - # arbitrary time for testing - base_time = datetime.strptime("1950-01-01", "%Y-%m-%d") - - schedule = Schedule( - waypoints=[ - Waypoint(location=Location(0, 0), time=base_time, instrument=None), - Waypoint( - location=Location(1, 1), - time=base_time + timedelta(hours=1), - instrument=None, - ), - ] - ) - schedule.to_yaml(out_path) - - schedule2 = Schedule.from_yaml(out_path) - assert schedule == schedule2 - - -def test_verify_schedule() -> None: - schedule = Schedule( - waypoints=[ - Waypoint(location=Location(0, 0), time=datetime(2022, 1, 1, 1, 0, 0)), - Waypoint(location=Location(1, 0), time=datetime(2022, 1, 2, 1, 0, 0)), - ] - ) - - ship_config = _get_ship_config(expedition_dir) - - schedule.verify(ship_config.ship_speed_knots, None) - - -def test_get_instruments() -> None: - schedule = Schedule( - waypoints=[ - Waypoint(location=Location(0, 0), instrument=["CTD"]), - Waypoint(location=Location(1, 0), instrument=["XBT", "ARGO_FLOAT"]), - Waypoint(location=Location(1, 0), instrument=["CTD"]), - ] - ) - - assert set(instrument.name for instrument in schedule.get_instruments()) == { - "CTD", - "XBT", - "ARGO_FLOAT", - } - - -@pytest.mark.parametrize( - "schedule,check_space_time_region,error,match", - [ - pytest.param( - Schedule(waypoints=[]), - False, - ScheduleError, - "At least one waypoint must be provided.", - id="NoWaypoints", - ), - pytest.param( - Schedule( - waypoints=[ - Waypoint(location=Location(0, 0)), - Waypoint( - location=Location(1, 0), time=datetime(2022, 1, 1, 1, 0, 0) - ), - ] - ), - False, - ScheduleError, - "First waypoint must have a specified time.", - id="FirstWaypointHasTime", - ), - pytest.param( - Schedule( - waypoints=[ - Waypoint( - location=Location(0, 0), time=datetime(2022, 1, 2, 1, 0, 0) - ), - Waypoint(location=Location(0, 0)), - Waypoint( - location=Location(1, 0), time=datetime(2022, 1, 1, 1, 0, 0) - ), - ] - ), - False, - ScheduleError, - "Waypoint\\(s\\) : each waypoint should be timed after all previous waypoints", - id="SequentialWaypoints", - ), - pytest.param( - Schedule( - waypoints=[ - Waypoint( - location=Location(0, 0), time=datetime(2022, 1, 1, 1, 0, 0) - ), - Waypoint( - location=Location(1, 0), time=datetime(2022, 1, 1, 1, 1, 0) - ), - ] - ), - False, - ScheduleError, - "Waypoint planning is not valid: would arrive too late at waypoint number 2...", - id="NotEnoughTime", - ), - pytest.param( - Schedule( - waypoints=[ - Waypoint( - location=Location(0, 0), time=datetime(2022, 1, 1, 1, 0, 0) - ), - Waypoint( - location=Location(1, 0), time=datetime(2022, 1, 2, 1, 1, 0) - ), - ] - ), - True, - ScheduleError, - "space_time_region not found in schedule, please define it to fetch the data.", - id="NoSpaceTimeRegion", - ), - ], -) -def test_verify_schedule_errors( - schedule: Schedule, check_space_time_region: bool, error, match -) -> None: - ship_config = _get_ship_config(expedition_dir) - - input_data = _load_input_data( - expedition_dir, - schedule, - ship_config, - input_data=Path("expedition_dir/input_data"), - ) - - with pytest.raises(error, match=match): - schedule.verify( - ship_config.ship_speed_knots, - input_data, - check_space_time_region=check_space_time_region, - ) diff --git a/tests/expedition/test_ship_config.py b/tests/expedition/test_ship_config.py deleted file mode 100644 index 6444e985..00000000 --- a/tests/expedition/test_ship_config.py +++ /dev/null @@ -1,126 +0,0 @@ -from pathlib import Path - -import pytest - -from virtualship.errors import ConfigError -from virtualship.models import Schedule, ShipConfig -from virtualship.utils import get_example_config, get_example_schedule - -expedition_dir = Path("expedition_dir") - - -@pytest.fixture -def schedule(tmp_file): - with open(tmp_file, "w") as file: - file.write(get_example_schedule()) - return Schedule.from_yaml(tmp_file) - - -@pytest.fixture -def schedule_no_xbt(schedule): - for waypoint in schedule.waypoints: - if waypoint.instrument and any( - instrument.name == "XBT" for instrument in waypoint.instrument - ): - waypoint.instrument = [ - instrument - for instrument in waypoint.instrument - if instrument.name != "XBT" - ] - - return schedule - - -@pytest.fixture -def ship_config(tmp_file): - with open(tmp_file, "w") as file: - file.write(get_example_config()) - return ShipConfig.from_yaml(tmp_file) - - -@pytest.fixture -def ship_config_no_xbt(ship_config): - delattr(ship_config, "xbt_config") - return ship_config - - -@pytest.fixture -def ship_config_no_ctd(ship_config): - delattr(ship_config, "ctd_config") - return ship_config - - -@pytest.fixture -def ship_config_no_ctd_bgc(ship_config): - delattr(ship_config, "ctd_bgc_config") - return ship_config - - -@pytest.fixture -def ship_config_no_argo_float(ship_config): - delattr(ship_config, "argo_float_config") - return ship_config - - -@pytest.fixture -def ship_config_no_drifter(ship_config): - delattr(ship_config, "drifter_config") - return ship_config - - -def test_import_export_ship_config(ship_config, tmp_file) -> None: - ship_config.to_yaml(tmp_file) - ship_config_2 = ShipConfig.from_yaml(tmp_file) - assert ship_config == ship_config_2 - - -def test_verify_ship_config(ship_config, schedule) -> None: - ship_config.verify(schedule) - - -def test_verify_ship_config_no_instrument(ship_config, schedule_no_xbt) -> None: - ship_config.verify(schedule_no_xbt) - - -@pytest.mark.parametrize( - "ship_config_fixture,error,match", - [ - pytest.param( - "ship_config_no_xbt", - ConfigError, - "Planning has a waypoint with XBT instrument, but configuration does not configure XBT.", - id="ShipConfigNoXBT", - ), - pytest.param( - "ship_config_no_ctd", - ConfigError, - "Planning has a waypoint with CTD instrument, but configuration does not configure CTD.", - id="ShipConfigNoCTD", - ), - pytest.param( - "ship_config_no_ctd_bgc", - ConfigError, - "Planning has a waypoint with CTD_BGC instrument, but configuration does not configure CTD_BGCs.", - id="ShipConfigNoCTD_BGC", - ), - pytest.param( - "ship_config_no_argo_float", - ConfigError, - "Planning has a waypoint with Argo float instrument, but configuration does not configure Argo floats.", - id="ShipConfigNoARGO_FLOAT", - ), - pytest.param( - "ship_config_no_drifter", - ConfigError, - "Planning has a waypoint with drifter instrument, but configuration does not configure drifters.", - id="ShipConfigNoDRIFTER", - ), - ], -) -def test_verify_ship_config_errors( - request, schedule, ship_config_fixture, error, match -) -> None: - ship_config = request.getfixturevalue(ship_config_fixture) - - with pytest.raises(error, match=match): - ship_config.verify(schedule) diff --git a/tests/expedition/test_simulate_schedule.py b/tests/expedition/test_simulate_schedule.py index 9eecd73d..bad8c9ad 100644 --- a/tests/expedition/test_simulate_schedule.py +++ b/tests/expedition/test_simulate_schedule.py @@ -7,7 +7,7 @@ ScheduleProblem, simulate_schedule, ) -from virtualship.models import Location, Schedule, ShipConfig, Waypoint +from virtualship.models import Expedition, Location, Schedule, Waypoint def test_simulate_schedule_feasible() -> None: @@ -15,16 +15,16 @@ def test_simulate_schedule_feasible() -> None: base_time = datetime.strptime("2022-01-01T00:00:00", "%Y-%m-%dT%H:%M:%S") projection = pyproj.Geod(ellps="WGS84") - ship_config = ShipConfig.from_yaml("expedition_dir/ship_config.yaml") - ship_config.ship_speed_knots = 10.0 - schedule = Schedule( + expedition = Expedition.from_yaml("expedition_dir/expedition.yaml") + expedition.ship_config.ship_speed_knots = 10.0 + expedition.schedule = Schedule( waypoints=[ Waypoint(location=Location(0, 0), time=base_time), Waypoint(location=Location(0.01, 0), time=base_time + timedelta(days=1)), ] ) - result = simulate_schedule(projection, ship_config, schedule) + result = simulate_schedule(projection, expedition) assert isinstance(result, ScheduleOk) @@ -34,23 +34,28 @@ def test_simulate_schedule_too_far() -> None: base_time = datetime.strptime("2022-01-01T00:00:00", "%Y-%m-%dT%H:%M:%S") projection = pyproj.Geod(ellps="WGS84") - ship_config = ShipConfig.from_yaml("expedition_dir/ship_config.yaml") - schedule = Schedule( + expedition = Expedition.from_yaml("expedition_dir/expedition.yaml") + expedition.ship_config.ship_speed_knots = 10.0 + expedition.schedule = Schedule( waypoints=[ Waypoint(location=Location(0, 0), time=base_time), Waypoint(location=Location(1.0, 0), time=base_time + timedelta(minutes=1)), ] ) - result = simulate_schedule(projection, ship_config, schedule) + result = simulate_schedule(projection, expedition) assert isinstance(result, ScheduleProblem) def test_time_in_minutes_in_ship_schedule() -> None: """Test whether the pydantic serializer picks up the time *in minutes* in the ship schedule.""" - ship_config = ShipConfig.from_yaml("expedition_dir/ship_config.yaml") - assert ship_config.adcp_config.period == timedelta(minutes=5) - assert ship_config.ctd_config.stationkeeping_time == timedelta(minutes=20) - assert ship_config.ctd_bgc_config.stationkeeping_time == timedelta(minutes=20) - assert ship_config.ship_underwater_st_config.period == timedelta(minutes=5) + instruments_config = Expedition.from_yaml( + "expedition_dir/expedition.yaml" + ).instruments_config + assert instruments_config.adcp_config.period == timedelta(minutes=5) + assert instruments_config.ctd_config.stationkeeping_time == timedelta(minutes=20) + assert instruments_config.ctd_bgc_config.stationkeeping_time == timedelta( + minutes=20 + ) + assert instruments_config.ship_underwater_st_config.period == timedelta(minutes=5) diff --git a/tests/instruments/test_adcp.py b/tests/instruments/test_adcp.py index 569f15a1..a2a5418a 100644 --- a/tests/instruments/test_adcp.py +++ b/tests/instruments/test_adcp.py @@ -6,7 +6,7 @@ import xarray as xr from parcels import FieldSet -from virtualship.instruments.adcp import simulate_adcp +from virtualship.instruments.adcp import ADCPInstrument from virtualship.models import Location, Spacetime @@ -77,17 +77,22 @@ def test_simulate_adcp(tmpdir) -> None: }, ) - # perform simulation + # dummy expedition and directory for ADCPInstrument + class DummyExpedition: + class instruments_config: + class adcp_config: + max_depth_meter = MAX_DEPTH + num_bins = NUM_BINS + + expedition = DummyExpedition() + + from_data = None + + adcp_instrument = ADCPInstrument(expedition, from_data) out_path = tmpdir.join("out.zarr") - simulate_adcp( - fieldset=fieldset, - out_path=out_path, - max_depth=MAX_DEPTH, - min_depth=MIN_DEPTH, - num_bins=NUM_BINS, - sample_points=sample_points, - ) + adcp_instrument.load_input_data = lambda: fieldset + adcp_instrument.simulate(sample_points, out_path) results = xr.open_zarr(out_path) diff --git a/tests/instruments/test_argo_float.py b/tests/instruments/test_argo_float.py index 3eda53ae..cbe25d76 100644 --- a/tests/instruments/test_argo_float.py +++ b/tests/instruments/test_argo_float.py @@ -6,7 +6,7 @@ import xarray as xr from parcels import FieldSet -from virtualship.instruments.argo_float import ArgoFloat, simulate_argo_floats +from virtualship.instruments.argo_float import ArgoFloat, ArgoFloatInstrument from virtualship.models import Location, Spacetime @@ -53,16 +53,18 @@ def test_simulate_argo_floats(tmpdir) -> None: ) ] - # perform simulation + # dummy expedition and directory for ArgoFloatInstrument + class DummyExpedition: + pass + + expedition = DummyExpedition() + from_data = None + + argo_instrument = ArgoFloatInstrument(expedition, from_data) out_path = tmpdir.join("out.zarr") - simulate_argo_floats( - fieldset=fieldset, - out_path=out_path, - argo_floats=argo_floats, - outputdt=timedelta(minutes=5), - endtime=None, - ) + argo_instrument.load_input_data = lambda: fieldset + argo_instrument.simulate(argo_floats, out_path) # test if output is as expected results = xr.open_zarr(out_path) diff --git a/tests/instruments/test_base.py b/tests/instruments/test_base.py new file mode 100644 index 00000000..29d319ba --- /dev/null +++ b/tests/instruments/test_base.py @@ -0,0 +1,121 @@ +from unittest.mock import MagicMock, patch + +from virtualship.instruments.base import Instrument +from virtualship.instruments.types import InstrumentType +from virtualship.utils import get_instrument_class + + +def test_all_instruments_have_instrument_class(): + for instrument in InstrumentType: + instrument_class = get_instrument_class(instrument) + assert instrument_class is not None, f"No instrument_class for {instrument}" + + +class DummyInstrument(Instrument): + """Minimal concrete Instrument for testing.""" + + def simulate(self, data_dir, measurements, out_path): + """Dummy simulate implementation for test.""" + self.simulate_called = True + + +@patch("virtualship.instruments.base.FieldSet") +@patch( + "virtualship.instruments.base._select_product_id", return_value="dummy_product_id" +) +@patch("virtualship.instruments.base.copernicusmarine") +def test_load_input_data(mock_copernicusmarine, mock_select_product_id, mock_FieldSet): + """Test Instrument.load_input_data with mocks.""" + mock_fieldset = MagicMock() + mock_FieldSet.from_netcdf.return_value = mock_fieldset + mock_FieldSet.from_xarray_dataset.return_value = mock_fieldset + mock_fieldset.gridset.grids = [MagicMock(negate_depth=MagicMock())] + mock_fieldset.__getitem__.side_effect = lambda k: MagicMock() + mock_copernicusmarine.open_dataset.return_value = MagicMock() + dummy = DummyInstrument( + expedition=MagicMock(schedule=MagicMock(space_time_region=MagicMock())), + variables={"A": "a"}, + add_bathymetry=False, + allow_time_extrapolation=False, + verbose_progress=False, + from_data=None, + ) + fieldset = dummy.load_input_data() + assert mock_FieldSet.from_xarray_dataset.called + assert fieldset == mock_fieldset + assert fieldset == mock_fieldset + + +def test_execute_calls_simulate(monkeypatch): + dummy = DummyInstrument( + expedition=MagicMock(schedule=MagicMock(space_time_region=MagicMock())), + variables={"A": "a"}, + add_bathymetry=False, + allow_time_extrapolation=False, + verbose_progress=True, + from_data=None, + ) + dummy.simulate = MagicMock() + dummy.execute([1, 2, 3], "/tmp/out") + dummy.simulate.assert_called_once() + + +def test_get_spec_value_buffer_and_limit(): + dummy = DummyInstrument( + expedition=MagicMock(schedule=MagicMock(space_time_region=MagicMock())), + variables={"A": "a"}, + add_bathymetry=False, + allow_time_extrapolation=False, + verbose_progress=False, + spacetime_buffer_size={"latlon": 5.0}, + limit_spec={"depth_min": 10.0}, + from_data=None, + ) + assert dummy._get_spec_value("buffer", "latlon", 0.0) == 5.0 + assert dummy._get_spec_value("limit", "depth_min", None) == 10.0 + assert dummy._get_spec_value("buffer", "missing", 42) == 42 + + +def test_generate_fieldset_combines_fields(monkeypatch): + dummy = DummyInstrument( + expedition=MagicMock(schedule=MagicMock(space_time_region=MagicMock())), + variables={"A": "a", "B": "b"}, + add_bathymetry=False, + allow_time_extrapolation=False, + verbose_progress=False, + from_data=None, + ) + dummy.from_data = None + + monkeypatch.setattr(dummy, "_get_copernicus_ds", lambda physical, var: MagicMock()) + + fs_A = MagicMock() + fs_B = MagicMock() + fs_B.B = MagicMock() + monkeypatch.setattr( + "virtualship.instruments.base.FieldSet.from_xarray_dataset", + lambda ds, varmap, dims, mesh=None: fs_A if "A" in varmap else fs_B, + ) + monkeypatch.setattr(fs_A, "add_field", MagicMock()) + dummy._generate_fieldset() + fs_A.add_field.assert_called_once_with(fs_B.B) + + +def test_load_input_data_error(monkeypatch): + dummy = DummyInstrument( + expedition=MagicMock(schedule=MagicMock(space_time_region=MagicMock())), + variables={"A": "a"}, + add_bathymetry=False, + allow_time_extrapolation=False, + verbose_progress=False, + from_data=None, + ) + monkeypatch.setattr( + dummy, "_generate_fieldset", lambda: (_ for _ in ()).throw(Exception("fail")) + ) + import virtualship.errors + + try: + dummy.load_input_data() + except virtualship.errors.CopernicusCatalogueError as e: + assert "Failed to load input data" in str(e) diff --git a/tests/instruments/test_ctd.py b/tests/instruments/test_ctd.py index 14e0a276..fff5fc4f 100644 --- a/tests/instruments/test_ctd.py +++ b/tests/instruments/test_ctd.py @@ -5,13 +5,12 @@ """ import datetime -from datetime import timedelta import numpy as np import xarray as xr from parcels import Field, FieldSet -from virtualship.instruments.ctd import CTD, simulate_ctd +from virtualship.instruments.ctd import CTD, CTDInstrument from virtualship.models import Location, Spacetime @@ -102,15 +101,42 @@ def test_simulate_ctds(tmpdir) -> None: ) fieldset.add_field(Field("bathymetry", [-1000], lon=0, lat=0)) - # perform simulation + # dummy expedition and directory for CTDInstrument + class DummyExpedition: + class schedule: + class space_time_region: + time_range = type( + "TimeRange", + (), + { + "start_time": fieldset.T.grid.time_origin.fulltime( + fieldset.T.grid.time_full[0] + ), + "end_time": fieldset.T.grid.time_origin.fulltime( + fieldset.T.grid.time_full[-1] + ), + }, + )() + spatial_range = type( + "SpatialRange", + (), + { + "minimum_longitude": 0, + "maximum_longitude": 1, + "minimum_latitude": 0, + "maximum_latitude": 1, + }, + )() + + expedition = DummyExpedition() + + from_data = None + + ctd_instrument = CTDInstrument(expedition, from_data) out_path = tmpdir.join("out.zarr") - simulate_ctd( - ctds=ctds, - fieldset=fieldset, - out_path=out_path, - outputdt=timedelta(seconds=10), - ) + ctd_instrument.load_input_data = lambda: fieldset + ctd_instrument.simulate(ctds, out_path) # test if output is as expected results = xr.open_zarr(out_path) @@ -132,6 +158,7 @@ def test_simulate_ctds(tmpdir) -> None: for var in ["salinity", "temperature", "lat", "lon"]: obs_value = obs[var].values.item() exp_value = exp[var] + assert np.isclose(obs_value, exp_value), ( f"Observation incorrect {ctd_i=} {loc=} {var=} {obs_value=} {exp_value=}." ) diff --git a/tests/instruments/test_ctd_bgc.py b/tests/instruments/test_ctd_bgc.py index c1213884..00f30077 100644 --- a/tests/instruments/test_ctd_bgc.py +++ b/tests/instruments/test_ctd_bgc.py @@ -5,13 +5,12 @@ """ import datetime -from datetime import timedelta import numpy as np import xarray as xr from parcels import Field, FieldSet -from virtualship.instruments.ctd_bgc import CTD_BGC, simulate_ctd_bgc +from virtualship.instruments.ctd_bgc import CTD_BGC, CTD_BGCInstrument from virtualship.models import Location, Spacetime @@ -163,15 +162,18 @@ def test_simulate_ctd_bgcs(tmpdir) -> None: ) fieldset.add_field(Field("bathymetry", [-1000], lon=0, lat=0)) - # perform simulation + # dummy expedition and directory for CTD_BGCInstrument + class DummyExpedition: + pass + + expedition = DummyExpedition() + from_data = None + + ctd_bgc_instrument = CTD_BGCInstrument(expedition, from_data) out_path = tmpdir.join("out.zarr") - simulate_ctd_bgc( - ctd_bgcs=ctd_bgcs, - fieldset=fieldset, - out_path=out_path, - outputdt=timedelta(seconds=10), - ) + ctd_bgc_instrument.load_input_data = lambda: fieldset + ctd_bgc_instrument.simulate(ctd_bgcs, out_path) # test if output is as expected results = xr.open_zarr(out_path) diff --git a/tests/instruments/test_drifter.py b/tests/instruments/test_drifter.py index ae230a87..9253c1a8 100644 --- a/tests/instruments/test_drifter.py +++ b/tests/instruments/test_drifter.py @@ -6,7 +6,7 @@ import xarray as xr from parcels import FieldSet -from virtualship.instruments.drifter import Drifter, simulate_drifters +from virtualship.instruments.drifter import Drifter, DrifterInstrument from virtualship.models import Location, Spacetime @@ -52,17 +52,19 @@ def test_simulate_drifters(tmpdir) -> None: ), ] - # perform simulation + # dummy expedition and directory for DrifterInstrument + class DummyExpedition: + pass + + expedition = DummyExpedition() + + from_data = None + + drifter_instrument = DrifterInstrument(expedition, from_data) out_path = tmpdir.join("out.zarr") - simulate_drifters( - fieldset=fieldset, - out_path=out_path, - drifters=drifters, - outputdt=datetime.timedelta(hours=1), - dt=datetime.timedelta(minutes=5), - endtime=None, - ) + drifter_instrument.load_input_data = lambda: fieldset + drifter_instrument.simulate(drifters, out_path) # test if output is as expected results = xr.open_zarr(out_path) diff --git a/tests/instruments/test_ship_underwater_st.py b/tests/instruments/test_ship_underwater_st.py index 9d44ee6d..e7ca18d1 100644 --- a/tests/instruments/test_ship_underwater_st.py +++ b/tests/instruments/test_ship_underwater_st.py @@ -6,14 +6,11 @@ import xarray as xr from parcels import FieldSet -from virtualship.instruments.ship_underwater_st import simulate_ship_underwater_st +from virtualship.instruments.ship_underwater_st import Underwater_STInstrument from virtualship.models import Location, Spacetime def test_simulate_ship_underwater_st(tmpdir) -> None: - # depth at which the sampling will be done - DEPTH = -2 - # arbitrary time offset for the dummy fieldset base_time = datetime.datetime.strptime("1950-01-01", "%Y-%m-%d") @@ -70,15 +67,20 @@ def test_simulate_ship_underwater_st(tmpdir) -> None: }, ) - # perform simulation + # dummy expedition and directory for Underwater_STInstrument + class DummyExpedition: + pass + + expedition = DummyExpedition() + + from_data = None + + st_instrument = Underwater_STInstrument(expedition, from_data) out_path = tmpdir.join("out.zarr") - simulate_ship_underwater_st( - fieldset=fieldset, - out_path=out_path, - depth=DEPTH, - sample_points=sample_points, - ) + st_instrument.load_input_data = lambda: fieldset + # The instrument expects measurements as sample_points + st_instrument.simulate(sample_points, out_path) # test if output is as expected results = xr.open_zarr(out_path) diff --git a/tests/instruments/test_xbt.py b/tests/instruments/test_xbt.py index 97e33ade..d218025a 100644 --- a/tests/instruments/test_xbt.py +++ b/tests/instruments/test_xbt.py @@ -5,13 +5,12 @@ """ import datetime -from datetime import timedelta import numpy as np import xarray as xr from parcels import Field, FieldSet -from virtualship.instruments.xbt import XBT, simulate_xbt +from virtualship.instruments.xbt import XBT, XBTInstrument from virtualship.models import Location, Spacetime @@ -96,15 +95,19 @@ def test_simulate_xbts(tmpdir) -> None: ) fieldset.add_field(Field("bathymetry", [-1000], lon=0, lat=0)) - # perform simulation + # dummy expedition and directory for XBTInstrument + class DummyExpedition: + pass + + expedition = DummyExpedition() + + from_data = None + + xbt_instrument = XBTInstrument(expedition, from_data) out_path = tmpdir.join("out.zarr") - simulate_xbt( - xbts=xbts, - fieldset=fieldset, - out_path=out_path, - outputdt=timedelta(seconds=10), - ) + xbt_instrument.load_input_data = lambda: fieldset + xbt_instrument.simulate(xbts, out_path) # test if output is as expected results = xr.open_zarr(out_path) diff --git a/tests/test_mfp_to_yaml.py b/tests/test_mfp_to_yaml.py index d242d30a..4eab16c2 100644 --- a/tests/test_mfp_to_yaml.py +++ b/tests/test_mfp_to_yaml.py @@ -3,7 +3,7 @@ import pandas as pd import pytest -from virtualship.models import Schedule +from virtualship.models import Expedition from virtualship.utils import mfp_to_yaml @@ -88,7 +88,7 @@ def test_mfp_to_yaml_success(request, fixture_name, tmp_path): """Test that mfp_to_yaml correctly processes a valid MFP file.""" valid_mfp_file = request.getfixturevalue(fixture_name) - yaml_output_path = tmp_path / "schedule.yaml" + yaml_output_path = tmp_path / "expedition.yaml" # Run function (No need to mock open() for YAML, real file is created) mfp_to_yaml(valid_mfp_file, yaml_output_path) @@ -97,9 +97,9 @@ def test_mfp_to_yaml_success(request, fixture_name, tmp_path): assert yaml_output_path.exists() # Load YAML and validate contents - data = Schedule.from_yaml(yaml_output_path) + data = Expedition.from_yaml(yaml_output_path) - assert len(data.waypoints) == 3 + assert len(data.schedule.waypoints) == 3 @pytest.mark.parametrize( @@ -138,7 +138,7 @@ def test_mfp_to_yaml_exceptions(request, fixture_name, error, match, tmp_path): """Test that mfp_to_yaml raises an error when input file is not valid.""" fixture = request.getfixturevalue(fixture_name) - yaml_output_path = tmp_path / "schedule.yaml" + yaml_output_path = tmp_path / "expedition.yaml" with pytest.raises(error, match=match): mfp_to_yaml(fixture, yaml_output_path) @@ -146,7 +146,7 @@ def test_mfp_to_yaml_exceptions(request, fixture_name, error, match, tmp_path): def test_mfp_to_yaml_extra_headers(unexpected_header_mfp_file, tmp_path): """Test that mfp_to_yaml prints a warning when extra columns are found.""" - yaml_output_path = tmp_path / "schedule.yaml" + yaml_output_path = tmp_path / "expedition.yaml" with pytest.warns(UserWarning, match="Found additional unexpected columns.*"): mfp_to_yaml(unexpected_header_mfp_file, yaml_output_path) diff --git a/tests/test_utils.py b/tests/test_utils.py index 4c6db8fc..8bd2338e 100644 --- a/tests/test_utils.py +++ b/tests/test_utils.py @@ -1,26 +1,264 @@ -from virtualship.models import Schedule, ShipConfig -from virtualship.utils import get_example_config, get_example_schedule +from pathlib import Path +import numpy as np +import pytest +import xarray as xr +from parcels import FieldSet -def test_get_example_config(): - assert len(get_example_config()) > 0 +import virtualship.utils +from virtualship.models.expedition import Expedition +from virtualship.utils import ( + _find_nc_file_with_variable, + _get_bathy_data, + _select_product_id, + _start_end_in_product_timerange, + add_dummy_UV, + get_example_expedition, +) -def test_get_example_schedule(): - assert len(get_example_schedule()) > 0 +@pytest.fixture +def expedition(tmp_file): + with open(tmp_file, "w") as file: + file.write(get_example_expedition()) + return Expedition.from_yaml(tmp_file) -def test_valid_example_config(tmp_path): - path = tmp_path / "test.yaml" - with open(path, "w") as file: - file.write(get_example_config()) +@pytest.fixture +def dummy_spatial_range(): + class DummySpatialRange: + minimum_longitude = 0 + maximum_longitude = 1 + minimum_latitude = 0 + maximum_latitude = 1 + minimum_depth = 0 + maximum_depth = 4 + + return DummySpatialRange() + + +@pytest.fixture +def dummy_time_range(): + class DummyTimeRange: + start_time = "2020-01-01" + end_time = "2020-01-02" + + return DummyTimeRange() + + +@pytest.fixture +def dummy_space_time_region(dummy_spatial_range, dummy_time_range): + class DummySpaceTimeRegion: + spatial_range = dummy_spatial_range + time_range = dummy_time_range + + return DummySpaceTimeRegion() + + +@pytest.fixture +def dummy_instrument(): + class DummyInstrument: + pass + + return DummyInstrument() + + +@pytest.fixture +def copernicus_no_download(monkeypatch): + """Mock the copernicusmarine `subset` and `open_dataset` functions, approximating the reanalysis products.""" - ShipConfig.from_yaml(path) + # mock for copernicusmarine.subset + def fake_download(output_filename, output_directory, **_): + Path(output_directory).joinpath(output_filename).touch() + def fake_open_dataset(*args, **kwargs): + return xr.Dataset( + coords={ + "time": ( + "time", + [ + np.datetime64("1993-01-01"), + np.datetime64("2022-01-01"), + ], # mock up rough renanalysis period + ) + } + ) -def test_valid_example_schedule(tmp_path): + monkeypatch.setattr("virtualship.utils.copernicusmarine.subset", fake_download) + monkeypatch.setattr( + "virtualship.utils.copernicusmarine.open_dataset", fake_open_dataset + ) + yield + + +def test_get_example_expedition(): + assert len(get_example_expedition()) > 0 + + +def test_valid_example_expedition(tmp_path): path = tmp_path / "test.yaml" with open(path, "w") as file: - file.write(get_example_schedule()) + file.write(get_example_expedition()) + + Expedition.from_yaml(path) + + +def test_instrument_registry_updates(dummy_instrument): + from virtualship import utils + + utils.register_instrument("DUMMY_TYPE")(dummy_instrument) + + assert utils.INSTRUMENT_CLASS_MAP["DUMMY_TYPE"] is dummy_instrument + + +def test_add_dummy_UV_adds_fields(): + fieldset = FieldSet.from_data({"T": 1}, {"lon": 0, "lat": 0}, mesh="spherical") + fieldset.__dict__.pop("U", None) + fieldset.__dict__.pop("V", None) + + # should not have U or V fields initially + assert "U" not in fieldset.__dict__ + assert "V" not in fieldset.__dict__ + + add_dummy_UV(fieldset) + + # now U and V should be present + assert "U" in fieldset.__dict__ + assert "V" in fieldset.__dict__ + + # should not raise error if U and V already present + add_dummy_UV(fieldset) + + +@pytest.mark.usefixtures("copernicus_no_download") +def test_select_product_id(expedition): + """Should return the physical reanalysis product id via the timings prescribed in the static schedule.yaml file.""" + result = _select_product_id( + physical=True, + schedule_start=expedition.schedule.space_time_region.time_range.start_time, + schedule_end=expedition.schedule.space_time_region.time_range.end_time, + username="test", + password="test", + ) + assert result == "cmems_mod_glo_phy_my_0.083deg_P1D-m" + + +@pytest.mark.usefixtures("copernicus_no_download") +def test_start_end_in_product_timerange(expedition): + """Should return True for valid range ass determined by the static schedule.yaml file.""" + assert _start_end_in_product_timerange( + selected_id="cmems_mod_glo_phy_my_0.083deg_P1D-m", + schedule_start=expedition.schedule.space_time_region.time_range.start_time, + schedule_end=expedition.schedule.space_time_region.time_range.end_time, + username="test", + password="test", + ) + + +def test_get_bathy_data_local(tmp_path, dummy_space_time_region): + """Test that _get_bathy_data returns a FieldSet when given a local directory for --from-data.""" + # dummy .nc file with 'deptho' variable + data = np.array([[1, 2], [3, 4]]) + ds = xr.Dataset( + { + "deptho": (("x", "y"), data), + }, + coords={ + "longitude": (("x", "y"), np.array([[0, 1], [0, 1]])), + "latitude": (("x", "y"), np.array([[0, 0], [1, 1]])), + }, + ) + nc_path = tmp_path / "bathymetry/dummy.nc" + nc_path.parent.mkdir(parents=True, exist_ok=True) + ds.to_netcdf(nc_path) + + # should return a FieldSet + fieldset = _get_bathy_data(dummy_space_time_region, from_data=tmp_path) + assert isinstance(fieldset, FieldSet) + assert hasattr(fieldset, "bathymetry") + assert np.allclose(fieldset.bathymetry.data, data) + + +def test_get_bathy_data_copernicusmarine(monkeypatch, dummy_space_time_region): + """Test that _get_bathy_data calls copernicusmarine by default.""" + + def dummy_copernicusmarine(*args, **kwargs): + raise RuntimeError("copernicusmarine called") + + monkeypatch.setattr( + virtualship.utils.copernicusmarine, "open_dataset", dummy_copernicusmarine + ) + + try: + _get_bathy_data(dummy_space_time_region) + except RuntimeError as e: + assert "copernicusmarine called" in str(e) + + +def test_find_nc_file_with_variable_substring(tmp_path): + # dummy .nc file with variable 'uo_glor' (possible for CMS products to have similar suffixes...) + data = np.array([[1, 2], [3, 4]]) + ds = xr.Dataset( + { + "uo_glor": (("x", "y"), data), + }, + coords={ + "longitude": (("x", "y"), np.array([[0, 1], [0, 1]])), + "latitude": (("x", "y"), np.array([[0, 0], [1, 1]])), + }, + ) + nc_path = tmp_path / "test.nc" + ds.to_netcdf(nc_path) + + # should find 'uo_glor' when searching for 'uo' + result = _find_nc_file_with_variable(tmp_path, "uo") + assert result is not None + filename, found_var = result + assert filename == "test.nc" + assert found_var == "uo_glor" + + +def test_data_dir_and_filename_compliance(): + """ + Test compliance of data directory structure and filename patterns as sought by base.py methods relative to as is described in the docs. + + Test that: + - Instrument._generate_fieldset and _get_bathy_data use the expected subdirectory names. + - The expected filename date pattern (YYYY_MM_DD) is used in _find_files_in_timerange. + + + ('phys', 'bgc', 'bathymetry') for local data loading, as required by documentation. + + To avoid drift between code implementation and what expectations are laid out in the docs. + """ + base_path = Path(__file__).parent.parent / "src/virtualship/instruments/base.py" + utils_path = Path(__file__).parent.parent / "src/virtualship/utils.py" + + base_code = base_path.read_text(encoding="utf-8") + utils_code = utils_path.read_text(encoding="utf-8") + + # Check for phys and bgc in Instrument._generate_fieldset + assert 'self.from_data.joinpath("phys")' in base_code, ( + "Expected 'phys' subdirectory not found in Instrument._generate_fieldset. This indicates a drift between docs and implementation." + ) + assert 'self.from_data.joinpath("bgc")' in base_code, ( + "Expected 'bgc' subdirectory not found in Instrument._generate_fieldset. This indicates a drift between docs and implementation." + ) + + # Check for bathymetry in _get_bathy_data + assert 'from_data.joinpath("bathymetry")' in utils_code, ( + "Expected 'bathymetry' subdirectory not found in _get_bathy_data. This indicates a drift between docs and implementation." + ) + + # Check for date_pattern in _find_files_in_timerange + assert 'date_pattern=r"\\d{4}_\\d{2}_\\d{2}"' in utils_code, ( + "Expected date_pattern r'\\d{4}_\\d{2}_\\d{2}' not found in _find_files_in_timerange. This indicates a drift between docs and implementation." + ) - Schedule.from_yaml(path) + # Check for P1D and P1M in t_resolution logic + assert 'if all("P1D" in s for s in all_files):' in utils_code, ( + "Expected check for 'P1D' in all_files not found in _find_files_in_timerange. This indicates a drift between docs and implementation." + ) + assert 'elif all("P1M" in s for s in all_files):' in utils_code, ( + "Expected check for 'P1M' in all_files not found in _find_files_in_timerange. This indicates a drift between docs and implementation." + )