Skip to content

Commit 368f788

Browse files
iurytpre-commit-ci[bot]VeckoTheGeckoerikvansebille
authored
Add fetch command to download expedition data based on space_time_region from schedule.yaml (#83)
* add bbox and time range * add AreaOfInterest class * export AreaOfInterest * add AreaOfInterest to Schedule class * fetch function for downloading data based on the area of interest * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * import copernicusmarine * import datetime * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix typo * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update src/virtualship/cli/commands.py Co-authored-by: Vecko <[email protected]> * Update src/virtualship/cli/commands.py Co-authored-by: Vecko <[email protected]> * patch `path` varname * update _get_ship_config and _get_schedule * Add credential config * Add note to obtain credentials * Add copernicusmarine dep * Patch types start_time and end_time already datetime objects * Update user messages in `virtualship fetch` * Adding minimum and maximum depth to area_of_interest * Setting None as default option for drifter endtime * Removing redundant import in text_drifter * Pin copernicusmarine < 2 * Validate domains for area_of_interest * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make depth optional in AreaOfInterest * Make AreaOfInterest optional in Schedule * Rename AreaOfInterest to SpaceTimeRegion * Rename area_of_interest.py file * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add fetch utility functions and tests * Patch attr name * pointing the right function to make it work * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove temporary code * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * ask for username and password if not provided * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * creates function that makes a sha256 from a dictionary * create a data folder based on the hash from area_of_interest * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * insert blank lines * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update hash to use _fetch utils * Avoid mixing matching creds * Download caching feature * Update input data loading with new data folder structure * Handle download_cleanup on wrong credentials * Copy schedule to download folde * Patch download command * Add `input_data` param for testing * Add tests * Add fetch test * Pin copernicusmarine >= 2 Fixes #90 * RTD explicit config key https://about.readthedocs.com/blog/2024/12/deprecate-config-files-without-sphinx-or-mkdocs-config/ * Delete download_data script * Update pyproject.toml * Improve filename to hash conversion * Rename to assert_complete_download * Error message when area of interest isn't defined * Add area of interest hash salting * Update 'area of interest' to 'space-time region' throughout * Avoid circular import * virtualship help documentation * Update help messages --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Vecko <[email protected]> Co-authored-by: Erik van Sebille <[email protected]>
1 parent 149e8bf commit 368f788

File tree

22 files changed

+761
-219
lines changed

22 files changed

+761
-219
lines changed

.readthedocs.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
33

44
version: 2
5-
5+
sphinx:
6+
configuration: docs/conf.py
67
build:
78
os: ubuntu-22.04
89
tools:

environment.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ dependencies:
1111
- pydantic >=2, <3
1212
- pip
1313
- pyyaml
14+
- copernicusmarine >= 2
1415

1516
# linting
1617
- pre-commit

pyproject.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ dependencies = [
3333
"numpy >=1, < 2",
3434
"pydantic >=2, <3",
3535
"PyYAML",
36+
"copernicusmarine >= 2",
3637
]
3738

3839
[project.urls]
@@ -104,6 +105,8 @@ ignore = [
104105
"D212",
105106
# one-blank-line-before-class
106107
"D203",
108+
# First line of docstring should be in imperative mood
109+
"D401",
107110

108111
# TODO: Remove later
109112
"D100", "D103"

scripts/download_data.py

Lines changed: 0 additions & 137 deletions
This file was deleted.

src/virtualship/cli/_creds.py

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
from __future__ import annotations
2+
3+
from pathlib import Path
4+
5+
import click
6+
import pydantic
7+
import yaml
8+
9+
CREDENTIALS_FILE = "credentials.yaml"
10+
11+
12+
class CredentialFileError(Exception):
13+
"""Exception raised for errors in the input file format."""
14+
15+
pass
16+
17+
18+
class Credentials(pydantic.BaseModel):
19+
"""Credentials to be used in `virtualship fetch` command."""
20+
21+
COPERNICUS_USERNAME: str
22+
COPERNICUS_PASSWORD: str
23+
24+
@classmethod
25+
def from_yaml(cls, path: str | Path) -> Credentials:
26+
"""
27+
Load credentials from a yaml file.
28+
29+
:param path: Path to the file to load from.
30+
:returns Credentials: The credentials.
31+
"""
32+
with open(path) as file:
33+
data = yaml.safe_load(file)
34+
35+
if not isinstance(data, dict):
36+
raise CredentialFileError("Credential file is of an invalid format.")
37+
38+
return cls(**data)
39+
40+
def dump(self) -> str:
41+
"""
42+
Dump credentials to a yaml string.
43+
44+
:param creds: The credentials to dump.
45+
:returns str: The yaml string.
46+
"""
47+
return yaml.safe_dump(self.model_dump())
48+
49+
def to_yaml(self, path: str | Path) -> None:
50+
"""
51+
Write credentials to a yaml file.
52+
53+
:param path: Path to the file to write to.
54+
"""
55+
with open(path, "w") as file:
56+
file.write(self.dump())
57+
58+
59+
def get_dummy_credentials_yaml() -> str:
60+
return (
61+
Credentials(
62+
COPERNICUS_USERNAME="my_username", COPERNICUS_PASSWORD="my_password"
63+
)
64+
.dump()
65+
.strip()
66+
)
67+
68+
69+
def get_credentials_flow(
70+
username: str | None, password: str | None, creds_path: Path
71+
) -> tuple[str, str]:
72+
"""
73+
Execute flow of getting credentials for use in the `fetch` command.
74+
75+
- If username and password are provided via CLI, use them (ignore the credentials file if exists).
76+
- If username and password are not provided, try to load them from the credentials file.
77+
- If no credentials are provided, print a message on how to make credentials file and prompt for credentials.
78+
79+
:param username: The username provided via CLI.
80+
:param password: The password provided via CLI.
81+
:param creds_path: The path to the credentials file.
82+
"""
83+
if username and password:
84+
if creds_path.exists():
85+
click.echo(
86+
f"Credentials file exists at '{creds_path}', but username and password are already provided. Ignoring credentials file."
87+
)
88+
return username, password
89+
90+
try:
91+
creds = Credentials.from_yaml(creds_path)
92+
click.echo(f"Loaded credentials from '{creds_path}'.")
93+
return creds.COPERNICUS_USERNAME, creds.COPERNICUS_PASSWORD
94+
except FileNotFoundError:
95+
msg = f"""Credentials not provided. Credentials can be obtained from https://data.marine.copernicus.eu/register. Either pass in via `--username` and `--password` arguments, or via config file at '{creds_path}'. Config file should be YAML along following format:
96+
### {creds_path}
97+
98+
{get_dummy_credentials_yaml().strip()}
99+
100+
###
101+
102+
Prompting for credentials instead...
103+
"""
104+
click.echo(msg)
105+
username = click.prompt("username")
106+
password = click.prompt("password", hide_input=True)
107+
return username, password

0 commit comments

Comments
 (0)