Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 59 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

The command line for [Source Cooperative](https://source.coop). Use it to:

- **Get data credentials.** Sign in through your browser once. AWS tools then get temporary S3 credentials for the Source Cooperative data proxy, and the CLI renews them when they expire.
- **Get data credentials.** Sign in through your browser once. AWS tools then get temporary S3 credentials for the Source Cooperative data proxy, and the CLI renews them when they expire. `exec` runs any S3 tool against the proxy, signed in or not.
- **Manage source.coop.** List, view, create, edit and delete products, members and service accounts, and create and edit organizations. The source.coop API applies the same rules as the web UI.

All commands work the same way:
Expand All @@ -31,6 +31,9 @@ source-coop product create

# after you add the AWS profile (below)
aws s3 ls s3://my-org/my-data --profile source-coop

# or with no profile at all
source-coop exec -- aws s3 ls s3://my-org/my-data/
```

`source-coop --help` lists the commands. `source-coop <command> --help` lists the flags of a command, with examples.
Expand Down Expand Up @@ -289,6 +292,50 @@ aws_session_token = ...
> [!TIP]
> These credentials are temporary. When they expire, run the command again. Each run adds a new section, and AWS uses the last one, so remove old sections occasionally. We recommend `credential_process` in `~/.aws/config` instead.

### Run a tool with `exec`

`exec` runs a command with the data proxy as its S3 endpoint and your credentials in its environment, as `aws-vault exec` does for AWS. It needs no profile, and works for tools that can't use one: GDAL (QGIS, rasterio, geopandas), Rust's `object_store` (obstore, Polars) and s5cmd.

```bash
source-coop exec -- aws s3 ls s3://my-org/my-data/
source-coop exec -- aws s3 cp s3://my-org/my-data/data.parquet .
source-coop exec -- s5cmd cp 's3://my-org/my-data/*.parquet' ./local/
source-coop exec -- gdalinfo /vsis3/my-org/my-data/image.tif
source-coop exec -- python script.py
```

Everything after `--` is the command, flags included. The command's exit code is passed on.

![`exec`: the AWS CLI and s5cmd with your credentials and the proxy, no profile](docs/demos/exec.svg)

The command gets:

- **Credentials:** `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` and `AWS_CREDENTIAL_EXPIRATION`.
- **The proxy as its S3 endpoint**, under each tool's name for it: `AWS_ENDPOINT_URL_S3` (the AWS CLI v2.13+ and SDKs), `S3_ENDPOINT_URL` (s5cmd), `AWS_ENDPOINT` (`object_store`, with `AWS_ALLOW_HTTP` for a plain-HTTP proxy), and `AWS_S3_ENDPOINT`, `AWS_HTTPS` and `AWS_VIRTUAL_HOSTING=FALSE` (GDAL's `/vsis3/`). Only S3 goes to the proxy: calls a script makes to other AWS services keep their own endpoints.
- **A region**, `AWS_REGION` and `AWS_DEFAULT_REGION` set to `us-west-2`, only if neither is set already, as signing needs one.

`AWS_PROFILE`, `AWS_DEFAULT_PROFILE`, `AWS_IGNORE_CONFIGURED_ENDPOINT_URLS` and `AWS_ENDPOINT_URL` are removed, so a profile can't lend other credentials and no other endpoint can take S3 requests away from the proxy. (`object_store` reads a general `AWS_ENDPOINT_URL` as its S3 endpoint.) Signed in, `AWS_NO_SIGN_REQUEST` and `AWS_SKIP_SIGNATURE` are removed too, so GDAL and `object_store` sign even if you exported them for public buckets. `--role-arn` and `--proxy-url` (or `SOURCE_ROLE_ARN` and `SOURCE_PROXY_URL`) select the role and environment, as for `creds`.

The credentials are fetched once, when the command starts. Cached ones are used if they have at least 15 minutes left. Otherwise `exec` gets new ones. A command still running when they expire (an hour after they're fetched, unless you signed in with `--duration`) must be run again. For long jobs, use the [`credential_process` profile](#reading-data-with-aws-tools), which renews them.

**Not signed in,** `exec` still runs the command, unsigned, so it can read public products. It says so on stderr, and:

- tells GDAL and `object_store` not to sign (`AWS_NO_SIGN_REQUEST=YES`, `AWS_SKIP_SIGNATURE=true`);
- hides every other AWS key from the command, because the proxy refuses keys it doesn't know, even for public data: it removes `AWS_ACCESS_KEY_ID` and the other credential variables, points `AWS_SHARED_CREDENTIALS_FILE` and `AWS_CONFIG_FILE` at an empty file, and sets `AWS_EC2_METADATA_DISABLED=true`.

The AWS CLI and s5cmd sign every request unless given `--no-sign-request`, and without credentials they stop with `Unable to locate credentials` or `NoCredentialProviders`. `exec` suggests the flag when it's missing:

```bash
source-coop exec -- aws --no-sign-request s3 ls s3://cholmes/admin-boundaries/
source-coop exec -- s5cmd --no-sign-request ls s3://cholmes/admin-boundaries/
```

A product that isn't public is refused with `AccessDenied`, as it is to anyone not signed in.

![`exec` signed out: unsigned, a public product listed, a restricted one refused](docs/demos/exec-signed-out.svg)

If you are signed in but the session can't be renewed (the refresh token expired or was revoked), `exec` stops with the error instead of running unsigned. Run `source-coop auth login` again, or `source-coop auth logout` to read public data unsigned.

### Multiple roles

> [!WARNING]
Expand Down Expand Up @@ -339,6 +386,8 @@ The data proxy accepts a write when all of these are true. Otherwise it answers

Name the object, or end the path with `/`. `aws s3 cp data.parquet s3://my-org/my-data` has no object key, and the proxy refuses it with `Missing object key`.

`exec` writes too, signed in: `source-coop exec -- aws s3 sync ./output s3://my-org/my-data/output/`.

To upload from a GitHub Actions workflow or a script, sign in as a [service account](#service-accounts) with write access to the product (see [Access to products](#access-to-products)). `service-account usage` prints a workflow that signs in (see [Use a service account](#use-a-service-account)). Replace its last step, `aws s3 ls`, with your `aws s3 sync`.

<!-- group: Manage source.coop -->
Expand Down Expand Up @@ -1023,7 +1072,7 @@ With `--json`:
- Field names are the API's. Times are RFC 3339.
- New fields can be added. Fields are not renamed or removed.

These commands do not accept `--json`, because their output is already for machines: `creds` (`--format`), `auth token` (a token), `api` (the API's response), `completions` (a shell script) and `service-account usage` (a workflow or shell exports). `auth login` does not accept it either, because it uses the browser. Use `auth status --json` after it.
These commands do not accept `--json`, because their output is already for machines: `creds` (`--format`), `auth token` (a token), `api` (the API's response), `completions` (a shell script), `service-account usage` (a workflow or shell exports) and `exec` (its command's output, and its command's exit code). `auth login` does not accept it either, because it uses the browser. Use `auth status --json` after it.

With `--json`, an error is also JSON, on stderr. If the error came from the API, it contains the data of the API's [RFC 9457](https://www.rfc-editor.org/rfc/rfc9457) problem:

Expand Down Expand Up @@ -1054,7 +1103,7 @@ All commands use the same exit codes:

### Read data

A product's data is at `s3://ACCOUNT/PRODUCT/` on the data proxy (`--proxy-url`, `https://data.source.coop` by default). Public data needs no sign-in; [Access Data](https://docs.source.coop/data-proxy) shows each tool's settings. Restricted data and writes need the credentials from `creds` (see [Reading data with AWS tools](#reading-data-with-aws-tools)).
A product's data is at `s3://ACCOUNT/PRODUCT/` on the data proxy (`--proxy-url`, `https://data.source.coop` by default). Public data needs no sign-in; [Access Data](https://docs.source.coop/data-proxy) shows each tool's settings. Restricted data and writes need the credentials from `creds` (see [Reading data with AWS tools](#reading-data-with-aws-tools)). `source-coop exec -- COMMAND` runs any S3 tool with the credentials and the proxy set, or unsigned when nobody is signed in (see [Run a tool with `exec`](#run-a-tool-with-exec)).

### Use the API without signing in

Expand All @@ -1070,7 +1119,7 @@ curl -H "Authorization: Bearer $(source-coop auth token)" https://source.coop/ap

![`auth token`: the session's access token, for `curl`](docs/demos/auth-token.svg)

To send a request through the CLI, see [Any API request](#any-api-request).
To send a request through the CLI, see [Any API request](#any-api-request). To read or write data, run the S3 tool through [`exec`](#run-a-tool-with-exec): `source-coop exec -- aws s3 ls s3://ACCOUNT/PRODUCT/`.

## Any API request

Expand Down Expand Up @@ -1117,7 +1166,7 @@ The CLI has no configuration file. Flags and environment variables set all value
| `SOURCE_PROXY_URL` | `--proxy-url` | `https://data.source.coop` | The environment: the data proxy (the S3 endpoint for AWS tools), and the session and credentials to use |
| `SOURCE_API_URL` | `--api-url` | `https://source.coop` | The source.coop site whose API (`/api/v1`) commands use |
| `SOURCE_TOKEN` | | | A bearer token for the API, used instead of the session's |
| `SOURCE_ROLE_ARN` | `--role-arn` | `_default` | The role for `creds`, `auth login`, `status` and `logout` |
| `SOURCE_ROLE_ARN` | `--role-arn` | `_default` | The role for `creds`, `exec`, `auth login`, `status` and `logout` |
| `SOURCE_OIDC_ISSUER` | `--issuer` | `https://auth.source.coop` | Where `auth login` signs in |
| `SOURCE_OIDC_CLIENT_ID` | `--client-id` | source.coop's | The OAuth2 client for `auth login` |
| `SOURCE_API_AUDIENCE` | `--audience` | `https://source.coop` | The audience of the API access token that `auth login` requests |
Expand Down Expand Up @@ -1211,7 +1260,11 @@ The session is in `source-coop/session.json`. Other environments use the same la

**`Your login has expired. Run 'source-coop auth login'.`** The session's refresh token expired or was revoked. Sign in again. `source-coop auth status` estimates when it expires.

**`Couldn't get new credentials (…). Run 'source-coop auth login'.`** `creds` could not renew a role's credentials. AWS tools show this message as their own error. The reason is in the brackets. Usually the session has expired, so sign in again. The message also says that public data needs no credentials (see [Access Data](https://docs.source.coop/data-proxy)).
**`Not signed in: run 'source-coop auth login'.`** Nobody is signed in to this environment (`--proxy-url`). `creds` exits with code 4; AWS tools that run it through `credential_process` show the message as their own error. The message also says that public data needs no credentials: sign in, or read it without them (see [Access Data](https://docs.source.coop/data-proxy)). `exec` doesn't fail here: it runs the command unsigned.

**`Couldn't get new credentials (…). Run 'source-coop auth login'.`** `creds` or `exec` could not renew a role's credentials. AWS tools show this message as their own error. The reason is in the brackets. Usually the session has expired, so sign in again. The message also says that public data needs no credentials (see [Access Data](https://docs.source.coop/data-proxy)).

**`Unable to locate credentials` or `NoCredentialProviders` under `exec`.** You are not signed in, so `exec` runs the command unsigned, and the AWS CLI and s5cmd then need `--no-sign-request`. Add it to read public data, or run `source-coop auth login`. See [Run a tool with `exec`](#run-a-tool-with-exec).

**macOS asks for your keychain password.** macOS asks the first time a new build reads the `source-coop-cli` keychain items, for example after an upgrade. Select **Always Allow** so that it does not ask again until the next upgrade.

Expand Down
8 changes: 4 additions & 4 deletions docs/demos/auth-logout.json
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
{
"duration": 14.566,
"duration": 14.474,
"steps": [
{
"time": 0.757,
"time": 0.758,
"label": "$ source-coop auth logout --json | jq -c"
},
{
"time": 6.045,
"time": 5.951,
"label": "$ source-coop auth status"
},
{
"time": 9.935,
"time": 9.832,
"label": "$ source-coop creds; echo \"exit $?\""
}
],
Expand Down
2 changes: 1 addition & 1 deletion docs/demos/auth-logout.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions docs/demos/exec-signed-out.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"duration": 22.863,
"steps": [
{
"time": 0.752,
"label": "$ source-coop exec -- aws s3 ls s3://partner-org/ocean-buoys/"
},
{
"time": 7.231,
"label": "$ source-coop exec -- aws --no-sign-request s3 ls s3://partner-org/ocean-buoys/"
},
{
"time": 14.67,
"label": "$ source-coop exec -- aws --no-sign-request s3 ls s3://demo-org/landsat-mirror/; echo \"exit $?\""
}
],
"keys": []
}
1 change: 1 addition & 0 deletions docs/demos/exec-signed-out.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions docs/demos/exec.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"duration": 20.086,
"steps": [
{
"time": 0.763,
"label": "$ source-coop exec -- aws s3 ls s3://demo-org/demo-data/"
},
{
"time": 6.768,
"label": "$ source-coop exec -- s5cmd ls 's3://demo-org/demo-data/readings/*'"
},
{
"time": 12.938,
"label": "$ source-coop exec -- env | grep -E '^AWS_(ACCESS_KEY_ID|ENDPOINT_URL_S3|REGION)='"
}
],
"keys": []
}
1 change: 1 addition & 0 deletions docs/demos/exec.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions docs/recorder/fixtures.json
Original file line number Diff line number Diff line change
Expand Up @@ -464,6 +464,24 @@
"key": "demo-data/readings/2026-09.parquet",
"size": 50331648,
"last_modified": "2026-09-14T08:30:00Z"
},
{
"bucket": "partner-org",
"key": "ocean-buoys/README.md",
"size": 3072,
"last_modified": "2026-08-02T10:00:00Z"
},
{
"bucket": "partner-org",
"key": "ocean-buoys/buoys.parquet",
"size": 7340032,
"last_modified": "2026-08-02T10:00:00Z"
},
{
"bucket": "partner-org",
"key": "ocean-buoys/observations/2026.parquet",
"size": 94371840,
"last_modified": "2026-10-01T10:00:00Z"
}
]
}
15 changes: 15 additions & 0 deletions docs/recorder/guide.py
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,19 @@ def aws(s):
s.run("source-coop creds --format env")


def exec_signed_in(s):
signed_in(s)
s.run(f"source-coop exec -- aws s3 ls s3://{ORG}/demo-data/"); s.prompt(); s.pause(1.5)
s.run(f"source-coop exec -- s5cmd ls 's3://{ORG}/demo-data/readings/*'"); s.prompt(); s.pause(1.5)
s.run("source-coop exec -- env | grep -E '^AWS_(ACCESS_KEY_ID|ENDPOINT_URL_S3|REGION)='")


def exec_signed_out(s):
s.run("source-coop exec -- aws s3 ls s3://partner-org/ocean-buoys/"); s.prompt(); s.pause(3)
s.run("source-coop exec -- aws --no-sign-request s3 ls s3://partner-org/ocean-buoys/"); s.prompt(); s.pause(2)
s.run(f"source-coop exec -- aws --no-sign-request s3 ls s3://{ORG}/landsat-mirror/; echo \"exit $?\"")


def prompts(s):
s.run("source-coop product create")
s.see("Account"); s.key(DOWN); s.enter()
Expand Down Expand Up @@ -397,6 +410,8 @@ def service_account_usage(s):
"auth-token": ("`auth token`: the session's access token, for `curl`", auth_token),
"auth-logout": ("`auth logout --json`, then `auth status` and `creds` signed out", auth_logout),
"aws": ("`aws s3 ls` through `credential_process`, then `creds --format env`", aws),
"exec": ("`exec`: the AWS CLI and s5cmd with your credentials and the proxy, no profile", exec_signed_in),
"exec-signed-out": ("`exec` signed out: unsigned, a public product listed, a restricted one refused", exec_signed_out),
"prompts": ("Prompts: a refused ID asked again, a draft kept after a failure, then Esc", prompts),
"completion": ("Shell completion: TAB lists products and service accounts from the API", completion),
"account-list": ("`account list`, `view`, then `list --json`", account_list),
Expand Down
36 changes: 33 additions & 3 deletions docs/recorder/stub.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@
it checks nothing, so it is no substitute for the API in tests.

Outside /api/v1 it is the data proxy, for `aws s3 ls`: a ListObjectsV2 of
fixtures.json's `objects`, signatures unchecked.
fixtures.json's `objects`, signatures unchecked. An unsigned listing of a
product that isn't public is refused, as the proxy refuses it. /oauth2/token
refuses to renew a session, as the sign-in server does once a refresh token
has expired.

python3 -I docs/recorder/stub.py PORT [FIXTURES]
"""
Expand Down Expand Up @@ -260,16 +263,43 @@ def list_objects(bucket, q):
+ tag("IsTruncated", "false") + body + "</ListBucketResult>")


def readable_unsigned(bucket, q):
"""Whether the proxy lists `bucket` and `prefix` for a request without a
signature: only inside a public product."""
product = q.get("prefix", "").split("/")[0]
return any(p["account_id"] == bucket and p["product_id"] == product and p["visibility"] == "public"
for p in products)


ACCESS_DENIED = ('<?xml version="1.0" encoding="UTF-8"?>'
"<Error><Code>AccessDenied</Code><Message>access denied</Message></Error>")


class Handler(BaseHTTPRequestHandler):
def log_message(self, *args):
pass

def answer(self, method):
url = urlparse(self.path)
q = {k: v[0] for k, v in parse_qs(url.query).items()}
if url.path == "/oauth2/token":
# The sign-in server, refusing to renew a session: a refresh
# token that expired or was revoked.
raw = json.dumps({"error": "invalid_grant",
"error_description": "The refresh token has expired."}).encode()
self.send_response(400)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
return
if not url.path.startswith("/api/v1"):
raw = list_objects(url.path.strip("/").split("/")[0], q).encode()
self.send_response(200)
bucket = url.path.strip("/").split("/")[0]
if self.headers.get("Authorization") or readable_unsigned(bucket, q):
status, raw = 200, list_objects(bucket, q).encode()
else:
status, raw = 403, ACCESS_DENIED.encode()
self.send_response(status)
self.send_header("Content-Type", "application/xml")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
Expand Down
7 changes: 4 additions & 3 deletions src/api.rs
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,9 @@ impl ApiError {
}
}

/// The error when there's no login session, for API calls and credentials.
pub const NOT_SIGNED_IN: &str = "Not signed in: run 'source-coop auth login'.";

/// The User-Agent sent to source.coop, so the server can tell CLI traffic,
/// and its version, from browsers and curl.
pub const USER_AGENT: &str = concat!("source-coop-cli/", env!("CARGO_PKG_VERSION"));
Expand Down Expand Up @@ -347,9 +350,7 @@ impl Client {
/// a member of.
pub async fn check_signed_in(&self) -> Result<serde_json::Value, ApiError> {
if self.token.is_none() {
return Err(ApiError::other(
"Not signed in: run 'source-coop auth login'.".into(),
));
return Err(ApiError::other(NOT_SIGNED_IN.into()));
}
self.get(self.url(&["whoami"])).await
}
Expand Down
Loading
Loading