Allow passing environment variables when creating Dask Gateway cluster

[rsignell]

On other Dask Gateway systems I've used, like Nebari, when I create a Dask Gateway cluster, I can specify environment variables, like the credentials that will allow the worker to access or write to s3:

On Pangeo-EOSC, however, we don't have the ability to specify environment variables:

Could the ability to supply environment variables to the workers please be configured for Pangeo-EOSC's Dask Gateway?

[rsignell]

ping @sebastian-luna-valero , just in case this has slipped out of sight...
We would love to have this enabled before our Pangeo training workshop that starts Oct 13

@jmunroe , you guys do this with your 2i2c hubs, right?

[sebastian-luna-valero]

It seems that we need to touch here:

pangeo-eosc/providers/kubernetes/daskhub.yaml

Lines 21 to 38 in c0d6234

	optionHandler: |
	from dask_gateway_server.options import Options, Integer, Float, String
	def options_handler(options):
	if ":" not in options.image:
	raise ValueError("When specifying an image you must also provide a tag")
	return {
	"worker_cores": options.worker_cores,
	"worker_memory": int(options.worker_memory * 2 ** 30),
	"image": options.image,
	}
	c.Backend.cluster_options = Options(
	Integer("worker_cores", default=1, min=1, max=8, label="Worker Cores"),
	Float("worker_memory", default=2, min=2, max=32, label="Worker Memory (GiB)"),
	String("image", default="quay.io/pangeo/pangeo-notebook:2025.06.02", label="Image"),
	handler=options_handler,
	)

Which is documented here: https://gateway.dask.org/cluster-options.html#examples

To add environment variables as requested, could you please share the link to an example? or the code that nebari uses for this?

Thanks!

[rsignell]

I'm hoping @jmunroe knows the answer!

[dharhas]

I found this. Hope it helps -

nebari-dev/nebari#599

[rsignell]

Thanks @dharhas! Hopefully this will be sufficient for @sebastian-luna-valero or others on the Pangeo-EOSC team to enable!

[ofk123]

@rsignell Just curious about how this option is used since I have not seen it in use before.

In my individual process I specify a target_mapper locally, which I submit to workers for their use..

But by copy-pasting my GCStoken.json into the Environment Variable input, each worker can fetch the token and create their own mapper?

[rsignell]

@ofk123 , I use this approach to pass AWS credentials as environment variables to the workers so they can write to object storage requiring authentication. I'm not sure I understand the method you are talking about. Can you provide an example?

[ofk123]

@rsignell I see, so you copy paste S3 Access- and Secret key's (for example) in to that input?

Okay, one example, but I use token-based access to a GCS-bucket, so probably not compareable:
If I locally initialize an empty store on my remote bucket with

with open(path) as f:
    token = json.load(f)
gcs = gcsfs.GCSFileSystem(token=token)
target_mapper = gcs.get_mapper(storepath)

empty_ds.to_zarr(
        target_mapper, 
        mode="w", 
        consolidated=True, 
        compute=False
    )

and then submit tasks to workers, and pass the same mapper,

submitted = client.submit( some.function,
            args, target_mapper,
            pure = False,
            workers = workerlist
        )

which allows each worker to write their output to the bucket with, in the same example,

output_ds.to_zarr(
        target_mapper,
        mode="r+",
        region={"latitude": slice(None), "longitude": slice(j,j+1,None), "time": slice(None), "pressure": slice(0,1)},
        compute=True
    )

I realize this might be off topic, but was curious. How do you usually get around this using AWS credentials, without this option?

[rsignell]

@ofk123, Our preferred method for writing to the Pangeo-EOSC buckets from Python is:

import fsspec
from dotenv import load_dotenv
import xarray as xr

bucket = 'rsignell4-protocoast'
_ = load_dotenv(f'/home/jovyan/dotenv/{bucket}_credentials.env')

fs = fsspec.filesystem('s3', anon=False, skip_instance_cache=True, use_listings_cache=False,
                       endpoint_url='https://pangeo-eosc-minioapi.vm.fedcloud.eu')

ds = xr.tutorial.load_dataset("air_temperature")
ds.to_zarr(fs.get_mapper(f'{bucket}/data/zarr/air_temperature.zarr'), zarr_format=3, mode='w')

where the credentials file looks like:

$ more ~/dotenv/rsignell4-protocoast_credentials.env 
AWS_ACCESS_KEY_ID=wg1oCNkZxxxxxxxxxx
AWS_SECRET_ACCESS_KEY=JyTJbxZz3jioZqj8N2IBwwxxxxxxxxxxxxxxxxxx

the anon=False tells fsspec to look for credentials via the usual s3 methods, one of which is looking for those AWS_.. environment vars!

Make sense?

[sebastian-luna-valero]

@rsignell could you please give it a go now?

[rsignell]

@sebastian-luna-valero , I gave it a try, but the environment variables don't seem to be getting set:
https://nbviewer.org/gist/rsignell/d33984a5fc784faff303efbec97c7e8b
Did I use the right options setting?

[sebastian-luna-valero]

Yes, you are using the right option.

I reproduced the issue. I am following what's shown in the old nebari commit, so I must be missing something obvious.

The other solution is to set env vars as in:

https://jacobtomlinson.dev/posts/2022/how-to-set-environment-variables-on-your-dask-workers/

i.e.:

def set_env():
    os.environ["FOO"] = "BAR"

client.run(set_env)

# check via
client.run(lambda: os.environ["FOO"])

[rsignell]

@aktech, any chance you could take a quick look and see if you can identify what we are doing wrong here?

We tried to follow the Nebari approach but we must be missing something... 😕