Making sure the Dask Scheduler and Dask Gateway workers have the same package versions

[rsignell]

I'm having a problem specifying the image for dask gateway workers to use. We need to have the Dask Workers using an image that has the same versions of Dask as the scheduler (the notebook), and the easiest way to ensure this is to specify that the Dask Workers use the same image as the notebook.

But when I do this, I sometimes find that even though I specify the image for the dask workers, the dask workers are using a different image.

Is this also getting cached somewhere? If so, we should disable the caching so that the Dask Workers actually use the image we specify!!

[rsignell]

@sebastian-luna-valero you should be able to run this notebook using the protocoast "develop" image:

https://nbviewer.org/gist/rsignell/d5015c604c9dc12ac50e452ad43f2d0e

[sebastian-luna-valero]

@rsignell I couldn't reproduce the issue.

Nonetheless, I have pushed this change: cc5454e

xref: https://artifacthub.io/packages/helm/dask/daskhub

Could you please try again and let me know?

[rsignell]

I just restarted my server, ran the landsat notebook and got this again:

[rsignell]

Perhaps the "options.image" isn't working either when we create a new cluster?
(we know that specifying environment variables via options is also not working)

[rsignell]

This is strange -- I just tried another notebook that creates a dask-gateway cluster and got this mismatch, which is different!

@sebastian-luna-valero so you were able to run the planetary computer notebook mentioned previously without problems?

[sebastian-luna-valero]

I am only executing the part related to dask gateway.

My first test worked because I was missing dask_client = cluster.get_client().

I am now able to reproduce the error.

Can you try running the attached notebook, please?

dask-gateway-versions-v3.ipynb

[rsignell]

I ran the dask-gateway-versions-v3.ipynb notebook and didn't have any errors.

But when I try to run a small test, I'm still having issues. Can you test this notebook using the protocoast-notebook:develop image?

https://nbviewer.org/gist/rsignell/ae0cec7da53245f626469123f912e49c

[sebastian-luna-valero]

I applied a change.

Could you re-run this test again, please?

https://nbviewer.org/gist/rsignell/ae0cec7da53245f626469123f912e49c

[rsignell]

Wow, what did you do!? It works great now!

[sebastian-luna-valero]

This config in the docs was causing problems, so I removed it.

[rsignell]

@sebastian-luna-valero I'm having these Dask scheduler/worker mismatch problems on Pangeo@EOSC again. And maybe some other problem with the Gateway cluster:

This notebook runs fine on another JupyterHub with Dask Gateway that I have access to.

When I run it on pangeo@EOSC, however, the attempt to register the plugin to the workers fails with:

CommClosedError: in <TLS (closed) ConnectionPool.register_worker_plugin local=tls://10.42.20.33:35740 remote=gateway://traefik-daskhub-dask-gateway.c-scale-pangeo-dask:80/c-scale-pangeo-dask.68ad96dc5ef846589ef05ac7005728d0>: Stream is closed

and when I look at the Dask Dashboard, I see the number of workers flipping back and forth between 15 and 16.

When I check the worker logs, I see:

2025-11-02 10:51:16,860 - distributed.worker - WARNING - Mismatched versions found +-------------+---------------------------------------------+----------------+---------------------------------------------------------+ | Package | Worker-22465dd2-2b74-46d3-8796-f5aa7cd29b65 | Scheduler | Workers | +-------------+---------------------------------------------+----------------+---------------------------------------------------------+ | dask | 2025.10.0 | 2025.10.0 | {'2025.7.0', '2025.9.1', '2025.10.0'} | | distributed | 2025.10.0 | 2025.10.0 | {'2025.7.0', '2025.9.1', '2025.10.0'} | | msgpack | 1.1.2 | 1.1.2 | {'1.1.1', '1.1.2'} | | numpy | 2.3.4 | 2.3.4 | {'2.3.4', '2.3.3', '2.2.6'} | | pandas | 2.3.3 | 2.3.3 | {'2.3.1', '2.3.3'} | | python | 3.13.9.final.0 | 3.13.9.final.0 | {'3.13.9.final.0', '3.12.11.final.0', '3.13.7.final.0'} | | toolz | 1.1.0 | 1.1.0 | {'1.0.0', '1.1.0'} | +-------------+---------------------------------------------+----------------+---------------------------------------------------------+

You should be able to run the reproducible notebook above on Pangeo@EOSC if you use the quay.io/globalcoast/protocoast-notebook:develop image.

[sebastian-luna-valero]

@rsignell I reproduced the error with the quay.io/globalcoast/protocoast-notebook:develop image.

However, it works with the quay.io/globalcoast/protocoast-notebook:main image.

Could you please investigate the latest changes made to the develop image? or what's the difference with the main image?

I am particularly interested in updates to dask* packages in the develop image.

[rsignell]

I still think it's because the worker image is getting cached somehow, and the notebook on develop is failing because the workers are getting the main image.

I'm trying now to build the main image using the develop environment, so they will be equal.