Strange behavior during dask-worker startup, the worker process is up but the Nanny is closed.

[mingmwang]

2023-09-20 20:08:16,017 - distributed.nanny - INFO - Start Nanny at: 'tcp://10.10.80.114:45085'
2023-09-20 20:08:16,810 - distributed.worker - INFO - Start worker at: tcp://10.10.80.114:43861
2023-09-20 20:08:16,810 - distributed.worker - INFO - Listening to: tcp://10.10.80.114:43861
2023-09-20 20:08:16,810 - distributed.worker - INFO - Worker name: xxxxxx-zbauyb
2023-09-20 20:08:16,810 - distributed.worker - INFO - dashboard at: 10.10.80.114:8787
2023-09-20 20:08:16,810 - distributed.worker - INFO - Waiting to connect to: tcp://yyyyy:8786
2023-09-20 20:08:16,810 - distributed.worker - INFO - -------------------------------------------------
2023-09-20 20:08:16,810 - distributed.worker - INFO - Threads: 21
2023-09-20 20:08:16,810 - distributed.worker - INFO - Memory: 0.93 GiB
2023-09-20 20:08:16,810 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-qt7920e0
2023-09-20 20:08:16,810 - distributed.worker - INFO - -------------------------------------------------
2023-09-20 20:08:17,351 - distributed.nanny - INFO - Closing Nanny at 'tcp://10.10.80.114:45085'. Reason: nanny-close
2023-09-20 20:08:19,034 - distributed.worker - WARNING - Mismatched versions found

+---------+---------------------------------------------+----------------+----------------+
| Package | Worker-fd9c1574-1992-4cb7-92ad-433b89175355 | Scheduler | Workers |
+---------+---------------------------------------------+----------------+----------------+
| python | 3.11.4.final.0 | 3.11.5.final.0 | 3.11.4.final.0 |
+---------+---------------------------------------------+----------------+----------------+
2023-09-20 20:08:19,034 - distributed.worker - INFO - Registered to: tcp://yyyyy:8786
2023-09-20 20:08:19,035 - distributed.worker - INFO - -------------------------------------------------
2023-09-20 20:08:19,036 - distributed.core - INFO - Starting established connection to tcp://yyyyy:8786

[mingmwang]

During that time, the dask-scheduler is restarting also. But looks like the nanny is not closed completely and becomes a zombie process.

[mingmwang]

Dask distribute version is 2023.5.0