Prepare ports failing in test workflow on MacOS

[Eric-Nitschke]

Checklist

• I am using the current main branch or the latest release. Please indicate.
• I am running on an up-to-date pypsa-earth environment. Update via conda env update -f envs/environment.yaml.

Describe the Bug

The test workflow on MacOS seems to fail rather often when trying to download the port data in prepare_ports. Since the filepath is hard-coded just above the download function, this is most likely unrelated to any specifications made in the config or elsewhere in the code.

The error occured in the following test workflows:

• 317
• 314
• 305 (though it doesn't show up anymore, since it rerun successfully since)

Error Message

[Tue Jan 14 16:39:18 2025]
rule prepare_ports:
    output: resources/ports.csv, resources/export_ports.csv
    jobid: 3
    reason: Missing output files: resources/export_ports.csv, resources/ports.csv
    resources: tmpdir=/var/folders/84/5079cppj57s0lzywhyzcxkjc0000gn/T

[Tue Jan 14 16:39:18 2025]
rule retrieve_cost_data:
    input: raw.githubusercontent.com/PyPSA/technology-data/v0.10.0/outputs/costs_2030.csv
    output: resources/sector/costs_2030.csv
    log: logs/sector/retrieve_cost_data_2030.log
    jobid: 4
    reason: Missing output files: resources/sector/costs_2030.csv
    wildcards: year=2030
    resources: tmpdir=/var/folders/84/5079cppj57s0lzywhyzcxkjc0000gn/T, mem_mb=5000, mem_mib=4769

[Tue Jan 14 16:39:18 2025]
rule prepare_transport_data_input:
    output: resources/transport_data.csv
    jobid: 39
    reason: Missing output files: resources/transport_data.csv
    resources: tmpdir=/var/folders/84/5079cppj57s0lzywhyzcxkjc0000gn/T

Restricted license - for non-production use only - expires 2026-11-23
/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pulp/apis/gurobi_api.py:170: UserWarning: GUROBI error: Unknown parameter '_test'.
  warnings.warn("GUROBI error: {}.".format(e))
/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
Traceback (most recent call last):
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/urllib/request.py", line 1348, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/http/client.py", line 1448, in connect
    super().connect()
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/http/client.py", line 942, in connect
    self.sock = self._create_connection(
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/socket.py", line 836, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/socket.py", line 967, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/runner/work/pypsa-earth/pypsa-earth/.snakemake/scripts/tmpr21wezi5.prepare_ports.py", line 91, in <module>
    df = download_ports().copy()
  File "/Users/runner/work/pypsa-earth/pypsa-earth/.snakemake/scripts/tmpr21wezi5.prepare_ports.py", line 35, in download_ports
    wpi_csv = pd.read_csv(fn, index_col=0)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pandas/io/common.py", line 728, in get_handle
    ioargs = _get_filepath_or_buffer(
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pandas/io/common.py", line 384, in _get_filepath_or_buffer
    with urlopen(req_info) as req:
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pandas/io/common.py", line 289, in urlopen
    return urllib.request.urlopen(*args, **kwargs)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
/Users/runner/micromamba/envs/pypsa-earth/lib/python3.10/site-packages/pypsa/networkclustering.py:16: UserWarning: The namespace `pypsa.networkclustering` is deprecated and will be removed in PyPSA v0.24. Please use `pypsa.clustering.spatial instead`. 
  warnings.warn(
Config file config.default.yaml is extended by additional config specified via the command line.
Config file configs/bundle_config.yaml is extended by additional config specified via the command line.
Config file configs/powerplantmatching_config.yaml is extended by additional config specified via the command line.
Config file config.yaml is extended by additional config specified via the command line.
[Tue Jan 14 16:39:22 2025]
Error in rule prepare_ports:
    jobid: 3
    output: resources/ports.csv, resources/export_ports.csv

[ekatef]

Hello @Eric-Nitschke and thank you for reporting!

A likely reason is that there have been some troubles with internet connection. "urlopen error [Errno 8] nodename nor servname provided, or not known" means that DNS resolution failed for some reason.

A stable solution would be probably to re-host the data instead requesting them, but licensing issues should be checked for that. A lighter solution is improving the error message to make it more clear.

[ekatef]

Hello @Eric-Nitschke, haven't you been able by any chance to check if the issue still persists?

[Eric-Nitschke]

Hey @ekatef, I didn't specifically check for it, but since the last pull requests dont seem to have any issues regarding the MacOS test workflow, I assume that it is solved by now.

[ekatef]

Hey @ekatef, I didn't specifically check for it, but since the last pull requests dont seem to have any issues regarding the MacOS test workflow, I assume that it is solved by now.

Hey @Eric-Nitschke, many thanks for the confirmation! I also didn't see this problem for quite some time so agree that we can count it as solved for now. I'm closing this issue for now but obviously feel free to re-open if you'd see any need for it.