Improve computational performance of build_osm_network

[davide-f]

Closes #806

and contributes to #445

Changes proposed in this Pull Request

This PR merges #807 #808 and #799

It completely revise the algorithm of set_lines_ids, set_substation_ids and fix_overpassing_nodes

Speedup increase is x100 for US with half the RAM consumption of the previous implementation.
PR coauthored with @mnm-matin

Checklist

• I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
• I tested my contribution locally and it seems to work fine.
• Code and workflow changes are sufficiently documented.
• Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
• Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
• Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
• Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
• A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

[davide-f]

@mnm-matin Performances of this PR on US leads to a x100 speed increase and to halve the RAM requirements.

Results are not exactly matching, mainly because the DBSCAN algorithm clusters together more nodes with the same tolerance hence leading to more lines to be dropped because starting and ending in the same node.
Maybe that is acceptable.

I think it is worth testing for a smaller country than US for the performances, then this PR would be ready I think.
@mnm-matin with your consent, I'd close the other PRs that are open on this topic and that are merged here.

Update:

• for NG, results perfectly match
• for ZA, some discrepancies apply mainly due to DBSCAN: for dense regions, a large number of buses gets clustered, hence with a setting of 5km tolerance, buses in a diameter of 15km may get merged. I think it can be acceptable

[davide-f]

By incorporating the other PRs on #650 and #829, the overall workflow for US takes less than 4h with a total memory compatible with a medium laptop :)

I'll be now testing the continent-wise analysis to test how pypsa-earth behaves with each country

[ollie-bell]

@davide-f would you not recommend using the changes in this PR? Not sure why it was closed.

[GbotemiB]

Hi all, I am currently using this PR to work on the US model run.

- start [Fri Aug  2 16:13:17 2024]
- stop [Fri Aug  2 16:16:49 2024]

My laptop runs MBP M1 32gb ram. It completed the build_osm_network process in about 5mins which is super fast. I have been running the US model now for days.

[yerbol-akhmetov]

Hi, @davide-f. This is a very great lifesaving PR. It would be fantastic if it can be merged.