Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster by arbitrary shapes (NUTS1, NUTS2, NUTS3, GADM) #263

Open
fneum opened this issue Jul 8, 2021 · 12 comments · May be fixed by #1502
Open

Cluster by arbitrary shapes (NUTS1, NUTS2, NUTS3, GADM) #263

fneum opened this issue Jul 8, 2021 · 12 comments · May be fixed by #1502
Assignees
@bobbyxng
Labels
design enhancement help wanted

Comments

@fneum
Copy link
Member

fneum commented Jul 8, 2021 •
edited
Loading

It is already possible to specify a custom busmap to the cluster_network rule to support arbitrary clustering. But that is so far a manual process.

There are many use cases for clustering according to administrative areas (NUTS, GADM, etc).

These are usually available as shapefiles (e.g. https://gisco-services.ec.europa.eu/distribution/v2/nuts/download/#nuts21).

I would like to automate clustering to arbitrary shapefiles. That could look something like this:

To create (note that you need not to supply a number of clusters):

snakemake -j all "networks/elec_s_shapes.nc"

Provide a shapefile in config.yaml:

clustering:
  optional_inputs:
    shapefile: "nuts/NUTS_RG_01M_2016_4326_LEVL_0.geojson"

That becomes and optional input to the snakemake rule in the Snakefile:

wildcard_constraints:
    clusters="[0-9]+m?|all|shapes",

rule cluster_network:
    input:
        **config["clustering"].get("optional_inputs", {})

I think the busmap should be relatively straightforward to create using geopandas.sjoin(). One would add a function to scripts/cluster_network.py:

def busmap_from_shapes():
    ...

Two difficulties I can think of right now:

  • The mapping of a bus' Voronoi cell from the base network to the regions datafile is not always 100% clear; i.e. the borders of regions will run through Voronoi cells which need to be split/disaggregated. Vice versa, it can also occur that a region is fully contained inside a Voronoi cell of the base network (presumably less likely).
  • Offshore nodes need to be mapped to onshore buses (that are covered by shapefile)
@fneum
Copy link
Member Author

fneum commented Jul 11, 2021 •
edited
Loading

Example for NUTS1 and region shapes from elec_s.nc (color codes the NUTS1 region):
image

@pz-max
Copy link
Collaborator

pz-max commented Sep 22, 2021 •
edited
Loading

Hi Fabian, the build_bus_region.py creates the Voronoi cells to which all information are aggregated.
Maybe it is worth adjusting the build_bus_region.py instead of the cluster_network.py function?

Or is there a particular reason for adjusting cluster_network.py? FYI, if we have sometime, we would like to implement the GADM zones (which would make NUTS3 probably also easy to add)

@FabianHofmann
Copy link
Contributor

But the Voronoi cells are build on basis of the ENTSOE gridmap, i.e. their nodes. I think the problem remains the same, as the nodes of the ENTOSE maps have to be aligned to the NUTS3 regions...

@pz-max
Copy link
Collaborator

pz-max commented Sep 22, 2021

Yes, true. The problem stays the same. What was nice with the Voronoi bus map technique is that each region had a clear connection. One node was connected to another one with a line. The main question is now with NUTS/GADM or other arbitrary shapes. How are these regions connected? Which region is linked to each with which magnitude?

In general what can happen:

  • A region is isolated (only transmission/distribution infrastructure in the region itself)
  • A region is connected with other regions

We would do the following:

  1. Create ID's for any shape i.e. Voronoi_busmap/GADM/NUTS.
  2. If buses/lines are in the same shape -> Associate the capacities and cost at the region as i.e. "distribution"
  3. If two buses, connected by a line, are in two different regions. Define a connection between the region with the given transmission capacity and costs.

One could then also develop some nice search algorithms i.e.

  • Connect from different locations to the isolated regions and assess if it reduces the total system costs (also DC lines connected from far distant regions)
    Or a non-search approach/option:
  • Define all regions as connected to the neighbouring ones (This is a good default which I guess is applied in Calliope)

@fneum
Copy link
Member Author

fneum commented Sep 22, 2021

Hmmm, I don't really see a good way of defining the clustered regions further upstream without a tremendous amount of work. Everything is tied around the ENTSO-E grid map.

  • The network simplification (aligning voltage level, removing stubs etc) in simplify_network would have to be moved further upstream, too.
  • If the initial regions become too large build_renewable_profiles will likely get a performance hit because the rasters need to be chunked for calculating the renewable potentials.

@FabianHofmann
Copy link
Contributor

  • If the initial regions become too large build_renewable_profiles will likely get a performance hit because the rasters need to be chunked for calculating the renewable potentials.

This should not be a problem, the performance with large regions is pretty solid. But anyhow, it would require a lot of rearrangement considering the simplify_network and correct assignment of load time series.

@pz-max pz-max mentioned this issue Sep 28, 2021
Open
@fneum
Copy link
Member Author

fneum commented May 1, 2022

@Hazem-IEG, were you planning to transfer this also to PyPSA-Eur at some point?

@hazemakhalek
Copy link

@fneum, yes, I already started working on it locally and did some initial tests a while back but stopped to focus on the sector coupled version of PyPSA-Earth.. soon I'll continue working on it

@fneum
Copy link
Member Author

fneum commented Feb 22, 2023

@Hazem-IEG any updates? This would be a nice feature to add to PyPSA-Eur.

@fneum
Copy link
Member Author

fneum commented Jul 23, 2023

This is in PyPSA-Earth:

Look for "alternative clustering":

https://github.com/pypsa-meets-earth/pypsa-earth/blob/dbca213b43bf96fa25ec5b4785cddadec3a85f5c/scripts/build_bus_regions.py#L200-L207

https://github.com/pypsa-meets-earth/pypsa-earth/blob/main/scripts/cluster_network.py

https://github.com/pypsa-meets-earth/pypsa-earth/blob/main/scripts/simplify_network.py

@fneum
Copy link
Member Author

fneum commented Aug 3, 2024 •
edited
Loading

Also https://www.geoboundaries.org/globalDownloads.html which is very good to download selectively.

@fneum fneum mentioned this issue Aug 6, 2024
Merged
6 tasks
@fneum
Copy link
Member Author

fneum commented Aug 23, 2024

Also @bobbyxng worked this out in PyPSA-Eur: https://github.com/bobbyxng/kodekiste/tree/main/scripts/cluster-by-nuts

@bobbyxng bobbyxng linked a pull request Jan 17, 2025 that will close this issue
Draft
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bobbyxng bobbyxng

Labels
design enhancement help wanted
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Cluster network by shapes (NUTS0, NUTS1, NUTS2, NUTS3, and ADM1) PyPSA/pypsa-eur
5 participants
@FabianHofmann @fneum @bobbyxng @pz-max @hazemakhalek