Update PyPSA version #1065

ekatef · 2024-07-21T14:22:53Z

Disentangles update of PyPSA version in #796 from changes needed to implement linopy into model construction.

Covers #787 and includes adaptation of the changed PyPSA implications for n.buses and n.lines structure to our current data structures.

As a note: the strategy towards processing "non-standard" columns has changed in the recent PyPSA versions. Non-standard means names which are not included into columns of components attributes (see lines attributes, for example).

Changes proposed in this Pull Request

The PR follows the approach of PyPSA-Eur, but introduces additional functionality to define custom aggregation strategies in a flexible way.

TODO It would be nice to have a more formalised definition of the components data structures in PyPSA. Currently, the custom columns are implicitly allowed, but lead to troubles during clustering.

Checklist

I consent to the release of this PR's code under the AGPLv3 license and non-code contributions under CC0-1.0 and CC-BY-4.0.
I tested my contribution locally and it seems to work fine.
Code and workflow changes are sufficiently documented.
Newly introduced dependencies are added to envs/environment.yaml and doc/requirements.txt.
Changes in configuration options are added in all of config.default.yaml and config.tutorial.yaml.
Add a test config or line additions to test/ (note tests are changing the config.tutorial.yaml)
Changes in configuration options are also documented in doc/configtables/*.csv and line references are adjusted in doc/configuration.rst and doc/tutorial.rst.
A note for the release notes doc/release_notes.rst is amended in the format of previous release notes, including reference to the requested PR.

This reverts commit 7700759.

ekatef · 2024-07-25T13:58:09Z

A testing note the PR and in particular specifying the aggregation strategy for v_nom leads to the changes in the objective function.

main

INFO:__main__:Objective function: 4567465309.0
INFO:__main__:Objective constant: 2127041137.1933956

update-pypsa_version

INFO:__main__:Objective function: 4185403898.0
INFO:__main__:Objective constant: 2126804937.1863549

update-pypsa_version + do not include v_nom into the aggregation strategy

INFO:__main__:Objective function: 5112254879.0
INFO:__main__:Objective constant: 2126771900.9923582

ekatef · 2024-07-25T14:48:45Z

An additional comment: update of PyPSA requites some adjustments in fetching isolated networks (merge_into_network() function). It may be worth to track it as an additional issue.

ekatef · 2024-07-25T16:12:01Z

@davide-f it would be perfect to have you review here. As a note, that is a re-implementation of #796

This reverts commit 20192e6.

ekatef · 2024-07-28T22:50:23Z

The reason of the weird behaviour is specifying v_nom for buses. Previously, the problem has been balanced by the fact that get_clustering_from_busmap( ) dropped some columns from the lines and buses dataframes. In the new configuration, v_nom was left in n.buses, and it's values were completelly different from base_voltage (100, 200, 500 kV instead of 380 kV).

Resuts of updated objective test

main

INFO:__main__:Objective function: 4567465309.0
INFO:__main__:Objective constant: 2127041137.1933956

update-pypsa_version

INFO:__main__:Objective function: 4507268645.0
INFO:__main__:Objective constant: 2126679587.453438

That still means ~1% difference for the objective function. It seems that the difference is linked with Stores for which e_nom-opt values differ significantly between the models. s_nom_opt for lines and p_nom_opt for generators are basically identical for the both models (the difference lower than 1e-7). There is also a small (~0.1%) difference in capital_cost for generators and marginal_cost for stores.

ekatef · 2024-07-28T22:53:31Z

Points to be tracked:

An effect of get_clustering_from_busmap( ) on the structure of the network dataframes. Previously, some columns have been silently discarded, while now they all seem to be kept in the component dataframes.
CSP buses may need additional checking of v_nom: it may be the case that currently a default 1kV value is used.
PyPSA update leads to some weird issue, when fetching the isolated network.

davide-f · 2024-08-01T13:52:47Z

Sorry, I got a bit confused among the PRs. Indeed I already reviewed it and it seemed fine, although there is a pending comment (specify minimum pypsa version)
To understand the differences, is the rest fine?
Small differences are no problem though.

I'm a bit surprised that linux ci passes although changes in linopy are not integrated yet...
Interestingly, mac ci now fails due to the weird problem we encountered in -sec; by checking the environment logs we may be able to track why [maybe]

davide-f

This looks nice to me :)

davide-f · 2024-07-22T09:12:13Z

envs/environment.yaml

@@ -12,7 +12,7 @@ dependencies:
 - pip
 - mamba   # esp for windows build

- pypsa>=0.24, <0.25
+- pypsa


We should use pypsa>=0.25 at least I believe, or shall it be higher?

To support in-built shapes, >=0.26 is needed

davide-f · 2024-08-01T13:57:10Z

scripts/build_osm_network.py

@@ -908,12 +911,33 @@ def built_network(

    sets_path_to_root("pypsa-earth")


This is not related to this PR, but can we drop this? [it can be a separate good first issue]
We should avoid using sets_path_to_root; it leads to problems when users rename the folder name

Happy to do :) But agree that it deserves a dedicated PR

As a coordination comment, the issue is tracked in #639, have updated the tags to make sure that it's marked as a good first issue 🙂

ekatef · 2024-08-01T18:33:03Z

Sorry, I got a bit confused among the PRs. Indeed I already reviewed it and it seemed fine, although there is a pending comment (specify minimum pypsa version) To understand the differences, is the rest fine? Small differences are no problem though.

I'm a bit surprised that linux ci passes although changes in linopy are not integrated yet... Interestingly, mac ci now fails due to the weird problem we encountered in -sec; by checking the environment logs we may be able to track why [maybe]

@davide-f thanks a lot for the review 😄 No problem and sorry for a non-linear development path. The PyPSA update keeps backward compatibility with lopf optimisation functions (thanks to @GbotemiB for discovering it!), so the PR works.

Regarding CI issue for mac, that is exactly the same issue as for the sec version (Illegal instruction: 4 snakemake --cores all build_test_configs). I wonder, if there may be some conflicts on the system level, triggered by PyPSA updates...

If having less than 1% differences in the objective function is enough to conclude that the problem is basically same, I think we can assume that it works more or less fine. The only issue which need fixes or additional tracking is some weird behaviour of fetching isolated network. But it can be probably resolved by using later PyPSA version.

I'd suggest the following steps to proceed:

Upgrade PyPSA version to 0.26 or 0.27 and test, also for fetching the isolated networks
Add a documentation PR to explain how to install a particular version of the model
Add linopy integration itself. May it be a good idea to stack the PRs for that? [Otherwise, there is not much benefits in updating PyPSA]

Does it sound reasonable for you? 🙂

ekatef · 2024-08-04T09:16:55Z

Results of the compatibility testing for updating PyPSA version: currently building the environment on mac seems to lead to some weird problem still in a dry run zsh: illegal hardware instruction snakemake -j 1 solve_all_networks -n. Most likely, that is caused by some troubles on the system level and can be connected with the recent errors for mac OS in CI.

Pinging all the versions in environment.yaml sounds like a very good idea...

GbotemiB · 2024-08-04T17:48:42Z

Results of the compatibility testing for updating PyPSA version: currently building the environment on mac seems to lead to some weird problem still in a dry run zsh: illegal hardware instruction snakemake -j 1 solve_all_networks -n. Most likely, that is caused by some troubles on the system level and can be connected with the recent errors for mac OS in CI.

Pinging all the versions in environment.yaml sounds like a very good idea...

I think the mac issue is related to some update that is needed with conda. I have read a few issues related to this. We can test to see if it works.

ekatef · 2024-08-04T18:33:32Z

Results of the compatibility testing for updating PyPSA version: currently building the environment on mac seems to lead to some weird problem still in a dry run zsh: illegal hardware instruction snakemake -j 1 solve_all_networks -n. Most likely, that is caused by some troubles on the system level and can be connected with the recent errors for mac OS in CI.
Pinging all the versions in environment.yaml sounds like a very good idea...

I think the mac issue is related to some update that is needed with conda. I have read a few issues related to this. We can test to see if it works.

Thanks for sharing! Do you have any ideas for fixing? I'm currently testing it, and any hints would be tremendously helpful

davide-f · 2024-08-04T19:19:04Z

Regarding the overall approach, the proposal is totally fine!
Agree that it is advisable to merge the two PRs and have one. <1% tolerance is acceptable.

I confirm that this issue with mac is not pypsa-version related.
We have experienced this also in the -sec environment, see
https://github.com/pypsa-meets-earth/pypsa-earth-sec/actions/runs/10195751150/job/28205079365#step:8:513
In the -sec we decided to ignore the problem as it was -sec related; but it seems it is not. It is likely an issue that will be solved automatically probably, but if we can solve it by fixing a version may be ok.
Is it reproducible locally?

An interesting option is to check the latest conda envs that worked and see if there are differences.

this works: https://github.com/pypsa-meets-earth/pypsa-earth-sec/actions/runs/9989449022/job/27608092287 with this being the conda list conda_working.txt
this does not: https://github.com/pypsa-meets-earth/pypsa-earth-sec/actions/runs/9993058180/job/27619557912 with this being the conda list: conda_not_working.txt

There are dissimilarities in some internal conda packages that may be explored.
More specifically:

left is working, right does not

ekatef · 2024-08-04T20:43:39Z

Regarding the overall approach, the proposal is totally fine! Agree that it is advisable to merge the two PRs and have one. <1% tolerance is acceptable.

I confirm that this issue with mac is not pypsa-version related. We have experienced this also in the -sec environment, see https://github.com/pypsa-meets-earth/pypsa-earth-sec/actions/runs/10195751150/job/28205079365#step:8:513 In the -sec we decided to ignore the problem as it was -sec related; but it seems it is not. It is likely an issue that will be solved automatically probably, but if we can solve it by fixing a version may be ok. Is it reproducible locally?

An interesting option is to check the latest conda envs that worked and see if there are differences.

this works: https://github.com/pypsa-meets-earth/pypsa-earth-sec/actions/runs/9989449022/job/27608092287 with this being the conda list conda_working.txt

this does not: https://github.com/pypsa-meets-earth/pypsa-earth-sec/actions/runs/9993058180/job/27619557912 with this being the conda list: conda_not_working.txt

There are dissimilarities in some internal conda packages that may be explored. More specifically: left is working, right does not

Super, thanks for the confirmation for the general approach. Then, I'll leave out the analysis of the reasons of the 1% difference, and will finalise the work by cleaning-up and merging both PRs to have a single one.

Regarding the environment issue, I confirm that the problem seems to be the same as in CI. When trying to run the workflow with an updated environment, I get zsh: illegal hardware instruction snakemake -j 1 solve_all_networks which looks very similar to the report of the CI. In my case, conda-package-handling 2.3.0 and conda-package-streaming 0.10.0 are working with conda 4.5.0, while the issue emerges with conda 24.7.1 + conda-libmamba-solver 24.7.0. Thanks a lot for the hints! Testing with a restriction on conda version.

ekatef · 2024-08-04T21:27:11Z

Update on the environment testing: have applied version restrictions to conda itself and conda-libmamba-solver, but it didn't resolved the issue. One environment with conda 24.5.0 and conda-libmamba-solver 24.1.0 is working, while another one fails. Versions of mamba and snakemake-minimal are also same in the both cases. Not sure what is the reason of the troubles.

ekatef · 2024-08-05T22:17:02Z

A comment regarding merging the isolated networks: PyPSA update leads to a weird issue in get_clustering_from_busmap( ), when being applied to the busmap used to merge an isolated network into a backbone network.

In particular, the troubles are triggered by aggregating of loads and storage units with this line:

pypsa-earth/scripts/simplify_network.py

Line 861 in 1bd7f84

aggregate_one_ports=["Load", "StorageUnit"],

The issue is linked with KeyError: 'p', which sounds quite puzzling, as neither n.loads, nor n.storage_units have p column.

The full log looks like that:

ERROR:_helpers:An error happened in module '~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexes/base.py', function 'get_loc': 'p'
Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'p'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "~/Documents/_github_/pypsa-earth/scripts/simplify_network.py", line 1153, in <module>
    n, fetched_nodes_map = merge_into_network(
  File "~/Documents/_github_/pypsa-earth/scripts/simplify_network.py", line 862, in merge_into_network
    clustering = get_clustering_from_busmap(
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pypsa/clustering/spatial.py", line 519, in get_clustering_from_busmap
    new_df, new_pnl = aggregateoneport(
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pypsa/clustering/spatial.py", line 266, in aggregateoneport
    data = n.get_switchable_as_dense(c, attr)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pypsa/descriptors.py", line 124, in get_switchable_as_dense
    vals = np.repeat([df.loc[fixed_i, attr].values], len(snapshots), axis=0)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1184, in __getitem__
    return self._getitem_tuple(key)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1368, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1065, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1431, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1381, in _get_label
    return self.obj.xs(label, axis=axis)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/generic.py", line 4287, in xs
    return self[key]
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/frame.py", line 4102, in __getitem__
    indexer = self.columns.get_loc(key)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: 'p'

GbotemiB · 2024-08-06T03:12:50Z

A comment regarding merging the isolated networks: PyPSA update leads to a weird issue in get_clustering_from_busmap( ), when being applied to the busmap used to merge an isolated network into a backbone network.

In particular, the troubles are triggered by aggregating of loads and storage units with this line:

pypsa-earth/scripts/simplify_network.py

Line 861 in 1bd7f84

aggregate_one_ports=["Load", "StorageUnit"],

The issue is linked with KeyError: 'p', which sounds quite puzzling, as neither n.loads, nor n.storage_units have p column.

The full log looks like that:

ERROR:_helpers:An error happened in module '~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexes/base.py', function 'get_loc': 'p'
Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3805, in get_loc
    return self._engine.get_loc(casted_key)
  File "index.pyx", line 167, in pandas._libs.index.IndexEngine.get_loc
  File "index.pyx", line 196, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7081, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7089, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'p'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "~/.vscode/extensions/ms-python.debugpy-2024.8.0-darwin-arm64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "~/Documents/_github_/pypsa-earth/scripts/simplify_network.py", line 1153, in <module>
    n, fetched_nodes_map = merge_into_network(
  File "~/Documents/_github_/pypsa-earth/scripts/simplify_network.py", line 862, in merge_into_network
    clustering = get_clustering_from_busmap(
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pypsa/clustering/spatial.py", line 519, in get_clustering_from_busmap
    new_df, new_pnl = aggregateoneport(
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pypsa/clustering/spatial.py", line 266, in aggregateoneport
    data = n.get_switchable_as_dense(c, attr)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pypsa/descriptors.py", line 124, in get_switchable_as_dense
    vals = np.repeat([df.loc[fixed_i, attr].values], len(snapshots), axis=0)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1184, in __getitem__
    return self._getitem_tuple(key)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1368, in _getitem_tuple
    return self._getitem_lowerdim(tup)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1065, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1431, in _getitem_axis
    return self._get_label(key, axis=axis)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexing.py", line 1381, in _get_label
    return self.obj.xs(label, axis=axis)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/generic.py", line 4287, in xs
    return self[key]
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/frame.py", line 4102, in __getitem__
    indexer = self.columns.get_loc(key)
  File "~/opt/miniconda3/envs/pypsa-earth-upg5/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3812, in get_loc
    raise KeyError(key) from err
KeyError: 'p'

I think I have come across this error before. Although, I cant clearly remember how I solved it.
There are a few suggestions, which is to delete the network and resource files, delete .snakemake directory, recopy the default config, then try to execute the command to solve the network.

ekatef · 2024-08-08T18:04:22Z

illegal hardware instruction

Hey @GbotemiB, thanks for sharing the experience! In this case, the problem is localised and reproducible. Probably, we'll be able to discover also the reason behind the previous troubles, when looking into it. But agree that the issue sound quite weird, and I'm thinking that it may be better to track it by a dedicated issue.

davide-f

Hello @ekatef :D
Just recommenting here to check the status of this PR [and retrigger the CI].
Added a minor comment but functionality is there :)

It would be good to update to main to check and aim to merge this and linopy transition.
They could also be merged the PR and tackled as one?!?!
I'm unsure the update of linopy works without the revision of the functions of extra functionalities

Please comment if you need support

davide-f · 2024-09-30T13:21:23Z

envs/environment.yaml

@@ -12,7 +12,7 @@ dependencies:
 - pip
 - mamba   # esp for windows build

- pypsa>=0.24, <0.25
+- pypsa>=0.25, <0.28


Can we increase the version further?

Yeas, I have been thinking about testing for <0.29. The main concern is that v0.29.0 drops n.lopf( ) which I'm not quite sure is really a great idea. What is your feeling about that?

Interesting; probably worth updating till <0.29 maybe?

=0.29 can be tested in the linopy PR with linopy; that PR should fix that too.
What do you think?

With 0.28 testing successful :)

davide-f · 2024-09-30T13:22:13Z

scripts/build_osm_network.py

+    # Keep only a predefined set of columns, as otherwise conflicts are possible
+    # e.g. the columns which names starts with "bus" are mixed up with
+    # the third-bus specification when executing additional_linkports()
+    lines_cols_standard = [


This could also be a global variable at the beginning with capitalized characters.

Done. Have I got your idea?

Done. Have I got your idea?

Great :D basically yes :) as a little comment, would you mind placing it around line 25-26ish?
Having it as a global variable could help as it may be loaded by other scripts if it will ever be necessary.

Makes sense :) Done!

for more information, see https://pre-commit.ci

…e_pypsa_version

ekatef · 2024-09-30T21:47:59Z

Hello @ekatef :D Just recommenting here to check the status of this PR [and retrigger the CI]. Added a minor comment but functionality is there :)

It would be good to update to main to check and aim to merge this and linopy transition. They could also be merged the PR and tackled as one?!?! I'm unsure the update of linopy works without the revision of the functions of extra functionalities

Please comment if you need support

@davide-f thanks a lot for looking into that! 😄

The idea to split both tasks was to disentangle the issue with non-documented changes in the network structure. As the next step, agree that the linopy PR should be merged into this one, and probably also linopy PR to the cross-sectoral model [not sure what would be the most elegant approach].

ekatef · 2024-09-30T21:54:26Z

As a technical comment, the CI currently fails for mac with conda: command not found. Not sure if it's related to replacement of conda with mamba in a CI

davide-f · 2024-10-01T12:00:05Z

As a technical comment, the CI currently fails for mac with conda: command not found. Not sure if it's related to replacement of conda with mamba in a CI

Let's rerun it and see what happens... that is a new as it worked before but it can be mamba related.
If it persists, we can update it with mamba agree.

Update: it persists; I'm fine with updating it. Would you like to tackle it here?

ekatef · 2024-10-03T14:33:58Z

As a technical comment, the CI currently fails for mac with conda: command not found. Not sure if it's related to replacement of conda with mamba in a CI

Let's rerun it and see what happens... that is a new as it worked before but it can be mamba related. If it persists, we can update it with mamba agree.

Update: it persists; I'm fine with updating it. Would you like to tackle it here?

I'd be more comfortable to see CI running before merging the PR. But we can probably tackle it in #1128 together with others dependencies fixes. What do you think?

ekatef · 2024-10-03T14:38:59Z

Hello @ekatef :D Just recommenting here to check the status of this PR [and retrigger the CI]. Added a minor comment but functionality is there :)

It would be good to update to main to check and aim to merge this and linopy transition. They could also be merged the PR and tackled as one?!?! I'm unsure the update of linopy works without the revision of the functions of extra functionalities

Please comment if you need support

Have re-tested the PR with PyPSA v0.28.0 and it works. The only trouble relates to some weird error in merging isolated networks. It has been seen before, and feels like something related to the environment. So, I'd focus on updating and stabilising the environment for now (which includes this PR and enabling linopy), and work on this fix for this particular problem after.

Otherwise, the testing seems to be fine with the difference in the objective function less 1%.

ekatef and others added 17 commits July 18, 2024 13:33

Add a zenodo link to natura.tiff

7700759

Update environment

a693644

Revise structure definition for lines

9dda0b3

Remove get_aggregation_strategies

ffd5d52

Fix typo aggregation_strategies

fa121ff

Replace aggregategenerators with aggregateoneport

0c6fa74

Add aggregation strategies as a parameter

67c3aba

Re-define aggregation strategies

e21a350

Update aggregation strategies

cf2ca44

Update aggregation strategies for lines

1ea9fe8

Update aggregation strategies for buses

edf211a

Fix typo

ab18c8d

Put aggregation strategies into a variable

62fe73e

Parametrize the aggregation strategies

07636af

Refactor update of the aggregation strategies

23983f9

Clean-up the code

eef8433

Revert "Add a zenodo link to natura.tiff"

729fecf

This reverts commit 7700759.

ekatef mentioned this pull request Jul 21, 2024

Linopy transition #796

Open

8 tasks

ekatef added 3 commits July 21, 2024 16:54

Define an explicit clustering strategy for v_nom

cefc09c

Add a release note

3567bab

Get glpk back

8f4f7ea

ekatef force-pushed the update_pypsa_version branch from 78cab98 to 8f4f7ea Compare July 21, 2024 15:30

ekatef added 3 commits July 28, 2024 23:53

Specify v_nom for buses explicitly

20192e6

Revert "Specify v_nom for buses explicitly"

383e253

This reverts commit 20192e6.

Merge branch 'main' into update_pypsa_version

bd90873

davide-f reviewed Aug 1, 2024

View reviewed changes

Add a version restriction to the environment specification

40513e7

ekatef mentioned this pull request Aug 12, 2024

Enable linopy ekatef/pypsa-earth#1

Open

8 tasks

Merge branch 'pypsa-meets-earth:main' into update_pypsa_version

ab6a4ef

davide-f reviewed Sep 30, 2024

View reviewed changes

ekatef and others added 5 commits September 30, 2024 22:54

Merge branch 'main' into update_pypsa_version

04653d7

[pre-commit.ci] auto fixes from pre-commit.com hooks

c05a71b

for more information, see https://pre-commit.ci

Adjust naming

9ff16d4

Move the variable definition

9c25cd1

Merge remote-tracking branch 'origin/update_pypsa_version' into updat…

cc34888

…e_pypsa_version

ekatef added 2 commits October 2, 2024 22:48

Move the variable

a3b20ce

Upgrade PyPSA version

dbef78a

		@@ -908,12 +911,33 @@ def built_network(

		sets_path_to_root("pypsa-earth")

Update PyPSA version #1065

Are you sure you want to change the base?

Update PyPSA version #1065

Conversation

ekatef commented Jul 21, 2024 • edited Loading

Changes proposed in this Pull Request

Checklist

ekatef commented Jul 25, 2024

main

update-pypsa_version

update-pypsa_version + do not include v_nom into the aggregation strategy

ekatef commented Jul 25, 2024

ekatef commented Jul 25, 2024

ekatef commented Jul 28, 2024

Resuts of updated objective test

main

update-pypsa_version

ekatef commented Jul 28, 2024 • edited Loading

davide-f commented Aug 1, 2024 • edited Loading

davide-f left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekatef Aug 12, 2024 • edited Loading

Choose a reason for hiding this comment

ekatef commented Aug 1, 2024

ekatef commented Aug 4, 2024 • edited Loading

GbotemiB commented Aug 4, 2024

ekatef commented Aug 4, 2024

davide-f commented Aug 4, 2024 • edited Loading

ekatef commented Aug 4, 2024

ekatef commented Aug 4, 2024 • edited Loading

ekatef commented Aug 5, 2024

GbotemiB commented Aug 6, 2024

ekatef commented Aug 8, 2024

davide-f left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davide-f Oct 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ekatef commented Sep 30, 2024

ekatef commented Sep 30, 2024

davide-f commented Oct 1, 2024 • edited Loading

ekatef commented Oct 3, 2024

ekatef commented Oct 3, 2024

ekatef commented Jul 21, 2024 •

edited

Loading

ekatef commented Jul 28, 2024 •

edited

Loading

davide-f commented Aug 1, 2024 •

edited

Loading

ekatef Aug 12, 2024 •

edited

Loading

ekatef commented Aug 4, 2024 •

edited

Loading

davide-f commented Aug 4, 2024 •

edited

Loading

ekatef commented Aug 4, 2024 •

edited

Loading

davide-f Oct 1, 2024 •

edited

Loading

davide-f commented Oct 1, 2024 •

edited

Loading