Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functionality for dH_abs in methods choice #81

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

nezzag
Copy link

@nezzag nezzag commented Jan 10, 2025

This adds an option to determine methods choice not based on dH (a relative measure of the difference between historical and modelled data), but on dH_abs, an absolute measure.

This is useful in cases where the component being harmonised is a small variable (e.g. harmonising bio-liquids demand in industry, or a minor sub-species of gas). Here dH can be large, but dH_abs is still small, and hence the 'difference' between modelled and historic can still be seen as small enough to enforce convergence.

Otherwise a constant_ratio approach can be enforced, even though dH_abs is small.

Code assesses what counts as a "small" absolute value by looking at the parent variable (which is the next level up in the IAMC hierarchy in this variable, e.g. Final Energy|Industry|Gases parent variable is Final Energy|Industry). dH_abs variables which are under half of the parent variable values are seen as small enough to still enforce convergence.

@gidden
Copy link
Member

gidden commented Jan 10, 2025

Thanks for the addition @nezzag ! Let me know when you're done with any edits and I can review.

@gidden
Copy link
Member

gidden commented Jan 21, 2025

Hey @nezzag - I realized that I have to (for some reason) approve the test suite, and it looks like there are failures. You can run these locally using pytest (let me know if you have any issues).

@gidden
Copy link
Member

gidden commented Jan 27, 2025

Looks like everything passes but the linters, @nezzag - great work! Let me know if you have any questions there.

@nezzag
Copy link
Author

nezzag commented Jan 28, 2025

Hi Matt,
Thanks! Yes I think it's all looking good now, the linting is passing.

In brief, what I have done is add a new column to the methods choice DataFrame, which is dH_abs, the absolute difference between m and h.

This is then compared to the "scale" of the data, which is calculated using the function calc_dh_abs_threshold. This looks for the 'parent variable', e.g. for Final Energy|Industry|Liquids, it would look for Final Energy|Industry (currently just dropping one leaf). If the difference is under 10% of the parent variable, then the difference is still seen as small.

In the methods choice, I've just added one decision gate. Which is if dH is large (>0.5) but dH_abs is small (<10% of the parent variable). This means that the historic data is <20% of the parent variable, and the modelled data to be harmonised is within 10% of this historic data. So this is where

  • The relative difference is large
  • But this absolute difference is small
  • And the variables in general are ~quite small

Here aneris now still uses the convergence approaches, rather than defaulting to a constant ratio. BUT, it decides whether to use ratio_converge_2080 or offset_converge_2080, depending on whether the offset would be positive or negative. If the offset is negative we use ratio, to avoid introducing negative values (basically because I'm moving to use aneris more for energy variable harmonisation where negatives don't make sense, although I guess that's the case for most gases also). If the offset is positive we use offset, because otherwise the ratio could be very large (e.g. 20-30x), and this can result in very odd harmonised pathways.

So the code should behave pretty much as before, except in the case where the data to be harmonised is ~small, and so the absolute difference is small, but the relative difference is large.

Let me know what you think!

@gidden
Copy link
Member

gidden commented Jan 31, 2025

Hey @nezzag - thanks again. I am about to dive into this and noticed there were no tests looking at the new values you've added. Would you mind adding at least one test for this?

I think the easiest way (and most useful for future use cases) is to add a test for the default_method_choice() function. This way you can mock up some input data that you know should each each branch of the tree and tests that it does. Would you mind tackling that?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants