Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[8pt] Hydrotable with row with hydroID but mostly nulls after it #1339

Open
RobHanna-NOAA opened this issue Nov 8, 2024 · 10 comments · May be fixed by #1414
Open

[8pt] Hydrotable with row with hydroID but mostly nulls after it #1339

RobHanna-NOAA opened this issue Nov 8, 2024 · 10 comments · May be fixed by #1414
Assignees
@hhs732
Labels
bug Something isn't working High Priority

Comments

@RobHanna-NOAA
Copy link
Contributor

RobHanna-NOAA commented Nov 8, 2024 •
edited
Loading

HydroVIS detected this and for now, it was only detected on one huc and one row in a hydrotable, but it is suspected to be happening more.

The Question:

  • In a Hydrotable, it is acceptable to have a row in there with a hydroid but mostly nulls after it? Or is this a bug in our code?
  • Is it a quick fix? Should we do a scan of other ones? What is the effort / reward on this fix. PS. I have a way from the HV side to get all records where this can be happening if it is needed, but only if it has value to this problem.

The case found:

Image
This is an image of a HydroVIS join table. If we check the huc and hydroID against our hydrotables, will we see some nulls in that row and should there be any?

HydroVIS has hardcoded put in a temp emergency fix with Derek's blessing to just drop all records when it see's that fix.

At a min, if we can figure out if it is a bug or just an acceptable condition, we can update HydroVIS code to put in a note of why it might happen. If we get a fix and when they get new code, we can tell them to drop that temp hardcoded fix in their code.

More details: Part of a redmine ticket and also recorded in HydroVIS isssue 950.

Update: Jan 7, 2025: A temp fix was put in for 2.1.8 via HydroVIS card 950.and part of a redmine ticket. 950 is now closed. But, when this fix comes into HV for the next release 2.2 ??), HydroVIS will need to take out the temp fix. It is now covered on the HydroVIS side by their card 1028.
@RobHanna-NOAA RobHanna-NOAA mentioned this issue Nov 8, 2024
Closed
@RobHanna-NOAA RobHanna-NOAA added bug Something isn't working Low Priority research labels Nov 8, 2024
@CarsonPruitt-NOAA
Copy link
Collaborator

This is definitely coming from our side, the number of cells is zero all the way down, so somehow this is a catchment with no pixels?
image

@CarsonPruitt-NOAA
Copy link
Collaborator

This particular hydroid's catchment looks to only be one 10m pixel.
image

There appears to be at least 5 other hydroids like this one in HUC 04040002. This is a great lake HUC, so I'm sure the lake masking has something to do with it.
image

@RobHanna-NOAA
Copy link
Contributor Author

Nudge.. to make sure we add it to the next Prod release. :)

@RobHanna-NOAA
Copy link
Contributor Author

When this is fixed in FIM, HV will also need a fix to take out their current temp fix. A new HV card is coming to join the task together.

@RobHanna-NOAA RobHanna-NOAA mentioned this issue Jan 7, 2025
Open
@hhs732
Copy link
Contributor

hhs732 commented Jan 15, 2025 •
edited
Loading

In HUC8 04040002, there are 7 feature-ids showing some issues in the hydrotable. Six of those have 2 outlets assign to them with 2 different hydroIDs which one of them is so small and sometimes far from the cathment. For example, feature-id 12170725 has two outlet hydroIDs of which 13820551 is so small and not even in the catchment:

Image

Image

Among these six feature-ids, 12169237 has the same issue. However, no catchment was assigned to that little hydro-id 13820520. That is why all rows are zero in the hydrotable for this one. Because that one was masked by the landsea_subset layer.

Image

Image

Image

@hhs732
Copy link
Contributor

hhs732 commented Jan 15, 2025 •
edited
Loading

One feature-id 12070743 was not capture in demDrived_reaches_split_filtured ... layer however it is in nwm_subset_streams:

Image

@hhs732
Copy link
Contributor

hhs732 commented Jan 15, 2025 •
edited
Loading

I checked some other HUC8s with lakes including 03100208, 02050103, 04030101, 04050002, 04040003, 04060101 and 16050101 to see if I can find the same issue. And I could not find any.

@hhs732
Copy link
Contributor

hhs732 commented Jan 17, 2025 •
edited
Loading

The issue is generated in split_flows.py, but is resolved in filter_catchments_and_add_attributes.py and gdal_rasterize for almost all HUC8 except 04040002. The following explains it in detail.

@hhs732
Copy link
Contributor

hhs732 commented Jan 24, 2025 •
edited
Loading

HUC8 04040002 -- DEV
demDerived_reaches_0.gpkg

Image

demDerived_reaches_split_0.gpkg (issue generated here)

Image

demDerived_reaches_split_filtered_0.gpkg (issue still here)

Image

HUC8 04040002 -- issue fixed

demDerived_reaches_split_filtered_0.gpkg (issue fixed)

Image

@hhs732
Copy link
Contributor

hhs732 commented Jan 24, 2025

By fixing the demDerived_reaches_split_filtered_0.gpkg, the nan values are also deleted from the hydrotables. The issue fixed was applied in filter_catchments_and_add_attributes.py

@hhs732 hhs732 linked a pull request Jan 27, 2025 that will close this issue
[2pt] PR: fixed hydrotable nan values #1414
Open
13 tasks
@hhs732 hhs732 linked a pull request Jan 27, 2025 that will close this issue
Open
13 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hhs732 hhs732

Labels
bug Something isn't working High Priority
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[2pt] PR: fixed hydrotable nan values NOAA-OWP/inundation-mapping
3 participants
@hhs732 @CarsonPruitt-NOAA @RobHanna-NOAA