not storing metadata for tiles without hash matched #278
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| if force_download: | ||
| logger.info(format_laz_log(fpath, "Forcing re-download")) | ||
| file_size, fpath, is_new, success, url_laz = download_laz( | ||
| file_size, fpath, is_new, nr_retries, success, url, url_laz, verify_ssl | ||
| ) | ||
| success = True |
There was a problem hiding this comment.
I moved all these because it was overwriting the success status and the is_new status when force_download is True > which was the case during our pipeline this means that a lot of tiles were set to "success=True" even if they were not.
| @@ -200,6 +200,13 @@ def compute_load_metadata( | |||
| """ | |||
| tile_id = context.partition_key | |||
| conn = context.resources.db_connection.connect | |||
| if not laz_files_ahn.success or laz_files_ahn.hash_name is None: | |||
There was a problem hiding this comment.
The filtering I mentioned above
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I found 2 main issues:
The success of the LAZ files was set to True even when we had the
force_downloadoption - which was the case in this runWhen storing the PDAL output to the metadata_ahn* tables, there was no filter for the unsuccessful LAZ tiles.
I added a filter to skip the metadata storage if the download or the hashes have failed.
This is what happened in this release's run:
Hashes are failing, the last download object gets a success=False and an empty path (.) but this is information is not used anywhere for filtering.
Then the lazdownload object is fed into pdal and of course there is no output:
The metadata is still stored in the metadata_table. Example from the metadata_ahn4 table (I have cleaned it now):
Specifically, these tiles had failed hashes:
AHN3 = 25gn2, 25hz2, 26cz1
AHN4 = 06hn2, 11fn2, 11fz2
I suggest we skip them and we ask Adriaan about it (why are the hashes diverging?)
For AHN5 the issue was different and it had to do with what I did in my previous commits.
I suggest we make a fresh download for all AHN5