Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

analyser_merge_shop_FR fails downloading Sirene data #2346

Open
jccomputing opened this issue Oct 1, 2024 · 0 comments
Open

analyser_merge_shop_FR fails downloading Sirene data #2346

jccomputing opened this issue Oct 1, 2024 · 0 comments

Comments

@jccomputing
Copy link

For about one week, the merge_shop_FR analyzer is failing because it can't dowload data. For example:

https://buildbot.osmose.openstreetmap.fr/#/builders/528/builds/1443

2024-10-01 03:11:36 france_alsace_bas_rhin : merge_shop_FR
2024-10-01 03:11:39   error: error on analyse merge_shop_FR...
2024-10-01 03:11:39     Traceback (most recent call last):
2024-10-01 03:11:39       File "/data/project/osmose/backend/./osmose_run.py", line 259, in execc
2024-10-01 03:11:39         if analyser_obj.timestamp() and remote_timestamp and analyser_obj.timestamp() <= remote_timestamp and analyser_obj.analyser_version() == remote_analyser_version:
2024-10-01 03:11:39            ^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge_Dynamic.py", line 51, in timestamp
2024-10-01 03:11:39         with self.analysers[0](self.config, self.error_file,  self.logger) as analyser_obj:
2024-10-01 03:11:39              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge_Dynamic.py", line 32, in __init__
2024-10-01 03:11:39         classs.__init__(self, config, error_file, logger, *args)
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/analyser_merge_shop_FR.py", line 56, in __init__
2024-10-01 03:11:39         self.init(
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge_Dynamic.py", line 67, in init
2024-10-01 03:11:39         Analyser_Merge_Point.init(self, url, name, source, load, mapping)
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge.py", line 1414, in init
2024-10-01 03:11:39         super().init(url, name, parser, load, conflate)
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge.py", line 1255, in init
2024-10-01 03:11:39         self.conflate.mapping.eval_static(self)
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge.py", line 1165, in eval_static
2024-10-01 03:11:39         self.eval_staticGroup(self.static2, analyser)
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge.py", line 1159, in eval_staticGroup
2024-10-01 03:11:39         r = colomn()
2024-10-01 03:11:39             ^^^^^^^^
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge.py", line 1270, in source
2024-10-01 03:11:39         return self.parser.source.as_tag_value()
2024-10-01 03:11:39                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge.py", line 454, in as_tag_value
2024-10-01 03:11:39         return " - ".join(filter(lambda x: x is not None, [self.attribution, self._get_millesime()]))
2024-10-01 03:11:39                                                                              ^^^^^^^^^^^^^^^^^^^^^
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge.py", line 440, in _get_millesime
2024-10-01 03:11:39         self.millesime = self.get_millesime()
2024-10-01 03:11:39                          ^^^^^^^^^^^^^^^^^^^^
2024-10-01 03:11:39       File "/data/project/osmose/backend/analysers/Analyser_Merge.py", line 532, in get_millesime
2024-10-01 03:11:39         downloader.requests_retry_session().head(self.fileUrl).headers['Last-Modified'],
2024-10-01 03:11:39         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
2024-10-01 03:11:39       File "/usr/lib/python3/dist-packages/requests/structures.py", line 52, in __getitem__
2024-10-01 03:11:39         return self._store[key.lower()][1]
2024-10-01 03:11:39                ~~~~~~~~~~~^^^^^^^^^^^^^
2024-10-01 03:11:39     KeyError: 'last-modified'

As seen here, it tries to access file on @cquest server but the last directory is not there anymore.

I don't know what's the proper fix to this problem and if the fix has to be on the source server or if osmose-backend code has to be adapted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant