Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python code not working due to conda error #762

Open
bobturneruk opened this issue Feb 9, 2021 · 19 comments
Open

Python code not working due to conda error #762

bobturneruk opened this issue Feb 9, 2021 · 19 comments
Assignees
@DennisReddyhoff
Labels
Python

Comments

@bobturneruk
Copy link
Member

... trying to work on the java API and get the python upload/download code to work with this.

The python code is currently not working because Conda uses S3FS version 5.2 instead of the required 4.2. After I run "conda install s3fs=4.2.0" it all works fine.
@bobturneruk bobturneruk self-assigned this Feb 9, 2021
@bobturneruk
Copy link
Member Author

bobturneruk commented Feb 11, 2021 •
edited
Loading

Finally got to looking at this @B0SKAMP! Current environment.yml pins s3fs=0.4.2:

https://github.com/ScottishCovidResponse/data_pipeline_api/blob/b4b2e51b02a09ef038aea41ff322ffc171325c80/environment.yml#L16

How are you creating your conda environment? Maybe recreating it will fix?

@bobturneruk
Copy link
Member Author

I see now this is with installing from conda cloud, not from using a clone of the repo. I suspect it's the same issue as @johnnonweiler experienced.

@bobturneruk
Copy link
Member Author

This runs without error on Windows 10:

conda create --force -n dpapi_test python=3.6
conda activate dpapi_test
conda install -c scottishcovidresponse data_pipeline_api

I'll check linux next. Raises some interesting questions on how to test conda packages!

@mjb3
Copy link

mjb3 commented Feb 11, 2021 via email

@johnnonweiler
Copy link
Contributor

johnnonweiler commented Feb 11, 2021 •
edited
Loading

This is a different issue to the one I reported previously. (The issue that I reported appears to have been a problem with the conda server, which was fixed a few hours after I had reported it in our Zulip chat.)

I didn't see this issue at the time, but then I did see it later, after running conda update --all (on Windows 10). However, I tried again this morning, and the problem seems to have gone away. I don't know what has changed.

@johnnonweiler
Copy link
Contributor

I do still see the same problem. (I had forgotten that I need to run the download script to see the error message.)

@bobturneruk
Copy link
Member Author

What command yields the error, please, @johnnonweiler?

@johnnonweiler
Copy link
Contributor

This command yields the error:
python -m data_pipeline_api.registry.download --config config.yaml

@bobturneruk
Copy link
Member Author

Thanks. I'm getting (Windows 10, for my own reference):

Traceback (most recent call last):
  File "C:\Users\Bob Turner\.conda\envs\dpapi_test\lib\runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\Users\Bob Turner\.conda\envs\dpapi_test\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "C:\Users\Bob Turner\Documents\data_pipeline_api\data_pipeline_api\registry\__init__.py", line 1, in <module>
    from data_pipeline_api.registry.downloader import Downloader
  File "C:\Users\Bob Turner\Documents\data_pipeline_api\data_pipeline_api\registry\downloader.py", line 16, in <module>
    from data_pipeline_api.registry.common import (
  File "C:\Users\Bob Turner\Documents\data_pipeline_api\data_pipeline_api\registry\common.py", line 25, in <module>
    from data_pipeline_api.file_api import FileAPI
  File "C:\Users\Bob Turner\Documents\data_pipeline_api\data_pipeline_api\file_api.py", line 6, in <module>
    from dataclasses import dataclass
ModuleNotFoundError: No module named 'dataclasses'

@bobturneruk
Copy link
Member Author

Similar to Linux @johnnonweiler, @B0SKAMP?

@johnnonweiler
Copy link
Contributor

I'm using Windows 10. I get:

Traceback (most recent call last):
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry\__init__.py", line 1, in <module>
    from data_pipeline_api.registry.downloader import Downloader
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry\downloader.py", line 16, in <module>
    from data_pipeline_api.registry.common import (
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\data_pipeline_api\registry\common.py", line 23, in <module>
    from s3fs import S3FileSystem
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\s3fs\__init__.py", line 1, in <module>
    from .core import S3FileSystem, S3File
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\s3fs\core.py", line 13, in <module>
    import aiobotocore
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\__init__.py", line 1, in <module>
    from .session import get_session, AioSession
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\session.py", line 6, in <module>
    from .client import AioClientCreator, AioBaseClient
  File "C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\aiobotocore\client.py", line 2, in <module>
    from botocore.client import logger, PaginatorDocstring, ClientCreator, \
ImportError: cannot import name 'S3ArnParamHandler' from 'botocore.client' (C:\Users\jnonweil\Miniconda3\envs\dpapi_test\lib\site-packages\botocore\client.py

(As Bram suggests, I believe this can be fixed with conda install s3fs=4.2.0.)

@B0SKAMP
Copy link

B0SKAMP commented Feb 11, 2021 via email

@bobturneruk
Copy link
Member Author

Thanks, I'll try and reproduce on my machine. (I'm sure you're right about the fix, I get same s3fs version, I'm thinking a bit about future-proofing and want to make sure I understand what's going on on my box as well!).

@bobturneruk
Copy link
Member Author

And I kind of want to know why it's using that version as we specify the correct one here https://github.com/ScottishCovidResponse/data_pipeline_api/blob/master/environment.yml

@bobturneruk
Copy link
Member Author

Related to #776

@DennisReddyhoff
Copy link

and also #770

@bobturneruk
Copy link
Member Author

Install from conda cloud is no longer supported, use:

pip install git+https://github.com/ScottishCovidResponse/data_pipeline_api.git

@bobturneruk
Copy link
Member Author

Currently the command yields a different error:

python -m data_pipeline_api.registry.download --config config.yaml

Traceback (most recent call last):
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 154, in <module>
    download_cli()
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\bobturner\.conda\envs\data_pipeline_api\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 148, in download_cli
    download_from_config_file(config_filename=config, token=token)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 129, in download_from_config_file
    download_from_configs(run_metadata, read_configs, token, root)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 99, in download_from_configs
    read_config, run_metadata.get(RunMetadata.default_input_namespace)
  File "C:\Users\bobturner\Documents\data_pipeline_api\data_pipeline_api\registry\download.py", line 53, in _parse_read_config
    raise ValueError(f"No namespace specified for read_config {read_config}")
ValueError: No namespace specified for read_config {'where': {'data_product': 'human/mixing-matrix', 'component': 'mixing-matrix'}}

@bobturneruk
Copy link
Member Author

bobturneruk commented May 27, 2021 •
edited
Loading

Perhaps this is a problem with my config.yaml. Not sure if this is worth fixing given ongoing changes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DennisReddyhoff DennisReddyhoff

Labels
Python
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mjb3 @B0SKAMP @johnnonweiler @bobturneruk @DennisReddyhoff