AutoDL not working with MinIO backend (migrating from Google Cloud) {google-cloud-merged branch}

[ObadaS]

When changing the .env file to use MinIO instead of Google Cloud on an existing deployment, then trying to download a file that's stored in MinIO (the files were copied from the Google cloud buckets into the MinIO buckets), we get this error.

Traceback:

File "/usr/local/lib/python2.7/site-packages/django/core/handlers/exception.py" in inner
  41.             response = get_response(request)

File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _legacy_get_response
  249.             response = self._get_response(request)

File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  187.                 response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in _get_response
  185.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py" in view
  68.             return self.dispatch(request, *args, **kwargs)

File "/usr/local/lib/python2.7/site-packages/django/views/generic/base.py" in dispatch
  88.         return handler(request, *args, **kwargs)

File "/app/codalab/apps/web/views.py" in get
  1293.         if not submission.detailed_results_ready:

File "/app/codalab/apps/web/models.py" in detailed_results_ready
  1529.         if self.detailed_results_file and self.detailed_results_file.url and self.detailed_results_file.file:

File "/usr/local/lib/python2.7/site-packages/django/db/models/fields/files.py" in _get_file
  51.             self._file = self.storage.open(self.name, 'rb')

File "/usr/local/lib/python2.7/site-packages/django/core/files/storage.py" in open
  38.         return self._open(name, mode)

File "/usr/local/lib/python2.7/site-packages/storages/backends/s3boto.py" in _open
  371.         f = self.file_class(name, mode, self)

File "/usr/local/lib/python2.7/site-packages/storages/backends/s3boto.py" in __init__
  64.         self.key = storage.bucket.get_key(self._storage._encode_name(name))

File "/usr/local/lib/python2.7/site-packages/boto/s3/bucket.py" in get_key
  193.         key, resp = self._get_key_internal(key_name, headers, query_args_l)

File "/usr/local/lib/python2.7/site-packages/boto/s3/bucket.py" in _get_key_internal
  200.                                                 query_args=query_args)

File "/usr/local/lib/python2.7/site-packages/boto/s3/connection.py" in make_request
  671.             retry_handler=retry_handler

File "/usr/local/lib/python2.7/site-packages/boto/connection.py" in make_request
  1071.                           retry_handler=retry_handler)

File "/usr/local/lib/python2.7/site-packages/boto/connection.py" in _mexe
  927.                 request.authorize(connection=self)

File "/usr/local/lib/python2.7/site-packages/boto/connection.py" in authorize
  377.         connection._auth_handler.add_auth(self, **kwargs)

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in add_auth
  755.                                                          **kwargs)

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in add_auth
  574.         string_to_sign = self.string_to_sign(req, canonical_request)

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in string_to_sign
  514.         sts.append(self.credential_scope(http_request))

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in credential_scope
  496.         region_name = self.determine_region_name(http_request.host)

File "/usr/local/lib/python2.7/site-packages/boto/auth.py" in determine_region_name
  690.         return region_name

Exception Type: UnboundLocalError at /my/competition/submission/43690/input.zip
Exception Value: local variable 'region_name' referenced before assignment

I am pretty sure that MinIO support was added later, starting (maybe) with this commit which is not present in the current branch for autodl.

[Didayolo]

Hi @ObadaS, can you put also the command you are trying to run? And a quick summary of what we are trying to achieve here? Thank you.

[ObadaS]

@Didayolo I updated the original post with more details

[ObadaS]

I made a new branch to try things out.
For now, all MinIO commits have been merged into this branch.
Commit: 687db4c
Branch: https://github.com/codalab/codalab-competitions/tree/autodlMinio

The docker images successfully builds, but the Django and Worker_Site containers crashes on startup with the following errors:

Django

Traceback (most recent call last):
  File "scripts/initialize.py", line 15, in <module>
    from configurations import importer
  File "/usr/local/lib/python2.7/site-packages/configurations/__init__.py", line 2, in <module>
    from .base import Settings, Configuration
  File "/usr/local/lib/python2.7/site-packages/configurations/base.py", line 7, in <module>
    from .utils import uppercase_attributes
  File "/usr/local/lib/python2.7/site-packages/configurations/utils.py", line 5, in <module>
    from django.utils.importlib import import_module
ImportError: No module named importlib
Traceback (most recent call last):
  File "scripts/initialize.py", line 15, in <module>
    from configurations import importer
  File "/usr/local/lib/python2.7/site-packages/configurations/__init__.py", line 2, in <module>
    from .base import Settings, Configuration
  File "/usr/local/lib/python2.7/site-packages/configurations/base.py", line 7, in <module>
    from .utils import uppercase_attributes
  File "/usr/local/lib/python2.7/site-packages/configurations/utils.py", line 5, in <module>
    from django.utils.importlib import import_module
ImportError: No module named importlib
/app/docker/run_django.sh: line 31: gunicorn: command not found

Worker_Site

Traceback (most recent call last):
  File "/usr/local/bin/celery", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/usr/local/lib/python2.7/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 309, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 469, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 489, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/usr/local/lib/python2.7/site-packages/celery/app/utils.py", line 235, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/usr/local/lib/python2.7/site-packages/celery/bin/base.py", line 492, in symbol_by_name
    return symbol_by_name(name, imp=imp)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/app/codalab/codalab/__init__.py", line 5, in <module>
    from .celery import app as celery_app
  File "/app/codalab/codalab/celery.py", line 12, in <module>
    from configurations import importer
  File "/usr/local/lib/python2.7/site-packages/configurations/__init__.py", line 2, in <module>
    from .base import Settings, Configuration
  File "/usr/local/lib/python2.7/site-packages/configurations/base.py", line 7, in <module>
    from .utils import uppercase_attributes
  File "/usr/local/lib/python2.7/site-packages/configurations/utils.py", line 5, in <module>
    from django.utils.importlib import import_module
ImportError: No module named importlib