Impala import errors in docker image

[Aakcht]

In new Hue docker images (gethue/hue:20221031-140101) I'm getting the following errors during docker image start:

[31/Oct/2022 22:51:59 -0700] query_api ERROR Some application are not enabled: No module named 'impala.server'
Traceback (most recent call last):
File "/usr/share/hue/apps/jobbrowser/src/jobbrowser/apis/query_api.py", line 45, in <module>
from impala.server import get_api as get_impalad_api, _get_impala_server_url
ModuleNotFoundError: No module named 'impala.server' .

Also UI can't load with the following error:

[31/Oct/2022 22:53:53 -0700] middleware INFO Processing exception: No module named 'impala.conf': Traceback (most recent call last):
File "/usr/share/hue/build/env/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/lib/python3.8/contextlib.py", line 75, in inner
return func(*args, **kwds)
File "/usr/share/hue/apps/jobbrowser/src/jobbrowser/views.py", line 117, in apps
return render('job_browser.mako', request, {
File "/usr/share/hue/desktop/core/src/desktop/lib/django_util.py", line 242, in render
return _render_to_response(
File "/usr/share/hue/desktop/core/src/desktop/lib/django_util.py", line 161, in _render_to_response
return django_mako.render_to_response(template, *args, **kwargs)
File "/usr/share/hue/desktop/core/src/desktop/lib/django_mako.py", line 127, in render_to_response
return HttpResponse(render_to_string(template_name, data_dictionary), **kwargs)
File "/usr/share/hue/desktop/core/src/desktop/lib/django_mako.py", line 116, in render_to_string_normal
result = template.render(**data_dict)
File "/usr/share/hue/build/env/lib/python3.8/site-packages/mako/template.py", line 439, in render
return runtime._render(self, self.callable_, args, data)
File "/usr/share/hue/build/env/lib/python3.8/site-packages/mako/runtime.py", line 874, in _render
_render_context(
File "/usr/share/hue/build/env/lib/python3.8/site-packages/mako/runtime.py", line 916, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/usr/share/hue/build/env/lib/python3.8/site-packages/mako/runtime.py", line 943, in _exec_template
callable_(context, *args, **kwargs)
File "/tmp/tmp64o8k9f0/jobbrowser/job_browser.mako.py", line 39, in render_body
from impala.conf import COORDINATOR_URL as IMPALA_COORDINATOR_URL
ModuleNotFoundError: No module named 'impala.conf'

Looks like this happens because for some reason Hue can't import /usr/share/hue/apps/impala/src . This can be tested by executing the following command in docker container:

 /usr/share/hue/build/env/bin/python3.8 -c 'from impala.conf import COORDINATOR_URL as IMPALA_COORDINATOR_URL' 

In older hue docker images (gethue/hue:20220524-140101) it works fine and I do not see any issues.

As a workaround it is possible to manually set PYTHONPATH env with all the packages, for example:

PYTHONPATH="/usr/share/hue:/usr/share/hue/apps/about/src:/usr/share/hue/apps/beeswax/gen-py:/usr/share/hue/apps/beeswax/src:/usr/share/hue/apps/filebrowser/src:/usr/share/hue/apps/hbase/gen-py:/usr/share/hue/apps/hbase/src:/usr/share/hue/apps/help/src:/usr/share/hue/apps/hive/src:/usr/share/hue/apps/impala/gen-py:/usr/share/hue/apps/impala/src:/usr/share/hue/apps/jobbrowser/src:/usr/share/hue/apps/jobsub/src:/usr/share/hue/apps/metastore/src:/usr/share/hue/apps/oozie/src:/usr/share/hue/apps/pig/src:/usr/share/hue/apps/proxy/src:/usr/share/hue/apps/rdbms/src:/usr/share/hue/apps/search/src:/usr/share/hue/apps/security/src:/usr/share/hue/apps/spark/src:/usr/share/hue/apps/sqoop/src:/usr/share/hue/apps/useradmin/src:/usr/share/hue/apps/zookeeper/src:/usr/share/hue/build/env/lib/python38.zip:/usr/share/hue/build/env/lib/python3.8:/usr/share/hue/build/env/lib/python3.8/lib-dynload:/usr/lib/python3.8:/usr/share/hue/build/env/lib/python3.8/site-packages:/usr/share/hue/desktop/core/src:/usr/share/hue/desktop/libs/aws/src:/usr/share/hue/desktop/libs/azure/src:/usr/share/hue/desktop/libs/hadoop/src:/usr/share/hue/desktop/libs/indexer/src:/usr/share/hue/desktop/libs/libanalyze/src:/usr/share/hue/desktop/libs/liboauth/src:/usr/share/hue/desktop/libs/liboozie/src:/usr/share/hue/desktop/libs/librdbms/src:/usr/share/hue/desktop/libs/libsaml/src:/usr/share/hue/desktop/libs/libsentry/src:/usr/share/hue/desktop/libs/libsolr/src:/usr/share/hue/desktop/libs/libzookeeper/src:/usr/share/hue/desktop/libs/metadata/src:/usr/share/hue/desktop/libs/notebook/src:/usr/share/hue/desktop/libs/dashboard/src:/usr/share/hue/desktop/libs/kafka/src:/usr/share/hue/build/env/local/lib/python3.8/dist-packages:/usr/local/lib/python3.8/dist-packages:/usr/lib/python3/dist-packages"

After this everything works fine in the new Hue docker image, but I'm not sure if it's correct way to fix the problem. Maybe some fix shoud be done in Hue Dockerfile?

[Harshg999]

Hey @Aakcht, thanks for reporting the issue.
We'll take a look!

[Harshg999]

Does this still happen even if you give the configs for Hive/Impala?

[Aakcht]

I can't test it with Impala config(since I don't have Impala available), but the issue still happens with Hive config on the new docker image.

[Aakcht]

To add to my previous comments, here is how it is possible to see the difference between the old and the new image with docker:

# for new image, this throws import error:
docker run --rm -it gethue/hue:20221031-140101 /usr/share/hue/build/env/bin/python3.8 -c 'from impala.conf import COORDINATOR_URL as IMPALA_COORDINATOR_URL'

# for old image, this works without errors:
docker run --rm -it gethue/hue:20220524-140101 /usr/share/hue/build/env/bin/python3.8 -c 'from impala.conf import COORDINATOR_URL as IMPALA_COORDINATOR_URL'

[Harshg999]

Update: An internal tracking JIRA has been created to look further into this issue. Thanks.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity and is not "roadmap" labeled or part of any milestone. Remove stale label or comment or this will be closed in 5 days.

[Harshg999]

Hey @Aakcht, the latest docker images should be good now.
We have removed shipping impyla package which was messing up with the Impala path I believe.