Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(ray_plugin): remove runtime_env from RayJobConfig #1690

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pingsutw
Copy link
Member

Although Ray can install Python packages at runtime, we should encourage users to build the image at compile time using imageSpec.

Note: Also find a bug while using runtime_env with the latest ray.

2024-06-10 21:42:03,280 INFO cli.py:304 -- Tailing logs until the job exits (disable with --no-wait):

Traceback (most recent call last):
  File "/usr/local/bin/pyflyte-execute", line 8, in <module>
    sys.exit(execute_task_cmd())
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 506, in execute_task_cmd
    _execute_task(
  File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 148, in f
    return outer_f(inner_f, args, kwargs)
  File "/usr/local/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 178, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 379, in _execute_task
    _task_def = resolver_obj.load_task(loader_args=resolver_args)
  File "/usr/local/lib/python3.10/site-packages/flytekit/core/utils.py", line 306, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/flytekit/core/python_auto_container.py", line 252, in load_task
    task_module = importlib.import_module(name=task_module)  # type: ignore
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root/ray_plugin/ray_example.py", line 47, in <module>
    @ray.remote
NameError: name 'ray' is not defined
2024-06-10 21:42:20,667 ERR cli.py:68 -- --------------------------------------------
2024-06-10 21:42:20,668 ERR cli.py:69 -- Job 'a6j6hhs26x5rksq8x4r5-n0-0-65w78' failed
2024-06-10 21:42:20,668 ERR cli.py:70 -- --------------------------------------------
2024-06-10 21:42:20,668 INFO cli.py:83 -- Status message: Job entrypoint command failed with exit code 1, last available logs (truncated to 20,000 chars):
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/root/ray_plugin/ray_example.py", line 47, in <module>
    @ray.remote
NameError: name 'ray' is not defined

@pingsutw pingsutw marked this pull request as draft June 10, 2024 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants