Poetry and Databricks

[cm-halfspace]

Does anyone have any experience with Poetry and Databricks?

We want to use it at my company where we have a databricks cluster running.
Currently we use an init script to install the python packages which is basically:

/databricks/python/bin/pip install -r /dbfs/mnt/bronze/libraries/requirements.txt

We want to move over to Poetry, however, we are having trouble getting Poetry up and running in the init scripts. It is a big mono-repo with sub-projects:

[tool.poetry]
name = "CompanyX"
version = "0.1.0"
description = ""
authors = []
readme = "README.md"
packages = []

[tool.poetry.group.main.dependencies]
python = ">=3.10,<3.12"

[tool.poetry.group.package1.dependencies]
package1 = { path = "PythonPackages/Package1", develop = true}

[tool.poetry.group.package2.dependencies]
package2 = { path = "PythonPackages/Package2", develop = true}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Does anyone have any experience in this area? I get almost zero google results on experiences with Databricks and Poetry combined.

[miltad]

Any progress on this topic so far?

[cm-halfspace]

From our side, unfortunately no. Right now we use Poetry for dependency management and then export a requirements.txt file that Databricks can then install, because we couldn't get Databricks to work with Poetry (virtual environments kept crashing). Super annoying since it just adds an unnecessary extra layer of complexity and brittleness.

[saraelshawa]

Hi, I was wondering if you found a fix for this? My environment keeps crashing as well when I try to get Databricks to work with Poetry. Any help would be appreciated, thanks!

[abhiaagarwal]

My company is using poetry and databricks in production. Check this out. https://github.com/databricks/bundle-examples/tree/main/knowledge_base/python_wheel_poetry.

[saraelshawa]

Thank you @abhiaagarwal !

[laigner]

Hello, I don't know if I get you right but what about using GitHub actions to automatically build a python wheel using poetry. Afterwards, use another GitHub action to upload the wheel to a location to which your Databricks service has access to. Then you can install your package as an own library, cluster wide, and use its functionality.

[abhiaagarwal]

Seems there's some work currently being done on this end for the Databricks Asset Bundles fyi. https://github.com/databricks/bundle-examples/tree/python-wheel-poetry/knowledge_base/python_wheel_poetry

[simonrmonk]

I'm leveraging a poetry plugin to run poetry build commands that play nicely with a mono-repo. Any idea if it will be possible to using poetry build in a databricks asset bundle that leverage a plugin? The plugin I have in mind is here: https://github.com/gerbenoostra/poetry-plugin-mono-repo-deps/

[pimenta888]

Same here, I was wondering if you found a fix for this?

[gilbertofp16]

Maybe a workaround how my team uses Poetry and Databricks
https://medium.com/p/6714af19c43d

[greatvovan]

@cm-halfspace I use poetry in notebook-scoped virtual environments. E.g.

1. Install poetry in the notebook environment:
   %pip install poetry

2. Create temp directory (optional if we have poetry.lock in the repo)

import os, tempfile
tmp_dir = tempfile.mkdtemp()
# We can pass env. variables only one way: Python -> %sh
os.environ['TMP_DIR'] = tmp_dir

3. Install libraries:

%sh
# cd <directory with poetry.toml> - if not in the root
if [ ! -f "poetry.lock" ]; then
  echo Copying to temp dir, because poetry.lock needs to be produced \
    and the current location is read-only.
  cp pyproject.toml $TMP_DIR
  cd $TMP_DIR
fi
poetry config virtualenvs.create false  # Optional, poetry should detect venv
poetry install --no-root --no-interaction --no-ansi

4. Lunch you code:

import my_module
my_module.entrypoint(...)

As you can see, we are using base poetry functionality, there is no magic here.

⬆ this works both in user notebooks and automated workloads (e.g. jobs under a service principal), though in jobs:

• the task need to be notebook-based,
• the task source needs to be source control.

I don't know how to achieve this in init scripts, because you would need the repository checked out at the init run time, which is hard to achieve due to authentication requirements, but I hope my example provides a reasonable alternative.