fix(python): Relative path resolution for plugin libraries

[anath2]

Following up on #21675.
The relative path plugin cannot be resolved because the root .venv is not included in the calculated path. This fixes it by calculating the path relative to the parent of sys.prefix

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 7 lines in your changes missing coverage. Please review.

Project coverage is 80.48%. Comparing base (0ca485a) to head (002d2d6).
Report is 47 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/polars-plan/src/dsl/function_expr/plugin.rs	30.00%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #21911      +/-   ##
==========================================
- Coverage   80.48%   80.48%   -0.01%     
==========================================
  Files        1636     1636              
  Lines      236700   236709       +9     
  Branches     2693     2693              
==========================================
+ Hits       190504   190507       +3     
- Misses      45563    45568       +5     
- Partials      633      634       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[amotzop]

I think this still assumes your virtual env is in your current working directory, which is not always the case

[anath2]

Thanks for pointing that out. Probably the correct approach should to be to find the
[path to virtual environment from cwd] / [path to lib.so from .venv]

[amotzop]

Someone can still change their cwd. Maybe a more foolproof solution would be to keep the path relative to the virtual environment and have polars concat that relative path to the full path of the venv when it tries to load a plugin?

[anath2]

What do you mean by change of cwd? os.path.relpath(path.resolve(), Path.cwd()) should still load correctly I think. At least on *nix systems

[amotzop]

I think in the current state the following code won't work:

import os
import polars as pl
import polars_hash as plh
ldf = pl.LazyFrame({"a": ["1", "2", "3"]}).with_columns(h=plh.col("a").chash.sha2_256())
os.chdir("..")
df = ldf.collect()

because the relative path calculation happens when the LazyFrame is created but the actual loading of the plugin happens only when it's collected. This might seem a bit convoluted, but in #21643 they talk about serializing and deserializing LazyFrames, which is exactly where such problems can occur.

I propose to save the plugin's path relative to the venv when it's created (as in your previous PR), but to change the code that actually loads the plugin (in polars-plan/src/dsl/function_expr/plugin.rs) to append the current venv path to the relative path. I think this should work in almost all cases.

[anath2]

Ah okay I see what you mean now. I did not consider that directory could change before collecting the LazyFrame

[mat-ej]

we just tested this solution and it seems to work well. I was able to move graphs from one .venv and run them in another .venv. Do you think this could get some attention and perhaps get finally merged ? @ritchie46

[ritchie46]

Alright. Thanks all!