ibis expressions being eagerly evaluated in positron

[boshek]

👋 thank you for working on positron. what a lovely development.

I've been using ibis with positron and have noticed an issue that I wanted to raise.

ibis is a bit like dbplyr for R in that it is an expressive data frame library with the capacity to use the same syntax across different backends. It is also similar to dbplyr in that you build up your query lazily without evaluating it. That part is done explicitly (execute: ibis; collect: dbplyr).

What I am noticing in positron is that when operating interactively with ibis, positron will eagerly fetch results (i suspect for the data explorer or variable views possibly via a repr called somewhere?). This negates one of the key advantages of the whole "lazily executed" query, especially when building up a complex query because that eager fetch will take quite a bit of time making interactive work much less useful.

This is reproduced on version 2024.09.0-1. A good way to see this is run this in a vscode notebook and then run the same notebook in positron. The chunk of code that creates the query below executes much more quickly in vscode. If one comments out this line ibis.options.interactive = True, it appears as though repr remains fast. However running ibis in interactive mode is definitely the optimal development experience hence reporting this for positron.

reprex

import ibis
from ibis import _
import time

ibis.options.interactive = True

con = ibis.duckdb.connect()
taxi = con.read_parquet("s3://voltrondata-labs-datasets/nyc-taxi")

start_time = time.time()
large_parties_with_cash = (
    taxi
    .filter(_.passenger_count > 3)
    .filter(_.payment_type == "Cash")
)
end_time = time.time()

print(f"Run the query in {end_time - start_time:.4f} seconds")

[jthomasmock]

Thanks for the great feedback here @boshek ! I'm very excited for the future of ibis and we're motivated to make sure that it works well (and as intended!) in Positron.

Let me dig in a bit with the dev team to see if there's an area of the IDE we're being too "eager" in evaluating ibis specifically.

[cpcloud]

👋🏻 @jthomasmock Happy to help out on the Ibis side as y'all need!

[boshek]

@jthomasmock - just wondering if this discussion has gone anywhere. I've been using ibis a bunch lately but having to choose between positron and ibis.options.interactive = True is a place I don't want to be :)

[jthomasmock]

Thanks for reaching out @boshek !

@wesm -- I believe that we made some of the Data Explorer + Python dataframe state checking operations less greedy but I'm not 100% sure if we've solved this problem.

[wesm]

Sorry that I missed this discussion up until now. I think it should be relatively straightforward to add some special handling for Ibis expressions so they do not get executed by the Variables pane.

[wesm]

I just opened #5544 to track this

[ntluong95]

This a very cool thing. I just know about ibis today and tested it in Positron. so if I set ibis.options.interactive = True, and I import the data by using ibis.read_csv(), I noticed several things

• The dataframe is displayed weirdly in the Variable pane. I guess it is not expected and is something you are discussing? Is this possible to have a way to view it as a normal df like with pandas and polars
• It this possible to have column code completion suggestion. I think this is a feature lacking in both pandas and polars