RFC: Adopt Narwhals for dataframe support

[FBruzzesi]

Description

Hey there 👋🏼

As you might have guessed from a few PRs I submitted, I started to get more comfortable with the codebase and spotted a few things while trying to see how Narwhals would look like in great-tables.

I have already something which is working in the current status: main...FBruzzesi:great-tables:feat/narwhalify-tbl-data-module

It's a bit of a messy branch, and I would definitely clean it up before opening a PR.

However I have a couple questions before finalizing the work:

1. Would this be something you are interested in and in scope in the first place? 😂
2. Would you prefer an incremental approach, or a end-to-end change/adoption?

For the incremental approach there is a "lazy" way of doing it, which is wrapping most functionalities in the _tbl_data.py module. However this would actually end up in many additional calls (from_native/to_native), while the final goal would be to convert once to narwhals when ingesting and keeping the data like that as long as possible.

On the other hand, there are some modules which can be "narwhalified" independently (such as _data_color/base.py).

Still the main advantages will be noticeable once the full changes are in (I know it's not a perfect metric, but the branch above has almost a -500 net lines of code and it avoids many conversions)

Last but not least, if you decide to move forward, I would suggest to support from narwhals 2.0: I recently fixed a couple of bugs we had, spotted while working on such branch.

Let me know your thoughts (I am also happy to have a call if needed)

[machow]

Hey--thanks for looking at the code and making a bunch of useful changes! I definitely think Narwhals would be really useful to implement.

I would lean towards putting narwhals inside _tbl_data.py for now, and even making it an optional dependency for now(in the way Pandas and Polars are). That way, if someone passes a Narwhals frame to GT, we can trigger all the logic, but we also ensure the constrained frame API in _tbl_data.py remains in place.

IMO cutting 500 lines of code, but introducing the interaction of a full DataFrame API across Great Tables would be a net increase in complexity. Today, I could implement Great Tables over a custom class (call it "SimpleDataFrame") that is just a dictionary of lists, by registering 15-ish implementations in _tbl_data.py. But if Narwhals is baked into Great Tables outside that interface, I would have to implement Narwhals over SimpleDataFrame (a more complex task).

In any event, I think my big thoughts are:

• it would be awesome to have support for Narwhals frames.
• implementing within _tbl_data.py as an optional dependency to start would ensure we constrain our internal frame interface to a small subset of everything DataFrames can do.

I know there are definitely some areas where DataFrames leak into Great Tables code, and I'm a big fan of Narwhals in general, so would love to hear your thoughts!

edit: in my mind the big benefit of an initial implementation as described above, is we could basically roll out the feature "Great Tables over ibis by passing in a narwhals frame", etc.., and kick the tires a bit!

[FBruzzesi]

I will try to summarize the discussion we had on Monday to keep track of all ideas and requirements.

I would argue that there could be different phases in the adoption of Narwhals.

1. Reach coverage and feature parity with all currently support dataframes (pandas, polars and pyarrow).
   1.1. test: refactor test_data_color_utils.py through parametrization #740 sneaks in dataframe constructors in conftest which would set the foundation for testing all dataframe backends more easily
   1.2. fix: Address slicing for pyarrow array in data_color #741 fixes one issue for pyarrow
   1.3. I can imagine there would be other instances in which a test is currently not run for all backends
2. Once the test suite is more robust, we can be more comfortable in the refactor.
3. Step 1 would be to replace the eager backends by using Narwhals to dispatch native functionalities whenever possible
   3.1. This might not always be the case (e.g. for custom callable/expression). In these cases the dispatch mechanism already in _tbl_data.py can be used
   3.2 Eventually we can support narwhals expressions as well
4. Step 2 we would like to experiment with lazy backends (such as duckdb and ibis), which are already supported in Narwhals. Here there might be a few limitations since the row order is not guaranteed. Therefore we need to figure out if either:
   • Let the user describe how to sort the dataframe
   • Pass a dataframe that already has a row index column to uniquely identify each row
5. Last but not least, the extension mechanism that Narwhals would like to support can come handy if a custom class (e.g. SimpleDataFrame) wants to be developed to address cases in which a dataframe backend is not available (e.g. for plain python dicts or lists).

---

As of now, I have a working solution for 3, but I would rather see 1 and 2 first to be more comfortable in the changes.

@machow @rich-iannone please integrate whatever I missed from our conversation and/or share your thoughts on this "plan".