refactor(typing): Add `_native.py` #3086

Thanks @dangotbanned - I didn't check line by line, I assumed that most of the changes were simply due to the different import source and somplifications. I double checked the new module and left a couple of comments. Let me know what your thoughts are

FBruzzesi · 2025-09-20T13:57:10Z

narwhals/_native.py

+### (1) `Native(*Frame|Series)`
+Minimal [`Protocol`]s for matching *almost any* supported native type of that group:
+
+    class NativeThing(Protocol):
+        def something_common(self, *args: Any, **kwargs: Any) -> Any: ...
+
+Note:
+    This group is primarily a building block for more useful types.
+
+### (2) `Into(*Frame|Series)`
+*Publicly* exported [`TypeAlias`]s of **(1)**:
+
+    IntoThing: TypeAlias = NativeThing
+
+**But**, occasionally, there'll be an edge-case which we can spell like:
+
+    IntoThing: TypeAlias = Union[<type that does not fit the protocol>, NativeThing]
+
+Tip:
+    Reach for these when there **isn't a need to preserve** the original native type.
+
+### (3) `Into(*Frame|Series)T`
+*Publicly* exported [`TypeVar`]s, bound to **(2)**:
+
+    IntoThingT = TypeVar("IntoThingT", bound=IntoThing)
+
+Important:
+    In most situations, you'll want to use these as they **do preserve** the original native type.
+
+Putting it all together, we can now add a *narwhals-level* wrapper:
+
+    class Thing(Generic[IntoThingT]):
+        def to_native(self) -> IntoThingT: ...


Mostly nitpicking on the usage of Thing. I am trying to put myself into the shoes of a newcomer and I wonder if something along the following lines would be more helpful:

Suggested change

### (1) `Native(*Frame|Series)`

Minimal [`Protocol`]s for matching *almost any* supported native type of that group:

class NativeThing(Protocol):

def something_common(self, *args: Any, **kwargs: Any) -> Any: ...

Note:

This group is primarily a building block for more useful types.

### (2) `Into(*Frame|Series)`

*Publicly* exported [`TypeAlias`]s of **(1)**:

IntoThing: TypeAlias = NativeThing

**But**, occasionally, there'll be an edge-case which we can spell like:

IntoThing: TypeAlias = Union[<type that does not fit the protocol>, NativeThing]

Tip:

Reach for these when there **isn't a need to preserve** the original native type.

### (3) `Into(*Frame|Series)T`

*Publicly* exported [`TypeVar`]s, bound to **(2)**:

IntoThingT = TypeVar("IntoThingT", bound=IntoThing)

Important:

In most situations, you'll want to use these as they **do preserve** the original native type.

Putting it all together, we can now add a *narwhals-level* wrapper:

class Thing(Generic[IntoThingT]):

def to_native(self) -> IntoThingT: ...

### (1) `Native(*Frame|Series)`

Minimal [`Protocol`]s for matching *almost any* supported native type of that group:

class NativeObject(Protocol):

def some_common_method(self, *args: Any, **kwargs: Any) -> Any: ...

Note:

This group is primarily a building block for more useful types.

### (2) `Into(*Frame|Series)`

*Publicly* exported [`TypeAlias`]s of **(1)**:

IntoNarwhalsObject: TypeAlias = NativeObject

**But**, occasionally, there'll be an edge-case which we can spell like:

IntoNarwhalsObject: TypeAlias = Union[<type that does not fit the protocol>, NativeObject]

Tip:

Reach for these when there **isn't a need to preserve** the original native type.

### (3) `Into(*Frame|Series)T`

*Publicly* exported [`TypeVar`]s, bound to **(2)**:

IntoNarwhalsObjectT = TypeVar("IntoNarwhalsObjectT", bound=IntoNarwhalsObject)

Important:

In most situations, you'll want to use these as they **do preserve** the original native type.

Putting it all together, we can now add a *narwhals-level* wrapper:

class NarwhalsObject(Generic[IntoNarwhalsObjectT]):

def to_native(self) -> IntoNarwhalsObjectT: ...

Thanks for the review @FBruzzesi!

Mostly nitpicking on the usage of Thing

I am tempted to bikeshed on the name 😄 ...

But regardless of what exactly it is - would some kind of intro like this be helpful?

narwhals/narwhals/_translate.py

Lines 1 to 11 in e92fcc9

"""[Protocols] defining conversion methods between representations.

These come in 3 flavors and are [generic] to promote reuse.

The following examples use the placeholder types `Narwhal` and `Other`:

- `Narwhal`: some class written in `narwhals`.

- `Other`: any other class, could be native, compliant, or a builtin.

## `To<Other>`

When we want to convert or unwrap a `Narwhal` into an `Other`,

we provide an **instance** method:

Yes I think it might help, but really I don't want to make a fuss out of it - I am not a fan of using a generic thing, and in this situation (i.e. a python context?!) *Class or *Object would be more appropriate 😂

I am tempted to bikeshed on the name 😄 ...
But regardless of what exactly it is

Sorry @FBruzzesi, I'm reading this back now and it isn't coming across the way that I intended.

My suggestion (#3086 (comment)) was supposed to be another way to help with

I am trying to put myself into the shoes of a newcomer

But not saying that should be instead of choosing a better name than *Thing 🙂

So the naming scheme in (#3086 (comment)) is slightly different from what I was trying to demonstrate

and in this situation (i.e. a python context?!) *Class or *Object would be more appropriate 😂

Here's what each of those look like:

Actual Current Proposed Alt 1 Alt 2

NativeSeries NativeThing NativeObject NativeObject NativeClass

IntoSeries IntoThing IntoNarwhalsObject IntoObject IntoClass

IntoSeriesT IntoThingT IntoNarwhalsObjectT IntoObjectT IntoClassT

Series Thing NarwhalsObject Object Class

Note
FWIW, I don't like any of them now - including Thing 😭

At the end of the day it's a developer documentation, not even visible to a user so I won't be too picky nor a blocker.

If everything else is okay - we could do one more round of brainstorming a better name than Thing to get it over the line ☺️

For iteration over it, I am quite open to anything which is not called Thing 😂

For iteration over it, I am quite open to anything which is not called Thing 😂

Alright I'll try to churn out a list of non-Things to choose from shortly 😉

Okay @FBruzzesi here we go:

Array

Data

Column

Matrix

Dog

Scalar

LinkedList

Box

Bear

@dangotbanned I am sold! Let's ship it with Bear's and Dog's 😂

/sarcam

Really feel free to merge after resolving the conflict

Thanks @FBruzzesi 😍

If you decide on one you prefer (including Bear and Dog 😉), feel free to update the doc in a follow up 🙌

narwhals/_native.py

Very related to #3086 (comment)

- Use similar syntax to #2455 - Introduce things like #2455 - #3086 (comment) - #3086 (comment) - Sub-divide the 5 cases - Avoid using the terms *wrap* and *match* outside of the two groups - TODOs on #3086 (comment)

dangotbanned · 2025-10-17T09:27:57Z

@FBruzzesi how do you feel about the rest of this besides (#3086 (comment))?

I really want to move forward on it to unblock #3125 and I also noticed (main...chore/refactor-pandas-like-pyarrow-branching) would be touching some of the changed bits as well

If everything else is okay - we could do one more round of brainstorming a better name than Thing to get it over the line ☺️

FBruzzesi · 2025-10-17T10:11:46Z

@FBruzzesi how do you feel about the rest of this besides (#3086 (comment))?

I really want to move forward on it to unblock #3125 and I also noticed (main...chore/refactor-pandas-like-pyarrow-branching) would be touching some of the changed bits as well

If everything else is okay - we could do one more round of brainstorming a better name than Thing to get it over the line ☺️

Sorry @dangotbanned I went on vacation and somehow thought this PR was done. I will take another look in the weekend

FBruzzesi

Thanks @dangotbanned - I left a nitpick suggestion, everything else looks good to me. Sorry for the headache over naming things

narwhals/_native.py

Co-authored-by: Francesco Bruzzesi <[email protected]>

narwhals/stable/v1/typing.py

MarcoGorelli · 2025-10-17T16:31:06Z

thanks for your PR!

if you're both happy with it feel free to take it forwards

it wasn't clear to me on reading why we're now returning Into* instead of Native* in the tests but perhaps it'll become clear if i expand all the convos

dangotbanned · 2025-10-17T16:45:17Z

thanks for your PR!

if you're both happy with it feel free to take it forwards

Thanks @MarcoGorelli 😍

it wasn't clear to me on reading why we're now returning Into* instead of Native* in the tests but perhaps it'll become clear if i expand all the convos

Hmm, I seem to have done a bad job on the docs then 😅

Here I was trying to explain ...

narwhals/narwhals/_native.py

Lines 34 to 44 in 3a004e8

    
           ### (2) `Into<Thing>` 
        
           *Publicly* exported [`TypeAlias`]s of **(1)**: 
        
               IntoThing: TypeAlias = NativeThing 
        
           **But**, occasionally, there'll be an edge-case which we can spell like: 
        
               IntoThing: TypeAlias = Union[<type that does not fit the protocol>, NativeThing] 
        
           Tip: 
        
               Reach for these when there **isn't a need to preserve** the original native type.

... which means we can write like this

narwhals/narwhals/stable/v1/typing.py

Line 27 in 3a004e8

IntoDataFrame: TypeAlias = Union["NativeDataFrame", "DataFrameLike"]

narwhals/narwhals/_native.py

Line 279 in 3a004e8

IntoDataFrame: TypeAlias = NativeDataFrame

narwhals/narwhals/stable/v1/typing.py

Line 40 in 3a004e8

IntoLazyFrame: TypeAlias = "NativeLazyFrame"

narwhals/narwhals/_native.py

Line 292 in 3a004e8

IntoLazyFrame: TypeAlias = Union[NativeLazyFrame, NativeIbis]

This wasn't the original purpose of the aliases, but I've kinda reclaimed them to work as a lil barrier between the protocols

MarcoGorelli · 2025-10-17T17:04:57Z

cool, thanks!

Hmm, I seem to have done a bad job on the docs then 😅

oh no, you've done an excellent job, i just hadn't spent much time on it, happy to trust you two on this one

dangotbanned added 11 commits September 4, 2025 11:32

refactor(typing): Add _native.py w/ protocols, aliases

42850e3

Follow-up to (#3016 (comment))

refactor: Add guards

38bcb0a

refactor: Replace _namespace definitions

88907a0

refactor: Replace imports from _namespace

9584778

refactor: Replace imports from typing

9acfd4e

refactor: Replace duplicated stable typing

d066eb8

#3016 (comment)

refactor: remove unused TypeVar

d1f45bb

refactor: Deduplicate most Into* typing

c5ada4a

`v1` is different for the frame cases #3016 (comment)

cov

9508dd7

refactor: okay but smaller

57b86c4

fix(typing): NativeIbis is a NativeFrame

227ab27

dangotbanned added documentation Improvements or additions to documentation internal typing labels Sep 4, 2025

docs(typing): wip module doc

c6e163e

dangotbanned commented Sep 5, 2025

View reviewed changes

narwhals/_native.py Show resolved Hide resolved

docs: Cherry-pick some links

153c8ee

dangotbanned added a commit that referenced this pull request Sep 5, 2025

chore: Rename _typing.*Cudf* aliases to *CuDF*

8d241c8

- Follow-up to #3016 - Doesn't quite fit into #3086

dangotbanned mentioned this pull request Sep 5, 2025

chore: Rename _typing.*Cudf* aliases to *CuDF* #3090

Merged

10 tasks

Merge branch 'main' into refac-native-module

1829dd9

FBruzzesi pushed a commit that referenced this pull request Sep 6, 2025

chore: Rename _typing.*Cudf* aliases to *CuDF* (#3090)

9ca3602

- Follow-up to #3016 - Doesn't quite fit into #3086

Merge remote-tracking branch 'upstream/main' into refac-native-module

161ab6b

dangotbanned commented Sep 6, 2025

View reviewed changes

narwhals/_native.py Show resolved Hide resolved

dangotbanned added 6 commits September 6, 2025 21:31

Merge remote-tracking branch 'upstream/main' into refac-native-module

892d9d8

Merge branch 'main' into refac-native-module

3dbb3cc

Merge branch 'main' into refac-native-module

ff74f37

Merge remote-tracking branch 'upstream/main' into refac-native-module

e238589

docs(typing): Finish filling out module doc

b2cc37a

#3086 (comment)

Merge remote-tracking branch 'upstream/main' into refac-native-module

b70de63

dangotbanned marked this pull request as ready for review September 9, 2025 17:31

dangotbanned mentioned this pull request Sep 12, 2025

fix(typing): Overhaul @overloads in nw.from_native #3125

Draft

20 tasks

Merge remote-tracking branch 'upstream/main' into refac-native-module

3040310

dangotbanned mentioned this pull request Sep 15, 2025

feat: explain or similar API for inspecting plans ibis-project/ibis#9276

Open

1 task

dangotbanned added 2 commits September 18, 2025 14:26

Merge remote-tracking branch 'upstream/main' into refac-native-module

c181819

docs: Add a link to ibis issue walkthrough

16b0074

ibis-project/ibis#9276 (comment)

dangotbanned requested a review from FBruzzesi September 18, 2025 17:24

FBruzzesi reviewed Sep 20, 2025

View reviewed changes

dangotbanned added 2 commits September 20, 2025 16:46

Merge remote-tracking branch 'upstream/main' into refac-native-module

d395e1e

Merge remote-tracking branch 'upstream/main' into refac-native-module

f1ac582

dangotbanned added a commit that referenced this pull request Oct 1, 2025

feat(typing): DataFrame.from_native series typing!?!?!?!

85af320

Very related to #3086 (comment)

dangotbanned added 4 commits October 6, 2025 10:11

Merge branch 'main' into refac-native-module

56aa417

Merge remote-tracking branch 'upstream/main' into refac-native-module

3c88f4d

docs: Start rewriting, introduce some feedback

3683b06

- Use similar syntax to #2455 - Introduce things like #2455 - #3086 (comment) - #3086 (comment) - Sub-divide the 5 cases - Avoid using the terms *wrap* and *match* outside of the two groups - TODOs on #3086 (comment)

docs: Fluff up "Matching to an Implementation"

d060577

Merge branch 'main' into refac-native-module

7e7be3c

FBruzzesi approved these changes Oct 17, 2025

View reviewed changes

narwhals/_native.py Outdated Show resolved Hide resolved

Update narwhals/_native.py

3a004e8

Co-authored-by: Francesco Bruzzesi <[email protected]>

MarcoGorelli reviewed Oct 17, 2025

View reviewed changes

narwhals/stable/v1/typing.py Show resolved Hide resolved

dangotbanned added 2 commits October 17, 2025 20:24

Merge branch 'main' into refac-native-module

a4fea11

Merge remote-tracking branch 'upstream/main' into refac-native-module

31cc6f2

dangotbanned merged commit 556c6a6 into main Oct 20, 2025
31 of 34 checks passed

dangotbanned deleted the refac-native-module branch October 20, 2025 09:10

dangotbanned added a commit that referenced this pull request Oct 20, 2025

chore: re-sync imports following (#3086)

46abfe6

	"""[Protocols] defining conversion methods between representations.

	These come in 3 flavors and are [generic] to promote reuse.

	The following examples use the placeholder types `Narwhal` and `Other`:
	- `Narwhal`: some class written in `narwhals`.
	- `Other`: any other class, could be native, compliant, or a builtin.

	## `To<Other>`
	When we want to convert or unwrap a `Narwhal` into an `Other`,
	we provide an instance method:

Actual	Current	Proposed	Alt 1	Alt 2
`NativeSeries`	`NativeThing`	`NativeObject`	`NativeObject`	`NativeClass`
`IntoSeries`	`IntoThing`	`IntoNarwhalsObject`	`IntoObject`	`IntoClass`
`IntoSeriesT`	`IntoThingT`	`IntoNarwhalsObjectT`	`IntoObjectT`	`IntoClassT`
`Series`	`Thing`	`NarwhalsObject`	`Object`	`Class`

refactor(typing): Add _native.py #3086

refactor(typing): Add _native.py #3086

Uh oh!

Conversation

dangotbanned commented Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this? (check all applicable)

Related issues

Checklist

If you have comments or can explain your changes, please do so below

Uh oh!

Uh oh!

Uh oh!

dangotbanned commented Sep 18, 2025

Uh oh!

FBruzzesi commented Sep 19, 2025

Uh oh!

FBruzzesi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dangotbanned commented Oct 17, 2025

Uh oh!

FBruzzesi commented Oct 17, 2025

Uh oh!

FBruzzesi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MarcoGorelli commented Oct 17, 2025

Uh oh!

dangotbanned commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcoGorelli commented Oct 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

refactor(typing): Add `_native.py` #3086

refactor(typing): Add `_native.py` #3086

dangotbanned commented Sep 4, 2025 •

edited

Loading

dangotbanned commented Oct 17, 2025 •

edited

Loading