CTRP-style bounded quantification and verifytypes

[erictraut]

Discussed in #9445

^{Originally posted by wence- November 12, 2024}

$ pyright --version
pyright 1.1.388

I have class hierarchy for a simple tree-structured DSL that uses CTRP-style generics to provide typing for the "children" of a node.

Distilled:

from __future__ import annotations

from typing import Any

__all__ = [
    "Expr", "NonTerminal", "Terminal", "OneLevel",
]

class Node[T: Node[Any]]:
    children: tuple[T, ...]


class Expr(Node["Expr"]):
    pass

class Other(Node["Other"]):
    pass

class NonTerminal(Expr):
    def __init__(self, *exprs: Expr):
        self.children = exprs


class Terminal(Expr):
    def __init__(self):
        self.children = ()


class OneLevel(Expr):
    def __init__(self, *terminal: Terminal):
        self.children = terminal

# reveal_type(Expr().children)
# reveal_type(NonTerminal().children)
# reveal_type(Terminal().children)
# reveal_type(OneLevel().children)

If I ask pyright to reveal_type for an instance of any of Expr, NonTerminal, OneLevel, or Terminal, I obtain tuple[Expr, ...] (which I expect). So there are no type errors here.

Additionally, as expected, if I introduce:

class OtherExpr(Node["OtherExpr"]):
   pass

class Bad(Expr):
   def __init__(self, *bad: OtherExpr):
       self.children = bad

Then pyright complains because tuple[OtherExpr, ...] is not a subtype of tuple[Expr, ...].

However, if I install this as a package, call the above file baseoverride.py, and include:

# __init__.py
from myexample.baseoverride import Node, Expr, NonTerminal, OneLevel

__all__ = ["Node", "Expr", "NonTerminal", "OneLevel"]

# pyproject.toml
[project]
name = "myexample"
requires-python = ">=3.12"
version = "1.0"

Then pyright --verifytypes myexample reports some errors:

$ pyright --verifytypes myexample
Module name: "myexample"
Package directory: "/home/coder/.conda/envs/rapids/lib/python3.12/site-packages/myexample"
Module directory: "/home/coder/.conda/envs/rapids/lib/python3.12/site-packages/myexample"
Path of py.typed file: "/home/coder/.conda/envs/rapids/lib/python3.12/site-packages/myexample/py.typed"

Public modules: 2
   myexample
   myexample.baseoverride

Symbols used in public interface:
myexample.baseoverride.NonTerminal.children
  /home/coder/.conda/envs/rapids/lib/python3.12/site-packages/myexample/baseoverride.py:21:14 - error: Ambiguous base class override
    Type declared in base class is "tuple[T@Node, ...]"
    Type inferred in child class is "tuple[Expr, ...]"
myexample.baseoverride.OneLevel.__init__
  /home/coder/.conda/envs/rapids/lib/python3.12/site-packages/myexample/baseoverride.py:30:9 - error: Type of parameter "terminal" is partially unknown
    Parameter type is "Terminal"
myexample.baseoverride.Terminal.children
  /home/coder/.conda/envs/rapids/lib/python3.12/site-packages/myexample/baseoverride.py:26:14 - error: Ambiguous base class override
    Type declared in base class is "tuple[T@Node, ...]"
    Type inferred in child class is "tuple[()]"
    Inferred child class type is missing type annotation and could be inferred differently by type checkers
myexample.baseoverride.OneLevel.children
  /home/coder/.conda/envs/rapids/lib/python3.12/site-packages/myexample/baseoverride.py:31:14 - error: Ambiguous base class override
    Type declared in base class is "tuple[T@Node, ...]"
    Type inferred in child class is "tuple[Terminal, ...]"

Symbols exported by "myexample": 17
  With known type: 8
  With ambiguous type: 9
  With unknown type: 0

Other symbols referenced but not exported by "myexample": 0
  With known type: 0
  With ambiguous type: 0
  With unknown type: 0

Symbols without documentation:
  Functions without docstring: 0
  Functions without default param: 0
  Classes without docstring: 10

Type completeness score: 47.1%

Completed in 0.552sec

I think these are all the same class of error, so let's consider the NonTerminal case:

myexample.baseoverride.NonTerminal.children
  /home/coder/.conda/envs/rapids/lib/python3.12/site-packages/myexample/baseoverride.py:21:14 - error: Ambiguous base class override
    Type declared in base class is "tuple[T@Node, ...]"
    Type inferred in child class is "tuple[Expr, ...]"

I think I agree that the type of the children property in NonTerminal is tuple[Expr, ...]. However, in the base class Expr, because the generic typevar T is bound (to Expr), I think the children property should also have type tuple[Expr, ...], whereas in verifytypes mode, pyright thinks it is T@Node. This is in contrast to the reveal_type result (where pyright thinks it is tuple[Expr, ...]).

For the other two cases, I suppose that the assignments to children are to tighter types than tuple[Expr, ...], but I think that this is OK, because () is-a tuple[Expr, ...] as is tuple[Terminal, ...].

My rationale for having this setup is to avoid having to explicitly provide type annotations for the children property in every child class, since the only thing that I want to advertise of it is that is a tuple[Expr, ...].