Merge branch 'master' into master

.github/workflows/test.yml

@@ -96,7 +96,7 @@ jobs:
           # We also run these checks with pre-commit in CI,
           # but it's useful to run them with tox too,
           # to ensure the tox env works as expected
-        - name: Formatting with Black + isort and code style with flake8
+        - name: Formatting with Black + isort and code style with ruff
           python: '3.10'
           arch: x64
           os: ubuntu-latest

.gitignore

@@ -57,6 +57,5 @@ test_capi
 *.o
 *.a
 test_capi
-/.mypyc-flake8-cache.json
 /mypyc/lib-rt/build/
 /mypyc/lib-rt/*.so

.pre-commit-config.yaml

@@ -13,14 +13,7 @@ repos:
     rev: 5.12.0  # must match test-requirements.txt
     hooks:
       - id: isort
-  - repo: https://github.com/pycqa/flake8
-    rev: 6.0.0  # must match test-requirements.txt
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.0.272  # must match test-requirements.txt
     hooks:
-      - id: flake8
-        additional_dependencies:
-          - flake8-bugbear==23.3.23  # must match test-requirements.txt
-          - flake8-noqa==1.3.1       # must match test-requirements.txt
-
-ci:
-  # We run flake8 as part of our GitHub Actions suite in CI
-  skip: [flake8]
+      - id: ruff

CONTRIBUTING.md

@@ -89,7 +89,7 @@ pytest -n0 -k 'test_name'
 pytest mypy/test/testcheck.py::TypeCheckSuite::check-dataclasses.test
 
 # Run the linter
-flake8
+ruff .
 
 # Run formatters
 black . && isort .

docs/source/extending_mypy.rst

@@ -159,7 +159,7 @@ This hook will be also called for instantiation of classes.
 This is a good choice if the return type is too complex
 to be expressed by regular python typing.
 
-**get_function_signature_hook** is used to adjust the signature of a function.
+**get_function_signature_hook()** is used to adjust the signature of a function.
 
 **get_method_hook()** is the same as ``get_function_hook()`` but for methods
 instead of module level functions.

docs/source/installed_packages.rst

@@ -119,7 +119,7 @@ The ``setup.py`` file could look like this:
 
 .. code-block:: python
 
-    from distutils.core import setup
+    from setuptools import setup
 
     setup(
         name="SuperPackageA",
@@ -129,11 +129,6 @@ The ``setup.py`` file could look like this:
         packages=["package_a"]
     )
 
-.. note::
-
-   If you use :doc:`setuptools <setuptools:index>`, you must pass the option ``zip_safe=False`` to
-   ``setup()``, or mypy will not be able to find the installed package.
-
 Some packages have a mix of stub files and runtime files. These packages also
 require a ``py.typed`` file. An example can be seen below:
 
@@ -150,7 +145,7 @@ The ``setup.py`` file might look like this:
 
 .. code-block:: python
 
-    from distutils.core import setup
+    from setuptools import setup
 
     setup(
         name="SuperPackageB",
@@ -180,7 +175,7 @@ The ``setup.py`` might look like this:
 
 .. code-block:: python
 
-    from distutils.core import setup
+    from setuptools import setup
 
     setup(
         name="SuperPackageC",

docs/source/protocols.rst

@@ -3,26 +3,24 @@
 Protocols and structural subtyping
 ==================================
 
-Mypy supports two ways of deciding whether two classes are compatible
-as types: nominal subtyping and structural subtyping.
-
-*Nominal* subtyping is strictly based on the class hierarchy. If class ``D``
-inherits class ``C``, it's also a subtype of ``C``, and instances of
-``D`` can be used when ``C`` instances are expected. This form of
-subtyping is used by default in mypy, since it's easy to understand
-and produces clear and concise error messages, and since it matches
-how the native :py:func:`isinstance <isinstance>` check works -- based on class
+The Python type system supports two ways of deciding whether two objects are
+compatible as types: nominal subtyping and structural subtyping.
+
+*Nominal* subtyping is strictly based on the class hierarchy. If class ``Dog``
+inherits class ``Animal``, it's a subtype of ``Animal``. Instances of ``Dog``
+can be used when ``Animal`` instances are expected. This form of subtyping
+subtyping is what Python's type system predominantly uses: it's easy to
+understand and produces clear and concise error messages, and matches how the
+native :py:func:`isinstance <isinstance>` check works -- based on class
 hierarchy.
 
-*Structural* subtyping is based on the operations that can be performed with an object. Class ``D`` is
-a structural subtype of class ``C`` if the former has all attributes
-and methods of the latter, and with compatible types.
+*Structural* subtyping is based on the operations that can be performed with an
+object. Class ``Dog`` is a structural subtype of class ``Animal`` if the former
+has all attributes and methods of the latter, and with compatible types.
 
-Structural subtyping can be seen as a static equivalent of duck
-typing, which is well known to Python programmers. Mypy provides
-support for structural subtyping via protocol classes described
-below.  See :pep:`544` for the detailed specification of protocols
-and structural subtyping in Python.
+Structural subtyping can be seen as a static equivalent of duck typing, which is
+well known to Python programmers. See :pep:`544` for the detailed specification
+of protocols and structural subtyping in Python.
 
 .. _predefined_protocols:
 
@@ -60,8 +58,7 @@ For example, ``IntList`` below is iterable, over ``int`` values:
 
 :ref:`predefined_protocols_reference` lists all protocols defined in
 :py:mod:`typing` and the signatures of the corresponding methods you need to define
-to implement each protocol (the signatures can be left out, as always, but mypy
-won't type check unannotated methods).
+to implement each protocol.
 
 Simple user-defined protocols
 *****************************
@@ -89,18 +86,12 @@ class:
        for item in items:
            item.close()
 
-   close_all([Resource(), open('some/file')])  # Okay!
+   close_all([Resource(), open('some/file')])  # OK
 
 ``Resource`` is a subtype of the ``SupportsClose`` protocol since it defines
 a compatible ``close`` method. Regular file objects returned by :py:func:`open` are
 similarly compatible with the protocol, as they support ``close()``.
 
-.. note::
-
-   The ``Protocol`` base class is provided in the ``typing_extensions``
-   package for Python 3.4-3.7. Starting with Python 3.8, ``Protocol``
-   is included in the ``typing`` module.
-
 Defining subprotocols and subclassing protocols
 ***********************************************
 
@@ -171,6 +162,13 @@ abstract:
    ExplicitSubclass()  # error: Cannot instantiate abstract class 'ExplicitSubclass'
                        # with abstract attributes 'attr' and 'method'
 
+Similarly, explicitly assigning to a protocol instance can be a way to ask the
+type checker to verify that your class implements a protocol:
+
+.. code-block:: python
+
+   _proto: SomeProto = cast(ExplicitSubclass, None)
+
 Invariance of protocol attributes
 *********************************
 

mypy-requirements.txt

@@ -1,5 +1,5 @@
 # NOTE: this needs to be kept in sync with the "requires" list in pyproject.toml
-typing_extensions>=3.10
+typing_extensions>=4.1.0
 mypy_extensions>=1.0.0
 typed_ast>=1.4.0,<2; python_version<'3.8'
 tomli>=1.1.0; python_version<'3.11'

mypy/build.py

@@ -31,22 +31,19 @@
     Callable,
     ClassVar,
     Dict,
-    Iterable,
     Iterator,
     Mapping,
     NamedTuple,
     NoReturn,
     Sequence,
     TextIO,
-    TypeVar,
 )
-from typing_extensions import Final, TypeAlias as _TypeAlias
-
-from mypy_extensions import TypedDict
+from typing_extensions import Final, TypeAlias as _TypeAlias, TypedDict
 
 import mypy.semanal_main
 from mypy.checker import TypeChecker
 from mypy.errors import CompileError, ErrorInfo, Errors, report_internal_error
+from mypy.graph_utils import prepare_sccs, strongly_connected_components, topsort
 from mypy.indirection import TypeIndirectionVisitor
 from mypy.messages import MessageBuilder
 from mypy.nodes import Import, ImportAll, ImportBase, ImportFrom, MypyFile, SymbolTable, TypeInfo
@@ -344,7 +341,9 @@ class CacheMeta(NamedTuple):
 
 
 # Metadata for the fine-grained dependencies file associated with a module.
-FgDepMeta = TypedDict("FgDepMeta", {"path": str, "mtime": int})
+class FgDepMeta(TypedDict):
+    path: str
+    mtime: int
 
 
 def cache_meta_from_dict(meta: dict[str, Any], data_json: str) -> CacheMeta:
@@ -2239,7 +2238,7 @@ def semantic_analysis_pass1(self) -> None:
         analyzer = SemanticAnalyzerPreAnalysis()
         with self.wrap_context():
             analyzer.visit_file(self.tree, self.xpath, self.id, options)
-        self.manager.errors.set_unreachable_lines(self.xpath, self.tree.unreachable_lines)
+        self.manager.errors.set_skipped_lines(self.xpath, self.tree.skipped_lines)
         # TODO: Do this while constructing the AST?
         self.tree.names = SymbolTable()
         if not self.tree.is_stub:
@@ -3077,7 +3076,7 @@ def load_graph(
             manager.errors.report(
                 -1,
                 -1,
-                "See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules "  # noqa: E501
+                "See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules "
                 "for more info",
                 severity="note",
             )
@@ -3165,7 +3164,7 @@ def load_graph(
                             manager.errors.report(
                                 -1,
                                 0,
-                                "See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules "  # noqa: E501
+                                "See https://mypy.readthedocs.io/en/stable/running_mypy.html#mapping-file-paths-to-modules "
                                 "for more info",
                                 severity="note",
                             )
@@ -3466,15 +3465,8 @@ def sorted_components(
     edges = {id: deps_filtered(graph, vertices, id, pri_max) for id in vertices}
     sccs = list(strongly_connected_components(vertices, edges))
     # Topsort.
-    sccsmap = {id: frozenset(scc) for scc in sccs for id in scc}
-    data: dict[AbstractSet[str], set[AbstractSet[str]]] = {}
-    for scc in sccs:
-        deps: set[AbstractSet[str]] = set()
-        for id in scc:
-            deps.update(sccsmap[x] for x in deps_filtered(graph, vertices, id, pri_max))
-        data[frozenset(scc)] = deps
     res = []
-    for ready in topsort(data):
+    for ready in topsort(prepare_sccs(sccs, edges)):
         # Sort the sets in ready by reversed smallest State.order.  Examples:
         #
         # - If ready is [{x}, {y}], x.order == 1, y.order == 2, we get
@@ -3499,100 +3491,6 @@ def deps_filtered(graph: Graph, vertices: AbstractSet[str], id: str, pri_max: in
     ]
 
 
-def strongly_connected_components(
-    vertices: AbstractSet[str], edges: dict[str, list[str]]
-) -> Iterator[set[str]]:
-    """Compute Strongly Connected Components of a directed graph.
-
-    Args:
-      vertices: the labels for the vertices
-      edges: for each vertex, gives the target vertices of its outgoing edges
-
-    Returns:
-      An iterator yielding strongly connected components, each
-      represented as a set of vertices.  Each input vertex will occur
-      exactly once; vertices not part of a SCC are returned as
-      singleton sets.
-
-    From https://code.activestate.com/recipes/578507/.
-    """
-    identified: set[str] = set()
-    stack: list[str] = []
-    index: dict[str, int] = {}
-    boundaries: list[int] = []
-
-    def dfs(v: str) -> Iterator[set[str]]:
-        index[v] = len(stack)
-        stack.append(v)
-        boundaries.append(index[v])
-
-        for w in edges[v]:
-            if w not in index:
-                yield from dfs(w)
-            elif w not in identified:
-                while index[w] < boundaries[-1]:
-                    boundaries.pop()
-
-        if boundaries[-1] == index[v]:
-            boundaries.pop()
-            scc = set(stack[index[v] :])
-            del stack[index[v] :]
-            identified.update(scc)
-            yield scc
-
-    for v in vertices:
-        if v not in index:
-            yield from dfs(v)
-
-
-T = TypeVar("T")
-
-
-def topsort(data: dict[T, set[T]]) -> Iterable[set[T]]:
-    """Topological sort.
-
-    Args:
-      data: A map from vertices to all vertices that it has an edge
-            connecting it to.  NOTE: This data structure
-            is modified in place -- for normalization purposes,
-            self-dependencies are removed and entries representing
-            orphans are added.
-
-    Returns:
-      An iterator yielding sets of vertices that have an equivalent
-      ordering.
-
-    Example:
-      Suppose the input has the following structure:
-
-        {A: {B, C}, B: {D}, C: {D}}
-
-      This is normalized to:
-
-        {A: {B, C}, B: {D}, C: {D}, D: {}}
-
-      The algorithm will yield the following values:
-
-        {D}
-        {B, C}
-        {A}
-
-    From https://code.activestate.com/recipes/577413/.
-    """
-    # TODO: Use a faster algorithm?
-    for k, v in data.items():
-        v.discard(k)  # Ignore self dependencies.
-    for item in set.union(*data.values()) - set(data.keys()):
-        data[item] = set()
-    while True:
-        ready = {item for item, dep in data.items() if not dep}
-        if not ready:
-            break
-        yield ready
-        data = {item: (dep - ready) for item, dep in data.items() if item not in ready}
-    assert not data, f"A cyclic dependency exists amongst {data!r}"
-
-
 def missing_stubs_file(cache_dir: str) -> str:
     return os.path.join(cache_dir, "missing_stubs")