Merge branch 'main' into pythongh-123923-defer-func

Android/README.md

@@ -12,8 +12,12 @@ approachable user experience:
 
 ## Prerequisites
 
-Export the `ANDROID_HOME` environment variable to point at your Android SDK. If
-you don't already have the SDK, here's how to install it:
+First, make sure you have all the usual tools and libraries needed to build
+Python for your development machine.
+
+Second, you'll need an Android SDK. If you already have the SDK installed,
+export the `ANDROID_HOME` environment variable to point at its location.
+Otherwise, here's how to install it:
 
 * Download the "Command line tools" from <https://developer.android.com/studio>.
 * Create a directory `android-sdk/cmdline-tools`, and unzip the command line
@@ -37,11 +41,6 @@ development tools, which currently means Linux or macOS. This involves doing a
 cross-build where you use a "build" Python (for your development machine) to
 help produce a "host" Python for Android.
 
-First, make sure you have all the usual tools and libraries needed to build
-Python for your development machine. The only Android tool you need to install
-is the command line tools package above: the build script will download the
-rest.
-
 The easiest way to do a build is to use the `android.py` script. You can either
 have it perform the entire build process from start to finish in one step, or
 you can do it in discrete steps that mirror running `configure` and `make` for
@@ -80,12 +79,15 @@ call. For example, if you want a pydebug build that also caches the results from
 
 ## Testing
 
-The tests can be run on Linux, macOS, or Windows, although on Windows you'll
-have to build the `cross-build/HOST` subdirectory on one of the other platforms
-and copy it over.
+The test suite can be run on Linux, macOS, or Windows:
+
+* On Linux, the emulator needs access to the KVM virtualization interface, and
+  a DISPLAY environment variable pointing at an X server.
+* On Windows, you won't be able to do the build on the same machine, so you'll
+  have to copy the `cross-build/HOST` directory from somewhere else.
 
-The test suite can usually be run on a device with 2 GB of RAM, though for some
-configurations or test orders you may need to increase this. As of Android
+The test suite can usually be run on a device with 2 GB of RAM, but this is
+borderline, so you may need to increase it to 4 GB. As of Android
 Studio Koala, 2 GB is the default for all emulators, although the user interface
 may indicate otherwise. The effective setting is `hw.ramSize` in
 ~/.android/avd/*.avd/hardware-qemu.ini, whereas Android Studio displays the

Android/android.py

@@ -259,8 +259,8 @@ def setup_testbed():
              f"{temp_dir}/{outer_jar}", "gradle-wrapper.jar"])
 
 
-# run_testbed will build the app automatically, but it hides the Gradle output
-# by default, so it's useful to have this as a separate command for the buildbot.
+# run_testbed will build the app automatically, but it's useful to have this as
+# a separate command to allow running the app outside of this script.
 def build_testbed(context):
     setup_sdk()
     setup_testbed()
@@ -376,6 +376,8 @@ async def find_pid(serial):
     shown_error = False
     while True:
         try:
+            # `pidof` requires API level 24 or higher. The level 23 emulator
+            # includes it, but it doesn't work (it returns all processes).
             pid = (await async_check_output(
                 adb, "-s", serial, "shell", "pidof", "-s", APP_ID
             )).strip()
@@ -407,6 +409,7 @@ async def logcat_task(context, initial_devices):
     serial = await wait_for(find_device(context, initial_devices), startup_timeout)
     pid = await wait_for(find_pid(serial), startup_timeout)
 
+    # `--pid` requires API level 24 or higher.
     args = [adb, "-s", serial, "logcat", "--pid", pid,  "--format", "tag"]
     hidden_output = []
     async with async_process(
@@ -421,11 +424,15 @@ async def logcat_task(context, initial_devices):
                 # such messages, but other components might.
                 level, message = None, line
 
+            # Exclude high-volume messages which are rarely useful.
+            if context.verbose < 2 and "from python test_syslog" in message:
+                continue
+
             # Put high-level messages on stderr so they're highlighted in the
             # buildbot logs. This will include Python's own stderr.
             stream = (
                 sys.stderr
-                if level in ["E", "F"]  # ERROR and FATAL (aka ASSERT)
+                if level in ["W", "E", "F"]  # WARNING, ERROR, FATAL (aka ASSERT)
                 else sys.stdout
             )
 
@@ -573,8 +580,9 @@ def parse_args():
     test = subcommands.add_parser(
         "test", help="Run the test suite")
     test.add_argument(
-        "-v", "--verbose", action="store_true",
-        help="Show Gradle output, and non-Python logcat messages")
+        "-v", "--verbose", action="count", default=0,
+        help="Show Gradle output, and non-Python logcat messages. "
+        "Use twice to include high-volume messages which are rarely useful.")
     device_group = test.add_mutually_exclusive_group(required=True)
     device_group.add_argument(
         "--connected", metavar="SERIAL", help="Run on a connected device. "
@@ -591,6 +599,13 @@ def parse_args():
 
 def main():
     install_signal_handler()
+
+    # Under the buildbot, stdout is not a TTY, but we must still flush after
+    # every line to make sure our output appears in the correct order relative
+    # to the output of our subprocesses.
+    for stream in [sys.stdout, sys.stderr]:
+        stream.reconfigure(line_buffering=True)
+
     context = parse_args()
     dispatch = {"configure-build": configure_build_python,
                 "make-build": make_build_python,

Doc/c-api/memory.rst

@@ -102,30 +102,38 @@ All allocating functions belong to one of three different "domains" (see also
 strategies and are optimized for different purposes. The specific details on
 how every domain allocates memory or what internal functions each domain calls
 is considered an implementation detail, but for debugging purposes a simplified
-table can be found at :ref:`here <default-memory-allocators>`. There is no hard
-requirement to use the memory returned by the allocation functions belonging to
-a given domain for only the purposes hinted by that domain (although this is the
-recommended practice). For example, one could use the memory returned by
-:c:func:`PyMem_RawMalloc` for allocating Python objects or the memory returned
-by :c:func:`PyObject_Malloc` for allocating memory for buffers.
+table can be found at :ref:`here <default-memory-allocators>`.
+The APIs used to allocate and free a block of memory must be from the same domain.
+For example, :c:func:`PyMem_Free` must be used to free memory allocated using :c:func:`PyMem_Malloc`.
 
 The three allocation domains are:
 
 * Raw domain: intended for allocating memory for general-purpose memory
   buffers where the allocation *must* go to the system allocator or where the
   allocator can operate without the :term:`GIL`. The memory is requested directly
-  to the system.
+  from the system. See :ref:`Raw Memory Interface <raw-memoryinterface>`.
 
 * "Mem" domain: intended for allocating memory for Python buffers and
   general-purpose memory buffers where the allocation must be performed with
   the :term:`GIL` held. The memory is taken from the Python private heap.
+  See :ref:`Memory Interface <memoryinterface>`.
 
-* Object domain: intended for allocating memory belonging to Python objects. The
-  memory is taken from the Python private heap.
+* Object domain: intended for allocating memory for Python objects. The
+  memory is taken from the Python private heap. See :ref:`Object allocators <objectinterface>`.
 
-When freeing memory previously allocated by the allocating functions belonging to a
-given domain,the matching specific deallocating functions must be used. For example,
-:c:func:`PyMem_Free` must be used to free memory allocated using :c:func:`PyMem_Malloc`.
+.. note::
+
+  The :term:`free-threaded <free threading>` build requires that only Python objects are allocated using the "object" domain
+  and that all Python objects are allocated using that domain. This differs from the prior Python versions,
+  where this was only a best practice and not a hard requirement.
+
+  For example, buffers (non-Python objects) should be allocated using :c:func:`PyMem_Malloc`,
+  :c:func:`PyMem_RawMalloc`, or :c:func:`malloc`, but not :c:func:`PyObject_Malloc`.
+
+  See :ref:`Memory Allocation APIs <free-threaded-memory-allocation>`.
+
+
+.. _raw-memoryinterface:
 
 Raw Memory Interface
 ====================
@@ -299,6 +307,8 @@ versions and is therefore deprecated in extension modules.
 * ``PyMem_DEL(ptr)``
 
 
+.. _objectinterface:
+
 Object allocators
 =================
 

Doc/c-api/type.rst

@@ -345,8 +345,12 @@ The following functions and structs are used to create
       The :c:member:`~PyTypeObject.tp_new` of the metaclass is *ignored*.
       which may result in incomplete initialization.
       Creating classes whose metaclass overrides
-      :c:member:`~PyTypeObject.tp_new` is deprecated and in Python 3.14+ it
-      will be no longer allowed.
+      :c:member:`~PyTypeObject.tp_new` is deprecated.
+
+   .. versionchanged:: 3.14
+
+      Creating classes whose metaclass overrides
+      :c:member:`~PyTypeObject.tp_new` is no longer allowed.
 
 .. c:function:: PyObject* PyType_FromSpecWithBases(PyType_Spec *spec, PyObject *bases)
 
@@ -362,8 +366,12 @@ The following functions and structs are used to create
       The :c:member:`~PyTypeObject.tp_new` of the metaclass is *ignored*.
       which may result in incomplete initialization.
       Creating classes whose metaclass overrides
-      :c:member:`~PyTypeObject.tp_new` is deprecated and in Python 3.14+ it
-      will be no longer allowed.
+      :c:member:`~PyTypeObject.tp_new` is deprecated.
+
+   .. versionchanged:: 3.14
+
+      Creating classes whose metaclass overrides
+      :c:member:`~PyTypeObject.tp_new` is no longer allowed.
 
 .. c:function:: PyObject* PyType_FromSpec(PyType_Spec *spec)
 
@@ -378,8 +386,12 @@ The following functions and structs are used to create
       The :c:member:`~PyTypeObject.tp_new` of the metaclass is *ignored*.
       which may result in incomplete initialization.
       Creating classes whose metaclass overrides
-      :c:member:`~PyTypeObject.tp_new` is deprecated and in Python 3.14+ it
-      will be no longer allowed.
+      :c:member:`~PyTypeObject.tp_new` is deprecated.
+
+   .. versionchanged:: 3.14
+
+      Creating classes whose metaclass overrides
+      :c:member:`~PyTypeObject.tp_new` is no longer allowed.
 
 .. raw:: html
 
@@ -492,14 +504,10 @@ The following functions and structs are used to create
          See :ref:`PyMemberDef documentation <pymemberdef-offsets>`
          for details.
 
-      The following fields cannot be set at all when creating a heap type:
+      The following internal fields cannot be set at all when creating a heap
+      type:
 
-      * :c:member:`~PyTypeObject.tp_vectorcall`
-        (use :c:member:`~PyTypeObject.tp_new` and/or
-        :c:member:`~PyTypeObject.tp_init`)
-
-      * Internal fields:
-        :c:member:`~PyTypeObject.tp_dict`,
+      * :c:member:`~PyTypeObject.tp_dict`,
         :c:member:`~PyTypeObject.tp_mro`,
         :c:member:`~PyTypeObject.tp_cache`,
         :c:member:`~PyTypeObject.tp_subclasses`, and
@@ -519,6 +527,12 @@ The following functions and structs are used to create
         :c:member:`~PyBufferProcs.bf_releasebuffer` are now available
         under the :ref:`limited API <limited-c-api>`.
 
+     .. versionchanged:: 3.14
+
+        The field :c:member:`~PyTypeObject.tp_vectorcall` can now set
+        using ``Py_tp_vectorcall``.  See the field's documentation
+        for details.
+
    .. c:member:: void *pfunc
 
       The desired value of the slot. In most cases, this is a pointer

Doc/c-api/typeobj.rst

@@ -2137,11 +2137,40 @@ and :c:data:`PyType_Type` effectively act as defaults.)
 
 .. c:member:: vectorcallfunc PyTypeObject.tp_vectorcall
 
-   Vectorcall function to use for calls of this type object.
-   In other words, it is used to implement
-   :ref:`vectorcall <vectorcall>` for ``type.__call__``.
-   If ``tp_vectorcall`` is ``NULL``, the default call implementation
-   using :meth:`~object.__new__` and :meth:`~object.__init__` is used.
+   A :ref:`vectorcall function <vectorcall>` to use for calls of this type
+   object (rather than instances).
+   In other words, ``tp_vectorcall`` can be used to optimize ``type.__call__``,
+   which typically returns a new instance of *type*.
+
+   As with any vectorcall function, if ``tp_vectorcall`` is ``NULL``,
+   the *tp_call* protocol (``Py_TYPE(type)->tp_call``) is used instead.
+
+   .. note::
+
+      The :ref:`vectorcall protocol <vectorcall>` requires that the vectorcall
+      function has the same behavior as the corresponding ``tp_call``.
+      This means that ``type->tp_vectorcall`` must match the behavior of
+      ``Py_TYPE(type)->tp_call``.
+
+      Specifically, if *type* uses the default metaclass,
+      ``type->tp_vectorcall`` must behave the same as
+      :c:expr:`PyType_Type->tp_call`, which:
+
+      - calls ``type->tp_new``,
+
+      - if the result is a subclass of *type*, calls ``type->tp_init``
+        on the result of ``tp_new``, and
+
+      - returns the result of ``tp_new``.
+
+      Typically, ``tp_vectorcall`` is overridden to optimize this process
+      for specific :c:member:`~PyTypeObject.tp_new` and
+      :c:member:`~PyTypeObject.tp_init`.
+      When doing this for user-subclassable types, note that both can be
+      overridden (using :py:func:`~object.__new__` and
+      :py:func:`~object.__init__`, respectively).
+
+
 
    **Inheritance:**
 

Doc/howto/argparse.rst

@@ -444,7 +444,7 @@ And the output:
 
    options:
      -h, --help            show this help message and exit
-     -v {0,1,2}, --verbosity {0,1,2}
+     -v, --verbosity {0,1,2}
                            increase output verbosity
 
 Note that the change also reflects both in the error message as well as the

Doc/howto/descriptor.rst

@@ -389,7 +389,9 @@ Here are three practical data validation utilities:
 
         def validate(self, value):
             if value not in self.options:
-                raise ValueError(f'Expected {value!r} to be one of {self.options!r}')
+                raise ValueError(
+                    f'Expected {value!r} to be one of {self.options!r}'
+                )
 
     class Number(Validator):
 
@@ -469,6 +471,7 @@ The descriptors prevent invalid instances from being created:
     Traceback (most recent call last):
         ...
     ValueError: Expected -5 to be at least 0
+
     >>> Component('WIDGET', 'metal', 'V')    # Blocked: 'V' isn't a number
     Traceback (most recent call last):
         ...
@@ -1004,7 +1007,6 @@ here is a pure Python equivalent that implements most of the core functionality:
             if doc is None and fget is not None:
                 doc = fget.__doc__
             self.__doc__ = doc
-            self.__name__ = ''
 
         def __set_name__(self, owner, name):
             self.__name__ = name
@@ -1303,8 +1305,8 @@ mean, median, and other descriptive statistics that depend on the data. However,
 there may be useful functions which are conceptually related but do not depend
 on the data.  For instance, ``erf(x)`` is handy conversion routine that comes up
 in statistical work but does not directly depend on a particular dataset.
-It can be called either from an object or the class:  ``s.erf(1.5) --> .9332`` or
-``Sample.erf(1.5) --> .9332``.
+It can be called either from an object or the class:  ``s.erf(1.5) --> 0.9332``
+or ``Sample.erf(1.5) --> 0.9332``.
 
 Since static methods return the underlying function with no changes, the
 example calls are unexciting:

Doc/howto/free-threading-extensions.rst

@@ -181,6 +181,8 @@ Some of these functions were added in Python 3.13.  You can use the
 to provide implementations of these functions for older Python versions.
 
 
+.. _free-threaded-memory-allocation:
+
 Memory Allocation APIs
 ======================
 

Doc/library/ctypes.rst

@@ -51,7 +51,7 @@ function call fails.
 
 
 Here are some examples for Windows. Note that ``msvcrt`` is the MS standard C
-library containing most standard C functions, and uses the cdecl calling
+library containing most standard C functions, and uses the ``cdecl`` calling
 convention::
 
    >>> from ctypes import *