HTTP requests in django api test sometimes never end from version 3.9.0

[kigawas]

I'm trying to reproduce but couldn't. But the new executor works with old sync.py (https://github.com/django/asgiref/blob/3.8.1/asgiref/sync.py)

The test code structure is like:

import pytest
from django.test.client import AsyncClient
from django.utils import timezone

from polls.models import Question

pytestmark = pytest.mark.django_db


@pytest.fixture
async def question():
    return await Question.objects.acreate(
        question_text="Sample Question", pub_date=timezone.now()
    )


@pytest.fixture
async def api_response(question, async_client: AsyncClient):
    response = await async_client.get("/api/questions/")
    assert response.status_code == 200
    return response.json()


async def test_index1(question, api_response, async_client: AsyncClient):
    response = await async_client.delete( 
        "/polls/", headers={"X-Test-Data": "".join(api_response["questions"])}
    )
    assert response.status_code == 200 # could not reach here
    assert response.content == b"Hello, world. You're at the polls index."


async def test_index2(question, api_response, async_client: AsyncClient):
    response = await async_client.get(
        "/polls/", headers={"X-Test-Data": "".join(api_response["questions"])}
    )
    assert response.status_code == 200
    assert response.content == b"Hello, world. You're at the polls index."

pyproject.toml

[project]
description = "Add your description here"
name = "asgiref-debug"
readme = "README.md"
requires-python = ">=3.13"
version = "0.1.0"

dependencies = [
  "asgiref>=3.9.1",
  "django>=5.2.4",
  "django-ninja>=1.4.3",
]

[dependency-groups]
dev = [
  "pytest>=8.4.1",
  "pytest-asyncio>=1.1.0",
  "pytest-django>=4.11.1",
  "ruff>=0.12.4",
]

[tool.pytest.ini_options]
DJANGO_SETTINGS_MODULE = "mysite.settings"
asyncio_default_fixture_loop_scope = "session"
asyncio_mode = "auto"

[carltongibson]

@kigawas Could you add a description of the behaviour you're observing please? "Stuck" from the title isn't enough to go on.

(You have to give me some chance of reproducing your issue. Given that you couldn't providing as much clarity as possible is going to be required.)

[kigawas]

@carltongibson

The problem is simple: http request in api test (like await async_client.delete(...)) hangs there and never ends.

I could not reproduce, but tried debugging the sync.py (SyncToAsync and AsyncToAsync's __call__ method, it might be related with coroutine loop or task executor issues.

[eugenenelou]

I have the same problem, but it's not "sometimes", it always get stuck.

With the stack-trace of KeyboardInterrupt, i'm able to see that the line it's stuck on is in the run_until_future at

self._work_ready.wait()

in the __call__ method of async_to_sync.

I'm running this in pytest and also using django-ninja.

edit:

actually the stack trace is one level deeper, in current_thread_executor.py:

        try:    # restore state no matter what (e.g., KeyboardInterrupt)
            if timeout is None:
>               waiter.acquire()
E               KeyboardInterrupt

[carltongibson]

@eugenenelou Are you able to reduce it to a minimal example so we can have a look?

[eugenenelou]

I'm trying to get an example, for now I don't reproduce the stuck with i minimal example, but running two tests that use async_to_sync do not work, there are errors with an unclosed event loop:

    | Traceback (most recent call last):
    |   File "~/projects/atlas/backend/app/.venv/lib/python3.13/site-packages/_pytest/unraisableexception.py", line 33, in gc_collect_harder
    |     gc.collect()
    |     ~~~~~~~~~~^^
    | ResourceWarning: unclosed <socket.socket fd=35, family=1, type=1, proto=0>
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "~/projects/atlas/backend/app/.venv/lib/python3.13/site-packages/_pytest/unraisableexception.py", line 67, in collect_unraisable
    |     warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
    |     ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=35, family=1, type=1, proto=0>
    | Enable tracemalloc to get traceback where the object was allocated.
    | See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "~/projects/atlas/backend/app/.venv/lib/python3.13/site-packages/_pytest/unraisableexception.py", line 33, in gc_collect_harder
    |     gc.collect()
    |     ~~~~~~~~~~^^
    | ResourceWarning: unclosed <socket.socket fd=33, family=1, type=1, proto=0>
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "~/projects/atlas/backend/app/.venv/lib/python3.13/site-packages/_pytest/unraisableexception.py", line 67, in collect_unraisable
    |     warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
    |     ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=33, family=1, type=1, proto=0>
    | Enable tracemalloc to get traceback where the object was allocated.
    | See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
    +---------------- 3 ----------------
    | Traceback (most recent call last):
    |   File "~/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 759, in __del__
    |     _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
    |     ~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File "~/projects/atlas/backend/app/.venv/lib/python3.13/site-packages/_pytest/unraisableexception.py", line 67, in collect_unraisable
    |     warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
    |     ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | pytest.PytestUnraisableExceptionWarning: Exception ignored in: <function BaseEventLoop.__del__ at 0x7c4c427caa20>
    | Enable tracemalloc to get traceback where the object was allocated.
    | See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
    +------------------------------------

I don't have much time right now to reduce to a minimal example, it already took me some time to find this issue. I'll see if I can do that later.

[andersk]

Without being able to look at any of the code I can only speculate, but as a total shot in the dark, I wonder if you’re forgetting a reference to a task somewhere and the garbage collector is deleting it? See this warning in the documentation for asyncio.create_task and asyncio.shield:

Important: Save a reference to the result of this function, to avoid a task disappearing mid-execution. The event loop only keeps weak references to tasks. A task that isn’t referenced elsewhere may get garbage collected at any time, even before it’s done.

[kigawas]

I think I've found a fix to get around this.

Hi @eugenenelou can you try my fix above to see whether it's working for you?

[eugenenelou]

@kigawas I confirm it unblocks the test, but there is a new warning that happens a lot:

ResourceWarning: unclosed <socket.socket fd=712, family=1, type=1, proto=0>

[kigawas]

@eugenenelou I didn't find any warnings like yours

[eugenenelou]

I found that I was using nest_asyncio.apply() and removing that fixed the hanging for me.

[andersk]

I don’t know anything about nest_asyncio, but I see that it’s unmaintained, that it had a number of open compatibility issues, and that it works via invasive monkey-patching of asyncio. I would not expect anything like that to work well with asgiref.

[eugenenelou]

I don’t know anything about nest_asyncio, but I see that it’s unmaintained, that it had a number of open compatibility issues, and that it works via invasive monkey-patching of asyncio. I would not expect anything like that to work well with asgiref.

Indeed I had tried it to be able to await coroutine from any breakpoint in the debugger, but forgot to remove it.