Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

asyncioreactor: make sure task isn't deleted midway #274

Merged
merged 1 commit into from
Nov 30, 2023

Conversation

fruch
Copy link

@fruch fruch commented Nov 27, 2023

in push function, self._loop.create_task is called and it's return value is ignored. While the tests may pass now, this code is not correct and this example is called out in docs as a source of bugs, as python docs suggests.

Ref: https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task

@fruch fruch requested a review from Lorak-mmk November 27, 2023 13:49
@fruch fruch self-assigned this Nov 27, 2023
@@ -176,7 +176,10 @@ def push(self, data):
)
else:
# avoid races/hangs by just scheduling this, not using threadsafe
self._loop.create_task(self._push_msg(chunks))
background_tasks = set()

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be a member, accessed by self.background_tasks, no? Otherwise it can be garbage collected as well (unless PYthon's garbage collector works in some non-obvious way)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

python GC works with ref counting, and since this is referenced in the now in the task callback it won't be evicted

saving it on the instance might be problematic, since this can be called multiple times, and cause deletion of all of the tasks added

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so Python GC can't deal with cyclic references?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it can handle cyclic references, but I was referring to what you suggested, of saving the list of task on the instance:

    def __init__(self):
            background_tasks = set()

    def push(self):
            ....

            # avoid races/hangs by just scheduling this, not using threadsafe
            task = self._loop.create_task(self._push_msg(chunks))
            
            self.background_tasks.add(task)
            task.add_done_callback(self.background_tasks.discard)

this can end up exact the same issue this PR is trying to elevate, of task object getting cleared midway, if we might have multiple calls to push, there would be multiple tasks in the set, and all of them would be discarded at the same time, when the first task would end.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... I don't follow why it would delete all items - set.discard removed specified item, and callback receives finished task as argument, so will remove only this task from set.

Even in python docs example background_tasks is a global variable

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

o.k., I've was confused, and thought .clear was used

changed it

cassandra/io/asyncioreactor.py Show resolved Hide resolved
@fruch fruch force-pushed the fix_asyncio_task_handle branch from af8d36e to e78ff33 Compare November 28, 2023 13:46
@fruch fruch requested a review from Lorak-mmk November 28, 2023 13:46
in push function, self._loop.create_task is called and it's return value is ignored.
While the tests may pass now, this code is not correct and this example is called out
in docs as a source of bugs, as python docs suggests.

Ref: https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task
@fruch fruch force-pushed the fix_asyncio_task_handle branch from e78ff33 to b80960f Compare November 29, 2023 21:22
@fruch fruch merged commit 679ad24 into scylladb:master Nov 30, 2023
18 of 20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants