Initial Asyncio Module PR [1/3] #741

yuce · 2025-09-19T16:56:49Z

This is the initial asyncio support.

Added the asyncio module which contains public asyncio API
Added the internal module and internal/asyncio_ modules, which contains the private asyncio API/implementation.
Added asyncio support for the Map proxy. Currently near cache, transactions, and lock releated-methods are not supported. The near cache support will come in another PR. Transactions will likely not be supported. Locks may need a different design.
I didn't include the API docs, in order to make the PR smaller. I'll add them in another PR.
The following tests are ported from the old API, and they work without changes (besides making them compatible with async/await syntax):
- tests/integration/asyncio/authentication_tests
- tests/integration/asyncio/backup_acks_tests
- tests/integration/asyncio/client_test (one test is not ported, due to its Topic DDS dependency)
- tests/integration/asyncio/proxy/map_test

Most of the code in this PR was duplicated to the internal module by prefixing them with asyncio_. For example ROOT/cluster.py was duplicated/modifed as internal/asyncio_cluster.py

Here is the diff between modules in this PR vs their counterparts in the old API:
https://gist.github.com/yuce/56e79a29a1d4d1d996788381d489c0a4

codecov-commenter · 2025-09-19T17:09:09Z

Codecov Report

❌ Patch coverage is 76.17091% with 580 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.41%. Comparing base (5c7a4d5) to head (6ced889).

Files with missing lines	Patch %	Lines
hazelcast/internal/asyncio_connection.py	76.16%	149 Missing ⚠️
hazelcast/internal/asyncio_proxy/map.py	78.26%	125 Missing ⚠️
hazelcast/internal/asyncio_invocation.py	72.41%	80 Missing ⚠️
hazelcast/internal/asyncio_cluster.py	63.88%	65 Missing ⚠️
hazelcast/internal/asyncio_compact.py	32.89%	51 Missing ⚠️
hazelcast/internal/asyncio_listener.py	79.23%	38 Missing ⚠️
hazelcast/asyncio/client.py	84.36%	33 Missing ⚠️
hazelcast/internal/asyncio_proxy/base.py	77.53%	31 Missing ⚠️
hazelcast/internal/asyncio_proxy/manager.py	90.00%	4 Missing ⚠️
hazelcast/internal/asyncio_reactor.py	96.49%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #741      +/-   ##
==========================================
- Coverage   95.33%   93.41%   -1.93%     
==========================================
  Files         378      389      +11     
  Lines       21992    24426    +2434     
==========================================
+ Hits        20967    22818    +1851     
- Misses       1025     1608     +583

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

yuce · 2025-09-26T10:42:46Z

I've added the diff between modules in this PR vs their counterparts in the old API to the PR description.

gbarnett-hz

Had a quick look.

For others: probably easier to use https://difftastic.wilfred.me.uk/introduction.html to make life easier.

Haven't studied the reactor in-detail and associated logic in detail. From the diff it looked like the changes were relatively minor, albeit many of them.

I can review those in-depth if needed (or you cannot find people).

gbarnett-hz · 2025-09-29T09:46:52Z

hazelcast/asyncio/client.py

+    _CLIENT_ID = AtomicInteger()
+
+    @classmethod
+    async def create_and_start(cls, config: Config = None, **kwargs) -> "HazelcastClient":


General note -- I'm sure there are others.

If we're trying to get this API nice w.r.t. typing as well then this will probably show some error by default as None is not applicable for Config. Might be more prominent here as I think this is entry point into client creation.

Same for __init__ -- config: Config | None.

Might be missing something in the diff: in client.py there's a load of documentation -- is it elsewhere, or why omit it for this one? (most likely doc applicable from __init__)

re: Docs, I mentioned that in the PR description:

I didn't include the API docs, in order to make the PR smaller. I'll add them in another PR.

If you use better diff tool it doesn't make a difference. Give that one I mentioned a go.

I can review those in-depth if needed (or you cannot find people).

Yes, please.

At least with PyCharm, the user gets the correct type annotation.
There is this project in case you'd like to try the asyncio module:
https://github.com/yuce/hazelcast-asyncio-sample

In any case, I pushed a PR that adds explicit |None:
baa3bc1

PyCharm may be more forgiving -- I've never used it. pyright or pyrefly are good tools to use to determine compliance to the type invariants specified using type annotations.

We check the typings with mypy.

gbarnett-hz · 2025-09-29T10:07:14Z

hazelcast/asyncio/client.py

+            raise
+        _logger.info("Client started")
+
+    async def get_map(self, name: str) -> Map[KeyType, ValueType]:


Intention is only to support also VC in near (immediate) term?

Map is the only proxy included, in order to make the PR small. Other proxies, except VC, may be excluded from the beta release.

gbarnett-hz · 2025-09-29T10:19:35Z

hazelcast/internal/asyncio_proxy/manager.py

+}
+
+
+class ProxyManager:


Note to others: is in proxy/__init__.py

In the diff that I provided in the description I've shown that, but maybe it wasn't easy to notice.

The diff was not good.

gbarnett-hz · 2025-09-29T10:33:09Z

hazelcast/internal/asyncio_reactor.py

+from hazelcast.internal.asyncio_connection import Connection
+from hazelcast.core import Address
+
+_BUFFER_SIZE = 128000


Best to define centrally if possible given it's used across reactors.

I think they should be independent with each other.

The other constant hasn't been changed for ~4 years, same value.

There's not a big probability that it will change.
But I would either have to import it from hazelcast.reactor, or refactor the code so both the asyncore and the asyncio reactor imported it from a common module.
That either introduces a dependency between those reactor modules, or require changes in the "old" Python code, which I tried to avoid.

gbarnett-hz · 2025-09-29T10:36:50Z

hazelcast/internal/asyncio_reactor.py

+    def shutdown(self):
+        if not self._is_live:
+            return
+        # TODO: cancel tasks


compared to reactor.py is this correct?

start, shutdown are not necessary for the AsyncioReactor.
Removed them in the 3rd PR with this commit:
58783dc

hazelcast/internal/asyncio_connection.py

yuce added 5 commits September 19, 2025 18:17

Initial asyncio commit

cbe98dc

Updates

162fd17

Updates

9931956

Merge branch 'master' into asyncio-module

856e3df

Black

35384bf

yuce added 3 commits September 19, 2025 20:12

Updates

fdda120

Updates

fee5b45

Updates

fc2c38b

yuce requested review from anthony-griffin-dev, emreyigit, gbarnett-hz and ihsandemir September 19, 2025 17:30

yuce added 7 commits September 19, 2025 20:39

Removed docs, include HazelcastClient/Map in public API

1772031

Updates

170cf89

Merge branch 'master' into asyncio-module

539c904

black

22449a8

Ignore incorrect mypy errors

5406bc6

Updates

a417a4a

Updates

d00c480

yuce added this to the 5.6.0 milestone Sep 25, 2025

gbarnett-hz reviewed Sep 29, 2025

View reviewed changes

yuce added 6 commits September 29, 2025 14:42

Annotate optional params

baa3bc1

black

ebfc9e2

Remove update to test util

6928837

black

3e03cbf

black

51ced7a

update

e635b94

yuce changed the title ~~Initial Asyncio Module PR~~ Initial Asyncio Module PR [1/3] Oct 1, 2025

yuce added 2 commits October 2, 2025 12:06

Merge branch 'master' into asyncio-module

a630706

Merge branch 'master' into asyncio-module

6ced889

		}


		class ProxyManager:

Uh oh!

Initial Asyncio Module PR [1/3] #741

Are you sure you want to change the base?

Initial Asyncio Module PR [1/3] #741

Conversation

yuce commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

yuce commented Sep 26, 2025

Uh oh!

gbarnett-hz left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuce commented Sep 19, 2025 •

edited

Loading

codecov-commenter commented Sep 19, 2025 •

edited

Loading

gbarnett-hz left a comment •

edited

Loading