Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement protocol #72

Closed
wants to merge 30 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
9b684f3
WIP add tests and dummy implementation for kafka protocol
Jun 16, 2019
6bdd53f
improve tests for kafka protocol
Jun 16, 2019
64a526a
finish typing for structs.py
Jun 16, 2019
9fa22e3
add basic implementation for struct serde
Jun 16, 2019
6196327
add test for array serde
Jun 16, 2019
abff002
switch to BinaryIO and use real varlen encoding for varint and varlong
Jun 16, 2019
a47b364
refactor and comment varlen/zigzag encoding
Jun 16, 2019
2608691
run black and optimize imports
Jun 16, 2019
cea8d86
import correct BinaryIO type
Jun 17, 2019
8e0c545
refactor structs.py a little
Jun 18, 2019
f710e75
refactor serializers
Jun 21, 2019
053114f
first working protocol request, refactoring required
Jun 21, 2019
451abfd
add dataclasses to requirements for python3.6
Jun 21, 2019
de41099
some refactoring
Jun 22, 2019
d25c30d
add beautifulsoup and requests to dev requirements
Jun 22, 2019
0ddd0b6
add gitignore for generated files
Jul 15, 2019
5f19963
WIP protocol code generator
Jul 15, 2019
1f885a6
add new api endpoint
Sep 23, 2019
5e74633
resolve api version compatibilities
Sep 23, 2019
1ead1af
Merge branch 'master' into implement-protocol
Sep 23, 2019
8e3fcc2
add dependency to dataclasses for python <3.7
Sep 23, 2019
0e472d5
create a few other helper files in schema_gernerator
Sep 23, 2019
262abd5
add generated api files
Sep 23, 2019
cf92c9b
reformat and refactor code
Sep 23, 2019
56888e6
run black with correct line length
Sep 24, 2019
52739db
fix api version query
Sep 24, 2019
3c73998
use test_config for protocol test
Sep 24, 2019
15938e1
run black on schema generator
Sep 24, 2019
3b7d907
optimize imports and suppress flake8 warnings in connection.py
Sep 24, 2019
5c84097
run black again on api files
Sep 24, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
162 changes: 92 additions & 70 deletions Pipfile.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 9 additions & 7 deletions esque/cli/output.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,21 +105,23 @@ def pretty_bytes(value: bytes) -> str:
return value.decode("UTF-8")


def pretty_duration(value: Any, *, multiplier: int = 1) -> str:
if not value:
def pretty_duration(orig_value: Any, *, multiplier: int = 1) -> str:
if not orig_value:
return ""

if type(value) != int:
value = int(value)
if type(orig_value) != int:
value = int(orig_value)
else:
value = orig_value

value *= multiplier

# Fix for conversion errors of ms > C_MAX_INT in some internal lib
if value > MILLISECONDS_PER_YEAR:
value = int(value / MILLISECONDS_PER_YEAR)
return pendulum.duration(years=value).in_words()
return f"{orig_value} ({pendulum.duration(years=value).in_words()})"

return pendulum.duration(milliseconds=value).in_words()
return f"{orig_value} ({pendulum.duration(milliseconds=value).in_words()})"


def pretty_topic_diffs(topics_config_diff: Dict[str, TopicDiff]) -> str:
Expand Down Expand Up @@ -173,7 +175,7 @@ def pretty_size(value: Any) -> str:
]
for sign, size in units:
if value >= size:
return f"{pretty_float(value / size)} {sign}"
return f"{value} ({pretty_float(value / size)} {sign})"


def bold(s: str) -> str:
Expand Down
100 changes: 100 additions & 0 deletions esque/protocol/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
from .connection import BrokerConnection

from .api import (
ApiKey,
ApiVersions,
Request,
RequestData,
ResponseData,
SUPPORTED_API_VERSIONS,
ProduceRequestData,
ProduceResponseData,
FetchRequestData,
FetchResponseData,
ListOffsetsRequestData,
ListOffsetsResponseData,
MetadataRequestData,
MetadataResponseData,
LeaderAndIsrRequestData,
LeaderAndIsrResponseData,
StopReplicaRequestData,
StopReplicaResponseData,
UpdateMetadataRequestData,
UpdateMetadataResponseData,
ControlledShutdownRequestData,
ControlledShutdownResponseData,
OffsetCommitRequestData,
OffsetCommitResponseData,
OffsetFetchRequestData,
OffsetFetchResponseData,
FindCoordinatorRequestData,
FindCoordinatorResponseData,
JoinGroupRequestData,
JoinGroupResponseData,
HeartbeatRequestData,
HeartbeatResponseData,
LeaveGroupRequestData,
LeaveGroupResponseData,
SyncGroupRequestData,
SyncGroupResponseData,
DescribeGroupsRequestData,
DescribeGroupsResponseData,
ListGroupsRequestData,
ListGroupsResponseData,
SaslHandshakeRequestData,
SaslHandshakeResponseData,
ApiVersionsRequestData,
ApiVersionsResponseData,
CreateTopicsRequestData,
CreateTopicsResponseData,
DeleteTopicsRequestData,
DeleteTopicsResponseData,
DeleteRecordsRequestData,
DeleteRecordsResponseData,
InitProducerIdRequestData,
InitProducerIdResponseData,
OffsetForLeaderEpochRequestData,
OffsetForLeaderEpochResponseData,
AddPartitionsToTxnRequestData,
AddPartitionsToTxnResponseData,
AddOffsetsToTxnRequestData,
AddOffsetsToTxnResponseData,
EndTxnRequestData,
EndTxnResponseData,
WriteTxnMarkersRequestData,
WriteTxnMarkersResponseData,
TxnOffsetCommitRequestData,
TxnOffsetCommitResponseData,
DescribeAclsRequestData,
DescribeAclsResponseData,
CreateAclsRequestData,
CreateAclsResponseData,
DeleteAclsRequestData,
DeleteAclsResponseData,
DescribeConfigsRequestData,
DescribeConfigsResponseData,
AlterConfigsRequestData,
AlterConfigsResponseData,
AlterReplicaLogDirsRequestData,
AlterReplicaLogDirsResponseData,
DescribeLogDirsRequestData,
DescribeLogDirsResponseData,
SaslAuthenticateRequestData,
SaslAuthenticateResponseData,
CreatePartitionsRequestData,
CreatePartitionsResponseData,
CreateDelegationTokenRequestData,
CreateDelegationTokenResponseData,
RenewDelegationTokenRequestData,
RenewDelegationTokenResponseData,
ExpireDelegationTokenRequestData,
ExpireDelegationTokenResponseData,
DescribeDelegationTokenRequestData,
DescribeDelegationTokenResponseData,
DeleteGroupsRequestData,
DeleteGroupsResponseData,
ElectPreferredLeadersRequestData,
ElectPreferredLeadersResponseData,
IncrementalAlterConfigsRequestData,
IncrementalAlterConfigsResponseData,
)
Loading