Customizable DataBlob encoding type #8222
Draft
+788
−622
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stephanos
force-pushed
the
customizable-serialization
branch
10 times, most recently
from
cf59fcf to
840ae27
Compare
stephanos
changed the title
stephanos
force-pushed
the
customizable-serialization
branch
4 times, most recently
from
ca150aa to
c70e7e4
Compare
stephanos
commented
Sep 3, 2025
| @@ -13,6 +13,7 @@ This document describes the project's testing setup, utilities and best practice | |||
| - `TEMPORAL_TEST_LOG_FORMAT`: Controls the output format for test logs. Available options: `json` or `console` | |||
| - `TEMPORAL_TEST_LOG_LEVEL`: Sets the verbosity level for test logging. Available levels: `debug`, `info`, `warn`, `error`, `fatal` | |||
| - `TEMPORAL_TEST_OTEL_OUTPUT`: Enables OpenTelemetry (OTEL) trace output for failed tests to the provided file path. | |||
| - `TEMPORAL_TEST_DATA_ENCODING`: If set, overrides the default data blob encoding. Available options: `json`, `proto3`. | |||
stephanos
force-pushed
the
customizable-serialization
branch
14 times, most recently
from
528435d to
b43ad9f
Compare
stephanos
force-pushed
the
customizable-serialization
branch
from
b43ad9f to
d02dff7
Compare
stephanos
commented
Sep 3, 2025
| func QueueStateFromBlob(data *commonpb.DataBlob) (*persistencespb.QueueState, error) { | ||
| result := &persistencespb.QueueState{} | ||
| return result, Decode(data, result) | ||
| } |
There was a problem hiding this comment.
^ moved all this into serializer.go.
stephanos
commented
Sep 3, 2025
| SerializeEvents(batch []*historypb.HistoryEvent) (*commonpb.DataBlob, error) | ||
| DeserializeEvents(data *commonpb.DataBlob) ([]*historypb.HistoryEvent, error) |
There was a problem hiding this comment.
Moved deserialize methods to own interface.
stephanos
commented
Sep 3, 2025
| func NewSerializer() Serializer { | ||
| return &serializerImpl{} | ||
| return NewSerializerWithEncoding(EncodingTypeFromEnv()) |
There was a problem hiding this comment.
The benefit from this approach that it'll always be the one from the env which I think is beneficial since there's no question of whether or not the env var takes effect.
stephanos
force-pushed
the
customizable-serialization
branch
5 times, most recently
from
069cc12 to
bffaf1b
Compare
stephanos
commented
Sep 4, 2025
| @@ -73,29 +78,25 @@ func (u *HistoryBranchUtilImpl) NewHistoryBranch( | |||
| func (u *HistoryBranchUtilImpl) ParseHistoryBranchInfo( | |||
| branchToken []byte, | |||
| ) (*persistencespb.HistoryBranch, error) { | |||
| return serialization.HistoryBranchFromBlob(&commonpb.DataBlob{Data: branchToken, EncodingType: enumspb.ENCODING_TYPE_PROTO3}) | |||
There was a problem hiding this comment.
So this was problematic since it hard-coded the encoding type as we are not storing it. I changed the interface instead to accept []byte and use the encoding type of the current serializer. This way it can also be JSON, if that's configured.
stephanos
commented
Sep 4, 2025
| historyEvents, err := h.payloadSerializer.DeserializeEvents(&commonpb.DataBlob{ | ||
| EncodingType: enumspb.ENCODING_TYPE_PROTO3, | ||
| Data: request.GetRequest().GetEvents().GetData(), | ||
| EncodingType: cmp.Or(eventsBlob.GetEncodingType(), enumspb.ENCODING_TYPE_PROTO3), |
There was a problem hiding this comment.
Changed this so it uses the stored encoding type, but if not defined falls back to the hard-coded one. It was too risky too remove this entirely since I don't know if it's safe to assume that all have an encoding type specified.
stephanos
force-pushed
the
customizable-serialization
branch
3 times, most recently
from
f0a8bd6 to
a9939a2
Compare
stephanos
commented
Sep 10, 2025
| func TestNewServerWithJSONEncoding(t *testing.T) { | ||
| t.Setenv(serialization.SerializerDataEncodingEnvVar, "json") | ||
| startAndVerifyServer(t) | ||
| } |
There was a problem hiding this comment.
Only new test I added. Basic validation of JSON encoding; that's why I extended startAndStopServer to do more than just start up.
stephanos
force-pushed
the
customizable-serialization
branch
3 times, most recently
from
2e73bfc to
6fd2bbb
Compare
stephanos
force-pushed
the
customizable-serialization
branch
from
6fd2bbb to
ed60ec6
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
Allow customizing the DataBlob encoding type via env variable.
Why?
Allow to switch from protobuf to JSON encoding for human-readable format to aid in debugging.
How did you test it?
Potential risks
No behaviour change is expected.