Blockchain updates for L2 extension #1430

esuwu · 2024-06-14T03:54:39Z

No description provided.

pkg/state/transaction_handler.go

nickeskov · 2024-06-25T23:04:04Z

pkg/state/appender.go

+
+	// write updates into the updatesChannel here
+	// TODO possibly run it in a goroutine? make sure goroutines run in order?
+	if a.bUpdatesExtension != nil {


I think having the dummy implementation of the "blockchain updates extension" looks simpler than ! = nil checks, on my mind.

What do you think?

It depends! For tests it was easier to send a nil, but maybe I should do what you suggested. How would you check if it's empty?

pkg/blockchainupdates/nuts_publisher.go

nickeskov · 2024-06-25T23:06:21Z

pkg/blockchainupdates/nuts_publisher.go

+	func(ctx context.Context, updatesChannel <-chan BUpdatesInfo) {
+		for {
+			select {
+			case updates := <-updatesChannel:


It would be wise to handle the closed channel case.

Thanks for handling it, I saw you added it

pkg/state/appender.go

nickeskov · 2024-07-16T15:28:21Z

pkg/state/accounts_data_storage.go

@@ -242,7 +242,7 @@ func (s *accountsDataStorage) entryBytes(addr proto.Address, entryKey string) ([
 func (s *accountsDataStorage) retrieveEntries(addr proto.Address) ([]proto.DataEntry, error) {
 	addrNum, err := s.addrToNum(addr)
 	if err != nil {
-		return nil, err
+		return nil, proto.ErrNotFound


err should be returned and wrapped with proto.ErrNotFound

Add check errors.Is(err, keyvalue.ErrNotFound)

Needs to be fixed

nickeskov · 2024-07-16T15:31:26Z

pkg/blockchaininfo/proto_converters.go

+			return L2ContractDataEntries{}, err
+		}
+		entry.SetKey(protoEntry.Key)
+		switch e := entry.(type) {


Use ProtobufConverter.Entry method from pkg/proto/protobuf_converters.go

nickeskov · 2024-11-07T01:34:51Z

cmd/node/node.go

+	ctx context.Context,
+	cfg *settings.BlockchainSettings,
+) (*state.BlockchainUpdatesExtension, error) {
+	const l2ContractAddr = "3Msx4Aq69zWUKy4d1wyKnQ4ofzEDAfv5Ngf"


This parameter must be configurable.

nickeskov · 2024-11-07T01:45:27Z

pkg/state/appender.go

+			Height:         blockInfo.Height,
+		},
+	}
+	a.bUpdatesExtension.BUpdatesChannel <- bUpdatesInfo


This operation can block the whole node. Maybe add a timeout for writing the update?

nickeskov · 2024-11-07T01:48:17Z

pkg/state/accounts_data_storage.go

@@ -242,7 +242,7 @@ func (s *accountsDataStorage) entryBytes(addr proto.Address, entryKey string) ([
 func (s *accountsDataStorage) retrieveEntries(addr proto.Address) ([]proto.DataEntry, error) {
 	addrNum, err := s.addrToNum(addr)
 	if err != nil {
-		return nil, err
+		return nil, proto.ErrNotFound


Needs to be fixed

nickeskov · 2024-11-07T01:48:58Z

pkg/blockchaininfo/topics.go

+/* Topics. */
+const (
+	BlockUpdates      = "block_topic"
+	microblockUpdates = "microblock_topic"


nickeskov · 2024-11-07T02:55:26Z

pkg/blockchaininfo/blockchaininfo_test.go

+		if entry.GetKey() == key {
+			// Use a type switch to check the type
+			switch entry.(type) {
+			case *proto.BinaryDataEntry:


GetValueType method can be used to determine the exact data type of the entry.

nickeskov · 2024-11-07T02:56:28Z

pkg/blockchaininfo/blockchaininfo_test.go

TestChangesGenerationNewEntries and TestChangesGenerationContainsPrevious are failing. Should be fixed.

nickeskov · 2024-11-07T04:07:17Z

pkg/blockchaininfo/types.go

+	ContractUpdatesInfo L2ContractDataEntries
+}
+
+// TODO wrap errors.


nickeskov · 2024-11-07T04:07:20Z

pkg/blockchaininfo/types.go

+	}
+
+	equal := true
+	// todo REMOVE POINTERS


nickeskov · 2024-11-07T04:08:46Z

pkg/grpc/l2/blockchain_info/blockchain_info.proto

+import "waves/block.proto";
+
+
+message BlockInfo {


I think txCount and Signature fields have to be present in the message.

nickeskov · 2024-11-07T04:09:59Z

pkg/state/accounts_data_storage.go

@@ -282,6 +282,79 @@ func (s *accountsDataStorage) retrieveEntries(addr proto.Address) ([]proto.DataE
 	return entries, nil
 }

+func (s *accountsDataStorage) retrieveEntriesAtHeight(addr proto.Address, height uint64) ([]proto.DataEntry, error) {


This is actually unused for now, because the caller stateManager.RetrieveEntriesAtHeight is also unused.

nickeskov · 2024-11-07T04:10:46Z

pkg/state/appender.go

+
+	// write updates into the updatesChannel here
+	// TODO possibly run it in a goroutine? make sure goroutines run in order?
+	if a.bUpdatesExtension != nil {


What do you think?

nickeskov · 2024-11-14T01:02:35Z

pkg/blockchaininfo/blockchaininfo.go

+	reader := bytes.NewReader(metaBlockValueBytes)
+
+	// Extract blockHeight and epoch.
+	readInt64(reader)


Suggested change

readInt64(reader)

_ = readInt64(reader)

nickeskov · 2024-11-14T01:03:06Z

pkg/blockchaininfo/blockchaininfo.go

+}
+
+// Decode base64 and extract blockHeight and height.
+func extractEpochFromBlockMeta(metaBlockValueBytes []byte) int64 {


I think it's better to create a blockMeta struct and UnmarshalBinary method for it.

nickeskov · 2024-11-14T01:05:21Z

pkg/blockchaininfo/nats_publisher.go

+const StoreBlocksLimit = 200
+const PortDefault = 4222
+const HostDefault = "127.0.0.1"
+const ConnectionsTimeoutDefault = 5 * server.AUTH_TIMEOUT
+
+const NatsMaxPayloadSize int32 = 1024 * 1024 // 1 MB
+
+const PublisherWaitingTime = 100 * time.Millisecond


I think these constants can be package private.

nickeskov · 2024-11-14T01:39:34Z

pkg/state/accounts_data_storage.go

+	key := accountsDataStorKey{addrNum: addrNum}
+
+	recordBytes, err := s.hs.entryDataAtHeight(key.bytes(), height)
+	if errors.Is(err, keyvalue.ErrNotFound) || errors.Is(err, errEmptyHist) || recordBytes == nil {


Replace recordBytes == nil to len(recordBytes) == 0.

nickeskov · 2024-11-14T01:40:16Z

pkg/blockchaininfo/types.go

+		}
+	}
+
+	equal := changes == nil


Replace changes == nil to len(changes) == 0

nickeskov · 2024-11-14T01:42:24Z

pkg/blockchaininfo/types.go

+	return equal, changes, nil
+}
+
+func statesEqual(state BUpdatesExtensionState, scheme proto.Scheme) (bool, BUpdatesInfo, error) {


Create a method from the function.

pkg/blockchaininfo/nats_publisher.go

nickeskov · 2024-11-14T02:36:52Z

pkg/blockchaininfo/types.go

+		if err != nil {
+			return false, nil, err
+		}
+		previousMap[dataEntry.GetKey()] = value


Maybe cache the previous map?

nickeskov · 2024-11-14T02:47:03Z

pkg/blockchaininfo/nats_publisher.go

+	currentState  *BUpdatesInfo
+	previousState *BUpdatesInfo // this information is what was just published


Maybe store info in a struct with BlockUpdatesInfo and transformed map of data entries (key -> marshaled value)?

nickeskov · 2024-11-14T04:08:55Z

pkg/blockchaininfo/nats_publisher.go

Please, replace all default log calls in the new code to zap logger calls.

Bumps [github.com/stretchr/testify](https://github.com/stretchr/testify) from 1.9.0 to 1.10.0. - [Release notes](https://github.com/stretchr/testify/releases) - [Commits](stretchr/testify@v1.9.0...v1.10.0) --- updated-dependencies: - dependency-name: github.com/stretchr/testify dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Return error instead of killing process with fatal log record.

Bumps [github.com/elliotchance/orderedmap/v2](https://github.com/elliotchance/orderedmap) from 2.4.0 to 2.5.0. - [Release notes](https://github.com/elliotchance/orderedmap/releases) - [Commits](elliotchance/orderedmap@v2.4.0...v2.5.0) --- updated-dependencies: - dependency-name: github.com/elliotchance/orderedmap/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Check on leasing state added. * Tests added. --------- Co-authored-by: Nikolay Eskov <[email protected]>

* Refactor and fix 'NewPeerInfoFromString' Add check for IP address parsing failure. Improved 'TestNewPeerInfoFromString'. * Improve 'NewPeerInfoFromString' Add hostname IP resolving. * Simplified 'NewTCPAddrFromString' * Refactoring and bugfix of 'resolveHostToIPv4' - Now it returns ipV4 address exactly in ipV4 form. - Fixed bug with skipping address for chesk after ipV6 removal. - Fixed bug with duplication of ipV4 address when resolving 'localhost'. * Optimized a bit 'filterToIPV4'. * Create 'NewPeerInfosFromString' function. * Use 'NewPeerInfosFromString' for resolving 'host:port' records by '-peers' CLI parameter. * Fix 'TestNewPeerInfoFromString' test. --------- Co-authored-by: Alexey Kiselev <[email protected]>

esuwu added 3 commits June 13, 2024 22:54

Initial draft

63b159c

Changed a comment

69e9dbd

moved a file

b35bea1

esuwu changed the title ~~Initial draft~~ Initial design draft Jun 14, 2024

esuwu added 4 commits June 13, 2024 22:59

rename a folder

27494ad

Sent the updates channel into block appplier

983babc

Added some design comments

5272899

Context in the updates function

4fef6e3

nickeskov reviewed Jun 25, 2024

View reviewed changes

pkg/state/transaction_handler.go Outdated Show resolved Hide resolved

nickeskov reviewed Jun 25, 2024

View reviewed changes

pkg/blockchainupdates/nuts_publisher.go Outdated Show resolved Hide resolved

nickeskov reviewed Jun 25, 2024

View reviewed changes

esuwu added 2 commits July 1, 2024 13:27

Merge branch 'master' into node-updates-plugin-l2

dcd18f3

Added protobuf

3cc00d5

github-advanced-security bot found potential problems Jul 3, 2024

View reviewed changes

pkg/state/appender.go Fixed Show fixed Hide fixed

esuwu changed the title ~~Initial design draft~~ Blockchain updates for L2 extension Jul 3, 2024

Moved proto, added conversion

6752f8a

nickeskov reviewed Jul 16, 2024

View reviewed changes

esuwu added 6 commits July 18, 2024 23:56

Added state comparison

bdc8854

Added a draft for retrieving history entries

19cc302

Renamed a function

b6fb33c

Fixed most of the linter issues

dca31ee

Merge branch 'master' into node-updates-plugin-l2

f524fe9

Added paging

18eab8c

nickeskov added wip This is a WIP, should not be merged right away do not merge The PR is not ready to be merged labels Aug 26, 2024

esuwu added 3 commits August 27, 2024 23:42

Added changes detector

e2da71e

Fixed a bug

da48829

Renamed functions

b9318c2

Make some types and functions package private.

fb56f6b

nickeskov reviewed Nov 7, 2024

View reviewed changes

nickeskov added 4 commits November 7, 2024 04:39

Update 'go.mod' go version.

89ffaf2

Change location of 'blockchaininfo_test.go'.

28086dc

Add new method 'PartialBlockHeader' for 'ProtobufConverter'.

2f8c3c0

Refactor 'BUpdatesInfoFromProto' function with unused functions removal.

5f1010c

nickeskov force-pushed the node-updates-plugin-l2 branch from 586cc1e to 5f1010c Compare November 7, 2024 03:58

Refactored a bit 'L2ContractDataEntriesFromProto'.

7dc0748

nickeskov requested changes Nov 7, 2024

View reviewed changes

nickeskov and others added 2 commits November 11, 2024 22:28

Merge branch 'master' into node-updates-plugin-l2

42bf0ff

Added a parse function

3f9bb83

nickeskov requested changes Nov 14, 2024

View reviewed changes

pkg/blockchaininfo/nats_publisher.go

Copy link

Member

nickeskov Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, replace all default log calls in the new code to zap logger calls.

nickeskov and others added 17 commits November 18, 2024 07:37

Merge branch 'master' into node-updates-plugin-l2

5d37195

Set 'NoSigs' option for nats sever to true.

cf23bdf

Fix 'runPublisher' in case when updates channel is closed.

d7dcd5e

Add nats sever shutdown.

d13b940

Fix deadlock and refactoring.

4dc94d5

Merge branch 'master' into node-updates-plugin-l2

214bd46

Fixed err not found handling

8fbe2b4

Fixed a toolchain not found issue (#1545)

bb23dbc

Handle blockchain import errors properly (#1548)

68db8b3

Return error instead of killing process with fatal log record.

Check on leasing state added. (#1550)

c598d67

* Check on leasing state added. * Tests added. --------- Co-authored-by: Nikolay Eskov <[email protected]>

Prepare timers for go1.23. (#1552)

bd889f5

Merge branch 'master' into node-updates-plugin-l2

6814098

Left a todo

b87f8ad

Merged from master

cdb6083

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Blockchain updates for L2 extension #1430

Blockchain updates for L2 extension #1430

esuwu commented Jun 14, 2024

nickeskov Jun 25, 2024

nickeskov Nov 7, 2024

esuwu Nov 20, 2024

nickeskov Jun 25, 2024

esuwu Nov 20, 2024

nickeskov Jul 16, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Jul 16, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 7, 2024

nickeskov Nov 14, 2024

nickeskov Nov 14, 2024

nickeskov Nov 14, 2024

nickeskov Nov 14, 2024

nickeskov Nov 14, 2024

nickeskov Nov 14, 2024

nickeskov Nov 14, 2024

nickeskov Nov 14, 2024

nickeskov Nov 14, 2024

		currentState *BUpdatesInfo
		previousState *BUpdatesInfo // this information is what was just published

Blockchain updates for L2 extension #1430

Are you sure you want to change the base?

Blockchain updates for L2 extension #1430

Conversation

esuwu commented Jun 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment