[Morse->Shannon Migration] state export/import - collect accounts

[bryanchriswhite]

Summary

Add poktrolld migrate collect-morse-accounts command to convert Morse state export into a MorseAccountState for import into Shannon.

Issue

• [Morse->Shannon Migration] Migration module #1034

Type of change

Select one or more from the following:

• New feature, functionality or library
• Consensus breaking; add the consensus-breaking label if so. See [Infra] Automatically add the consensus-breaking label #791 for details
• Bug fix
• Code health or cleanup
• Documentation
• Other (specify)

Testing

• Documentation: make docusaurus_start; only needed if you make doc changes
• Unit Tests: make go_develop_and_test
• LocalNet E2E Tests: make test_e2e
• DevNet E2E Tests: Add the devnet-test-e2e label to the PR.

Sanity Checklist

• I have tested my changes using the available tooling
• I have commented my code
• I have performed a self-review of my own code; both comments & source code
• I create and reference any new tickets, if applicable
• I have left TODOs throughout the codebase, if applicable

[bryanchriswhite]

@Reviewer do we need to consider these fields?

[Olshansk]

Yes, but add the same qualifiers you have elsewhere. E.g. [(gogoproto.jsontag) = "tokens"];

[bryanchriswhite]

I think my highlighting was off, I meant to call out jailed and status. If we do need these, what role should they play in the migration?

[bryanchriswhite]

Regarding the json option, will add. 👍 It needs to be staked_tokens though as this structure is derived from the Morse export data structure (Morse proto types).

[bryanchriswhite]

Seems to bottom out around 10^5 on my machine.

[bryanchriswhite]

Subsequent observations seem to indicate that performance actually seems to degrade with concurrency.

[bryanchriswhite]

I took a look at the number of accounts, apps, and suppliers on Morse mainnet; based on this information I don't think it's worthwhile to spend any effort on optimization/parallization here:

object	count
accounts	56808
apps	2298
suppliers	13269

$ curl -X POST https://pocket-rpc.liquify.com/v1/query/accounts -d '{"page":1,"per_page":100000}' -o ./morse_accounts.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6671k    0 6671k  100    28  2788k     11  0:00:02  0:00:02 --:--:-- 2790k
 $ jq ".result|length" ./morse_accounts.json
56808
# curl -X POST https://pocket-rpc.liquify.com/v1/query/apps -d '{"opts": {"page":1,"per_page":100000}}' -o ./morse_apps.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  600k    0  600k  100    38  1331k     84 --:--:-- --:--:-- --:--:-- 1333k
$ jq ".result|length" ./morse_apps.json
2298
$ curl -X POST https://pocket-rpc.liquify.com/v1/query/nodes -d '{"opts": {"page":1,"per_page":100000}}' -o ./morse_suppliers.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6172k    0 6172k  100    38  5039k     31  0:00:01  0:00:01 --:--:-- 5043k
$ jq ".result|length" ./morse_suppliers.json   
13269

[Olshansk]

Given that this is all new code, let's use autocli instead.

[bryanchriswhite]

AutoCLI does not apply here because there is no gRPC service, message, or query.

The purpose of this command is to facilitate the deterministic (i.e. reproducible) transformation from the Morse export data structure (MorseStateExport) into the Shannon import data structure (MorseAccountState). It does not interact with the network directly.

[Olshansk]

Can we rename this to morse.proto?

[bryanchriswhite]

My rationale here was that we depend on multiple types from Morse that can be divided into two categories:

1. Those which MUST be persisted on-chain in Shannon
2. Those which are only required for the export/import state transform CLI command.

Given these categories, both of which I would characterize as "morse". I chose to go with "types.proto" for the on-chain types, for consistency with other modules, and then chose "legacy" for the other types.

[bryanchriswhite]

Upon further reflection, I've realized that the ONLY types that are needed on-chain in Shannon are MorseAccountState, MorseAccount, and MorsePublicKey. I've moved MorseApplication and MorseValidator to legacy.proto reflect this.