[Initiative] Repackage existing record files in the Block Stream.

[jsync-swirlds]

Epic Goal

Repackage record file data in a series of uncompressed ZIP files containing ZStd compressed block files. Each block contains a single record file, its sidecars, and all of its signature files. These block files must be stored in cloud storage buckets, and Mirror Node must be able to read the contained data and verify the contents.

Required Actions

• Create a tool to convert existing record files to block files #359
• Run the conversion tool on groups of record files.
• Work with Mirror Node team to verify ability to read the ZIP files.

Create a tool to convert existing record files to block files

• Read a record file
• (Optional) Split the record file into one entry per round (may not be technically possible).
• Read and verify all available signature files
• wrap the record file data and all signatures in a RecordFile BlockItem
• Wrap the BlockItem in a Block.
• Write the Block to disk, using zstd compression
• Verify the written block
• Repeat for all record files (roughly 160 million)

Run the conversion tool on groups of record files.

• Launch a VM in GCP or AWS local to the record file archive(s).
• Run the conversion tool on A group of 100,000 record files.
• Create an uncompressed ZIP file for each group
• Upload the new ZIP file to a new storage bucket

Work with Mirror Node team to read the ZIP files

• Read a single ZIP from cloud storage
• extract block files
• Uncompress a single block file using ZStd decompression
• Create an input stage in the Mirror Node import to unpack the block file to retrieve the original record file, sidecar file(s), and signature files.
• Verify that the existing mirror node import process can consume the data extracted from the "Block" file.