[repostore] Retrieve empty blocks

[emizzle]

Empty blocks were not retrievable by the RepoStore, and this was causing integration tests to fail when K > 1, as empty blocks were used for padding protected manifests. The failures, however, were only being witnessed on ARM64 (Apple silicon M2's and likely M1's, though untested), and not on AMD64, for reasons that are still unknown. @dryajov was looking into checking that the erasure coding was indeed being done correctly.

[emizzle]

Once I added the empty block tests, which calls hasBlock, there was an ambiguous call:

Error: ambiguous call; both repostore.hasBlock(self: RepoStore, cid: Cid) [proc declared in /Users/egonat/repos/status-im/nim-codex/codex/stores/repostore.nim(339, 6)] and repostore.hasBlock(self: RepoStore, cid: Cid) [method declared in /Users/egonat/repos/status-im/nim-codex/codex/stores/repostore.nim(244, 8)] match for: (RepoStore, Cid)

It appears that hasBlock was duplicated in repostore but was not being called anywhere else in the tests, or maybe even the codebase, so adding the hasBlock test for empty blocks was a good thing.

[veaceslavdoina]

@emizzle, in case you need in the future other ARM64/AMD64, we can continue our discussion here - https://github.com/codex-storage/infra-codex/issues/14.

[dryajov]

For context, in addition to this fix, there is an additional erasure coding issue:

Currently we don't support erasure coding a single block, there has to be at least two blocks to be able to erasure code them.

Leopard will not encode a single buffer, it will simply copy it back. Blocks are treated as a list of symbols and each block occupies one buffer. So, for all intents and purposes, when using one block, we're sending a vector of L*K elements, but K=1 in this case, so leopard simply copies it back.

The options are:

Split the block into two and erasure code the two pieces together
Erasure code with a dummy block
???

In short, it's not a big deal, its a corner case that is probably not worth addressing right now, and when we need to address it, the fix should be relatively straightforward.

For your tests to work, just generate at least two random blocks, other than that it should works as expected.

[dryajov]

btw, is there an issue to track this already, I couldn't find it.

[emizzle]

Erasure code with a dummy block

@dryajov, isn't the K=1 case already being padded with an empty block, hence the issue arising where empty blocks were not retrievable from the RepoStore? Or by "dummy" block, do you mean something different than an empty block?

The issue that seems out of place is that AMD CPUs are not padding with an empty block in the K=1 case (and therefore no retrieval of empty blocks needed), whereas this issue was present on ARM silicon.

Would the fix for the K=1 case be limited to AMD CPUs?

[dryajov]

Or by "dummy" block, do you mean something different than an empty block?

Other than empty block. Yeah, I'm guessing it was a combination of several factors, but I've been able to reproduce it with a single block, when there are more than one blocks, K=2, it works as expected, at least with my experiments.

[emizzle]

Issue created: #522