Resolve open questions

[casey]

See DESIGN.md.

[mebious-dev]

Below is a bullet list summarising my thoughts on some of the questions:

• Empty Directories: Empty directories should remain invalid for the create and verify commands as I would imagine including entries without hashes in the manifest may complicate things in the long run. It also wouldn't be useful as the manifest is intended for files that are able to be signed and verified. One possible option is to list the index and hashes in separate sections within the manifest so that empty directories can be recorded without interfering with the hashing and verification. RHash implements this using comments, as shown here.
• Entry Nesting: I would say that this is dependent on the selected serialisation format.
• Manifest Format: A migration to and/or inclusion of CBOR may have the least friction as it is the closest to JSON in addition to having a formal specification. Human readability could be addressed by providing commands to filter and parse the manifest. As the manifest doesn't seem to require direct modifications from the user, I do not believe that human readability is a high priority.
• Metadata: A separate metadata file would help avoid user error from affecting the automatically generated data.

Other brainstorming:

• Would database file formats be considered for the manifest(s)?
• Could file indexing in filepack also be implemented as a separate repository/project so that it can be used in other projects? Something I've noted is that currently available local file indexing utilities aren't particularly ideal.

[casey]

Thanks for your thoughts!

I'm planning on switching hashes to something that recursively hashes directories. If I do this, empty directories would no longer be a special case, so I think no harm in making them work.

I definitely like the idea of using CBOR, and converting to JSON would still be possible, which, as you say, filepack could support. JSON is just so widely used though 🤔

I've definitely settled on using a separate file for metadata. The thing which pushed this over the edge was the desire to verify directories without a manifest, by verifying against the final root hash (which I'm calling the "fingerprint"). You would do:

filepack verify --fingerprint HASH

And it would calculate the recursive directory hash, and compare it against HASH. If the manifest contained metadata, and thus contributed to HASH, you could not do this without the metadata, whereas if metadata is just another file, it's no problem.

I wouldn't want to use database formats just because they're so complicated, and offer so many degrees of freedom.

Could you elaborate on what you mean by file indexing? What kinds of thing do you think would be useful?