Missing pieces for cat-file --remote

[quark-zju]

Hi, I'm trying to implement something like git cat-file --batch but talks to the remote server.

More specifically, I'd like to:

• Re-use ssh connection using Git Protocol V2, to avoid ssh handshake overhead.
• Fetch by object ids. Avoid over fetch (no ancestors of commits, no sub-trees). This seems achievable by using wants, filter("tree:0") to avoid sub-tree traversal, and shallow to prevent commit traversal.

The gix crates are pleasant to use and mostly just work. But I run into some issues. I wonder if they are designed as expected:

• In gix-protocol/src/fetch/arguments/blocking_io.rs. It has assert!(add_done_argument, "If there are no haves, is_done must be true.");. In my case there are a few wants but no have. I have to remove this assertion. Should this assertion be changed somehow?
• It seems gix_protocol::fetch expects receive_pack to be called once and exits the 'negotiation loop. In my case I'd like to try multiple round-trips - some wants followed by receive_pack, then repeat wants and receive_pack. Should the fetch API be changed to support multiple receive_pack somehow? Or am I expected to call fetch multiple times on a same transport (in this case should fetch take a reference to transport instead of consuming it)?

There is another minor issue. It's not blocking, though:

• Is there a way to parse a bundle in memory, or even process it in a streaming fashion? I noticed that I can pass the receive_pack's input: impl BufRead to gix_pack::Bundle::write_to_directory then use Bundle::at(path) to read it. It'd be nice if the input can be parsed in-memory or even better in a streaming fashion (so memory usage is bounded).

The test script I use can be found here. I'm using gixtoxide at a3c283f. Note I removed this assert! for it to run.

[Byron]

Thanks for sharing and for the detailed description of the feature - I think I have a pretty good understanding of what you are trying to achieve.

Fetch by object ids. Avoid over fetch (no ancestors of commits, no sub-trees). This seems achievable by using wants, filter("tree:0") to avoid sub-tree traversal, and shallow to prevent commit traversal.

On protocol level, you should be able to specify filter options already. shallow support, however, isn't implemented yet, but I will coincidentally start with that tomorrow for my work on cargo.

But I run into some issues. I wonder if they are designed as expected: [..]

Generally, I am going by the documentation mostly and what's observable when looking at git traffic, and it's not always entirely clear to me how the protocol is supposed to work. Thus, some assertions are mostly a test of my current understanding, and running into them would mean my understanding was wrong. Thanks to your work, this should be smoothened out over time, as the code is run under different conditions.

With that in mind, all I needed this code to do is to do typical fetches as described. Doing multiple fetches in a row should work with V2 due to the command architecture, even though some APIs might still be geared towards a certain use-case. This can be considered a shortcoming that you are invited to fix as you see fit (while providing test-cases that motivate these fixes).

My recommendation is to make it work in your fork and then we see how these changes can be integrated.

Is there a way to parse a bundle in memory, or even process it in a streaming fashion? I noticed that I can pass the receive_pack's input: impl BufRead to gix_pack::Bundle::write_to_directory then use Bundle::at(path) to read it. It'd be nice if the input can be parsed in-memory or even better in a streaming fashion (so memory usage is bounded

This operation is already fully streaming and step-wise, so progress reporting is granular and interruption work as well. That said, it does need access to disk space to write the pack to disk while streaming for later mapping and lookup during the resolution stage. If this should be in memory too you would have to take the parts of the Bundle::write_to_directory() implementation and adjust them to fit your use-case - it should be no problem to write the pack into memory instead even though that would defeat the streaming to some extend.

I hope that helps.

[quark-zju]

Thanks for your prompt feedback! I think I can fix the assertion and fetch loop with relatively simple changes. shallow and filter already work in my testing. For in-memory bundle handling I understand buffering everything is undesirable. I was thinking about producing the result in a streaming way. But that doesn't seem possible if the bundle allows the use of deltas.

[Byron]

I was thinking about producing the result in a streaming way. But that doesn't seem possible if the bundle allows the use of deltas.

Indeed, packs are seen as a unit and in the end, one will have to see them in their entirety to produce an index, and to do connectivity checks to validate the internal consistency of the object graph it contains (the latter isn't done yet by gitoxide).

[quark-zju]

Indeed, packs are seen as a unit and in the end, one will have to see them in their entirety to produce an index, and to do connectivity checks to validate the internal consistency of the object graph it contains (the latter isn't done yet by gitoxide).

If we only need to iterate through all objects and pass them to the application, then there is no need for an index. If we can do any of these, there would be a streaming way of handling the traffic:

• Tell the server to not use delta.
• The server can use delta but must sort objects topologically, i.e. delta is always before delta base. The receiving end only buffers "unresolved" deltas and once they are resolved, drop from memory.
• The server can use delta and can sort objects arbitrarily, but provides extra "ref count' information about how many times a delta base is used by deltas. So the receiving end can drop delta base once it knows referred deltas are already proceeded.
• The server can use delta and can sort objects arbitrarily, the receiving end persists objects in another way format (ex. just insert them into sqlite keyed by oid), so it can apply deltas on request.

Assuming it's hard to change the server, the last item might be interesting to implement.

[Byron]

I see what you mean now with streaming.

Not using deltas would be one way for sure. And when using deltas, knowing how many more pack entries are using a particular base would be the most interesting metric to allow dropping it earlier. Note that delta-entries are already encoded to refer to there base by an offset to the prior pack.

Once gitoxide has a server, it could indeed open up all kinds of experiments. Personally, I am not interested in streaming more, but in having faster packs to increase object access performance. ZLib is a major bottleneck and merely changing the compressor to zstd would have massive performance benefits.

[quark-zju]

It turns out that no code change is needed!

• The assertion in question can be avoided by returning Action::Cancel from negotiate. It does not cancel the following receive_pack.
• &mut Transport, can be passed to fetch to reuse connection. I didn't realize &mut Transport impls Transport.