Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve how we handle two different set of CARs being uploaded for the same root CID #184

Open
dchoi27 opened this issue Mar 31, 2023 · 0 comments

Comments

@dchoi27
Copy link

dchoi27 commented Mar 31, 2023

Context (from @vasco-santos):

Reading from freeway after second upload will mean 3 CAR CIDs to go through via freeway.
Freeway does a R2 list with prefix in DUDEWHERE ${rootCid}/ and gets all CAR files ever associated with given Root
We will read all indexes from Satnav for given CARs (1 extra…) and create an in memory Map index, where we keep track of all index files
Then we start the actual read from root. We read from root block and start unixfs export by going through all links from root block we get until full dag
TLDR:
In case there are more side indexes than needed, we will read all the extra side indexes and they will count towards our sub requests limit (and max shards protection)
When actually reading, we try to find a given block CID in multiple indexes in parallel and once we get one CAR CID where CID is we return back…. So, in practise we won’t do extra reads by having more CARs associated with Root CID

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant