Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

spade v0 retrieval worker #33

Closed
wants to merge 23 commits into from
Closed

spade v0 retrieval worker #33

wants to merge 23 commits into from

Conversation

jcace
Copy link
Contributor

@jcace jcace commented Sep 14, 2023 •
edited
Loading

closes #32

  • Modifies the bitswap pkg to handle the "multi-layer" traversal required for spade CIDs (ex, like this)
  • Modifies the Bitswap worker to call the new Spade traversal code when it encounters a spade task

jcace
jcace commented Sep 14, 2023
View reviewed changes
pkg/spade-test/spade-test.go Outdated Show resolved Hide resolved
jcace
jcace commented Sep 14, 2023
View reviewed changes
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
jcace
jcace commented Sep 14, 2023
View reviewed changes
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
jcace
jcace commented Sep 14, 2023
View reviewed changes
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
jcace
jcace commented Sep 14, 2023
View reviewed changes
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
@jcace
Copy link
Contributor Author

jcace commented Sep 14, 2023

Sample run output:

DEBUG   spade-test      spade-spike/spade-spike.go:258  starting spade-test
DEBUG   spade-test      spade-spike/spade-spike.go:212  retrieving bafybeib62b4ukyzjcj7d2h4mbzjgg7l6qiz3ma4vb4b2bawmcauf5afvua

INFO    spade-test      spade-spike/spade-spike.go:164  Connecting to target peer...
DEBUG   spade-test      spade-spike/spade-spike.go:173  Retrieving block...
DEBUG   spade-test      spade-spike/spade-spike.go:230  retrieved bafybeib62b4ukyzjcj7d2h4mbzjgg7l6qiz3ma4vb4b2bawmcauf5afvua which has 3 links

DEBUG   spade-test      spade-spike/spade-spike.go:212  retrieving bafybeighnvpjaoood3figbrdugd2zk3kf3y7hlrezk56ttfsw2gkk43sq4

INFO    spade-test      spade-spike/spade-spike.go:164  Connecting to target peer...
DEBUG   spade-test      spade-spike/spade-spike.go:173  Retrieving block...
DEBUG   spade-test      spade-spike/spade-spike.go:230  retrieved bafybeighnvpjaoood3figbrdugd2zk3kf3y7hlrezk56ttfsw2gkk43sq4 which has 1 link

DEBUG   spade-test      spade-spike/spade-spike.go:212  retrieving bafybeifqros243kzvhf72n6urhfydtzgt6yosgoxkpthsaksqpm6psc5pm

INFO    spade-test      spade-spike/spade-spike.go:164  Connecting to target peer...
DEBUG   spade-test      spade-spike/spade-spike.go:173  Retrieving block...
DEBUG   spade-test      spade-spike/spade-spike.go:230  retrieved bafybeifqros243kzvhf72n6urhfydtzgt6yosgoxkpthsaksqpm6psc5pm which has 1 link

DEBUG   spade-test      spade-spike/spade-spike.go:212  retrieving bafybeibhbx3y6tnn7q4gpsous6apnobft5jybvroiepdsmvps2lmycjjxu

INFO    spade-test      spade-spike/spade-spike.go:164  Connecting to target peer...
DEBUG   spade-test      spade-spike/spade-spike.go:173  Retrieving block...
DEBUG   spade-test      spade-spike/spade-spike.go:220  retrieved data cid bafybeibhbx3y6tnn7q4gpsous6apnobft5jybvroiepdsmvps2lmycjjxu which contains 1692 bytes

success:  true

jcace
jcace commented Sep 14, 2023
View reviewed changes
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
xinaxu
xinaxu suggested changes Sep 14, 2023
View reviewed changes
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
This was referenced Sep 15, 2023
Closed
Closed
Closed
jcace
jcace commented Sep 15, 2023
View reviewed changes
pkg/spade-spike/spade-spike.go Outdated Show resolved Hide resolved
@jcace jcace marked this pull request as ready for review October 10, 2023 17:24
@jcace jcace requested a review from xinaxu October 10, 2023 17:24
@jcace jcace changed the title WIP spade v0 retrieval worker spade v0 retrieval worker Oct 10, 2023
stephen-pl
stephen-pl reviewed Oct 10, 2023
View reviewed changes
integration/spadev0/util.go Outdated
"retrieve_size": "1048576",
// todo: specify # of cids to test per layer of the tree TBD
"retrieve_type": string(task.Spade),
"retrieve_size": "1048576",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the extra space here

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's the default gofmt formatter, which is part of the Go plugin for vscode - it's aligning all of the values to start at the vertical line (ie, because "traverse_depth" is a slightly longer string, it pushes them out an extra space)

xinaxu
xinaxu reviewed Oct 11, 2023
View reviewed changes
integration/spadev0/util.go Outdated Show resolved Hide resolved
pkg/net/bitswap.go Outdated Show resolved Hide resolved
pkg/net/bitswap.go Outdated Show resolved Hide resolved
@jcace jcace requested a review from xinaxu October 12, 2023 20:22
ribasushi
ribasushi reviewed Oct 12, 2023
View reviewed changes
pkg/net/bitswap.go Outdated Show resolved Hide resolved
ribasushi
ribasushi reviewed Oct 12, 2023
View reviewed changes
pkg/net/bitswap.go Outdated Show resolved Hide resolved
@jcace
Copy link
Contributor Author

jcace commented Oct 13, 2023 •
edited
Loading

In the interest of getting this merged in now, I created this issue for a future improvement: data-preservation-programs/spade#34

@jcace jcace mentioned this pull request Oct 17, 2023
Merged
@jcace
Copy link
Contributor Author

jcace commented Oct 17, 2023

Closing this PR as it has been moved to the forked SpadeRetrievalBot repo instead.

@jcace jcace closed this Oct 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ribasushi ribasushi ribasushi left review comments

@stephen-pl stephen-pl stephen-pl left review comments

@xinaxu xinaxu Awaiting requested review from xinaxu

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Spade - Bitswap Retrieval Worker
4 participants
@jcace @ribasushi @xinaxu @stephen-pl