Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Badger2 integration #7705

Closed
wants to merge 2 commits into from
Closed

Badger2 integration #7705

wants to merge 2 commits into from

Conversation

gammazero
Copy link
Contributor

This is a provisional PR and should not be merged until dependencies are merged into their master branches.

  • Add badger2 plugin
  • Update docs to describe badger2
  • Add tests for badger2
  • Set dependencies to pending PRs with necessary updates

@welcome
Copy link

welcome bot commented Sep 30, 2020

Thank you for submitting this PR!
A maintainer will be here shortly to review it.
We are super grateful, but we are also overloaded! Help us by making sure that:

  • The context for this PR is clear, with relevant discussion, decisions
    and stakeholders linked/mentioned.

  • Your contribution itself is clear (code comments, self-review for the
    rest) and in its best form. Follow the code contribution
    guidelines
    if they apply.

Getting other community members to do a review would be great help too on complex PRs (you can ask in the chats/forums). If you are unsure about something, just leave us a comment.
Next steps:

  • A maintainer will triage and assign priority to this PR, commenting on
    any missing things and potentially assigning a reviewer for high
    priority items.

  • The PR gets reviews, discussed and approvals as needed.

  • The PR is merged by maintainers when it has been approved and comments addressed.

We currently aim to provide initial feedback/triaging within two business days. Please keep an eye on any labelling actions, as these will indicate priorities and status of your contribution.
We are very grateful for your contribution!

aschmahmann
aschmahmann requested changes Oct 2, 2020
View reviewed changes
docs/datastores.md Show resolved Hide resolved
test/sharness/t0025-datastores.sh Outdated Show resolved Hide resolved
plugin/plugins/badger2ds/badger2ds.go Show resolved Hide resolved
@gammazero gammazero requested a review from aschmahmann October 6, 2020 17:39
@aschmahmann
Copy link
Contributor

@gammazero your go.mod files are conflicting with master. Could you rebase when you get a chance?

aschmahmann
aschmahmann requested changes Oct 6, 2020
View reviewed changes
Copy link
Contributor

@aschmahmann aschmahmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Added a couple small comments and a suggestion on the compression + blockCacheSize parameters.

plugin/plugins/badger2ds/badger2ds.go Outdated Show resolved Hide resolved
repo/fsrepo/config_test.go Show resolved Hide resolved
docs/datastores.md Outdated Show resolved Hide resolved
Stebalien
Stebalien approved these changes Oct 12, 2020
View reviewed changes
Copy link
Member

@Stebalien Stebalien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

aschmahmann
aschmahmann approved these changes Oct 12, 2020
View reviewed changes
Copy link
Contributor

@aschmahmann aschmahmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to go pending a release of go-ds-badger2 and upgrading go-ipfs-config to v0.10.0

@aschmahmann aschmahmann added the status/blocked Unable to be worked further until needs are met label Oct 12, 2020
@gammazero
Badger2 integration
f3945f7 
This is a provisional PR and should not be merged until dependencies are merged into their master branches.

- Add badger2 plugin
- Update docs to describe badger2
- Add tests for badger2
- Add options to configure compression and blockCacheSize
- Document vlogFileSize config param
- Update go-ipfs-config
- Update go-ds-badger2
@gammazero
Update to latest release of go-ds-badger and go-ipfs-config
914d390
@aschmahmann
Copy link
Contributor

This is blocked by the badger folks figuring out if there latest breaking changes will be handled within badger v2 or whether there will be a badger v3. The go-ipfs-config PR adding badger 2 support has been reverted until this is resolved.

@dokterbob
Copy link
Contributor

@aschmahmann There's now a Badger v3 https://github.com/dgraph-io/badger/releases/tag/v3.2011.1

However, go-ds-badger2 is actively maintained. Requirements mentioned by you have been met a long time ago and, even as an experimental feature, it could gain great benefit and insight to have something less broken to test with. This might also yield useful feedback while a move towards badger 3 is made.

We currently have a badgerdb in IPFS which is broken in obvious ways making it unsuitable for production.

@dokterbob
Copy link
Contributor

In short; all that's necessary to have a badger v2, based on actively maintained code, in master, is to resolve these merge conflicts. Getting v3 support is probably a fair amount of work and might delay having something working by several months/up to a year.

It would be great to have this in user's hands!

@godcong
Copy link
Contributor

godcong commented Apr 16, 2021

I tried to use V3, but it will panic in some cases.

@dokterbob
Copy link
Contributor

@aschmahmann Poke!

@dokterbob
Copy link
Contributor

At ipfs-search.com we currently have a script running from a cron job which stops IPFS, flattens the datastore using the badger binary and then restarts IPFS.

I know you've got a lot on your hands, but it would be great to see some progress on this one!

@dokterbob
Copy link
Contributor

Poke!

@feld
Copy link

feld commented Aug 23, 2022 •
edited
Loading

flatfs is not good when you've got 10s to 100s of gigs in IPFS. One of my use cases can survive with a manual flatten as suggested by @dokterbob but it would be nice if we had badger v2 or v3 to work with

edit: i may just rebase this on kubo v14.0 and badger v2.2007.4 and see what happens...

edit2: also learned that flatten is not the same as garbage collection; flatten is a different thing in Badger where it consolidates storage layers but has nothing to do with freeing up space. But freeing up some space could be a side effect.

@feld
Copy link

feld commented Aug 26, 2022 •
edited
Loading

@dokterbob I have a branch with Badger2 working on IPFS v0.14.0 and garbage collection does work properly

https://github.com/feld/kubo/tree/badger2-rebase-v0.14.0

@feld feld mentioned this pull request Aug 26, 2022
Closed
@lidel lidel marked this pull request as draft September 17, 2024 14:24
@lidel
Copy link
Member

lidel commented Sep 17, 2024 •
edited
Loading

Closing this one due to badverv2, and latest one being badgerv3.

iirc long term we will more likely introduce pebble (#10347) and short-to-mid term discourage use of badgerv1 (#10517)

@lidel lidel closed this Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aschmahmann aschmahmann aschmahmann approved these changes

@Stebalien Stebalien Stebalien approved these changes

Assignees
No one assigned
Labels
status/blocked Unable to be worked further until needs are met
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@gammazero @aschmahmann @dokterbob @godcong @feld @lidel @Stebalien