Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: updated DataStore [2/n] #141

Closed
wants to merge 4 commits into from

Conversation

gmaclennan
Copy link
Member

@gmaclennan gmaclennan commented Aug 2, 2023

From #139:

A DataStore should be an abstraction over a particular "Mapeo Store", e.g. a collection of hypercores that store a particular datatype. Responsibilities:

  • Write a document to the write core of the store.
  • Read a document based on versionId.
  • Manage read indexing and return current indexing state.
  • Handle decoding/encoding of data

With this we can isolate the boundary with hypercore / core store to this module (apart from within the blob store), and largely isolate indexing: The write method can resolve when the document is indexed.

const store = new DataStore({
  coreManager,
  namespace: 'auth' | 'data',
  // DataStore will consider a block "indexed" once this function resolves.
  indexDocs: (blocks: Buffer[]) => Promise<void>
})

// Guarantee that `doc` is indexed once this resolves
store.write(doc: Exclude<MapeoDoc, 'versionId'>): Promise<MapeoDoc>

store.read(versionId: string): Promise<MapeoDoc>

store.getIndexState() // return whether is currently indexing, and estimate of indexing remaining to be done

store.on('index-state') // same as above.

@gmaclennan gmaclennan changed the base branch from main to feat/jsonschema-to-sql-schema August 2, 2023 20:38
@gmaclennan
Copy link
Member Author

Now largely complete, will be best to review once all items in #139 are done, because this might still change.

@gmaclennan gmaclennan self-assigned this Aug 3, 2023
@gmaclennan gmaclennan linked an issue Aug 4, 2023 that may be closed by this pull request
@gmaclennan gmaclennan force-pushed the feat/jsonschema-to-sql-schema branch from 981070a to 4da0146 Compare August 7, 2023 15:14
@gmaclennan gmaclennan changed the title feat: updated DataStore feat: updated DataStore [2/n] Aug 7, 2023
@achou11 achou11 self-requested a review August 7, 2023 18:35
Copy link
Member

@achou11 achou11 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple of minor suggestions. also need to remove a test.solo call to see if all the added tests pass, but otherwise seems good to me

Is the plan to eventually use this as the datastore implementation (currently named to avoid collision with existing one)

lib/core-manager/index.js Outdated Show resolved Hide resolved
Comment on lines +65 to +69
if (['newListener', 'removeListener'].includes(eventName)) return
this.#coreIndexer.on(eventName, listener)
})
this.on('removeListener', (eventName, listener) => {
if (['newListener', 'removeListener'].includes(eventName)) return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extract these events to a constant? i.e. const LISTENER_EVENTS = ['newListener', 'removeListener'] (you can come up with a better variable name 😄 )

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer not to because I'd like this to be explicit in the code, rather than the indirection of a constant - we'd end up naming it "newListenerAndRemoveListenerEventNames". Making is a constant might be a slight performance improvement, but this is not somewhere that is a bottleneck (events are added rarely).

lib/datastore/data-store-new.js Outdated Show resolved Hide resolved
tests/datastore-new.js Outdated Show resolved Hide resolved
@gmaclennan gmaclennan marked this pull request as ready for review August 7, 2023 20:08
@gmaclennan
Copy link
Member Author

Rebased onto changes in #140

@gmaclennan
Copy link
Member Author

Merged in #149

@gmaclennan gmaclennan closed this Aug 9, 2023
@gmaclennan gmaclennan deleted the feat/new-datastore branch August 31, 2023 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Updated DataStore class
2 participants