Skip to content

Conversation

@ognyanstoimenov
Copy link
Collaborator

@ognyanstoimenov ognyanstoimenov commented Sep 16, 2025

Reference Issues/PRs

What does this implement or fix?

Add staged data notebook to the website.

Preview of the notebook: https://github.com/man-group/ArcticDB/blob/1ae8d5a2c6a75ce6a33b853b81401d0ab40af303/docs/mkdocs/docs/notebooks/ArcticDB_staged_data_with_tokens.ipynb

Any other comments?

Checklist

Checklist for code changes...
  • Have you updated the relevant docstrings, documentation and copyright notice?
  • Is this contribution tested against all ArcticDB's features?
  • Do all exceptions introduced raise appropriate error messages?
  • Are API changes highlighted in the PR description?
  • Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

@ognyanstoimenov ognyanstoimenov added the patch Small change, should increase patch version label Sep 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments aiming to make this more user-centric.

Add:

  • An introduction to Staging data, and description of the "Why" use-cases e.g. parallel loading of data. Then describe the high-level flow: stage data in parallel or sequentially, collect the tokens / cookies and finalise the staged data.

Remove:

  • Old API: If we want the new way to be standard then no point referencing the old way

Change:

  • New API -> API
  • Document "stage result": i.e. every staged write returns a 'cookie/token' that is used for the finalisation stage.

Wy do we need this: -> Move to the introduction. Likely doesn't require code just description of how the API avoids pitfalls. Staging while finalising: similar just requires a comment rather than code.

How do you clean-up unstaged data in future runs? Should there be a clear or equivalent method to cleanup before running a large backfill.

@ognyanstoimenov ognyanstoimenov force-pushed the notebook_staged_data_demo branch from 5e7ac40 to 94b6dba Compare September 30, 2025 13:01
Copy link

@jamesblackburn jamesblackburn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notebook looks good to me!

@ognyanstoimenov ognyanstoimenov merged commit c5c5c1a into master Oct 15, 2025
143 of 144 checks passed
@ognyanstoimenov ognyanstoimenov deleted the notebook_staged_data_demo branch October 15, 2025 11:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

patch Small change, should increase patch version

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants