Add Incremental Indexing v1 #1318
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add cli and api entrypoints for update index * Semver * Update docs * Run tests on feature branch main * Better /main handling in tests
* Calculate new inputs and deleted inputs on update * Semver * Clear ruff checks * Fix pyright * Fix PyRight * Ruff again
* Remove extraneous param * Add community report mocking assertions * Collapse primary report generation * Collapse embeddings * Format * Semver * Remove extraneous check * Move option set
* Collapse create_base_entity_graph * Format/typing * Semver * Fix smoke tests * Simplify assignment
* Collapse entity summarize * Semver
* Set up base assertions * Replace entity_extract * Finish collapsing workflow * Semver * Update snoke tests
* Update final text units * Format * Address comments
* Add naive community merge using time period * formatting * Query fixes * Add descriptions from merged_entities * Add summarization and embeddings * Use iso format * Ruff * Pyright and smoke tests * Pyright * Pyright * Update parquet for verb tests * Fix smoke tests * Remove sorting * Update smoke tests * Smoke tests * Smoke tests * Updated verb test to ack for latest changes on covariates
* Add config for incremental index + Bug fixes * Ruff * Fix smoke tests
andresmor-ms
previously approved these changes
Oct 24, 2024
andresmor-ms
approved these changes
Oct 30, 2024
|
Thank you for this feature update ! Based on the instructions , i belive all one has to do is to add a new file to the 'input' folder and update the settings.yaml file with the new output file name. Looking at the issue , the proposed fix suggest that a graphrag.append command would be added to work with increamental indexing but that doesnot seem to be present. I might be missing something here. Can someone kindly point out the correct way to do incremental indexing ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add Incremental Indexing support.
Usage
Drop new files into the already existing input folder and update settings.yaml
In settings.yaml
Related Issues
#741
Checklist
Additional Notes
[Add any additional notes or context that may be helpful for the reviewer(s).]