Skip to content

[SPO] Fix bugs when handling drive_items in incremental sync#1827

Merged
jedrazb merged 3 commits intomainfrom
jedrazb-spo-incremental-sync-issues-investigation-and-fix
Oct 23, 2023
Merged

[SPO] Fix bugs when handling drive_items in incremental sync#1827
jedrazb merged 3 commits intomainfrom
jedrazb-spo-incremental-sync-issues-investigation-and-fix

Conversation

@jedrazb
Copy link
Copy Markdown
Contributor

@jedrazb jedrazb commented Oct 23, 2023

Closes https://github.com/elastic/enterprise-search-team/issues/6051

This fixed the issue with changed content of a file (driveItem) not being picked up by an incremental sync.

The simple solution is to disable timestamp checks for site_drives and then rely on efficient delta api to get all changes for drive items.

I verified that the drive_item <-> drive_site is the only scenario when change to an object doesn't update its "parent" lastModifiedDateTime field. Interestingly, drive item updates would result in timestamp updates for its parent "list" and "site", so we are safe at keeping timestamp checks for sites.

For other object types in SPO there were no "unexpected" behaviour when it comes to updating timestamps after edits.

Bonus bug fix

I realized that with fetch_drive_item_permissions enabled, the incremental sync will fail at handling DELETE operations since we are trying to fetch ACLs for a deleted document - it will result in 404 error. I added a logic to handle that scenario and manually tested that it works.

Checklists

Pre-Review Checklist

  • this PR has a meaningful title
  • this PR links to all relevant github issues that it fixes or partially addresses
  • if there is no GH issue, please create it. Each PR should have a link to an issue
  • this PR has a thorough description
  • Covered the changes with automated tests
  • Tested the changes locally
  • Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
  • Considered corresponding documentation changes
  • Contributed any configuration settings changes to the configuration reference

@github-actions
Copy link
Copy Markdown

💚 Backport PR(s) successfully created

Status Branch Result
pre-8.10-stable #1828
8.11 #1829

The backport PRs will be merged automatically after passing CI.

jedrazb added a commit that referenced this pull request Oct 23, 2023
jedrazb added a commit that referenced this pull request Oct 24, 2023
…ntal sync (#1827) (#1828)

Co-authored-by: Jedr Blaszyk <jedrazb@gmail.com>
seanstory pushed a commit that referenced this pull request Dec 14, 2023
@seanstory
Copy link
Copy Markdown
Member

💚 All backports created successfully

Status Branch Result
8.10

Questions ?

Please refer to the Backport tool documentation

seanstory added a commit that referenced this pull request Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants