Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation. Provide catalog health check and surgery for four existing issues. #1

Merged
merged 60 commits into from
Jul 8, 2019

Commits on May 23, 2019

  1. Drop unnecessary empty test.

    deiferni committed May 23, 2019
    Configuration menu
    Copy the full SHA
    1983559 View commit details
    Browse the repository at this point in the history
  2. Fix tests for plone 5.

    deiferni committed May 23, 2019
    Configuration menu
    Copy the full SHA
    e2b6bea View commit details
    Browse the repository at this point in the history

Commits on Jul 8, 2019

  1. Add catalog checkup for rid/uid and data.

    Check consistency of mapping from path to rid and vice versa, also check
    catalog metadata for its contents.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    c92f16d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f9d1ccd View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    08fba6d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    a9eae33 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    85894cf View commit details
    Browse the repository at this point in the history
  6. Allow writing to a log.

    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    635e963 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    d4f0f05 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    b00244c View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    a9dae03 View commit details
    Browse the repository at this point in the history
  10. Improve naming in various places.

    - Split catalog module into surgery and checkup.
    - Use unhealthy instead of abberation
    - Use healthcheck instead of checkup
    - shorten zopectl command name
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    a380f49 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    8cd7185 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    36fc03f View commit details
    Browse the repository at this point in the history
  13. Add surgery for orphaned rid.

    The object that was used to catalog these entries is gone and cannot be found
    at path any longer.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    14c4310 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    04e96a5 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    6cef3fa View commit details
    Browse the repository at this point in the history
  16. Add commit and post-surgery healthcheck.

    Only commit in surgery and only if the post-surgery health check is successful.
    Also add '--dry-run` argument to prevent committing for a testrun.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    60ccfb3 View commit details
    Browse the repository at this point in the history
  17. Add UID index length to healthcheck.

    The UID index contains one UID per plone object. The UID for each object is
    unique, thus we know that for each object in the catalog there will also be one
    entry in the UID index. Thus we can use this index as a sample to check the
    catalog index health.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    130e2a9 View commit details
    Browse the repository at this point in the history
  18. Add command tests, slightly rework arguments.

    Default dryrun to true for checkup. Checkup must never cause persistent changes
    and thus never commit.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    271b2f5 View commit details
    Browse the repository at this point in the history
  19. Improve surgery index removal.

    Remove manually from indices as unindex_object aborts too
    early if the internal data is inconsistent.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    879b7f3 View commit details
    Browse the repository at this point in the history
  20. Rename index_data to catalog_data.

    We mostly check the catalogs internal data strucuture and not the indices. I
    think not mentioning indices in the method name is better.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    5ca56ee View commit details
    Browse the repository at this point in the history
  21. Consistently call index uuid_index.

    This hopefully helps a bit to avoid confusion between the uuid_index, an index
    in the catalog and the catalogs internal uid attribute which contains the
    path->rid mapping.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    a80d0be View commit details
    Browse the repository at this point in the history
  22. Provide get_physical_path and get_rid methods.

    Consistently provide these methods as helper methods on the base test class.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    7edcd7d View commit details
    Browse the repository at this point in the history
  23. Add healthcheck for uuid index data.

    Check rids in index and unindex against each other and agains the catalogs
    uid mapping (path->rid).
    This seems to change some of our assumptions regarding how we should update
    catalog index length.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    9be3479 View commit details
    Browse the repository at this point in the history
  24. Add helper to make unhealty extra rid after move.

    Creates an unhealthy catalog the same way as happens in production. Should then
    be fixable by surgery.
    
    Also document in what way the catalog is unhealthy and provide entry points for
    further analysis while developing fixes.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    5e45e2d View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    b648547 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    e581946 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    c4c7544 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    00c9df9 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    4a5d787 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    1c470ec View commit details
    Browse the repository at this point in the history
  31. Add surgery for unindex_object call.

    Some indices seem to be ablte to maintain their internal integrity thus we can
    use the official `unindex_object` method to remove a rid from these indices.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    189ce41 View commit details
    Browse the repository at this point in the history
  32. Configuration menu
    Copy the full SHA
    1850ae6 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    ae77525 View commit details
    Browse the repository at this point in the history
  34. Configuration menu
    Copy the full SHA
    9efc04a View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    7d91876 View commit details
    Browse the repository at this point in the history
  36. Pass a linked Length to decrease during removal.

    Allow to pass a `Length` that has to be decreased once an index row is empty.
    Slightly refactors coupling of `index._length` to the index which is most
    reliably used to determine the correct length as this varies heavily between
    different index implementations.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    5eeed1d View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    72557a3 View commit details
    Browse the repository at this point in the history
  38. Configuration menu
    Copy the full SHA
    612f06d View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    a482db9 View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    77dda25 View commit details
    Browse the repository at this point in the history
  41. Configuration menu
    Copy the full SHA
    ff66ab7 View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    f25ff7e View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    c58b2ec View commit details
    Browse the repository at this point in the history
  44. Configuration menu
    Copy the full SHA
    eb45986 View commit details
    Browse the repository at this point in the history
  45. Add plone 5.1 classifier.

    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    18008c9 View commit details
    Browse the repository at this point in the history
  46. Add surgery to drop extra rid not in indexes.

    Re-use existing surgery for extra rid, that is not in the UUID index, but in
    rid-->path mapping and catalog metadata.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    db45c54 View commit details
    Browse the repository at this point in the history
  47. Process indexing queue after fixes.

    Some fixes reindex objects, to guarantee an up-to date catalog before
    post-surgery healthcheck we process the indexing queue.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    a897511 View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    1f8bb48 View commit details
    Browse the repository at this point in the history
  49. Configuration menu
    Copy the full SHA
    cf7e754 View commit details
    Browse the repository at this point in the history
  50. Fix for acquired objects moved into their parents.

    Correct special case handling for objects that were moved into one of their
    parent objects and thus can still be found via acquisition when their former
    path is used as their id was not changed.
    We assumed that the redirector is involved, this is not correct. Thus drop
    redirector check and instead make sure that the acquisition chain of the wrapped
    and unwrapped object is different.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    a802308 View commit details
    Browse the repository at this point in the history
  51. Configuration menu
    Copy the full SHA
    381f8af View commit details
    Browse the repository at this point in the history
  52. Always doom transaction for healthcheck.

    Explicitly doom transaction in healthcheck command, factor out healthcheck
    helper function that can be called from surgery.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    99e5af3 View commit details
    Browse the repository at this point in the history
  53. Configuration menu
    Copy the full SHA
    f95437e View commit details
    Browse the repository at this point in the history
  54. Configuration menu
    Copy the full SHA
    894a421 View commit details
    Browse the repository at this point in the history
  55. Rename IndexSurgery to SurgeryStep.

    Improves readability by avoiding to overload the name surgery.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    e5ec455 View commit details
    Browse the repository at this point in the history
  56. Configuration menu
    Copy the full SHA
    83c668d View commit details
    Browse the repository at this point in the history
  57. Drop unused imports.

    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    ff140bd View commit details
    Browse the repository at this point in the history
  58. Test surgery lookup by doctor is correct.

    Also improve testing and add another assertion here and there.
    deiferni committed Jul 8, 2019
    Configuration menu
    Copy the full SHA
    9f2195b View commit details
    Browse the repository at this point in the history