Speed up indexing with cursor_tuple_fraction pg option #11382

[jacobtylerwalls]

Types of changes

• Bugfix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)

Description of Change

See #11382

Issues Solved

Closes #11382

Checklist

• I targeted one of these branches:
  • dev/8.0.x (under development): features, bugfixes not covered below
• I added a changelog in arches/releases
• I submitted a PR to arches-docs (if appropriate)
• Unit tests pass locally with my changes
• I added tests that prove my fix is effective or that my feature works
• My test fails on the target branch

Ticket Background

• Sponsored by: Farallon

[aarongundel]

I don't understand what this is doing - I don't see this option listed https://docs.djangoproject.com/en/5.1/ref/databases/#postgresql-connection-settings in here. Can you expand on this or point me to some documentation?

[jacobtylerwalls]

Sorry, the link was buried in the issue:

We use QuerySet.iterator() in a couple places for indexing and deletion, and we tend to fully exhaust those iterators, so we would probably benefit from setting the cursor_tuple_fraction option:

By default, PostgreSQL assumes that only the first 10% of the results of cursor queries will be fetched. The query planner spends less time planning the query and starts returning results faster, but this could diminish performance if more than 10% of the results are retrieved.

Django link
PG link

This a little bit getting out in front of the issue since I don't have an example in hand where we're being bitten by this (e.g. where we're suffering with a slower query plan), but it seems good to go ahead and bias this toward fetching the whole result, as every iterator() we've used is always fully exhausted.

PS -- I'm already using this in the lingo settings, where the concept tree is built up using an iterator().

[aarongundel]

LGTM