Skip to content

feat(allocation): dynamically enforce allocation policies via load thresholds - #8527

Merged
pbhandari merged 2 commits into
masterfrom
prajjwalbhandari/eap-831-dynamically-enforce-allocation-policies
Oct 8, 2026
Merged

pbhandari merged 2 commits into
masterfrom
prajjwalbhandari/eap-831-dynamically-enforce-allocation-policies

Conversation

@pbhandari

Copy link
Copy Markdown
Contributor

Closes EAP-831.

Replaces the two storage_routing boolean flags with per-policy load thresholds that gate dynamic pardoning.

Changes (one commit each)

  1. Always fetch cluster load info — drop storage_routing.enable_get_cluster_loadinfo (option + schema key + the gates in get_cluster_loadinfo and db_query); load info is now fetched unconditionally so dynamism is always configurable.
  2. Configurable per-policy thresholds — add a block-level thresholds key to the allocation_policies config (sibling of policies/match/remove). The resolver copies the block's thresholds onto every policy it constructs, so each policy owns its reject/throttle ceilings at runtime. Keys are enumerated in sentry-options/schemas/snuba/schema.json, and a test asserts those keys are a subset of LoadInfo's dataclass fields.
  3. Dynamic enforcement — drop storage_routing.enable_dynamic_allocation_policy. LoadInfo.should_pardon() is replaced by LoadInfo.exceeds(thresholds), and the pardon path in QuotaAllowance.decision now pardons a rejection only when the policy is pardonable, load info exists, reject thresholds are configured, and the cluster is below all of them. No thresholds / missing load info means keep rejecting.

Config shape:

allocation_policies:
  EAP:
    - match: {...}
      policies: [...]
      thresholds:
        reject:   { cluster_load: 90, concurrent_queries: 100 }
        throttle: { ... }   # helper built now, consumed later

is_enforced / is_pardonable

  • is_enforced=False -> dynamism skipped; policy always passes through (existing short-circuit).
  • is_enforced=True + no thresholds -> always enforced (never pardoned).
  • is_pardonable is unchanged — still the per-policy toggle for participating in dynamic pardoning.

Out of scope

  • The generic-exception path in _get_cluster_loadinfo returns LoadInfo() with all -1s, which compares below any threshold and would wrongly pardon. Fail-safe handling of that case is deferred.
  • Wiring thresholds.throttle into throttling decisions (the helper supports it; consumption comes later).

Tests

  • tests/query/allocation_policies/test_thresholds_schema.py (new): schema threshold keys subset of LoadInfo fields.
  • LoadInfo.exceeds unit tests.
  • Updated existing pardon tests (test_allocation_policy_base, test_db_query, test_storage_routing, test_cluster_loadinfo) to the thresholds API.

@linear-code

linear-code Bot commented Oct 8, 2026

Copy link
Copy Markdown

EAP-831

@pbhandari
pbhandari force-pushed the prajjwalbhandari/eap-831-dynamically-enforce-allocation-policies branch 7 times, most recently from 8dd1ba8 to ecef84b Compare October 8, 2026 19:39
@pbhandari
pbhandari marked this pull request as ready for review October 8, 2026 19:40
@pbhandari
pbhandari requested review from a team as code owners October 8, 2026 19:40
Comment on lines +409 to +414
"cluster_load": { "type": "number", "optional": true },
"concurrent_queries": { "type": "number", "optional": true },
"cgroup_user_time_normalized": { "type": "number", "optional": true },
"disk_inflight_ops": { "type": "number", "optional": true },
"memory_allocated": { "type": "number", "optional": true },
"part_mutation": { "type": "number", "optional": true }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't need all these signals. But this is fine as a temp measure while we figure out what signals to monitor.

@pbhandari
pbhandari force-pushed the prajjwalbhandari/eap-831-dynamically-enforce-allocation-policies branch from ecef84b to b1f9212 Compare October 8, 2026 20:04
@pbhandari
pbhandari merged commit 4e49389 into master Oct 8, 2026
72 checks passed
@pbhandari
pbhandari deleted the prajjwalbhandari/eap-831-dynamically-enforce-allocation-policies branch October 8, 2026 23:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants