Skip to content

Proposed work: consolidate shared public JSON fetch helper for maintenance scripts #1144

Description

@SueJianjian

Summary

Several maintenance/report scripts currently implement their own small public-JSON fetch helper around urllib.request.urlopen(...), but they do not agree on request headers, timeout/error wrapping, or decode behavior.

This shows up today in at least these files on main:

  • scripts/check_bounty_issue_states.py
  • scripts/check_live_bounty_closing_refs.py
  • scripts/claim_inventory.py
  • scripts/proposed_work_triage.py
  • scripts/submission_quality_gate.py

Current examples:

  • claim_inventory.py uses _get_json(...) with accept: application/json and wraps only URLError / TimeoutError as RuntimeError("public API request failed: ...").
  • submission_quality_gate.py uses _load_json_url(...) with no explicit JSON Accept header and wraps HTTPError, OSError, URLError, and JSONDecodeError as "{description} unavailable: ...".
  • proposed_work_triage.py uses _load_public_json(...) with only a custom User-Agent, then callers each decide how to degrade on network / JSON failures.
  • check_live_bounty_closing_refs.py and check_bounty_issue_states.py each have another _fetch_json(...) copy with their own Accept header and error text.

Why it is useful

This is a small maintainability gap with real regression risk:

  • a future bugfix to timeout handling, JSON decode failures, or required headers has to be copied across multiple scripts
  • scripts that should behave similarly when public API reads fail currently produce different error wording and slightly different failure classes
  • contributors and maintainers cannot rely on one consistent contract for "best-effort public JSON load" across report/readiness helpers

A bounded cleanup would be to introduce one shared read-only helper for public JSON fetches and migrate the current scripts to it, while preserving each script's higher-level behavior.

Smallest useful scope

  • add a shared helper for public JSON fetches used by maintenance/report scripts
  • standardize the baseline request headers and JSON decode path
  • standardize which network / HTTP / JSON parse failures are wrapped at the helper layer
  • keep per-script policy decisions at the call site (fail hard vs warning/degrade), rather than changing product behavior

Acceptance criteria

  • one shared helper replaces the duplicated urllib JSON fetch helpers in the scripts listed above, or an equivalent same-scope set on current main
  • the helper documents the read-only contract and the baseline exception semantics
  • focused tests cover at least success, malformed JSON, and network/HTTP failure paths
  • existing script-specific behavior stays intact apart from the intended internal helper consolidation

Evidence

  • scripts/check_bounty_issue_states.py defines _fetch_json(...)
  • scripts/check_live_bounty_closing_refs.py defines another _fetch_json(...)
  • scripts/claim_inventory.py defines _get_json(...)
  • scripts/proposed_work_triage.py defines _load_public_json(...)
  • scripts/submission_quality_gate.py defines _load_json_url(...)

These helpers all perform the same narrow job of read-only public JSON loading, but with drift in headers, decoding, and exception wording.

Duplicate search

Checked open proposed-work issues and recent submissions around:

  • shared public JSON helper
  • urllib/json loader for maintenance scripts
  • public API fetch helper consolidation

Related but distinct:

  • #1140 covers shared read-only gh CLI subprocess handling, not HTTP/urllib public JSON fetches
  • #1142 covers gh decoded JSON shape validation, not network fetch helper consolidation
  • #1143 covers response-shape validation policy after data is loaded, not the shared HTTP/JSON loader itself

Out of scope

  • no payment execution, treasury mutation, wallet custody, bridge/exchange/off-ramp, or MRWK price behavior
  • no change to public API payload schemas themselves
  • no broad refactor of unrelated script parsing or reporting logic

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions