Skip to content

Latest commit

 

History

History
44 lines (30 loc) · 4.78 KB

Krystal-Coded.md

File metadata and controls

44 lines (30 loc) · 4.78 KB

Krystal-Coded

Purpose

This update makes the scheduled platform-stat sync more trustworthy under throttling, temporary upstream outages, and manual workflow runs. The design deliberately keeps the existing Supabase schema and scoring model: the changes improve how data is collected and protected, not how members earn points.

CodeChef: first-party profile data

  • Decision: remove the dependency on codechef-stats.tashif.codes. Its rate limiting had produced intermittent empty/zero results, while the public CodeChef profile exposed the required values.
  • Implementation: scripts/fetchers/codechef.py now makes one request to https://www.codechef.com/users/{handle} and extracts current rating, highest rating, total solved problems, and contest count from the returned HTML.
  • Protection: the fetcher recognizes a real profile page before reporting success, reports failure explicitly when it receives an invalid/challenge page, serializes all CodeChef requests, and leaves a two-second gap before the next request.
  • Documentation: api base url.md now records the first-party CodeChef URL.

Rate-limit and retry strategy

  • Codeforces: scripts/fetchers/codeforces.py uses one shared semaphore and now keeps a two-second gap between all Codeforces API calls. This matches Codeforces' public API guidance and replaces the former one-second interval.
  • CodeChef: scripts/fetchers/codechef.py uses the same serialized approach to reduce Cloudflare throttling risk without adding a proxy service or paid credential.
  • Shared retry behavior: scripts/fetchers/utils.py adds retry_delay(). A 429 response now waits for the server's numeric or HTTP-date Retry-After value when present. If the server supplies no usable value, the client uses exponential backoff plus small random jitter so concurrent retries do not all restart together.
  • Callers updated: the shared JSON fetcher plus the Codeforces and CodeChef-specific retry paths use the new delay behavior.

Snapshot integrity

GitHub Actions protection

  • Decision: prevent a manually dispatched run and the twice-daily schedule from querying every platform at the same time.
  • Implementation: .github/workflows/sync.yml now uses the member-stats-sync concurrency group with cancel-in-progress: false. One sync runs at a time; a manual run waits rather than doubling external request traffic.

Verification

  • Live CodeChef check: verified direct profile values for the supplied accounts, including zesty_wasp_34 at 161 solved problems and a 1255 rating.
  • Live Codeforces check: verified successful accepted-problem retrieval from a high-volume public profile.
  • Unit tests: scripts/tests/test_codechef.py covers profile parsing and server-directed retry timing. scripts/tests/test_sync_safety.py covers skipped first snapshots and preservation of an existing CodeChef record after a failed fetch.
  • Validation command: python -B -m unittest scripts.tests.test_codechef scripts.tests.test_sync_safety -v.

Intentionally not added

  • No ScraperAPI/proxy integration: direct CodeChef profile requests worked after pacing was added, so there is no new vendor cost, credential, or data-processing dependency.
  • No database migration: _fetch_ok is runtime-only and never written to Supabase.
  • No scoring changes: the existing leaderboard formula continues to use the same CodeChef, LeetCode, Codeforces, GitHub, GFG, and HackerRank fields.