Skip to content

Fix HighMemoryUsage alert ratio - #28

Open
Zhaoziyuan2024 wants to merge 2 commits into
weilixiong:mainfrom
Zhaoziyuan2024:fix-high-memory-alert-ratio
Open

Fix HighMemoryUsage alert ratio#28
Zhaoziyuan2024 wants to merge 2 commits into
weilixiong:mainfrom
Zhaoziyuan2024:fix-high-memory-alert-ratio

Conversation

@Zhaoziyuan2024

Copy link
Copy Markdown

Summary

Fixes #1.

Fix the HighMemoryUsage Prometheus alert so it divides process RSS by machine memory instead of dividing process_resident_memory_bytes by itself. This also adds a focused validation path that rejects self-dividing alert expressions before dry-run or upload.

Changes

  • Updated HighMemoryUsage to use process_resident_memory_bytes / on(instance) group_left machine_memory_bytes > 0.9.
  • Aligned the instance:memory_usage:ratio recording rule with the same machine-memory denominator.
  • Added alert-rule validation for direct same-metric division.
  • Added unittest coverage for the recommended alert rules, the fixed memory alert, and an invalid self-dividing alert expression.
  • Added the required diagnostic build JSON and .logd chunks generated from the code commit.

Testing

  • python -m unittest discover -v
    • Passes: 4 tests.
  • python -m py_compile tools/monitoring_setup.py tests/__init__.py tests/test_monitoring_setup.py
    • Passes.
  • python tools/monitoring_setup.py --alerts --dry-run
    • Passes and prints the fixed HighMemoryUsage expression.
  • PYTHONIOENCODING=utf-8 python build.py
    • Generates diagnostic/build-fb0df9c3.json and three .logd chunks.
    • Exits 1 in this Windows Codex runtime because the runtime is missing the repo's cross-language build tools: Rust, Go, GCC/G++, CMake, Make, Java, Ruby, Lua, and GHC. The bundled Node runtime also does not include npm, so frontend is recorded as failed in the diagnostic metadata.

Checklist

  • Relevant modules affected by these changes build locally
  • Tests pass locally
  • Diagnostic build log is committed in this PR
  • Documentation has been updated, if applicable
  • Configuration or schema changes are documented, if applicable
  • No generated build artifacts are committed, except the required diagnostic build log
  • Changes are scoped to the PR purpose and avoid unrelated cleanup
  • Security, privacy, and error-handling implications have been considered

  • I would like to request that my diagnostic build log is removed before merging

@Zhaoziyuan2024

Copy link
Copy Markdown
Author

/claim #1

Ready for review/merge for bounty #1.

This fixes the HighMemoryUsage alert so it no longer divides process_resident_memory_bytes by itself, aligns the recording rule with the same machine-memory denominator, and adds validation/tests to catch direct same-metric division.

Validation performed:

  • python -m unittest discover -v - PASS, 4 tests
  • python -m py_compile tools/monitoring_setup.py tests/__init__.py tests/test_monitoring_setup.py - PASS
  • python tools/monitoring_setup.py --alerts --dry-run - PASS
  • PYTHONIOENCODING=utf-8 python build.py generated diagnostic/build-fb0df9c3.json and the .logd chunks; the build exits 1 in my Windows runtime because required cross-language tools are missing, which is recorded in the diagnostic metadata.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[$30 BOUNTY] [Python] Fix HighMemoryUsage alert ratio expression

1 participant