Skip to content

feat(health): allow excluding noisy LogServices in periodic log collection#3349

Open
cdraman wants to merge 1 commit into
NVIDIA:mainfrom
cdraman:HAAS-169
Open

feat(health): allow excluding noisy LogServices in periodic log collection#3349
cdraman wants to merge 1 commit into
NVIDIA:mainfrom
cdraman:HAAS-169

Conversation

@cdraman

@cdraman cdraman commented Jul 10, 2026

Copy link
Copy Markdown

When forge-hw-health runs in periodic log collection mode, it enumerates
every Redfish LogService on the BMC — including the bmcweb Journal, which
is the BMC web server's own HTTP access log.

This creates a self-reinforcing noise loop: every Redfish poll we make is
itself recorded as a Journal entry. The next poll collects those entries,
which generates more entries. Volume grows as a function of our own polling
rate, not real hardware events, drowning out meaningful signal like GPU XID
faults and CPER records.

Add exclude_services to collectors.logs.periodic — a list of substrings
matched against each LogService @odata.id. Matching services are skipped
at discovery time before any entries are fetched.

Related issues

Closes #3344

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

Additional Notes

@cdraman cdraman requested a review from a team as a code owner July 10, 2026 00:44
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 79772738-9d36-49f5-b875-bdf03001bb63

📥 Commits

Reviewing files that changed from the base of the PR and between a0baf02 and 38d9f1a.

📒 Files selected for processing (5)
  • crates/health/example/config.bmc-mock-periodic.toml
  • crates/health/example/config.example.toml
  • crates/health/src/collectors/logs/periodic.rs
  • crates/health/src/config.rs
  • crates/health/src/discovery/spawn.rs
🚧 Files skipped from review as they are similar to previous changes (5)
  • crates/health/example/config.bmc-mock-periodic.toml
  • crates/health/src/config.rs
  • crates/health/src/discovery/spawn.rs
  • crates/health/example/config.example.toml
  • crates/health/src/collectors/logs/periodic.rs

Summary by CodeRabbit

  • New Features
    • Added configurable, case-sensitive exclusion of Redfish LogService entries during periodic log collection using substring patterns matched against each service’s @odata.id.
    • Discovery now reports both the total number of services found and how many were excluded.
    • By default, “Journal” log noise is skipped; setting the list to empty includes all discovered log services.
  • Documentation
    • Updated example TOML configurations to show how to use exclude_services.
  • Tests
    • Added unit tests for empty exclusions, substring matching, multi-pattern exclusion, and case sensitivity.

Walkthrough

Periodic log collection now supports configurable, case-sensitive substring filters for Redfish LogService OData IDs. Matching services are skipped during discovery, exclusion counts are logged, Journal is excluded by default, and examples plus unit tests cover the behavior.

Changes

Periodic LogService Exclusion

Layer / File(s) Summary
Configuration contract and examples
crates/health/src/config.rs, crates/health/example/config.example.toml, crates/health/example/config.bmc-mock-periodic.toml
Periodic configuration adds exclude_services, defaulting to Journal filtering; examples document and configure the setting.
Collector configuration wiring
crates/health/src/discovery/spawn.rs, crates/health/src/collectors/logs/periodic.rs
Periodic startup passes exclusion patterns into LogsCollector, which stores them for discovery.
Service discovery filtering and validation
crates/health/src/collectors/logs/periodic.rs
Manager, chassis, and system LogService discovery filters matching OData IDs, de-duplicates retained services, reports exclusion counts, and tests matching behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant PeriodicLogConfig
  participant spawn_generic_redfish_collectors
  participant LogsCollector
  participant RedfishBMC
  PeriodicLogConfig->>spawn_generic_redfish_collectors: provide exclude_services
  spawn_generic_redfish_collectors->>LogsCollector: create LogsCollectorConfig
  LogsCollector->>RedfishBMC: discover LogServices
  LogsCollector->>LogsCollector: filter matching OData IDs
  LogsCollector-->>LogsCollector: log total and excluded service counts
Loading
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR changes the default to exclude Journal, but the linked issue requires an empty default that preserves existing behavior when omitted. Default exclude_services to an empty list and keep Journal exclusion as an example-only setting, so omitted config preserves current behavior.
Docstring Coverage ⚠️ Warning Docstring coverage is 55.56% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately reflects the main feature: excluding noisy LogServices during periodic health log collection.
Description check ✅ Passed The description matches the code change and explains the periodic LogService exclusion feature.
Out of Scope Changes check ✅ Passed All changes stay within periodic LogService filtering, config wiring, examples, and tests.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

🔍 Container Scan Summary

Service Total Critical High Medium Low Other
boot-artifacts-aarch64 3 0 0 3 0 0
boot-artifacts-x86_64 3 0 0 3 0 0
forge-admin-cli-x86_64 255 13 30 79 7 126
machine-validation-runner 800 40 234 292 36 198
machine_validation 800 40 234 292 36 198
machine_validation-aarch64 800 40 234 292 36 198
nvmetal-carbide 800 40 234 292 36 198
TOTAL 3461 173 966 1253 151 918

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.

@cdraman

cdraman commented Jul 10, 2026

Copy link
Copy Markdown
Author

@coderabbitai full review

@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/health/src/collectors/logs/periodic.rs (1)

427-481: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Prefer check_values over ad-hoc #[test] functions for this matcher.

service_is_excluded is a pure validator with two inputs (pattern list, service id) and a boolean output — exactly the case the style guide calls out for table-driven tests. As per coding guidelines, "Use check_cases / check_values directly when a macro would obscure a table with several inputs or several expected fields per row in table-driven tests." A single check_values table covering all five current scenarios (empty list, substring match, non-match, multi-pattern, case sensitivity) would replace the five separate #[test] functions with one enumerable table and make it trivial to add future cases.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/collectors/logs/periodic.rs` around lines 427 - 481,
Replace the five ad-hoc tests in the tests module with one table-driven
check_values test covering empty exclusions, substring matches, non-matches,
multiple patterns, and case sensitivity. Keep each case’s pattern list, service
ID, and expected boolean explicit, and use the project’s existing check_values
convention while continuing to exercise service_is_excluded.

Sources: Coding guidelines, Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/health/src/collectors/logs/periodic.rs`:
- Around line 417-425: Update service_is_excluded to ignore empty
exclude_services patterns before performing substring matching, ensuring only
non-empty patterns are checked against service_id.

---

Nitpick comments:
In `@crates/health/src/collectors/logs/periodic.rs`:
- Around line 427-481: Replace the five ad-hoc tests in the tests module with
one table-driven check_values test covering empty exclusions, substring matches,
non-matches, multiple patterns, and case sensitivity. Keep each case’s pattern
list, service ID, and expected boolean explicit, and use the project’s existing
check_values convention while continuing to exercise service_is_excluded.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 515f362c-71e1-4934-99a0-26ac9837da9e

📥 Commits

Reviewing files that changed from the base of the PR and between 699951d and 790262c.

📒 Files selected for processing (5)
  • crates/health/example/config.bmc-mock-periodic.toml
  • crates/health/example/config.example.toml
  • crates/health/src/collectors/logs/periodic.rs
  • crates/health/src/config.rs
  • crates/health/src/discovery/spawn.rs

Comment thread crates/health/src/collectors/logs/periodic.rs
@yoks

yoks commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Lets make exclude Journal by default, seems like this behavior is not desired in vast majority of cases

# Typically used to drop the bmcweb "Journal" HTTP-access log (high-volume noise,
# and self-referential since polling generates Journal entries). Empty (default)
# collects from every discovered LogService. Periodic mode only.
exclude_services = []

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to also define this config attrubute on L198.

Auto mode accepts both sse and periodic config variables for fallback behavior

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/health/example/config.bmc-mock-periodic.toml`:
- Around line 55-60: Update the periodic log collection comment to describe an
empty exclude_services list as an explicit opt-out that collects from every
discovered LogService, and identify ["Journal"] as the configured default shown
in this example. Remove the contradictory “Empty (default)” wording while
preserving the existing explanation of the exclusion behavior.

In `@crates/health/example/config.example.toml`:
- Around line 213-217: Align the documented and runtime defaults for periodic
log-service exclude_services. Update the exclude_services setting in the example
configuration or the corresponding PeriodicLogConfig::default implementation so
both use the same fallback value, including the comment describing the default.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 4c0c4310-d7ae-485a-894d-ad85ef8a92da

📥 Commits

Reviewing files that changed from the base of the PR and between a985af8 and a0baf02.

📒 Files selected for processing (5)
  • crates/health/example/config.bmc-mock-periodic.toml
  • crates/health/example/config.example.toml
  • crates/health/src/collectors/logs/periodic.rs
  • crates/health/src/config.rs
  • crates/health/src/discovery/spawn.rs
🚧 Files skipped from review as they are similar to previous changes (2)
  • crates/health/src/discovery/spawn.rs
  • crates/health/src/collectors/logs/periodic.rs

Comment thread crates/health/example/config.bmc-mock-periodic.toml Outdated
Comment thread crates/health/example/config.example.toml
…ction

When forge-hw-health runs in periodic log collection mode, it enumerates
every Redfish LogService on the BMC — including the bmcweb Journal, which
is the BMC web server's own HTTP access log.

This creates a self-reinforcing noise loop: every Redfish poll we make is
itself recorded as a Journal entry. The next poll collects those entries,
which generates more entries.

Add exclude_services to collectors.logs.periodic (and auto-mode fallback)
— a list of substrings matched against each LogService odata.id. Matching
services are skipped at discovery time before any entries are fetched.

Defaults to [\"Journal\"] since suppressing the bmcweb HTTP-access log is
the right behavior in the vast majority of deployments. Set to [] to
collect from every discovered LogService."

Signed-off-by: Dasa Chandramouli <dchandramoul@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Allow excluding noisy LogServices in periodic log collection in health

3 participants