Feature: Fine‑grained control over summary reporting information scope

[Erpim]

Feature request

Propose two dimensions of configuration to control the scope of collected and reported summary data in rl‑insight:

1. Level‑based tiering: Introduce configurable detail levels for collected data. Higher levels correspond to more detailed information, while lower levels only retain core minimal metrics. Users can set a level threshold, and data items with a higher level than the threshold will be skipped. More detailed high‑level data may introduce increased disk write overhead.
2. Module‑label based per‑module level control: Support module labels such as agent, train, rollout, trace. Allow independent level configuration for each module label. Users can either completely disable data collection for uninterested modules, or enable high‑detail level only for specific modules while keeping others at minimal level.

Proposed mechanism:

• A global default level as fallback for modules without explicit configuration.
• Per‑module label level override to take precedence over global setting.
• Each data entry is tagged with its required level and module label; collector filters entries according to matching module‑level config.
• Configuration is adjustable via config file or runtime parameters.

Motivation

Currently rl‑insight tends to collect full‑volume data by default, which brings non‑trivial overhead in large‑scale RL training workloads. I'm always frustrated when collecting full‑detail summary data leads to slow disk I/O, increased write latency and redundant large log storage, even when users only care about a subset of modules or metrics.

Different users have different observation needs:

• For production runs: only core monitoring metrics are required, and detailed trace/rollout data is unnecessary.
• For debugging: users may only need high‑detail data for one specific module (e.g. rollout), while other modules can stay at basic level to save I/O cost.

This feature enables users to trade‑off between observation granularity and runtime performance. It avoids unnecessary data collection overhead, while still retaining the capability to capture full‑detail information on demand for targeted modules.

Use‑case examples

1. Debug scenario: set high level only for rollout module, keep train/agent at basic level to reduce disk overhead.
2. Production scenario: use low global level, disable irrelevant modules, keep only essential monitoring metrics.
3. Investigation scenario: enable high‑detail level for trace module for deep problem analysis.

Your contribution

Provide actual scenario as input.

[tardis-key]

I think this feature will become increasingly important as the rl-insight scenario expands! I also agree that it should be implemented, and I'll add it to my to-do list.

Of course, if you'd like to give it a try, you're more than welcome.