Prometheus scrape targets are lost after `rl-insight server start`, leaving all dashboards

[tardis-key]

Summary

Restarting the monitor stack via rl-insight server start wipes the dynamically registered Prometheus scrape targets, so Prometheus stops scraping and every dashboard (agent_loop_trajectory, quick_start_demo, verl_tainer_* ) goes empty (e.g. run_id variable has no values).

Root cause

• The repo's prometheus.yml is an empty template (scrape_configs: []); scrape targets are meant to be registered at runtime.
• Trainers / the monitor hub register themselves at startup via update_prometheus_config() → PrometheusTargetStore.register(), which writes targets directly into the runtime prometheus.yml.
• On startup, _render_prometheus_config() overwrites the runtime prometheus.yml from the repo template, discarding all dynamically registered targets.
• Prometheus stops scraping; instant queries return empty because samples go stale after ~5 minutes. The historical data itself is NOT lost (still in the TSDB).

Impact

• All dashboards sharing the Prometheus datasource are affected at once.
• Also affects anything relying on node_exporter / trainer metrics registered the same way.
• Broken ordering: hub started before server start loses its registration; server restart after hub startup also loses it.

Suggested fix

Decouple static template from dynamic registration, e.g.:

1. file_sd: put a fixed scrape_configs entry with file_sd_configs pointing to a targets file (e.g. runtime/targets/trainer.yml); registration only updates that targets file, which server start does not touch.
2. Or make _render_prometheus_config() merge/keep existing runtime targets instead of overwriting.

Repro

1. Start data generation (generate_trace_data.py --metrics-report-port 9094) → dashboards show data.
2. Run rl-insight server start (restart).
3. curl http://localhost:9090/api/v1/targets → no active targets; all dashboards empty.

Environment

• Server: 178.139.22.6, container hxb_verl_0810
• rl-insight dev branch (tardis-key/rl-insight), Grafana 13.0.0 / Prometheus 2.54.1
• Runtime dir: /root/.rl-insight/runtime, TSDB: /root/.rl-insight/data/prometheus (retention 30d)

[cyyecao-lappland]

I would like to take this bug.

I plan to use the file_sd direction so static server rendering and dynamic target registration have separate ownership:

• keep a stable file_sd_configs entry in the rendered Prometheus config;
• move dynamically registered trainer / monitor-hub targets into a dedicated runtime target file;
• update that target file atomically during registration;
• ensure rl-insight server start can regenerate static configuration without deleting registered targets;
• preserve clean first-install behavior when no runtime target file exists yet.

Regression coverage would include registering targets before startup/restart, rendering the server configuration, verifying that targets survive, checking idempotent re-registration, and confirming that a fresh empty runtime still produces valid Prometheus configuration. I will also run a local Prometheus integration smoke test rather than relying only on YAML assertions.

I checked the currently open PRs and did not find an implementation of this restart-persistence fix. If file_sd is the preferred direction, I am happy to own the patch and tests.

[mayunaise]

Thanks @cyyecao-lappland for outlining the file_sd approach.

We independently reproduced this issue while validating RL-Insight against a live verl training job and have submitted #153 using the same general direction. The PR separates the generated Prometheus configuration from persistent target state, and also includes legacy target migration, atomic cross-process updates, restart recovery, and regression tests.

I'd appreciate your review and feedback on #153, especially if you see any gaps compared with the integration smoke test you planned. I'd also welcome suggestions on robustness and target lifecycle management, such as ownership, explicit unregister semantics, stale-target detection and cleanup, TTL/expiry behavior, and recovery after abnormal process termination. Those lifecycle concerns could be handled as a focused follow-up without expanding this restart-persistence bugfix.

I'm happy to incorporate improvements so we can avoid duplicating the implementation effort.

[cyyecao-lappland]

Thanks for letting me know. I’ll take a look at #153 instead of working on a duplicate fix. It looks like you’ve already covered the main points I had in mind. I’ll leave any specific feedback on the PR.

I agree that target cleanup and unregister support are better handled separately.