[BUG] memtrace multicore: DFE no-forward-progress assert

[iconstan01]

Title

memtrace multicore: DFE no-forward-progress assert, then teardown asserts (FTQ/Lookahead not drained)

Body

Hi all,

I found a reproducible multicore memtrace liveness failure.

Baseline repro (first issue observed)

• Commit tested: cc8a749
• Frontend: memtrace
• Cores: 2

Command:

./src/scarab --frontend memtrace --num_cores=2 \
  --cbp_trace_r0=/home/iconst01/SCARAB_traces/geekbench/single_core/navigation/traces/simp/4190.zip \
  --cbp_trace_r1=/home/iconst01/SCARAB_traces/geekbench/single_core/html5_browser/traces/simp/780.zip \
  --inst_limit=100000000

Observed:

No forward progress for 1000000 cycles
src/decoupled_frontend.cc:368: ASSERT FAILED ... : 0

Immediately after abort, teardown asserts also fire:

src/lookahead_buffer.cc:45: ASSERT FAILED ... : ft_buffer_count == 0
src/decoupled_frontend.cc:190: ASSERT FAILED ... : ftq.empty()

Follow-up investigation

1. I changed DFE forward-progress accounting from global to per-core.
   The failure persisted.

2. I enabled debug prints and saw DFE frequently breaking due to FTQ full (Break due to full FTQ), i.e. no effective progress.

3. I then removed the DFE break on FTQ-full just for debugging.
   This avoids the original DFE forward-progress assert path, but the run still fails later with:

   • Exact message: What prevents proceeding? Node stage is empty!
   • src/sim.c:344 forward-progress assert (last_forward_progress:0, node stage empty)
   • and teardown asserts (lookahead_buffer.cc:45, decoupled_frontend.cc:190)

Interpretation

The initial user-visible failure is DFE no-forward-progress.
The later lookahead_buffer / ftq.empty() asserts appear to be fallout during shutdown after the liveness failure.

I can provide:

• full logs
• debug log snippets
• temporary debug-only patch (not proposed as final fix)

Suggestion (process)

To catch this class of issues earlier (Or any multicore issues), it may help to add a small multicore memtrace regression to CI
(for example, 2 cores with a short inst_limit) and fail the run if forward progress stops.

[iconstan01]

Additional regression-window context:

I did not observe this failure on an older base commit (d2853c8) while running many multicore memtrace experiments (same workflow class, including long multicore batches).
On latest tested commit (cc8a749 ), the issue reproduces with the command in the issue body.

So this appears to be a regression introduced somewhere in:

d2853c8 .. cc8a749

I have not fully bisected this range yet.

[yql5510718]

This is probably because of the default lookahead buffer enabling. Currently, exec_driven mode does not support the lookahead buffer feature.

[iconstan01]

Thanks, that helps. One clarification so I can understand your comment clearly: I’m reproducing in --frontend memtrace, but LB appears enabled/used there as well in current code:

• lookahead_buf_size default is 1,
• sim.c unconditionally initializes LB,
• DFE on-path path unconditionally pops from LB.

So memtrace seems to still traverse the LB path currently.
If the intended behavior is “LB only for exec_driven,” should memtrace/PT bypass LB (or allow lookahead_buf_size=0 without assert)?

[yql5510718]

So we make LB enabled by default for both trace and execution mode. But it's not tested for execution-driven. If you don't need the lookahead buffer, the fastest way to solve this is to revive the FT build calling before the LV integration like this.

[iconstan01]

Thanks for the clarification.

I understand LB is enabled in both trace/memtrace and execution-driven modes, and that it has only been validated in trace mode so far.

From what I reported in this issue, the current LB path is fine in single-core memtrace, but not robust in multicore memtrace.

So I think the missing piece is multicore validation/coverage for the LB path.

Also, to confirm LB is the source of this behavior: on top of the commit I referenced above, I implemented a local LB bypass/disable path, and with that change the multicore memtrace reproducer finishes normally.

[aperais]

Dear @yql5510718 , I'm working with @iconstan01 on this project and multi core support is definitely something we will need down the line, hence the pipecleaning work done by @iconstan01

There is no debate that maintaining such a piece of software is a great timesink and a service to the community, but the multicore functionalities appear less tested than the single core ones. Perhaps we could contribute some tests for the multicore part, but looking into the "Actions" tabs I suspect committers run smoketests by hand rather than through the CI, and so am not sure how to go on about this.

[hlitz]

Our CI indeed is only running memtrace single core but we are planning to add multi core and exec driven. We will have another patch to disable the LB for exec soon