fills up RAM after 2 generations leading to PC lockup

[screwballl]

Describe the bug
Every generation reserves system RAM, then does not release it after finishing generation. With 32GB RAM it fills up and hard locks the PC by the 3rd or 4th gen attempt. Tested with 0.6B and 1.7B LM model. First gen increases system RAM usage from 1.5GB to 13.3GB, 2nd generation jumps to 16.5 to 21.4GB usage, 3rd generation jumps to 30 to 32GB+.
Almost every time after 2nd gen the default GUI in the browser keeps the prior gens in the results section instead of adding the new gens.

Expected behavior
Code should clear/free RAM at least partially and reuse the RAM loaded processes instead of starting new ones every generation.

OS and hardware
Linux Mint 22.3, Ryzen 9600X, 32GB DDR5-6000, 16GB RTX5060Ti

context
closing and restarting it, initialize service, and refresh page sometimes allows up to 4th gen but it is rare. Close and restart and refresh GUI in browser clears settings and lyrics entered, and language chosen.

[imsarang]

Hi is this issue still there, can take this up!
@jaretburkett @saltchicken @fspecii @xmhu @screwballl

[screwballl]

Updated with latest files, issue persists

[imsarang]

ok, gonna take this up!
@screwballl @jaretburkett @saltchicken @fspecii @xmhu

[rkfg]

This is indeed an issue. I run the Gradio UI with the following parameters: --server-name 0.0.0.0 --service_mode true --offload_to_cpu true --offload_dit_to_cpu true --enable-api
The .env file contains this:

ACESTEP_CONFIG_PATH=acestep-v15-sft
ACESTEP_LM_MODEL_PATH=acestep-5Hz-lm-4B

Each generation adds about 8 GB to the python process (the DiT size?). Even though I have 128 GB of RAM it fills up quickly because I also run other memory hungry programs like browser. I tried to exclude the LM as much as possible to narrow the range of possible culprits. Turned off CoT options, used a pre-generated audio codes, disabled "Think" of course. My workflow is regenerating the same batch of two songs with two fixed seeds and changing the lyrics a little between the runs. The log shows a lot of offloading of VAE, text encoder, and main model.

It's very easy to reproduce but I struggle with python tooling to find memory leaks, it's surprisingly poor. I tried making and comparing heap snapshots with tracemalloc but the result is unsatisfactory, it shows a few very small objects (about 20 Mb) while the process grows by 8 GB as I said before. It could be native torch memory, I'm not very experienced in these low level things in python.

I have a 5090 and use Debian testing, the program runs in a Docker container. I need offloading to let my other services use GPU as well.

[rkfg]

I added RSS memory tracking to init_service_offload_context.py: rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss and then print it in the log. The biggest offender is offloading the model during decoding audio codes to latents:

2026-03-06 06:46:33.531 | INFO     | acestep.core.generation.handler.init_service_offload_context:_load_model_context:57 - [_load_model_context] Loaded model to cuda in 0.4656s RSS: 13969964
2026-03-06 06:46:33.534 | INFO     | acestep.core.generation.handler.init_service_offload_context:_load_model_context:63 - [_load_model_context] Offloading model to CPU, RSS: 13969964
2026-03-06 06:46:34.873 | INFO     | acestep.core.generation.handler.init_service_offload_context:_load_model_context:74 - [_load_model_context] Offloaded model to CPU in 1.3391s, RSS: 18111156
  File "/opt/conda/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/opt/conda/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/app/acestep/core/generation/handler/generate_music_execute.py", line 59, in _service_target
    _result["outputs"] = self.service_generate(
  File "/app/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
  File "/app/acestep/core/generation/handler/service_generate.py", line 96, in service_generate
    batch = self._prepare_batch(
  File "/app/acestep/core/generation/handler/conditioning_batch.py", line 74, in _prepare_batch
    self._prepare_target_latents_and_wavs(batch_size, target_wavs, audio_code_hints)
  File "/app/acestep/core/generation/handler/conditioning_target.py", line 42, in _prepare_target_latents_and_wavs
    decoded_latents = self._decode_audio_codes_to_latents(code_hint)
  File "/app/acestep/core/generation/handler/audio_codes.py", line 58, in _decode_audio_codes_to_latents
    with self._load_model_context("model"):
  File "/opt/conda/lib/python3.11/contextlib.py", line 144, in __exit__
    next(self.gen)
  File "/app/acestep/core/generation/handler/init_service_offload_context.py", line 75, in _load_model_context
    traceback.print_stack()

The stack trace is printed on every move (to CUDA and to CPU) so I can see what caused it exactly. Here the RSS jumped from 14 GB to 18 GB because of offloading. This happens every time I generate. I also noticed that this code path is executed again, the model is loaded to GPU and then offloaded back to CPU and this time RSS does NOT increase. Then the model is yet again loaded and offloaded during generation and RSS doesn't move at all. So it only happens for the first time the model is offloaded, VAE and text encoder also seem to increase RSS when they're offloaded for the first time but it's less noticeable.

So the problem seems to be that these models (DiT, VAE, Qwen text encoder) are loaded from scratch on every generation (clicking the "Generate Music" button that is) and only persist during it, but then somehow lost or leaked.

[ChuxiJ]

Thank you for this incredibly detailed bug report! Special thanks to @rkfg for the thorough investigation and pinpointing the exact cause.

Confirmed Issue

We've identified a serious memory leak in the model offloading logic. The problem occurs when using --offload_to_cpu or --offload_dit_to_cpu flags:

Root cause: In acestep/core/generation/handler/init_service_memory_transfer.py, the _recursive_to_device method creates new torch.nn.Parameter objects during device transfer (lines 29-36, 91-98) without explicitly deleting the old parameters. This causes the model to be duplicated in memory rather than moved, leading to ~8GB RAM increase per generation.

Location: The issue is in both:

• init_service_offload_context.py - Context manager that triggers the moves
• init_service_memory_transfer.py - The actual parameter transfer logic

Temporary Workaround

Until this is fixed, you can try:

1. Avoid offloading if you have sufficient VRAM - run without --offload_to_cpu flags
2. Restart service after every 2-3 generations to clear accumulated memory
3. Use smaller models (0.6B instead of larger ones) to reduce memory pressure

Fix Plan

The fix will involve:

1. Add explicit cleanup when creating new parameters:

old_param = module._parameters[param_name]
module._parameters[param_name] = new_param
del old_param  # Explicit cleanup

2. Use in-place operations where possible instead of creating new Parameter objects

3. Add memory monitoring to track and warn about high RAM usage

4. Force garbage collection after offload operations

Related Issues

This is related to:

• API server silently loads LLM on first request and retains ~4 GB VRAM permanently #198 (LLM memory retention in API server)
• PR Fix cumulative GPU/CPU memory leak in generation workflow #781 (may already contain related fixes)

We'll prioritize this fix as it causes system instability. I'll create a PR to address this issue. Thank you again for the detailed analysis!