Skip to content

fix(checkpoint): FTW output drops the PLE n-gram shards, so qwen4_exp checkpoints cannot be served from it #370

Description

@sgzxs
With no index and no safetensors present, this returns an empty list, so the `shards` dict in
`ple_disk.py:63-87` stays empty and `ple_disk.py:88-89` raises:

```python
if sorted(shards) != list(range(len(shards))) or not shards:
    raise ValueError(f"PLE shard indices are not contiguous 0..N-1: {sorted(shards)[:8]}")

Note the message is misleading for this failure mode: [] means "found no PLE shards in this
folder at all", but it reads as "found them and they are out of order", which points an operator at
checkpoint corruption rather than at a missing file.

Restoring only some of the 10 files produces the genuinely useful form, but the preview is still
confusing, because the shards are scattered across the files in lexicographic key order rather than
numeric order - which is what weight.py:229-231 documents ("scattered over the model-plefp8-*
shards in header (lexicographic) order"). Verified from the file headers:

model-plefp8-00000.safetensors -> shards [0, 1, 10, 100..109]
model-plefp8-00001.safetensors -> shards [11, 12, 110..120]
model-plefp8-00002.safetensors -> shards [13..18, 121..127]
model-plefp8-00003.safetensors -> shards [2, 3, 19..29]
...
model-plefp8-00009.safetensors -> shards [9, 90..99] + weight_scale

So with files 00005-00008 still outstanding the check reports
[0, 1, 2, 3, 4, 9, 10, 11], and with 00007-00008 outstanding it reports
[0, 1, 2, 3, 4, 5, 6, 9]. Both are simply the first eight indices present, not a
contiguous-prefix failure - which is worth saying in the message too.

Evidence

Converted output directory as ft checkpoint left it, before restoring anything (73 GiB, 223
files; it is 121 GiB now that the 10 PLE shards are back):

  • freetoken-00000.ftw .. freetoken-00009.ftw, 72.70 GiB total
  • freetoken_weight.json: 794 weight + 288 experts_bank tensors, and zero keys matching
    ngram_embedding
  • 0 files matching *.safetensors; no model.safetensors.index.json
  • all non-weight metadata was copied through, including the 192 layer-*-experts-*.complete.json
    sidecars (they end in .json, so _WEIGHT_SUFFIXES does not skip them)

That last row is the tell: the copy step is working exactly as written. Only the weight-class-3
shards are missing, because only they are both .safetensors and not represented in the FTW.

What the 10 dropped shards contain, verified against the published model.safetensors.index.json
Independent of which is chosen, the ple_disk.py:88-89 message should distinguish "no PLE shards
found under <folder>" from "shards found but non-contiguous". The former is what a dropped-file
bug looks like, and the current wording sent me looking for checkpoint corruption first.

Workaround

Keep the 10 model-plefp8-*.safetensors in the directory that --model points at. The index is
not required: with only the PLE shards present, the iter_weight_files() fallback resolves them
correctly and does not pick up the .ftw files. I confirmed that directly - _ple_table_files()
on a directory holding 3 of the 10 shards and no index returned exactly those 3 paths, with
includes any .ftw? False.

With all 10 restored (each byte-exact against the published size, 47.68 GiB total) the table
resolves:

from freetoken.models.qwen4_exp.ple_disk import resolve_row_source
s = resolve_row_source(r"E:\LLM\models\RadixArk--Qwen3.8-Flash-Next-NVFP4")
len(s.paths), len(s.extent_base), s.total_rows, s.row_bytes, s.scale
(10, 128, 320001536, 160, 0.00019931793212890625)

and ft serve on the FTW directory now starts and generates, on the default --ple-backend disk:

$ curl -s http://127.0.0.1:1919/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"RadixArk--Qwen3.8-Flash-Next-NVFP4","messages":[{"role":"user","content":"The capital of France is"}],"max_tokens":48,"temperature":0.0}'
{"id":"chatcmpl-3","object":"chat.completion","created":1788456738,"model":"RadixArk--Qwen3.8-Flash-Next-NVFP4","choices":[{"index":0,"message":{"role":"assistant","content":"Paris","reasoning_content":"We need to respond to user: \"The capital of France is\". Likely wants completion. Need final concise. Ensure no hidden issues. Answer: Paris. Could complete sentence."},"finish_reason":"stop"}],"usage":{"prompt_tokens":57,"completion_tokens":40,"total_tokens":97}}

GET /health returns 200 and GET /v1/models reports context_length: 262144. That request took
14.5 s wall for 40 completion tokens (cold); one earlier request issued during engine warm-up
returned a JSON error body with no choices. The per-table scalar weight_scale is read from
model-plefp8-00009.safetensors and applied at lookup (ple_disk.py:261-262), so this path is
not the silent wrong-embedding case the checkpoint's qualification notes warn about.

So the only thing standing between a successful ft checkpoint and a servable directory is those
10 files, which the converter had already read once and then declined to carry over.

On a 128 GiB host, stay on the default --ple-backend disk. The pinned path puts the full
47.7 GiB table in host RAM on top of the offload expert banks; #293 reports 113 GiB worker RSS at
ready for that configuration, which does not fit here.

Disclosure

Drafted with AI assistance. Every file and line reference, byte count, tensor name, dtype, shape,
shard-to-file mapping, version string, log line and HTTP response above was read directly from the
installed 0.1.2+g0ee42b72c sources, from main at 03c28d2b via the GitHub API, from the
checkpoint's published model.safetensors.index.json and the restored shard headers, or from the
running server. The resolve_row_source tuple and the /v1/chat/completions response in the
workaround section are measured, not projected: the 10 shards are complete and byte-exact, and the
request was served by this host.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions