Repository navigation
[Bug]: read_only=True close() modifies persisted Flat index bytes in v0.7.0 #815
Description
Description
In zvec 0.7.0, opening a persisted FP32 Flat/COSINE collection with CollectionOption(read_only=True) and then calling close() changes bytes in its vector-index file.
A snapshot taken immediately after open() is unchanged; the snapshot after close() differs. There are no queries, inserts, updates, deletes, or flushes between the read-only open and close. This reproduces with only two 4-dimensional vectors; no scalar indexes, FTS, application adapters, or external data are needed.
Steps to reproduce
Install the published package with python -m pip install zvec==0.7.0, save the following as repro.py, and run python repro.py.
import hashlib
import tempfile
import time
from pathlib import Path
import zvec
def snapshot(path):
return {str(p.relative_to(path)): p.read_bytes()
for p in path.rglob("*") if p.is_file()}
with tempfile.TemporaryDirectory(prefix="zvec-readonly-") as tmp:
path = Path(tmp) / "collection"
schema = zvec.CollectionSchema(
name="readonly_repro",
vectors=zvec.VectorSchema(
"embedding", zvec.DataType.VECTOR_FP32, 4,
index_param=zvec.FlatIndexParam(metric_type=zvec.MetricType.COSINE),
),
)
collection = zvec.create_and_open(str(path), schema)
collection.insert([
zvec.Doc(id="a", vectors={"embedding": [1.0, 0.0, 0.0, 0.0]}),
zvec.Doc(id="b", vectors={"embedding": [0.0, 1.0, 0.0, 0.0]}),
])
collection.flush()
collection.close()
print("zvec:", zvec.__version__)
for attempt in range(1, 4):
# Immediate reopen/close cycles do not always change the file.
time.sleep(1.1)
before = snapshot(path)
collection = zvec.open(str(path), zvec.CollectionOption(read_only=True))
print(f"attempt={attempt} unchanged_after_open={snapshot(path) == before}")
collection.close() # No queries or writes between open and close.
after = snapshot(path)
changes = 0
for name in sorted(before.keys() | after.keys()):
a, b = before.get(name, b""), after.get(name, b"")
if a == b:
continue
changes += 1
offsets = [i for i, (x, y) in enumerate(zip(a, b)) if x != y]
print(f" changed={name} sizes={len(a)}/{len(b)}")
print(f" changed_bytes={len(offsets)} offsets={offsets[:16]}")
print(" sha256_before=" + hashlib.sha256(a).hexdigest())
print(" sha256_after =" + hashlib.sha256(b).hexdigest())
print(f" changed_files_after_close={changes}")Observed result
In a clean environment containing only zvec and its NumPy dependency, all three iterations changed 0/embedding.index.1.proxima after close. The file size remained 5,324,800 bytes; six bytes changed at the same offsets in this run:
zvec: 0.7.0
attempt=1 unchanged_after_open=True
changed=0/embedding.index.1.proxima sizes=5324800/5324800
changed_bytes=6 offsets=[1064832, 1064833, 1064834, 1064835, 1064880, 1081352]
sha256_before=ba49ed3b75510868e0c260c55ddef977a8af844dc745a453332139c30ef8657c
sha256_after =04d8e31399a76b03b6e132683eb04bb5b05c7f124502c38b9d0f2870df6ecbc4
changed_files_after_close=1
The short delay matters: immediate repeated open/close cycles did not always produce a difference. The same behavior also reproduced when each read-only open/close ran in a fresh Python subprocess. File hashes and exact byte counts may vary between runs.
Expected behavior and impact
Closing a read-only collection should leave persisted index contents unchanged. If rewriting internal metadata is intentional even in read-only mode, clarification of that contract would be helpful.
The demonstrated impact is that content-hash/immutability checks detect index changes caused solely by a reader's lifecycle. This report does not demonstrate data loss or incorrect query results: a separate fetch check after three read-only cycles returned both original vectors unchanged.
Environment
- macOS 26.6.2 (25G83), Darwin 25.6.0, Apple Silicon arm64
- Python 3.12.13
- zvec 0.7.0, installed from the published Python package
- NumPy 2.5.4
- Tested index: FP32, Flat, COSINE
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
thlurte commentedon Oct 14, 2026
Hi @feihongxu0824 , I would like to investigate and work on a fix for this issue to ensure
read_only=Trueindices remain completely unmodified acrossclose().If you are not working, could you assign this to me? Thanks!
egolearner commentedon Oct 14, 2026
补充确认：HNSW 也存在只读关闭改写索引文件的问题，影响范围不止 Flat。
在 macOS arm64、
zvec-rust = 0.7.2的真实原生存储测试中可以复现。这里的版本号是 Rust binding 版本，并非本 issue 中 Python 包的版本号。应用创建CollectionOptions后确实调用了set_read_only(true)，应用层关闭只读存储时也没有显式调用flush()。复现过程：
[1.0, 0.0, 0.0]，使用 HNSW / COSINE；完成 flush、optimize 后关闭 writer，确认向量索引已经构建完成。.proxima文件的完整 SHA256，等待 1.1 秒，避免秒级时间戳相同掩盖问题。另一个测试复用常驻只读缓存：连续查询和释放每次查询的 lease 后，哈希仍然相同；关闭缓存、实际释放原生句柄后，哈希才变化。这是存储层生命周期测试，尚未测量完整 server 进程中的触发频率或额外写入的性能影响。
此前在实际语料上逐字节比较，两次重复查询后的 HNSW 文件均保持 13,352,960 字节，每次只有 6 个字节变化，位于以下字段：
HnswChunkMeta.update_timeMetaFooter.update_timeMetaFooter.footer_crc字段共占 20 字节，但短时间内时间戳通常只有低位变化。该次比较中，其余字节未变；这些证据不表明向量或图结构被修改，也不表明发生了重建，但确实存在查询关闭时的持久化写入副作用。
HNSW 关闭路径有两处需要一起检查。以下链接固定到当前 main 的
d7b478d，用于对照源码：Index::close()虽然在只读模式跳过自己的flush()，但仍会调用streamer_->cleanup()。HnswStreamer::close()无条件调用entity_->set_index_meta(meta_)；set_index_meta()会执行IndexHelper::SerializeToStorage()。ChunkBroker::close()无条件调用flush(0UL)，后者更新 chunk 时间戳，再 refresh / flush 底层存储。做过一个隔离诊断实验：只给
ChunkBroker::close()加 dirty 判断，完整哈希测试仍失败；同时跳过无修改时的索引元数据序列化后，直接关闭和缓存关闭两个测试均通过。因此修复时不能只关注最后一次 flush，还需覆盖上层 metadata serialization。这个实验仅验证了问题定位，并非已经完成写入持久性、异常恢复等全套验证的补丁。thlurte commentedon Oct 15, 2026
Thanks a lot @egolearner for the detailed root cause analysis and confirming the behavior on HNSW!
I investigated the implementation across both Flat and HNSW (as well as related graph variants in
hnsw_rabitq,hnsw_sparse, andvamana).Here is what I found and the proposed fix:
FlatStreamer::close()unconditionally calledentity_->flush_linear_meta(). We can add clean/dirty state tracking toFlatStreamerEntity(marked dirty on initial storage creation and vector writes, clean on existing storage load/flush) and guardflush_linear_meta()on close.entity_->set_index_meta(meta_)inHnswStreamer::close()when!entity_->dirty().flush(0UL)inChunkBroker::close()when!dirty_, and ensurebroker_->flush()resetsdirty_ = falseupon flush completion.hnsw_rabitq,hnsw_sparse, andvamana.virtual bool is_dirty() constonIndexStreamersoIndex::is_dirty()queries both the streamer and underlying storage.Index::close(), skip callingflush()whenis_read_only_ || !is_dirty(). (When dirty writes exist on a writable index,flush()is still executed to preserve write persistence).storage_->is_dirty()staysfalse, which naturally preventsMmapFileStorage::close_index()from invokingrefresh_index(0)and mutatingMetaFooter.update_timeandfooter_crc.close()cycles for read-only and un-dirtied collections (with > 1.1s delay across cycles to catch timestamp/CRC changes).iaojnh commentedon Oct 21, 2026
Thanks for the minimal reproducer and the additional HNSW investigation. The fix has been merged into
mainvia [#834](#834).Read-only close now skips unnecessary metadata writes, including timestamp and checksum updates. Regression tests verify byte-for-byte file immutability across repeated open/close cycles and confirm that writable indexes still persist changes correctly.