[Bug]: File descriptor exhaustion during segment rollover duplicates documents and leaves the collection unwritable #811

Description

When creating the next writing segment fails, the old segment can remain both in the segment manager and in writing_segment_. stats() and query() then count its documents twice. Later inserts fail, and the broken state can survive a close and reopen. Some documents whose insert returned an error become visible through a forward scan after reopening.

switch_to_new_segment_for_writing() dumps the old segment and adds it to the segment manager before Segment::CreateAndOpen() creates its replacement. A replacement-creation failure returns without undoing either step. get_all_segments_unsafe() lists both owners.

Steps to Reproduce

Reproduced on unmodified 98aa126, macOS arm64, through the C++ API with default collection options:

1. Create a collection with an inverted-indexed string field cls and a plain integer field n. Insert 1,000 documents with cls = 'a' and n = 1, then close it.
2. Open it and lower the process's soft RLIMIT_NOFILE to the number of open descriptors plus a small allowance. The verifier swept allowances from 4 to 30 in fresh processes.
3. Call optimize(). Opening the new segment's scalar index fails with EMFILE ("Too many open files"). Restore the descriptor limit.
4. Compare stats().doc_count with the row count and distinct primary keys from query(cls = 'a'). Try another insert and another optimize(), then close and reopen.
5. Query both cls = 'a' and the plain field n. Try more inserts and query their plain-field values.

Logs / Stack Trace

At allowance 4 on 98aa126:

optimize: Failed to open scalar indexer
stats().doc_count: 2000
query(cls = 'a'): 2000 rows, 1000 distinct primary keys
insert(x0): failed, empty error message
second optimize: Segment has been dumped.
close: OK
reopen: doc_count=1001, query(cls = 'a') returns 1000 rows
query(n = 7): returns x0, whose insert failed
insert(y0), insert(y1), insert(y2): failed, empty error messages
query(n = 101): returns y1, whose insert failed

Expected: a failed replacement creation leaves each acknowledged document visible once, does not publish a rejected insert, and permits writes and a rollover retry after resources are available. The same consistency must hold after reopening.

Operating System

macOS 27.0.1, arm64. Linux and Windows were not run.

Build & Runtime Environment

C++ API, Apple clang, CMake Release. The independent reproduction used unmodified 98aa126. The fix tests used the cached 98aa126 source with all six #790 files and the rollover addition checked byte-for-byte against the PR branch. Current main was not built because its unrelated RaBitQ 0.3.8 dependency was unavailable locally.

Additional Context

The fix creates the replacement before dumping the old segment, and adds the old segment to the segment manager only when the replacement is ready. It retires failed replacement creations through the existing segment lifecycle, leaves pre-existing conflicting directories untouched, and adds no on-disk format, public API or persistent structure.

A test-only RLIMIT_NOFILE regression covers rollover from a full-segment insert, optimize(), and create_index(). It checks two successive failed creations, cleanup of their unused directories, indexed queries, an unconditional forward scan after index creation, unique primary keys, retry, writes, and reopening. A seventh case checks the results of a batch whose rollover fails after a successful prefix. The updated regression fails all seven cases on 8944774 and passes all seven with the finished fix. The final targeted run passed 293 cases, including the seven regressions; one exhaustive manifest-truncation case timed out at 60 seconds, as it also did on unchanged 8944774. No timeout is counted as a pass. Linux and Windows were not run.

The fix is part of #790.