database corruption after eviction

[bplotnick]

Note: I faced db corruption a few times and used AI to diagnose this. It wrote the following text. I am not sure the accuracy - it seems plausible, but it is above my current level of expertise to be sure. I typically I would not be a meat proxy like this, but the issue definitely is real and the corruption appears to not be happening after the suggested fix, so I figured I would post this.

Summary

celld can reuse a preserved local SQLite database under the same ownership
epoch after an on-demand or sticky eviction. The eviction path deletes the
database's local LTX metadata. When the cell wakes, Db::open() therefore sees
an empty LTX history and starts again at transaction ID 1, while replication
continues writing into the existing remote epoch prefix.

The new transaction objects overwrite the beginning of the old epoch, but any
old objects above the new transaction high-water mark remain. A later remote
restore treats the combined objects as one transaction stream. Depending on
the pages changed in each generation, this causes acknowledged data to roll
back or produces a structurally malformed SQLite database.

This violates the fencing invariant documented in docs/fencing.md: one epoch
must describe one append-only writer generation.

Affected version

Observed on celld v0.2.1 (ae8fac0) with the in-process LTX replicator.

Relevant code

• crates/logic/lib.rs:3442-3446: on-demand eviction calls
  begin_eviction(id, false), retaining ownership and the current epoch.
• crates/celld/ltx_repl.rs:601-614: eviction preserves db.sqlite as
  db.evicted, but unconditionally removes db.sqlite-litestream.
• crates/celld/ltx_repl.rs:403-436: activation accepts and reuses a
  same-epoch db.evicted snapshot.
• crates/ltx/src/db.rs:450-457: with no local LTX files, Db::pos() returns
  transaction position zero.
• The next sync consequently writes 0000000000000001-0000000000000001.ltx
  into a prefix that may already contain transactions 1 through N.

Minimal regression test

The following unit test uses the existing in-memory object-store constructor.
It demonstrates deterministic acknowledged-data rollback without requiring a
large enough workload to damage a b-tree. It belongs in
crates/celld/ltx_repl.rs under a celld_internal_tests test module.

#[tokio::test]
async fn same_epoch_eviction_must_not_restart_ltx_history() -> anyhow::Result<()> {
    use object_store::memory::InMemory;
    use rusqlite::Connection;
    use std::sync::Arc;

    let local = tempfile::tempdir()?;
    let store = Arc::new(InMemory::new());
    let repl = LtxRepl::start_with_store_for_test(local.path(), store);
    let cell = "Counter:repro";
    let epoch = 7;

    let active = repl
        .activate(ActivationOptions {
            cell,
            epoch,
            fresh: true,
            took_over: false,
            resume_local: false,
        })
        .await?;

    let db = Connection::open(&active.path)?;
    db.execute_batch(
        "CREATE TABLE counter (value INTEGER NOT NULL);\
         INSERT INTO counter VALUES (0);",
    )?;
    repl.await_durable(cell, epoch, 1).await?;

    // Leave a remote tail above transaction 1.
    for value in 1..=4 {
        db.execute("UPDATE counter SET value = ?1", [value])?;
        repl.await_durable(cell, epoch, value as u64 + 1).await?;
    }
    drop(db);

    // This is the problematic combination: retain the epoch and SQLite file,
    // but discard its LTX metadata.
    repl.evict(cell, epoch, true).await;

    let active = repl
        .activate(ActivationOptions {
            cell,
            epoch, // same epoch
            fresh: false,
            took_over: false,
            resume_local: false,
        })
        .await?;
    let db = Connection::open(&active.path)?;
    db.execute("UPDATE counter SET value = 100", [])?;
    repl.await_durable(cell, epoch, 100).await?;
    drop(db);

    // Remove the healthy local copy so the next activation must consume the
    // now-mixed remote stream.
    repl.evict(cell, epoch, false).await;

    let restored = repl
        .activate(ActivationOptions {
            cell,
            epoch: epoch + 1,
            fresh: false,
            took_over: true,
            resume_local: false,
        })
        .await?;
    let db = Connection::open(&restored.path)?;
    let value: i64 = db.query_row("SELECT value FROM counter", [], |row| row.get(0))?;

    // Expected: 100. Before the fix, the stale tail from the first generation
    // is applied after the replacement transaction 1 and restores 4.
    assert_eq!(value, 100);
    Ok(())
}

To reproduce structural corruption instead of value rollback, replace the
counter updates with transactions that repeatedly insert enough rows to split
b-tree pages, delete them, and create/drop indexed tables. After the final
remote restore, PRAGMA integrity_check reports orphaned pages, missing index
entries, or duplicate page references.

Production evidence

The affected ControlPlane cell had a clean local epoch 69 snapshot. Its remote
epoch 69 prefix contained transaction objects from three different wake
generations:

• transaction 1: last modified 2026-08-18;
• transactions 2 through 11: last modified 2026-08-17;
• transactions 12 and above: last modified 2026-08-15.

The local epoch 69 database passed PRAGMA integrity_check. Epoch 70, created by
a remote restore from the mixed epoch 69 prefix, was the first local snapshot
to fail integrity checking. Subsequent epochs inherited the damage. The failure
included duplicate page references, orphaned pages, and rows missing from all
indexes.

This timestamp stratification cannot be produced by a single append-only LTX
generation. It exactly matches transaction numbering restarting at 1 on each
same-epoch wake and overwriting only the prefix reached by that generation.

Expected behavior

Every new LTX writer generation must use a fresh epoch, or a same-epoch local
resume must retain and validate the complete local LTX position. A remote epoch
must never contain objects from separate transaction-ID-zero lineages.

Actual behavior

A preserved same-epoch SQLite snapshot is reopened without its LTX metadata.
The next write begins at transaction 1 and unconditionally overwrites objects
in the existing epoch prefix, leaving an incompatible tail.

Suggested fix

The simplest safe rule is to advance the ownership epoch for every activation,
including on-demand and sticky eviction wakes. The preserved previous-epoch
SQLite snapshot can still be reused locally when ownership rules prove it safe.

If same-epoch wake must remain supported, eviction must preserve the associated
LTX metadata atomically with the SQLite snapshot. If either part is missing,
activation must acquire a new epoch rather than writing into the old prefix.
Remote restoration into an already-used epoch also needs an explicit baseline
position; it must not restart at transaction 1.

Defense in depth:

1. Before uploading, reject a local position lower than the highest remote L0
   transaction for the same epoch.
2. Validate LTX lineage/checksum continuity during restore.
3. Run PRAGMA quick_check on a private restored image before publishing the
   cell runtime.

These checks should fail activation rather than publish a malformed database,
but they do not replace the epoch/position fix.

Recovery note

Fixing the writer prevents recurrence but does not repair an already mixed
remote prefix. Recovery must start from the last clean local snapshot or export
readable tables into a fresh SQLite image, then publish that image under a new
epoch. The poisoned prefix must not be selected as a restore source again.

[ry]

Thank you for the bug report. We were able to repeat the error and are working on a fix.