Validity: an integral float in a validity timestamp is silently coerced to microseconds — the write path stamps rows at 1970 (silent data corruption)

[shuruheel]

Disclosure up front. We maintain mnestic, a hard fork
of CozoDB (fork point 481af05, 2024-12-04). We hit this bug in our own tree, traced it, and
fixed it in mnestic 0.12.2. Three of the four affected sites — and the worst one, the write path —
are verbatim upstream code, so the bug is upstream's too, and it affects every CozoDB database
with a Validity column. We are filing it here because Cozo users have a silent corruption they
have no way of knowing about. Everything below was reproduced against a clean checkout of
upstream 481af05 (cozo 0.7.6, mem engine), not against our fork.

Summary

A Validity timestamp is an integer number of microseconds since the epoch. now() and
parse_timestamp() return a float number of seconds. Num::get_int accepts any
whole-numbered float (if f.round() == *f { Some(*f as i64) }), and every validity site funnels
through it — so a float seconds value is silently reinterpreted as microseconds, a factor of 1e6
too small, and lands in 1970.

On the write path this is data corruption, not an ergonomics wart. A :put of
[parse_timestamp('2024-06-01T00:00:00Z'), true] into a Validity column succeeds, and
permanently stamps the row at 1970-01-01T00:28:37Z. Ordinary queries read the row back and look
fine; the damage shows only under time travel, which is exactly where a bitemporal database is
supposed to be trustworthy. The fact leaks backwards: an as-of query for a date years before
the fact was true returns it, and the correctly-written row that should answer that query does
not appear.

On the read path the same coercion makes @ parse_timestamp(...) and @ now() return zero
rows and no error — the selector lands before any row was asserted, which is indistinguishable
from "no data yet".

The coercion itself is perfectly reasonable in isolation — accepting 3.0 where an integer is
wanted is a kindness almost everywhere else in the language. It becomes a corruption only because
validity is denominated in a different unit than the datetime builtins that naturally feed it.
Nothing in the type system or the error messages marks that boundary, which is why it has gone
unnoticed for years — including, as below, in Cozo's own test suite.

Minimal reproduction

Paste into a fresh REPL (mem engine is enough). Each block is one script.

:create emp {name: String, at: Validity => title: String}

Write one row the way a user naturally would — from a timestamp — and one row correctly, in
microseconds:

?[name, at, title] := name = 'ada',
                      at = [parse_timestamp('2024-06-01T00:00:00Z'), true],
                      title = 'CTO'
:put emp {name, at => title}

?[name, at, title] := name = 'grace',
                      at = [1717200000000000, true],
                      title = 'CTO'
:put emp {name, at => title}

Both succeed. Now look at what is actually on disk:

?[name, ts] := *emp{name, at}, ts = to_int(at)

ada    1717200000          <-- microseconds. 1717 seconds after the epoch: 1970-01-01T00:28:37Z
grace  1717200000000000    <-- 2024-06-01, as intended

ada's row is stamped in 1970. The corruption is now permanent — the value on disk is a plain
integer, so no amount of later validation can tell it from a deliberate 1970 stamp.

The fact leaks backwards in time. Ask who was CTO in 2020 — four years before either fact was
true:

?[name, title] := *emp{name, title, @ '2020-01-01T00:00:00Z'}

ada  CTO      <-- wrong: not true until 2024
                 (and `grace`, the correct row, is correctly absent)

Every as-of query between 1970 and 2024 is silently wrong in both directions: it returns a fact
that was not yet true, and it will not return the correct row.

The read path fails just as quietly. Against correctly-stored data (grace), a float
selector matches nothing:

?[name, title] := *emp{name, title, @ parse_timestamp('2024-07-01T00:00:00Z')}

(grace absent — the selector coerced to 1970. Zero rows, no error.)

The same happens via the validity(...) constructor
(at = validity(parse_timestamp('2024-06-01T00:00:00Z'), true) → identical 1970 stamp), and an
out-of-range float selector (@ 1e300) is accepted too, saturating to i64::MAX.

The four sites (paths at 481af05)

	file:line	what it does
funnel	cozo-core/src/data/value.rs:511	Num::get_int — Num::Float(f) => if f.round() == *f { Some(*f as i64) }. Every site below reaches the corruption through this one accessor.
write	cozo-core/src/data/relation.rs:371	the DataValue::List(l) arm of ColType::Validity in coerce: let o_ts = l[0].get_int(); — this is the one that corrupts data.
constructor	cozo-core/src/data/functions.rs:2572	op_validity: args[0].get_int().ok_or_else(|| miette!("'validity' expects an integer"))? — the message already promises an integer; the code does not hold it to that.
read	cozo-core/src/parse/query.rs:1086	expr2vld_spec: let microseconds = n.get_int()... — the @ selector.

And the two functions that feed them, both returning float seconds:
op_now (data/functions.rs:2449, as_secs_f64()) and op_parse_timestamp
(data/functions.rs:2509, as_secs_f64()).

Cozo's own test suite has been writing 1970

The single most persuasive piece of evidence, and we mean it without any snark — it is precisely
how easy this bug is to write:

cozo-core/src/runtime/tests.rs:1544, in hnsw_index:

last_accessed_at: Validity default [floor(now()), true],

floor(now()) is a whole-numbered float of seconds, which is exactly the input the coercion
accepts. Reproduced on 481af05, inserting a row today with that default:

?[belief, ts] := *beliefs{belief, last_accessed_at}, ts = to_int(last_accessed_at)

b1  1783955856     <-- 1970-01-01T00:29:44Z. A row written in 2026.

?[belief] := *beliefs{belief, @ '1970-02-01T00:00:00Z'}

b1               <-- a row written today is visible as-of February 1970

The test never asserted on the value, so nothing ever went red. That idiom is the natural way to
write "stamp this row now", it is in the test suite, and anyone who copied it from there has a
schema quietly writing 1970.

(Amusingly, bare [now(), true] — without floor — already errors, because now() returns a
fractional float and the coercion only ever accepted whole-numbered ones. The failure mode is
worse for the more careful-looking spelling.)

Why it is silent in both directions

• Write: succeeds. The value is a plausible i64. It reads back on ordinary queries, because
  an ordinary query resolves at "now" and a 1970 assertion is still in force. Only a time-travel
  query is positioned to notice, and when it does, it does not report an error — it reports a
  different, wrong, answer.
• Read: returns zero rows. The misread stamp always lands before any real row was asserted,
  so the empty result is indistinguishable from "nothing was true yet at that time" — a perfectly
  legitimate answer that a user has no reason to question.

Neither direction produces a diagnostic, and the two failure modes hide each other: a corrupt
database queried with a corrupt selector (both 1970) will happily return rows.

What we did in mnestic 0.12.2, and what we deliberately did not do

What we did. We added a strict accessor and used it at all four sites:

// cozo-core/src/data/value.rs (mnestic)
/// Integer with **no float coercion**: `Some` only for [`Num::Int`].
///
/// Use this, never [`Num::get_int`], wherever the integer carries a *unit*.
pub(crate) fn get_int_strict(&self) -> Option<i64> {
    match self {
        Num::Int(i) => Some(*i),
        Num::Float(_) => None,
    }
}

Num::get_int itself is untouched — the coercion is right everywhere it does not carry a unit.
Each of the four sites now rejects a float with a message that says what to write instead
(to_int(<expr> * 1000000) for seconds; and a note that round() returns a float and will not
convert it). The schema still compiles; it is the next write that fails, which is the point.
Public Rust API is byte-identical.

What we deliberately did not do, and why. The obvious next step is a magnitude check — "a
validity stamp below ~1e12 is obviously wrong, reject it". We considered it and rejected it, and
we think upstream should too.

Valid time in Cozo is an abstract, user-settable logical clock, not a wall clock. Cozo's own
tutorial queries @ 2019. The test suite uses @ 250. Both are correct and legitimate uses of
the feature. On such a relation a magnitude gate would reject valid data, and — worse — a
repair keyed on magnitude would multiply 250 into 250000000 and destroy it. Nothing in the
engine can tell an abstract-clock relation from a microsecond-clock one, because that distinction
lives entirely in the user's head. So:

• An integer in seconds (@ 1704067200) is still accepted, and still silently returns
  nothing. We did not fix that and do not believe it is fixable at this layer.
• Rejecting the float is fixable, and is unambiguously right, because a float is never a
  legitimate validity stamp under any clock convention: the type is i64.

The real answer for wall-clock users is a typed path — a Validity-valued conversion from a
datetime, so the unit is carried by the type rather than by the user's memory. That is what we are
building. If upstream wants a design sketch we are happy to write one up.

Repair recipe for affected users

Rows already on disk are plain integers. A fixed engine cannot see them, so a fix does not repair
them; existing databases stay wrong until someone repairs them by hand. This works on stock
upstream (verified on 481af05), and needs no fork.

Read this before you run anything. The < 1e12 threshold below is a heuristic you must
authorize, per relation — for exactly the reason a magnitude check does not belong in the
engine. If a relation's validity axis is an abstract clock (the tutorial's @ 2019, the test
suite's @ 250) rather than wall-clock microseconds, this recipe will "repair" 250 into
250000000 and destroy your data. Only you know which kind of clock each relation uses.

Take a backup (::backup). Run the detection query alone. Look at the rows. Then repair
one relation at a time.

1. Detect. (emp and the threshold are yours to choose; a wall-clock microsecond stamp for
any date after 2001 exceeds 1e15, so anything under 1e12 cannot be a real one.)

?[name, ts] := *emp{name, at}, ts = to_int(at), ts < 1000000000000

2. Repair — put the corrected versions first, so the fact is never absent from the relation:

bad[name, ts, ok, title] := *emp{name, at, title},
                            ts = to_int(at), ok = to_bool(at), ts < 1000000000000
?[name, at, title] := bad[name, ts, ok, title], at = [ts * 1000000, ok]
:put emp {name, at => title}

3. Then remove the corrupt versions:

?[name, at] := *emp{name, at}, to_int(at) < 1000000000000
:rm emp {name, at}

Verified end-to-end on 481af05, including the two cases that break a naive repair:

• A bare binding *emp{name, at} sees every validity version, so the scan is complete.
• It preserves is_assert: a corrupt retraction is moved to the right instant and stays a
  retraction.
• It handles a key with both a corrupt and a correct version (a corrupt assert followed by a
  correct retract).

Before repair, an as-of query for 2020-06 returned the successor — whose appointment did not
begin for another four years — alongside the incumbent who really did hold the job then. After
repair, as-of 2019 → nobody; 2021 → the one person then asserted; 2024-07 → the successor;
2025 → nobody (both retracted). All correct.

Environment

Reproduced on a clean checkout of 481af05 (cozo 0.7.6), mem engine, macOS, cargo test
against cozo-core. Nothing in the reproduction depends on a storage engine, on our fork, or on
any feature flag beyond the default.