Usage of "non-deterministic" floating point operations is misleading

[JLErvin]

Location (URL)

https://doc.rust-lang.org/std/primitive.f32.html#unspecified-precision-1

Summary

There are a number of floating point operations which are documented with the following language (e.g. f32::exp):

The precision of this function is non-deterministic. This means it varies by platform, Rust version, and can even differ within the same execution from one invocation to the next.

The language of "non-determinism" was introduced in this pull request: #124609. I would argue, however, that the usage of "non-determinism", paired with the language "... can even differ within the same execution from one invocation to the next" does not fully reflect the nuance that this comment is trying to document.

Backtracking through some PRs and issues, it seems that the root cause of this work is from #109118 and #71355. These highlight that the same core floating point operation can produce different results depending on the platform, release/debug mode, and whether or not the operation is performed at runtime or compilation time.

However, there is an important nuance, which I expect to be true, that the same compiled binary shall always produce the same outputs bits when given identical inputs bits for the same floating point operation. If this claim is not true, I would be extremely interested in examples where this is the case.

My understanding is that this claim is true, however, in which case I think the documentation should be updated to reflect it. I'm no wordsmith myself, but something like: "The precision of this function can vary depending on the platform, Rust version, compilation mode, and whether or not the value is generated at compile-time or runtime. For a given binary running on identical hardware, however, bit-for-bit identical inputs shall always produce identical bit-for-bit outputs."

For some context, a project I am working on runs a number of single-threaded, redundant computing operations on distributed, identical hardware, and then exchanges information to come to a consensus on the output. I am not aware of any instances where identical inputs to the internal floating-point calculations would produce different outputs. The documentation seems to suggest that it could.

I'm happy to open any PRs to make these changes, but I'd like some feedback first on the nuances before doing so.

cc @RalfJung who wrote the current documentation

[theemathas]

For an example of a similar non-determinism within the same execution, see #134417

[theemathas]

For an example of similar nondeterminism in C++, see https://isocpp.org/wiki/faq/newbie#floating-point-arith2

[JLErvin]

@theemathas those examples are fascinating (and terrifying). Although I think they're slightly orthogonal to the issue I'm trying to get clarity on.

Suppose I have an extremely trivial Rust program, written as so:

fn main() {
    let x = 3.0f32;
    let y = 5.0f32;

    println!("x * y = {}", (x * y));
}

Let's say I compile this in release mode and run it numerous times on identical hardware. Would I expect those runs to produce different bit-for-bit outputs?

It seems to me that that answer is no. It seems that most of the discussions linked are related to slightly different invocations of "equivalent" values which result in non-equality (e.g. whether or not the floating-point values are parsed in #134417)). But there is a slightly different discussion to be had of if the exact same lines of code will actually produce different output when executed multiple times.

[Fulgen301]

For an example of similar nondeterminism in C++, see https://isocpp.org/wiki/faq/newbie#floating-point-arith2

I'd be careful with that link since it's a very x87-centric view of floating point:

floating point calculations and comparisons are often performed by special hardware that often contain special registers, and those registers often have more bits than a double.

Not that floating-point math doesn't have dragons, but it has much improved since then, to the point that we now look down on x87 and pretty much have it as an open soundness issue.

[RalfJung]

that the same compiled binary shall always produce the same outputs bits when given identical inputs bits for the same floating point operation

This is not true (if I interpreted all your words correctly). The results can depend on whether the optimizer got knowledge of some of the inputs of the operation and simplified it accordingly. So for instance, (2.0f32).exp(2.5f32) and std::hint::black_box(2.0f32).exp(std::hint::black_box(2.5f32)) both executed in the same program can return different results if the host that compiled the program uses a different exp implementation than the target that runs it.

So to be concrete, what I mean is that this assertion can fail:

assert_eq!(
  (2.0f32).exp(2.5f32),
  std::hint::black_box(2.0f32).exp(std::hint::black_box(2.5f32))
);

I don't know enough about how exactly LLVM optimizes functions like exp to give a concrete example, but to my knowledge LLVM is totally in its right to constant-fold such operations without knowing exactly which libm will be used to execute the program.

So from what I can tell, the language we use in the docs exactly captures the nuance it is trying to convey.

Cc @tgross35

Suppose I have an extremely trivial Rust program, written as so:

This example uses *, which we indeed specify as fully deterministic and behaving the same across all targets (except when the result is NaN). There are targets where different answers can occur; we treat that as a bug (#114479).