Do not fail fault checker solely on materialized ambiguous PUT timeouts

[GatewayJ]

Summary

s3chaos currently treats unknown_writes_materialized as a hard checker failure. In a network-loss run, this can misclassify valid S3 ambiguous write behavior as a product correctness issue.

When a client-side PUT times out during a network fault, the server may still have received the full request body, committed the write, and lost or delayed only the response. A later successful GET returning that exact timed-out value is ambiguous, but it is not by itself evidence of data corruption, data loss, or committed-object unavailability.

Evidence

gh-1 run:

• RustFS image: docker.io/rustfs/rustfs:fault-6b96422b-1a39540-glibc-entryfix
• RustFS commit: 6b96422b1c3d4c9115e01d6808124d45c35a9f32
• s3chaos commit: 1a3954067391ac506d80ab68ac45226459a64df0
• scenario: network-loss
• workload: 4000 objects, concurrency=18, prefill_concurrency=16, total_payload_bytes=20971520000, request_timeout_seconds=30
• artifact dir: /data/rustfs/fault-artifacts/20260712T093651Z-pr4684-6b96422b-entryfix-all-chaos/gh1-all-chaos-4k18-20g-pr4684-rerun/suite-eec607c0-ba3f-4c33-96a0-058f6247e2ac/005-network-loss-r1/fault_network_loss_preserves_object_model

checker-pre-recommit-report.json showed:

{
  "passed": false,
  "committed_puts": 2973,
  "expected_live_objects": 2317,
  "verified_live_objects": 2316,
  "missing_committed_objects": [],
  "unavailable_committed_objects": [],
  "unknown_committed_read_failures": [],
  "hash_mismatches": [],
  "successful_corrupted_reads": [],
  "unknown_writes_materialized": [
    "object-001621: op-007732 Put Timeout ... materialized as the same sha",
    "object-002900: op-006059 Put Timeout ... materialized as the same sha"
  ],
  "unknown_write_value_conflicts": [],
  "final_list_warning_count": 0,
  "list_history_warning_count": 165,
  "tenant_recovered": true
}

The two sampled histories were:

• object-001621: pre-fault PUT succeeded and GET verified the old value. During the fault, an overwrite PUT timed out. After recovery, GET returned the timed-out overwrite value with matching SHA.
• object-002900: during the fault, a new PUT timed out. After recovery, GET returned that timed-out value with matching SHA.

The recovery stability report also had:

{
  "classification": "ambiguous_write_materialized",
  "still_unavailable_keys": [],
  "hash_mismatches": [],
  "data_corruption_evidence": [],
  "final_list_warning_count": 0,
  "harness_errors": []
}

Current behavior

At commit 1a3954067391ac506d80ab68ac45226459a64df0, checker success requires unknown_writes_materialized.is_empty():

• src/fault/checker.rs: report.passed = ... && report.unknown_writes_materialized.is_empty() ...
• classify_without_reread() classifies any non-empty unknown_writes_materialized as ambiguous_write_materialized.
• src/fault/reporting.rs maps ambiguous_write_materialized to needs_investigation, while also setting corruption=false.

This makes a run fail even when there is no missing object, no unavailable committed object, no hash mismatch, no successful corrupt read, no unknown write value conflict, and no final LIST inconsistency.

Expected behavior

A timed-out PUT that later materializes with the same payload should be treated as an ambiguous write observation, not as a hard correctness failure by itself.

Suggested policy:

• Do not fail the checker solely because unknown_writes_materialized is non-empty.
• Keep it visible as a warning/degraded/ambiguous evidence field.
• Continue to fail for hard correctness signals, including:
  • unknown_write_value_conflicts
  • hash_mismatches
  • successful_corrupted_reads
  • missing_committed_objects
  • unavailable_committed_objects
  • final LIST content inconsistencies
  • version/delete-marker correctness violations

Why this matters

The current behavior can produce false product failures under network loss. A client timeout is not a definitive failed write in S3 semantics; it is an unknown result. If the server later exposes exactly that timed-out value, the harness should report ambiguity, but should not classify it as a RustFS data correctness failure without a value conflict or other hard evidence.