2012-08-20 benchmark loss: adversarial GPS outliers at blackout boundary

[manankharwar]

Summary

FusionCore ATE RMSE 3D: 98.3 m. robot_localization EKF: 10.6 m. This is the larger of the two FC losses and the most instructive failure mode.

Root cause

The raw GPS stream for this sequence contains 105 mode-3 GPS fixes that are 720-840 m off the RTK ground truth. These appear in gps.csv as valid mode-3 fixes but are excluded from gps_rtk.csv (the ground-truth preprocessor rejects them). They cluster in a 24-second window at the end of the second GPS blackout (t=66 min).

This is an adversarial pattern for any chi2-based gating scheme.

Blackout structure

The sequence has two GPS blackouts:

• Blackout 1: 228 seconds at t=42.2 min
• Blackout 2: 211 seconds at t=62.5 min

During blackouts, FusionCore's coast mode relaxes the chi2 gate (coast_q_factor) to allow re-acquisition when GPS returns. This relaxation is necessary: after 3-4 minutes of dead-reckoning, the predicted GPS covariance is large and the Mahalanobis distance of a valid returning fix can exceed the normal gate threshold.

The 105 corrupt fixes arrive at exactly the moment the relaxed gate is open for re-acquisition after Blackout 2. The gate accepts them. The filter jumps 720+ meters. It then recovers over the next 2 minutes as subsequent valid fixes pull the estimate back.

Per-minute trajectory analysis

Time window	FC error	Status
0 - 42 min	1 - 10 m	Normal GPS coverage
43 - 46 min	spike to ~100 m, recovers in 2-3 min	Blackout 1 (228s): boundary errors up to ~70 m
47 - 62 min	3 - 10 m	Full recovery
63 - 67 min	spike to ~788 m, recovers in 2 min	Blackout 2 (211s): 105 fixes 720-840 m off RTK
68 - 82 min	5 - 10 m	Full recovery for remaining 15 minutes

The 98 m ATE RMSE is driven almost entirely by those two transients. Outside those windows FC achieves 5-10 m, on par with RL-EKF.

Why RL wins here

RL-EKF also sees the same corrupt GPS stream. It wins for an unrelated structural reason: RL-EKF's GPS covariance handling (trusting the raw position_covariance from the driver) keeps the Mahalanobis gate tighter during normal operation. On this particular sequence, the corrupt fixes do not land inside RL-EKF's narrower gate at that moment.

This is not evidence that RL-EKF is better at outlier rejection in general. On three other sequences (2012-02-04, 2012-03-31, 2012-09-28), RL-EKF's tight gate causes it to reject valid GPS fixes and lose GPS for long stretches, producing 55-265 m ATE. FusionCore wins those sequences precisely because its gate is calibrated to real sensor noise.

The fundamental tradeoff

coast_q_factor controls how much the chi2 gate relaxes during blackout recovery. It is a direct tradeoff:

• Lower coast_q_factor: tighter gate during recovery, more likely to reject corrupt fixes at the boundary, but also more likely to reject valid returning fixes after long blackouts.
• Higher coast_q_factor: looser gate, accepts valid fixes reliably after long blackouts, but also accepts adversarial clusters like this one.

The 24-second cluster at exactly the blackout boundary is a worst-case input for this tradeoff. Any gate calibrated to accept valid fixes after a 211-second blackout will accept 105 corrupt fixes 720 m off if they arrive at the same moment.

What would fix this

Two directions under investigation:

1. Cluster-based gating. A single fix 720 m off is suspicious. 105 fixes in 24 seconds all clustered 720-840 m off the dead-reckoned position is a distinguishable pattern. A secondary consistency check (e.g., reject if last N fixes all land more than K meters from the predicted position) could catch this without affecting single-fix outlier behavior.

2. Velocity-consistency check. These 105 fixes imply the robot teleported 720 m in one GPS epoch (200 ms). The implied velocity is ~3600 m/s. A GPS-reported velocity check (where available) or a position-delta rate check would flag this before the position update.

Neither is implemented yet. Contributions welcome.

Reproduce

bash benchmarks/run_one.sh 2012-08-20

Results write to benchmarks/nclt/2012-08-20/results_full/.

[Whatsonyourmind]

Your framing of the tradeoff is exactly right, and I think it also points to the way out of it. The reason coast_q_factor is stuck in a no-win tradeoff is that every gate in the discussion — chi2, cluster-distance, position-delta — is measured against the dead-reckoned prediction, and that prediction's covariance is precisely the thing that has blown up during the blackout. So any threshold on "distance from prediction" is forced to choose between rejecting valid re-acquisition and accepting the corrupt cluster. You can't escape that with a better threshold on the same quantity.

The escape is that the IMU/wheel-odometry is an independent witness whose relative trajectory stays bounded even while absolute position drifts unboundedly. Three things follow, roughly in increasing order of robustness:

1. Replace coast_q_factor with a physically-grounded gate radius. The honest bound on how far the true position could have moved from the dead-reckoned estimate after a blackout is the integral of the odometry/IMU error model over the blackout duration — not a tuned multiplier. 720 m of accumulated error over 211 s implies a sustained dead-reckoning drift of ~3.4 m/s, which a ground robot with wheel odometry does not produce (your actual drift rate over the 228 s Blackout 1 is the empirical bound — the 43–46 min boundary error only reached ~70 m). So the corrupt cluster sits outside any physically plausible drift bound even though it's inside the relaxed chi2 gate. This dissolves the tradeoff: the gate radius grows with blackout length automatically, tracks real odometry drift, and 720 m never falls inside it.

2. Robust consensus fit against the inertial shape, not the prediction. The corrupt fixes are mutually consistent (they agree with each other 720–840 m off), so consensus among GPS fixes alone can't reject them — during that 24 s window the corrupt fixes may be the only fixes present. But de-trend the candidate cluster by the inertial relative trajectory and a valid re-acquisition resolves to a single constant offset (= your accumulated dead-reckoning error, bounded by #1); the corrupt cluster does not (its implied offset exceeds the drift bound, or its residual against the inertial shape is large). Concretely: fit gps_candidate ≈ inertial_relative_trajectory + constant_offset over the window (a 2–3 parameter robust/RANSAC fit) and accept only if the residual is small and the fitted offset is within the #1 bound.

3. Make re-acquisition probationary — never collapse covariance on the first accepted fix. The 720 m jump happens because the filter hard-commits on a single epoch. Buffer candidate fixes through a short probationary window, run the fit in #2, and only inject the re-acquisition (shrink covariance) once you have K inliers consistent with the inertial shape. This costs a few seconds of delayed re-acquisition but removes the single-epoch teleport entirely.

One note on the velocity-consistency direction (your #2): it catches the first corrupt fix (720 m / 200 ms → ~3600 m/s), but the subsequent 104 are only ~120 m apart from each other across 24 s (~5 m/s between consecutive fixes), so a delta-rate check chained through accepted fixes passes them once the filter has jumped. The velocity check has to be against the inertial prediction each epoch, not against the previous GPS fix — and where GPS Doppler is available, Doppler-velocity vs INS-velocity is a strong independent check because a corrupt position fix often still carries a plausible (or independently corrupt) Doppler.

Net: the unifying idea is to gate re-acquisition on consistency with the inertial witness and its bounded drift, not on distance from a prediction whose covariance the blackout has destroyed. That's a single mechanism that resolves the tradeoff instead of trading one failure for another.

Happy to prototype the #1 drift-bound gate against benchmarks/run_one.sh 2012-08-20 if that'd be useful — it's the smallest change (a derived gate radius replacing the coast_q_factor knob) and should be measurable on that sequence in isolation. This is squarely the robust-estimation / independent-witness side of things I work on.