fix: check LN payment status before resetting on dev fee timeout

docs/DEV_FEE_TIMEOUT_SAFETY.md

@@ -0,0 +1,94 @@
+# Dev Fee Timeout Safety — Duplicate Payment Prevention
+
+## Overview
+
+When a dev fee Lightning payment times out, Mostrod queries the LN node for
+the actual payment status before deciding whether to reset and retry. This
+prevents duplicate payments caused by race conditions between timeouts and
+in-flight payments.
+
+## Problem
+
+Originally, when the 50-second timeout expired during a dev fee payment, the
+code unconditionally reset `dev_fee_paid = false` and cleared the payment hash.
+However, a timeout does not mean the payment failed — the Lightning payment
+could still be in-flight or may have succeeded after the timeout window.
+
+This created a race condition:
+
+1. Payment initiated, times out locally (but still in-flight on LN)
+2. Code resets order to unpaid state
+3. Original payment succeeds on LN (but local state already reset)
+4. Next scheduler cycle finds the order as "unpaid" and initiates a second payment
+5. **Result: double payment**
+
+See [Issue #568](https://github.com/MostroP2P/mostro/issues/568) for full details.
+
+## Solution
+
+### Two-phase payment flow (`src/app/release.rs`, `src/scheduler.rs`)
+
+The dev fee payment is split into two phases:
+
+1. **Resolve phase** (`resolve_dev_fee_invoice`): LNURL resolution + invoice
+   decode to extract the real LN payment hash.
+2. **Send phase** (`send_dev_fee_payment`): Sends the pre-resolved invoice via
+   LND.
+
+The real payment hash is stored in `dev_fee_payment_hash` **before** the payment
+is dispatched. This ensures that on timeout or crash, the hash is always
+available for querying LND.
+
+### `LndConnector::check_payment_status()` (`src/lightning/mod.rs`)
+
+Queries the LN node for the current status of a payment using `TrackPaymentV2`.
+Returns the LND `PaymentStatus` enum (Succeeded, InFlight, Failed, Unknown).
+
+### `check_dev_fee_payment_status()` (`src/scheduler.rs`)
+
+Helper that:
+1. Extracts the payment hash from the order (skips `PENDING-` markers, which
+   are legacy placeholders that cannot be tracked on LND)
+2. Decodes the hex hash to bytes
+3. Queries LND with a 10-second timeout
+4. If payment succeeded: marks order as paid in DB
+5. Returns a `DevFeePaymentState` enum for the caller
+
+With the two-phase flow, new payments always have a real hash stored before
+sending, so step 1 passes through to the LND query. The `PENDING-` guard
+remains only for backward compatibility with legacy markers from before this
+change — those correctly return `DevFeePaymentState::Unknown` since there is
+genuinely no trackable hash.
+
+### Timeout handler in `job_process_dev_fee_payment()` (`src/scheduler.rs`)
+
+Instead of unconditionally resetting on timeout:
+
+| LN Payment Status | Action |
+|-------------------|--------|
+| **Succeeded** | Mark as paid in DB, do NOT reset |
+| **InFlight** | Skip reset, leave state intact (payment may still complete) |
+| **Failed** | Safe to reset `dev_fee_paid = false` and retry |
+| **Unknown** | Skip reset to err on the side of caution (avoid duplicate) |
+
+### Stale real-hash cleanup (`src/scheduler.rs`)
+
+A cleanup pass runs each cycle for orders that have `dev_fee_paid = true` and a
+real (non-PENDING) payment hash. This handles crash recovery: if the process
+crashes between storing the hash and receiving LND confirmation, the cleanup
+queries LND and resets failed payments for retry.
+
+### Design Principle
+
+**When in doubt, don't retry.** A missed dev fee payment can be recovered
+manually, but a duplicate payment is money lost. The code errs on the side of
+caution — only resetting when the LN node confirms the payment definitively
+failed.
+
+## Related
+
+- Issue: [#568](https://github.com/MostroP2P/mostro/issues/568)
+- Dev fee invoice resolution: `src/app/release.rs` (`resolve_dev_fee_invoice`)
+- Dev fee payment send: `src/app/release.rs` (`send_dev_fee_payment`)
+- Dev fee scheduler: `src/scheduler.rs` (`job_process_dev_fee_payment`)
+- Dev fee documentation: `docs/DEV_FEE.md`

src/app/release.rs

@@ -2,12 +2,13 @@ use crate::config;
 use crate::config::constants::DEV_FEE_LIGHTNING_ADDRESS;
 use crate::config::MOSTRO_DB_PASSWORD;
 use crate::db::{self};
+use crate::lightning::invoice::decode_invoice;
 use crate::lightning::LndConnector;
 use crate::lnurl::resolv_ln_address;
 use crate::nip33::{new_order_event, order_to_tags};
 use crate::util::{
-    enqueue_order_msg, get_keys, get_nostr_client, get_order, settle_seller_hold_invoice,
-    update_order_event,
+    bytes_to_string, enqueue_order_msg, get_keys, get_nostr_client, get_order,
+    settle_seller_hold_invoice, update_order_event,
 };
 
 use argon2::password_hash::SaltString;
@@ -553,31 +554,30 @@ async fn payment_success(
     Ok(())
 }
 
-/// Send development fee payment via Lightning Network
+/// Resolve the dev fee invoice and extract the real payment hash.
 ///
-/// Attempts to pay the configured development fee for a completed order.
-/// Uses LNURL resolution to get payment invoice, then sends payment via LND.
+/// Performs LNURL resolution to get a BOLT11 invoice, then decodes it
+/// to extract the payment hash. This allows storing the real hash in the
+/// database *before* sending the payment, enabling LN status checks on
+/// timeout or crash recovery.
 ///
 /// # Timeouts
 /// - LNURL resolution: 15 seconds
-/// - send_payment call: 5 seconds
-/// - Payment result wait: 25 seconds
-/// - Total: 45 seconds maximum
 ///
 /// # Returns
-/// - `Ok(String)`: Payment hash on successful payment
-/// - `Err(MostroError)`: Error if payment fails or times out
-pub async fn send_dev_fee_payment(order: &Order) -> Result<String, MostroError> {
+/// - `Ok((payment_request, payment_hash_hex))` on success
+/// - `Err(MostroError)` if resolution or decoding fails
+pub async fn resolve_dev_fee_invoice(order: &Order) -> Result<(String, String), MostroError> {
     info!(
-        "Initiating dev fee payment for order {} - amount: {} sats to {}",
+        "Resolving dev fee invoice for order {} - amount: {} sats to {}",
         order.id, order.dev_fee, DEV_FEE_LIGHTNING_ADDRESS
     );
 
     if order.dev_fee <= 0 {
         return Err(MostroInternalErr(ServiceError::WrongAmountError));
     }
 
-    // Step 1: LNURL resolution (15s timeout)
+    // LNURL resolution (15s timeout)
     let payment_request = tokio::time::timeout(
         std::time::Duration::from_secs(15),
         resolv_ln_address(DEV_FEE_LIGHTNING_ADDRESS, order.dev_fee as u64),
@@ -598,14 +598,53 @@ pub async fn send_dev_fee_payment(order: &Order) -> Result<String, MostroError>
         e
     })?;
 
-    // Step 2: Create LND connector
+    // Decode invoice and extract payment hash
+    let invoice = decode_invoice(&payment_request)?;
+    let payment_hash_hex = bytes_to_string(invoice.payment_hash().as_ref());
+
+    info!(
+        "Resolved dev fee invoice for order {} - hash: {}",
+        order.id, payment_hash_hex
+    );
+
+    Ok((payment_request, payment_hash_hex))
+}
+
+/// Send development fee payment via Lightning Network
+///
+/// Sends a pre-resolved invoice payment via LND. The caller must first
+/// call `resolve_dev_fee_invoice` to obtain the payment request and store
+/// the payment hash in the database.
+///
+/// # Timeouts
+/// - send_payment call: 5 seconds
+/// - Payment result wait: 25 seconds
+/// - Total: 30 seconds maximum
+///
+/// # Returns
+/// - `Ok(String)`: Payment hash on successful payment
+/// - `Err(MostroError)`: Error if payment fails or times out
+pub async fn send_dev_fee_payment(
+    order: &Order,
+    payment_request: &str,
+) -> Result<String, MostroError> {
+    info!(
+        "Sending dev fee payment for order {} - amount: {} sats",
+        order.id, order.dev_fee
+    );
+
+    if order.dev_fee <= 0 {
+        return Err(MostroInternalErr(ServiceError::WrongAmountError));
+    }
+
+    // Step 1: Create LND connector
     let mut ln_client = LndConnector::new().await?;
     let (tx, mut rx) = channel(100);
 
-    // Step 3: Send payment (5s timeout to prevent hanging)
+    // Step 2: Send payment (5s timeout to prevent hanging)
     tokio::time::timeout(
         std::time::Duration::from_secs(5),
-        ln_client.send_payment(&payment_request, order.dev_fee, tx),
+        ln_client.send_payment(payment_request, order.dev_fee, tx),
     )
     .await
     .map_err(|_| {
@@ -625,7 +664,7 @@ pub async fn send_dev_fee_payment(order: &Order) -> Result<String, MostroError>
         e
     })?;
 
-    // Step 4: Wait for payment result (25s timeout)
+    // Step 3: Wait for payment result (25s timeout)
     // Loop to receive multiple status messages from LND until terminal status
     let payment_result = tokio::time::timeout(std::time::Duration::from_secs(25), async {
         while let Some(msg) = rx.recv().await {
@@ -671,7 +710,7 @@ pub async fn send_dev_fee_payment(order: &Order) -> Result<String, MostroError>
         MostroInternalErr(ServiceError::LnPaymentError("result timeout".to_string()))
     })??; // Double ? to unwrap both timeout Result and inner Result
 
-    // Step 5: Log and return the successful payment hash
+    // Step 4: Log and return the successful payment hash
     info!(
         "Dev fee payment succeeded for order {} - amount: {} sats, hash: {}",
         order.id, order.dev_fee, payment_result

src/db.rs

@@ -903,7 +903,6 @@ pub async fn find_unpaid_dev_fees(pool: &SqlitePool) -> Result<Vec<Order>, Mostr
           WHERE (status = 'settled-hold-invoice' OR status = 'success')
             AND dev_fee > 0
             AND dev_fee_paid = 0
-            AND (dev_fee_payment_hash IS NULL OR dev_fee_payment_hash NOT LIKE 'PENDING-%')
         "#,
     )
     .fetch_all(pool)

src/lightning/mod.rs

@@ -262,6 +262,43 @@ impl LndConnector {
         Ok(())
     }
 
+    /// Query the current status of a payment by its hash.
+    ///
+    /// Returns the LND `PaymentStatus` if the payment is found, or an error
+    /// if the payment cannot be tracked (e.g., unknown hash).
+    pub async fn check_payment_status(
+        &mut self,
+        payment_hash: &[u8],
+    ) -> Result<fedimint_tonic_lnd::lnrpc::payment::PaymentStatus, MostroError> {
+        let track_req = TrackPaymentRequest {
+            payment_hash: payment_hash.to_vec(),
+            no_inflight_updates: false,
+        };
+
+        let mut stream = self
+            .client
+            .router()
+            .track_payment_v2(track_req)
+            .await
+            .map_err(|e| MostroInternalErr(ServiceError::LnPaymentError(e.to_string())))?
+            .into_inner();
+
+        // Get the first (current) status update
+        if let Ok(Some(payment)) = stream.message().await {
+            fedimint_tonic_lnd::lnrpc::payment::PaymentStatus::try_from(payment.status).map_err(
+                |_| {
+                    MostroInternalErr(ServiceError::LnPaymentError(
+                        "Unknown payment status".to_string(),
+                    ))
+                },
+            )
+        } else {
+            Err(MostroInternalErr(ServiceError::LnPaymentError(
+                "No payment status received".to_string(),
+            )))
+        }
+    }
+
     pub async fn get_node_info(&mut self) -> Result<GetInfoResponse, MostroError> {
         let info = self.client.lightning().get_info(GetInfoRequest {}).await;