APay outbox retries send_outbound_payment forever on PaymentHashAlreadyUsed

[bandrivskiy]

Observed

2026/08/03 09:29:44 cron async_order_outbox job 14 send_outbound_payment/f38f9778e065ba445a5586cec0279ad5f40deff2359b6efa25d118cbfe962094 failed: RGB Lightning API error 400: {"error":"Payment hash already used","code":400,"name":"PaymentHashAlreadyUsed"} (HTTP 400)

This repeats every cron tick (CRON_EVERY, default 30s — config.go:85) indefinitely.

Matching line on the LSP node:

ERROR request{uri=/sendpayment request_id=16fef20a-...}: rgb_lightning_node::error: APIError: Payment hash already used

Why the node returns 400

APay is a same-hash relay: the outbound invoice is requested with the inbound payment hash (api.go:396). At send time, the LSP’s own node already has that hash in inbound_payments as the sender’s HODL in Claimable, so RLN’s duplicate-payment guard rejects the send (rgb-lightning-node/src/routes.rs:4882):

if rgb_payment.is_none() {
    if let Some(existing) = unlocked_state.inbound_payments().get(&payment_hash) {
        if matches!(
            existing.status,
            HTLCStatus::Pending | HTLCStatus::Claimable | HTLCStatus::Claiming
        ) {
            return Err(APIError::PaymentHashAlreadyUsed);
        }
    }
}

The guard is inside rgb_payment.is_none(), so only sat-only APay orders hit it. RGB-asset invoices skip it.

Whether that guard should exempt the APay relay case is a separate question for the rgb-lightning-node repo. This issue is about the LSP retrying a response that cannot change.

Why it never stops

• The inbound HODL stays Claimable by design until the LSP claims it at outbound_claimed, which happens after the very send being retried. So the guard cannot clear while the flow is progressing normally.
• MarkAsyncRotatingInvoiceOutboxRetry (async_order.go:249) only sets available_at = +15 seconds.
• attempts is incremented in ClaimAsyncRotatingInvoiceOutboxJob (async_order.go:154), but it is never read. There is no cap and no dead-letter status.
• aPaySendOutboundPaymentJob keeps running while the invoice is outbound_pending (api.go:459), and nothing moves it out of that state on send failure.
• MarkAsyncRotatingInvoiceFailed is only called from the earlier request_outbound_invoice job (api.go:341), not from the send job.
• The outbox row is stored in SQLite, so the retry loop survives restarts.

Net effect: one stuck order logs forever until someone edits the DB manually.

Error classification is inverted

• Routing failures are not retried. RLN returns HTTP 200 with status: Failed on routing errors. sendLNByInvoice maps that to errPaymentReportedFailed (api.go:1462), and api.go:470 swallows it, then marks the invoice as outbound_paid. So a payment that never left can be recorded as delivered.
• Permanent 4xx errors are retried forever. Errors like PaymentHashAlreadyUsed, InvalidInvoice, or InvalidAmount arrive as transport-level errors and hit the generic return err branch.

node_client.APIError.Code already carries the HTTP status (pkg/node_client/client.go:43), so errors.As should be enough to split the cases:

Class	Retry?	Today
HTTP 200 + status: Failed — no route, no liquidity	yes, bounded	swallowed, marked paid
HTTP 4xx — PaymentHashAlreadyUsed, InvalidInvoice, InvalidAmount	no, terminal	retried forever
HTTP 5xx / 503 / transport	yes	retried

Proposed fix

1. Classify errors by HTTP status in aPaySendOutboundPaymentJob: terminal on 4xx, retry on 5xx/transport, and bounded retry on node-reported routing failures instead of swallowing them.
2. Read attempts in MarkAsyncRotatingInvoiceOutboxRetry and move the job to a dead status once the cap is reached.
3. Add a terminal transition out of outbound_pending on send failure, so the job stops re-arming. This is related to the existing TODO at api.go:474.