blip-0036: on-the-fly channel funding

[t-bast]

Payments sent to mobile wallets often fail because the recipient doesn't have enough inbound liquidity to receive it. we add a mechanism to create an on-chain transaction on-the-fly before relaying such payments, which allow them to be relayed once the on-chain transaction is accepted by both peers.

This protocol uses dual-funding, splicing and liquidity ads, leveraging liquidity ads' extensions for paying funding fees.

It is recommended to use 0-conf, to avoid keeping upstream HTLCs held for a long time and locking up liquidity in the network.

[tnull]

Should this reuse/build on top of #25 ?

[t-bast]

Having the funding_txid field is actually quite important to more easily check consistency. We could use the TLV field from #25 and create a new one that only contains the funding_txid and require that both of them are set, but that requires writing code to handle the case when only one of them is set, which is a bit clunky and can be avoided.

But we can easily keep both TLV fields: the one from #25 is actually more general and can be used for other scenarios that don't involve a funding tx, so it has value on its own?

[tnull]

Having the funding_txid field is actually quite important to more easily check consistency. We could use the TLV field from #25 and create a new one that only contains the funding_txid and require that both of them are set, but that requires writing code to handle the case when only one of them is set, which is a bit clunky and can be avoided.

But we can easily keep both TLV fields: the one from #25 is actually more general and can be used for other scenarios that don't involve a funding tx, so it has value on its own?

Mhh, given we already have a dedicated bLIP (and at least on the LDK end implementation support) for exactly that use case I'd prefer the former variant: use the 65537 type from #25, and here only add what we need on top of that.

[t-bast]

It would be clunky though: since we need to add a new TLV anyway (because we want to include the funding_txid), why not include all the data we need in this TLV? Why add more unnecessary code to handle the case where only one of the TLV fields is set? I'd rather go with the simplest implementation here, and it's fine if we have two similar-but-not-exactly-the-same TLVs, since they're not used by the same protocols anyway?

[tnull]

Well, I think at least in LDK it would trigger exactly the same code paths, would expose the same API, etc. Just that we would run some additional checks on the funding_txid. So as this is seems like a special casing of #25, it might be a bit more overhead to treat them entirely differently, at least for implementations that would support 'both' variants.

[t-bast]

an LSP could choose to forward an HTLC with some skimmed fee on top for every HTLC on the channel (presumably at a lower fee) rather than only on funding, and the spec should support that so that LSPs can play around with whatever fees they want.

Agreed, that's a model that people may (or may not) want to use, and they're free to do so! But it requires additional fields to properly negotiate than the default ones offered by liquidity ads, a different mechanism for fee accounting, and this isn't the direction we're personally pursuing, so I'm not specifying this model here but rather a more restrictive one that suits our needs.

But I designed liquidity ads to easily allow extensions like this: as I described here, the model you're mentioning can be added by another bLIP with a dedicated payment_type.

I don't think there's a simple way to universally support all payment models: each one of them requires a few dedicated fields and some custom logic, that's why I introduced this payment_type abstraction that lets us support various payment models.

They can't (practically) go dig through the mempool (and UTXO set) to compare the on-chain fee of funding_txid against what the fee being skimmed here

That's not at all what I recommend doing, you're misunderstanding me. I'll explain below with an example.

presumably the recipient has pre-negotiated some fee with the LSP and can just compare the fee given here with what they expected.

Yes, exactly, but there can be multiple unpaid funding operations. If you only use bLIP 25, you cannot distinguish which one is paid by which HTLCs: so you're basically losing information that is useful to let users know what they've paid for (unless you rely on timestamps and just consider that you pay things in-order, but I'd rather rely on explicit fields - which also simplifies debugging/support).

As you mentioned, fees are negotiated before building the transaction. Once the transaction is signed, both sides can store in their DB the fees that are owed for this specific transaction. Then when relaying/receiving payments, they can check how much fee is owed and whether the collected fees match.

When chaining splice transactions, I'd like to be able to tell users that they've fully paid for the first splice, but the second splice hasn't yet been paid for (or has been partially paid). Having the funding_txid along with the fees deduced for the HTLC makes this trivial: if you don't have that, you can only rely on heuristics that may be wrong when we start composing with other protocols that also leverage extra fees.

If for example we start collecting fees from HTLCs for a completely separate reason (e.g. subscriptions for data storage), I don't think this should use the same TLV, otherwise it's impossible to track what is being paid for. But if on the contrary we use distinct TLV fields, it becomes obvious, easy to track, easy to debug, and compose perfectly: a single HTLC can contain multiple "extra fees" fields if it's paying for multiple unrelated things.

WDYT?

[TheBlueMatt]

Yes, exactly, but there can be multiple unpaid funding operations...

Ah, okay, I was not considering the case of having multiple potential funding transactions which you could be paying for. I am, however, still somewhat confused why this is ambiguous in your model, given the HTLC is tied to a channel already. Presumably you're talking about some case where we started funding a channel, it was insufficient, and we need to RBF the funding and thus a second HTLC might be paying for the second RBF variant? Or maybe you're talking about some splicing case where a channel has already been funding and is open and we're splicing in to receive a payment, but we receive a second payment which requires a greater splice before the first confirms, at which point we have two pending splices and some previous HTLC paid for the first one but this new HTLC is paying for a second one?

In both models, however, I fail to see any ambiguity - presumably we can just track the total fee that is needed to pay for the currently-pending splices and expect the next HTLC to charge that fee. Tracking a single "fee deficit" and being willing to let HTLCs pay for that deficit until its exhausted seems much simpler and doesn't have any ambiguity either, AFAICT.

In any case, I'd still prefer we add another TLV for the ambiguity resolving field, using the bLIP 25 field for the fee taken since that's already out there and there's no need to be stingy about TLVs.

Agreed, that's a model that people may (or may not) want to use, and they're free to do so! But it requires additional fields to properly negotiate than the default ones offered by liquidity ads, a different mechanism for fee accounting, and this isn't the direction we're personally pursuing, so I'm not specifying this model here but rather a more restrictive one that suits our needs.

It needs new fields in the negotiation of fees charged by the LSP/liquidity ad, but it shouldn't need additional fields at this level. AFICT the "fee deficit"-based accounting I mention above should work for ~any fee structure, and the update_add_htlc logic can rather simply capture that.

That said, IMO we should support the "fixed %/msat fees on each HTLC" model in liquidity ads. Its a pretty straightforward fee model and not supporting at least a handful of fee models (rather than just one) seems short-sighted.

[t-bast]

Tracking a single "fee deficit" and being willing to let HTLCs pay for that deficit until its exhausted seems much simpler and doesn't have any ambiguity either, AFAICT.

In any case, I'd still prefer we add another TLV for the ambiguity resolving field, using the bLIP 25 field for the fee taken since that's already out there and there's no need to be stingy about TLVs.

But that wouldn't let you:

• mark individual operations as "fully paid" unless you rely on timestamps, which can be incorrect for concurrent operations
• support concurrent operations (an HTLC that pays the fees for a previous splice and a subscription for something - or anything we may want to pay through HTLCs fees in the future): both payments would be bundled in a single extra_fee field, so we're losing valuable information?

It seems much more future-proof to me to have dedicated fee fields for each payment: this guarantees that we can stack them without any ambiguity and correctly settle what has been fully paid for. And I like being explicit about what is happening, every time we've tried to be clever and relied on things being implicit, we've regretted it when adding more use-cases...

That said, IMO we should support the "fixed %/msat fees on each HTLC" model in liquidity ads. Its a pretty straightforward fee model and not supporting at least a handful of fee models (rather than just one) seems short-sighted.

Can you start a thread on the liquidity ads spec PR then? I'd like to make some progress on it, but it needs feedback like this. Since this is completely different from a one-time flat fee, it will need a different set of fields that we should allow supporting.

[TheBlueMatt]

mark individual operations as "fully paid" unless you rely on timestamps, which can be incorrect for concurrent operations

Not sure I quite understand this requirement. You can just FIFO it if you want to display it in some UI?

support concurrent operations (an HTLC that pays the fees for a previous splice and a subscription for something - or anything we may want to pay through HTLCs fees in the future): both payments would be bundled in a single extra_fee field, so we're losing valuable information?

Not sure I understand why a fee deficit model doens't accomplish this? All pending fees you owe will be in the deficit and a single HTLC can pay for the full deficit, if it has enough balance. Rather, the current design looks like it doesn't allow an HTLC to pay for multiple things - the HTLC can only specify one txid that its paying for, so presumably if we want to pay for multiple in one HTLC it cant?

Can you start a thread on the lightning/bolts#1153 then? I'd like to make some progress on it, but it needs feedback like this. Since this is completely different from a one-time flat fee, it will need a different set of fields that we should allow supporting.

Sure.

[t-bast]

It's going to be easier to reason about this with an example:

• Alice purchases some inbound liquidity using from_future_htlc, resulting in funding_txid for which she owes 10_000 sats to Bob
• Alice has a subscription with Bob for additional services (could be anything) where she agreed that 1_000 sats will be collected from every incoming HTLC
• Alice made a one-time purchase of some additional service (could be anything as well) where she agrees that 1_000 sats will be collected from any future HTLC

With your proposal, the relayed HTLC will contain:

• A single extra_fees TLV containing amount = 12_000 sats
• A single on_the_fly_funding_txid TLV containing funding_txid

With my proposal, the relayed HTLC will contain:

• A single funding_fee TLV containing amount = 10_000 sat and funding_txid
• A single subscription_fee TLV containing amount = 1_000 sat and the subscription details
• A single one_time_purchase_fee TLV containing amount = 1_000 sat and details about the purchase

With my proposal, we know exactly what is being paid for. With yours, we may need to rely on heuristics: if only 11_000 sats were paid, which of the two purchases hasn't been paid for (and may need to be cancelled)? My example may not be great, but the high-level idea is that we should separate payments for unrelated things/protocols, because that is more explicit and we're sure that it can compose/stack properly.

Rather, the current design looks like it doesn't allow an HTLC to pay for multiple things - the HTLC can only specify one txid that its paying for, so presumably if we want to pay for multiple in one HTLC it cant?

That can't happen because one on-the-fly funding is tied to a specific set of payment hashes in from_future_htlc. Another on-the-fly funding that comes later can only occur if new HTLCs (with different payment hashes) come in, so there will never be a need to support paying two on-the-fly funding attempts from the same HTLC.

But there is definitely a need to support paying an on-the-fly funding attempt and something else in the same HTLC, which works seamlessly if "something else" uses a different TLV (since it's a different feature).

I hope my point is a bit clearer 😅

[tnull]

Suggested change

	so. Fees are instad paid from HTLCs that will be relayed once liquidity has
	so. Fees are instead paid from HTLCs that will be relayed once liquidity has

[t-bast]

Thanks, will fix!

[tnull]

FWIW ensuring the initial HTLCs remain forwardable in the face of concurrent forwards and htlc_minimum_msat/dust exposure limits potentially changing mid-flow can be non-trivial in our experience. Also, it's important to make sure the LSP is neither over- nor underpaid.

We previously solved this by batched sequencing of HTLCs while pausing other forwards until the channel open has been paid for. We found this is required as otherwise concurrent HTLCs might 'steal' liquidity from the HTLCs that should pay for the channel open, or the LSP might erroneously withhold or not withhold fees from additionally incoming HTLCs, all leading to arbitrary failures of the flow and rather poor/confusing UX.

Do think recommending such a sequencing model would be helpful? Or more generally, do we need to provide more guidance on how the LSP should handle additional HTLCs that come in after the channel has been opened but not paid for yet?

[t-bast]

I agree, this is the annoying detail to get right, it's probably worth a comment!

the LSP might erroneously withhold or not withhold fees from additionally incoming HTLCs

For simplicity, I chose to make that impossible in the protocol, which ensures that the LSP is never overpaid. This is done by specifying the payment_hashes associated with the on-the-fly funding directly when purchasing the liquidity (in the liquidity ads payment_types): this "commits" the LSP to claim its fee only from those payment_hashes, which greatly simplifies state tracking.

The drawback is that if other HTLCs stole the liquidity, the LSP must keep holding the HTLCs related to the on-the-fly funding attempt and retry relay when liquidity is available. If the HTLC expiry is reached before the LSP had the opportunity to relay them again, the LSP doesn't get paid its liquidity fees. To avoid that, it is indeed important to sequence HTLCs like you suggest. I'll add a paragraph for this, thanks for your comment!

[tnull]

For simplicity, I chose to make that impossible in the protocol, which ensures that the LSP is never overpaid. This is done by specifying the payment_hashes associated with the on-the-fly funding directly when purchasing the liquidity (in the liquidity ads payment_types): this "commits" the LSP to claim its fee only from those payment_hashes, which greatly simplifies state tracking.

Ah, right, good point. While this matches our current (LSPS2) model, it seems this would prohibit any other models that would pay for the channel by withholding from any/all HTLCs over time? But then again, not sure if anybody wants this model and I guess such a model could simply be implemented via forwarding fees?

[t-bast]

it seems this would prohibit any other models that would pay for the channel by withholding from any/all HTLCs over time?

That's true, this wouldn't be supported based on the current state of this bLIP alone. But it would be trivial to introduce a new payment_type for it if we think it makes sense: from_any_future_htlc that wouldn't reference a specific payment_hash?

[tnull]

That's true, this wouldn't be supported based on the current state of this bLIP alone. But it would be trivial to introduce a new payment_type for it if we think it makes sense: from_any_future_htlc that wouldn't reference a specific payment_hash?

Yeah that makes sense. I generally like the extensible approach, although want to note that at some point we should likely see that everybody converges on some set of payment types as otherwise we may end up with a landscape in which different clients and LSPs implement disjoint sets of features/payment types, resulting effectively in incompatibilities even though all speak 'liquidity ads'.

[ZmnSCPxj-jr]

I'm betting that it's a very rare case

My objection is that this is the sort of weird race-condition case that is hard to debug in practice, and that we should decide our systems to avoid such race conditions in the first place.

But nobody supports me here, so who cares anyway

[t-bast]

My objection is that this is the sort of weird race-condition case that is hard to debug in practice

This case isn't hard to debug at all? It's obvious on the recipient side what happens and why they're rejecting the payment, and it's trivial to display a notification to the user that this is why the payment was rejected.

I'd be all for a protocol that would perfectly handle all cases, but I don't think that can be achieved, because MPP introduces that kind of complexity.

[ZmnSCPxj-jr]

payers should try sending payments with a single HTLC at least once

This may not be as common eventually:

• Privacy-focused wallets will want multiple different LSPs and will split outgoing payments randomly right from the first attempt to reduce information leakage (and will use different node IDs for each LSP to reduce correlation, especially once PTLCs hit and we can drop the hashes); basically you do not want a single LSP to have accurate information about how much you send at each point in time, and MPP lets you split right from the first attempt.
• Multinut Cashu

Sure, you can argue that the recipient side can see what is happening, but the splitting is done on the sender side, and the sender side may have restrictions on what it can do. The problem has been pointed out and you have been informed about it.

[tnull]

and payers should try sending payments with a single HTLC at least once

I agree with Zman here: for a JIT channel the sender-side is really out of our control. So even if we were to establish some kind of community standard, we can't reliably lean on any assumption of (otherwise valid) sender-side behavior. IMO, we hence need to take precautions on the receiver side and the spec should a) at the very minimum allow for them b) preferably give clear guidance how to implement them.

[t-bast]

Let's take a step back, because I think you're giving too much weight to a rare edge case, which we can quantify.

First of all, there is a fundamental limitation that will apply to any on-the-fly funding protocol where fees are paid from HTLCs that couldn't be relayed: if the payment amount is smaller than the on-the-fly funding fees, we cannot accept it. This means that there will always be some payment attempts that fail because they cannot pay for an on-chain transaction. This is true of any such protocol variant, do we agree on that point? We're of course not considering additional features like fee credit or taking fees from unrelated future HTLCs, because we wouldn't be comparing apples to apples since it changes the trust model.

A payment made with an ideal on-the-fly funding protocol will fail if:

• the total payment amount is smaller than the on-chain fees
• and the user's inbound liquidity is smaller than this payment amount

A payment made with the current protocol will obviously fail if the above conditions are met. The additional case where it will fail is if:

• the total payment amount is higher than the on-chain fees
• the total payment amount is higher than the user's inbound liquidity
• the difference between the payment's total amount and the user's inbound liquidity is smaller than the on-chain fees AND it is split in a way that consumes all of the inbound liquidity
• it's important to note that if the payment's total amount is greater than the user's inbound liquidity + on-chain fees, the current protocol will always succeed, regardless of how the payment is split

The probability for the failures to happen depends on:

• the probability distribution of funds in the channel
• the ratio between on-chain fees and the channel size
• the probability distribution of incoming payment amounts
• the probability distribution of payment splitting

Detailed results will vary depending on what is chosen for those distributions. It could be interesting to compute the results for some standard distributions, but it's easy to see that in the worst case (where payments are always split in tiny chunks to fall into the failing scenario), the probability that the current protocol fails is at most twice the probability that the ideal protocol fails (because the additional failing case has the same "amount range" where it fails as the failing case for the ideal protocol).

More interestingly, if we assume that payments are split randomly by senders, this is driven by the ratio on-chain-fees / inbound_liquidity: if the channel capacity is much larger than the on-chain fees (which doesn't seem unreasonable), it's easy to see that the increase in failure probability becomes quite small.

I'm not saying that the currently proposed protocol is perfect, far from it. What I'm saying is that we must carefully consider the ratio between protocol complexity and (expected) reliability. If you can come up with a protocol that is as simple as the one I'm proposing with a better reliability, I'd be very happy to use it! But if the amount of additional complexity is large, I think it's important to understand how often it will perform better in theory before choosing it over a simpler variant (because we all know that a more complex protocol will in practice fail a lot more often because of bugs and unforeseen edge cases).