Filter Soroban Transactions from fee_stats

[sydneynotthecity]

What problem does your feature solve?

Since fee_stats profiles the inclusion fee over all transactions in a ledger, the data is skewed if a Soroban transaction is present in the ledger range. Soroban transactions have a different fee model and separate auction ,which should not be considered alongside classic transactions.

For example, this response from fee_stats includes Soroban transaction data, which artificially inflates the max inclusion fee:

{
  "last_ledger": "50944799",
  "last_ledger_base_fee": "100",
  "ledger_capacity_usage": "0.69",
  "fee_charged": {
    "max": "118842", <-- Soroban transaction
    "min": "100",
    "mode": "100",
    "p10": "100",
    "p20": "100",
    "p30": "100",
    "p40": "100",
    "p50": "100",
    "p60": "100",
    "p70": "100",
    "p80": "100",
    "p90": "100",
    "p95": "100",
    "p99": "100"
  },
  "max_fee": {
    "max": "369981750",
    "min": "100",
    "mode": "51234",
    "p10": "100",
    "p20": "101",
    "p30": "1000",
    "p40": "2073",
    "p50": "51234",
    "p60": "51234",
    "p70": "51234",
    "p80": "100159",
    "p90": "100772",
    "p95": "250005",
    "p99": "353225214"
  }
}

Soroban transaction: 03c05276692f663c5cb7d0cf8c3b25245b820a83fdc39bbb02b2ae85b3c2fe3a which was charged a fee of 118842 even though surge pricing did not occur for classic or Soroban tx during this ledger range. Since Horizon uses fee_charged, it does not calculate an accurate inclusion fee for the tx.

What would you like to see?

The current fee_stats endpoint should filter out Soroban transactions so that the statistics represent inclusion fee for classic transactions only. RPC or another service should provide a separate endpoint for Soroban fee_stats, and that should not be included in Horizon.

Example filters:

select *
from history_transactions
where resource_fee <= 0

What alternatives are there?

None, keep the Soroban transactions in the aggregate.

[janewang]

For Soroban transactions and Stellar transactions, they each have their own inclusion fees and own surge pricing. For example, classic transactions can be not non-surge price / base fee of 100, while Soroban fee go up during its surge pricing. Thus we need one data set for Soroban transactions and one data set for Stellar transactions in the lookback period. It doesn’t make sense to mix these two types of transactions. For Horizon, we should show only Stellar transaction and exlude Soroban transactions.

[Shaptic]

Soroban transactions have a [...] separate auction

Wait, really? For inclusion alone, too? @sydneynotthecity can you (or someone who knows, e.g. Core?) elaborate on how Soroban transactions are chosen to be included in the ledger? Is it something like "X txs of every ledger can be Soroban"? Or do they compete with non-Soroban transactions in the same auction? This is news to me since I haven't followed the fee discussion at all.

[sydneynotthecity]

They compete in separate auctions, but I do not know how core decides to allocate resources for Soroban vs non-Soroban transactions. If you want an example, ledger 50951287 charged a surge pricing inclusion fee of 5,000 stroops for classic transactions, but the two Soroban transactions were charged inclusion fees of 100 stroops:

select ledger_sequence,
  transaction_hash,
  operation_count,
  fee_charged,
  new_max_fee,
  inclusion_fee_charged,
  fee_charged / operation_count
from `crypto-stellar.crypto_stellar.history_transactions`
where batch_run_date >= '2024-03-26'
  and ledger_sequence = 50951287
order by inclusion_fee_charged desc

Devrel is working on providing more clarity on fees in our docs. I haven't read through the docs yet, so idk if it's at the level of detail you want.

@dmkozh will be able to provide better details on how core decides to split ledger load between Soroban and classic transactions

[dmkozh]

will be able to provide better details on how decides to split leader load between Soroban and classic transactions

Currently Core does not split load between Classic and Soroban in any way, i.e. these are completely separate from each other. Classic capacity is limited by 1000 operations/ledger and surge pricing triggers when there are Classic transactions in mempool that don't fit into that limit. Soroban capacity is limited by several per-ledger resource limits (such as 500M instructions, 70 KB of data written etc.) and surge pricing trigger condition is based only on Soroban transaction that doesn't fit into one of these limits being present in mempool.

[Shaptic]

Got it, so the inclusion fee is basically the auction that facilitates the fight for resources in the ledger. So even if you have all resources being used, if there is no add'l competition (contrived example: 2 txs using 50% of the capacity each, no other txs pending), then the inclusion fee is just the base fee. But if there are e.g. 4 txs that each want 50% of the capacity, they will get preference based on the inclusion fee.

I think that makes sense - thanks @sydneynotthecity and @dmkozh!