feat(balances): Adjust balance and alert threhold automation process #5361

Mo-Hussain · 2025-02-03T14:51:25Z

Description

Suggest changing the minimum number of daily txs that need to be supported to 100 from 200. Main motivation for this change is that it reduces the monetary cost of maintaining high dollar value relayer balance for BTC and ETH gas token chains. Table below shows the chains where the 200 tx calculations results in a significant dollar increase in relayer balance needed

┌────────────────┬─────────────────┬──────────────────────────────┬─────────────────────────────┐
│     chain      │ proposedDailyBurn│ proposedRelayerBallanceDollars│ currentRelayerBallanceDollars│
├────────────────┼─────────────────┼──────────────────────────────┼─────────────────────────────┤
│ bitlayer       │     0.00205     │            1620              │            198              │
│ bsquared       │     0.00358     │            2830              │            198              │
│ ethereum       │     0.269       │            6590              │           1530              │
│ morph          │     0.276       │            6750              │            306              │
│ rootstockmainnet│     0.00259    │            2040              │            197              │
└────────────────┴─────────────────┴──────────────────────────────┴─────────────────────────────┘

I haven't throughly checked all balances to see if they are kosher, I will do this once we have agreement on the 100 to 200 tx change

update-balance-threshold-config

Support desiredRelayerBalanceOverrides.json, allow us to override this threshold for new chains, chains were we need to maintain a temporary low balance and special cases like osmosis. Alert thresholds are also adjusted based on this override. The overrides also allows us to respond to cases where the current thresholds are significantly larger than the proposed values and we want to reset
Inform user if the proposed thresholds or daily burn estimation is 50% less than the current values, prompt user for manual review

If we want to reduce the thresholds, we can add an entry into the desiredRelayerBalanceOverrides.json and call update-balance-threshold-config.ts the reset all thresholds.

I haven't done any manual threshold reductions yet, I think this would be good to do in a quick follow up do the changes are more obvious

Add validation which checks if all the thresholds makes sense lowUrgencyKeyFunderBalance[chain] > desiredRelayerBalances[chain] > lowUrgencyEngKeyFunderBalance[chain] > highUrgencyRelayerBalance[chain]
Add lowUrgencyEngKeyFunderBalance.json config

write-alert.ts

run validation for all configs before pushing changes to grafana

Next Steps

Manually reduce thresholds based on them being more than 50% below and throughly review all thresholds
supporting a promQL query for sealevel balances and ata-payer
automating IGP payments
Roll out

Testing

Manual

codecov · 2025-02-03T14:58:58Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.53%. Comparing base (ef08c53) to head (e7a25c8).
Report is 8 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5361   +/-   ##
=======================================
  Coverage   77.53%   77.53%           
=======================================
  Files         103      103           
  Lines        2110     2110           
  Branches      190      190           
=======================================
  Hits         1636     1636           
  Misses        453      453           
  Partials       21       21

Components	Coverage Δ
core	`87.80% <ø> (ø)`
hooks	`79.39% <ø> (ø)`
isms	`83.68% <ø> (ø)`
token	`91.27% <ø> (ø)`
middlewares	`79.80% <ø> (ø)`

Mo-Hussain · 2025-02-03T17:12:29Z

typescript/infra/config/environments/mainnet3/balances/dailyRelayerBurn.json

  "tangle": 3.13,
  "telos": 22.7,
  "torus": 5.82,
-  "treasure": 789,


Not sure why treasure was so high initially, this is the current burn in the 10 look back window

Mo-Hussain · 2025-02-03T17:15:36Z

typescript/infra/config/environments/mainnet3/balances/desiredRelayerBalanceOverrides.json

@@ -0,0 +1,15 @@
+{


overrides should include new chains, chains with temporarily low balances until we're able to fund more and special cases likes osmosis

Mo-Hussain · 2025-02-04T09:44:45Z

typescript/infra/scripts/funding/calculate-relayer-daily-burn.ts


 const DAILY_BURN_PATH =
  './config/environments/mainnet3/balances/dailyRelayerBurn.json';

 const LOOK_BACK_DAYS = 10; // the number of days to look back for average destination tx costs
-const MIN_NUMBER_OF_TXS = 200; // the minimum number of txs to consider for daily burn
+const MIN_NUMBER_OF_TXS = 100; // the minimum number of txs to consider for daily burn


suggested change to 100

Mo-Hussain · 2025-02-04T13:52:15Z

typescript/infra/scripts/funding/calculate-relayer-daily-burn.ts

+    }
+
+    // push to burnInfoTable
+    burnInfoTable.push({


show user more info about daily burn, specifically dollar values

Mo-Hussain · 2025-02-04T15:49:17Z

typescript/infra/config/environments/mainnet3/balances/dailyRelayerBurn.json

@@ -1,124 +1,124 @@
 {


Majority of the reductions are due to 1) 200 min tx => 100 min tx and 2) desiredRelayerBalanceOverrides where we are temporarily keeping the balance low

Mo-Hussain · 2025-02-04T15:50:28Z

typescript/infra/config/environments/mainnet3/balances/dailyRelayerBurn.json

@@ -1,124 +1,124 @@
 {
-  "abstract": 0.00291,


Will do a through review of all thresholds and manual reduction based on relayer burn after 200 min tx => 100 min tx changes is agreed upon

changeset-bot · 2025-02-04T15:53:47Z

⚠️ No Changeset found

Latest commit: e7a25c8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Mo-Hussain · 2025-02-04T15:57:47Z

typescript/infra/scripts/funding/update-balance-threshold-config.ts

-    withBalanceThresholdConfig(yargs(process.argv.slice(2))),
-  ).argv;
-
-  const configToUpdate: BalanceThresholdType[] = all


Made a change to always update all thresholds at once, this enables us to validate and avoid easily falling in different configs being out of whack. It does sacrifice some flexibility, which has been helpful during development, but it is not expected to be required for the production process

Mo-Hussain · 2025-02-04T16:00:14Z

typescript/infra/src/funding/grafana.ts

-      orderedThresholds[key] = newThresholds[key];
-    });
-  return orderedThresholds;
+  return Object.fromEntries(


Decided to revert to something I consider more reliable lol

Mo-Hussain self-assigned this Feb 3, 2025

Mo-Hussain commented Feb 3, 2025

View reviewed changes

Mo-Hussain commented Feb 4, 2025

View reviewed changes

Mo-Hussain marked this pull request as ready for review February 4, 2025 16:46

Mo-Hussain requested review from tkporter and paulbalaji as code owners February 4, 2025 16:46

Mo-Hussain added 5 commits February 6, 2025 09:34

feat(balances): Adjust balance and alert threhold automation process

7aa446b

add validation

1a1d454

nit

d8aa145

read sealevel burn using prom query, refactor code and reog files

c90cd1e

rerun to include berachain

ef8b905

Mo-Hussain force-pushed the mo/balance-threshold-automation-2 branch from e7a25c8 to ef8b905 Compare February 6, 2025 09:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(balances): Adjust balance and alert threhold automation process #5361

feat(balances): Adjust balance and alert threhold automation process #5361

Mo-Hussain commented Feb 3, 2025 •

edited

Loading

codecov bot commented Feb 3, 2025 •

edited

Loading

Mo-Hussain Feb 3, 2025

Mo-Hussain Feb 3, 2025 •

edited

Loading

Mo-Hussain Feb 4, 2025

Mo-Hussain Feb 4, 2025

Mo-Hussain Feb 4, 2025

Mo-Hussain Feb 4, 2025 •

edited

Loading

changeset-bot bot commented Feb 4, 2025 •

edited

Loading

Mo-Hussain Feb 4, 2025

Mo-Hussain Feb 4, 2025

feat(balances): Adjust balance and alert threhold automation process #5361

Are you sure you want to change the base?

feat(balances): Adjust balance and alert threhold automation process #5361

Conversation

Mo-Hussain commented Feb 3, 2025 • edited Loading

Description

Testing

codecov bot commented Feb 3, 2025 • edited Loading

Codecov Report

Mo-Hussain Feb 3, 2025

Choose a reason for hiding this comment

Mo-Hussain Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

Mo-Hussain Feb 4, 2025

Choose a reason for hiding this comment

Mo-Hussain Feb 4, 2025

Choose a reason for hiding this comment

Mo-Hussain Feb 4, 2025

Choose a reason for hiding this comment

Mo-Hussain Feb 4, 2025 • edited Loading

Choose a reason for hiding this comment

changeset-bot bot commented Feb 4, 2025 • edited Loading

⚠️ No Changeset found

Mo-Hussain Feb 4, 2025

Choose a reason for hiding this comment

Mo-Hussain Feb 4, 2025

Choose a reason for hiding this comment

Mo-Hussain commented Feb 3, 2025 •

edited

Loading

codecov bot commented Feb 3, 2025 •

edited

Loading

Mo-Hussain Feb 3, 2025 •

edited

Loading

Mo-Hussain Feb 4, 2025 •

edited

Loading

changeset-bot bot commented Feb 4, 2025 •

edited

Loading