feat: Multiple json rpc urls

[Qwerty5Uiop]

Changelog

Introduced an ability to specify json rpc urls as a list. Each json rpc request will pick a random url from this list. All json rpc url env variables have been replaced with analogues for multiple use:

ETHEREUM_JSONRPC_HTTP_URL -> ETHEREUM_JSONRPC_HTTP_URLS
ETHEREUM_JSONRPC_TRACE_URL -> ETHEREUM_JSONRPC_TRACE_URLS
ETHEREUM_JSONRPC_ETH_CALL_URL -> ETHEREUM_JSONRPC_ETH_CALL_URLS
ETHEREUM_JSONRPC_FALLBACK_HTTP_URL -> ETHEREUM_JSONRPC_FALLBACK_HTTP_URLS
ETHEREUM_JSONRPC_FALLBACK_TRACE_URL -> ETHEREUM_JSONRPC_FALLBACK_TRACE_URLS
ETHEREUM_JSONRPC_FALLBACK_ETH_CALL_URL -> ETHEREUM_JSONRPC_FALLBACK_ETH_CALL_URLS

All previous variables names are kept for backwards compatibility.

Fallback urls usage logic now follows these rules:

• If some of primary json rpc url is unavailable, it is disabled until it will be available again (grouped by type: http, trace, eth_call)
• If all of the primary urls of some group are unavailable, fallback urls for this group are used

Docs update: blockscout/docs#332

[vbaranov]

I observe this error in the logs while running Blockscout locally with ETHEREUM_JSONPRC_HTTP_URL is set only:

2024-11-08T14:04:27.427 fetcher=coin_balance_catchup count=100 [error] Task #PID<0.13367.0> started from Indexer.Fetcher.CoinBalance.Catchup terminating
** (ArgumentError) errors were found at the given arguments:

  * 1st argument: not an already existing atom

    (erts 15.1) :erlang.binary_to_existing_atom("fallback__urls", :utf8)
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/utility/common_helper.ex:40: EthereumJSONRPC.Utility.CommonHelper.url_type_to_urls/3
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/http.ex:243: EthereumJSONRPC.HTTP.url/2
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/http.ex:69: EthereumJSONRPC.HTTP.chunked_json_rpc/3
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/request_coordinator.ex:89: anonymous fn/4 in EthereumJSONRPC.RequestCoordinator.perform/4
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/request_coordinator.ex:109: EthereumJSONRPC.RequestCoordinator.trace_request/2
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc.ex:532: EthereumJSONRPC.json_rpc/2
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc.ex:227: EthereumJSONRPC.fetch_balances/4
    (indexer 6.9.0) lib/indexer/fetcher/coin_balance/helper.ex:58: Indexer.Fetcher.CoinBalance.Helper.run/3
    (indexer 6.9.0) lib/indexer/fetcher/coin_balance/catchup.ex:65: Indexer.Fetcher.CoinBalance.Catchup.run/2
    (elixir 1.17.3) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
    (elixir 1.17.3) lib/task/supervised.ex:36: Task.Supervised.reply/4
Function: &Indexer.BufferedTask.log_run/1
    Args: [%{metadata: [fetcher: :coin_balance_catchup], batch: [{<<174, 155, 186, 150, 191, 236, 98, 235, 165, 31, 112, 202, 168, 227, 145, 211, 126, 149, 71, 134>>, 6746362}, {<<174, 167, 99, 69, 14, 49, 60, 134, 98, 102, 78, 89, 172, 223, 83, 176, 27, 73, 242, 194>>, 6746366}, {<<174, 170, 249, 204, 26, 226, 6, 50, 101, 35, 186, 211, 75, 146, 171, 75, 149, 109, 144, 217>>, 6746361}, {<<174, 222, 77, 102, 12, 89, 45, 163, 142, 233, 134, 223, 78, 249, 65, 181, 172, 196, 234, 26>>, 6746362}, {<<175, 61, 252, 46, 53, 250, 108, 155, 193, 5, 203, 209, 23, 236, 159, 204, 85, 152, 187, 226>>, 6746359}, {<<175, 138, 252, 39, 16, 39, 137, 42, 111, 186, 211, 126, 59, 242, 7, 22, 231, 198, 94, 23>>, 6746360}, {<<175, 203, 184, 126, 82, 170, 98, 232, 100, 53, 45, 253, 226, 250, 27, 31, 24, 42, 17, 1>>, 6746366}, {<<175, 237, 166, 29, 185, 225, 98, 41, 59, 46, 242, 194, 188, 90, 128, 11, 55, 187, 94, 74>>, 6746359}, {<<175, 237, 166, 29, 185, 225, 98, 41, 59, 46, 242, 194, 188, 90, 128, 11, 55, 187, 94, 74>>, 6746361}, {<<175, 237, 166, 29, 185, 225, 98, 41, 59, 46, 242, 194, 188, 90, 128, 11, 55, 187, 94, 74>>, 6746364}, {<<176, 113, 141, 90, 238, 179, 4, 127, 88, 22, 138, 123, 160, 195, 170, 167, 212, 14, 192, 102>>, 6746362}, {<<176, 126, 120, 87, 141, 182, 179, 255, 155, 220, 8, 187, 248, 194, 46, 90, 217, 65, 39, 72>>, 6746360}, {<<177, 123, 99, 54, 70, 116, 123, 6, 245, 34, 46, 91, 26, 27, 122, 26, 223, 60, 97, 227>>, 6746358}, {<<177, 232, 35, 151, 71, 94, 50, 200, 215, 28, 89, 99, 246, 124, 42, 251, 211, 214, 179, 61>>, 6746360}, {<<178, 129, 96, 207, 233, 213, 72, 216, 50, 101, 235, 76, 177, 205, 75, 235, 138, 98, 41, 30>>, 6746359}, {<<178, 139, 152, 226, 106, 5, 198, 154, 178, 224, 230, 136, 133, 125, 252, 36, 149, 120, 239, 116>>, 6746357}, {<<178, 139, 152, 226, 106, 5, 198, 154, 178, 224, 230, 136, 133, 125, 252, 36, 149, 120, 239, 116>>, 6746358}, {<<178, 245, 148, 98, 55, 4, 137, 209, 104, 222, 242, 223, 156, 239, 3, 228, 195, 48, 152, 142>>, 6746360}, {<<179, 101, 42, 103, 92, 212, 166, 10, 33, 10, 128, 77, 24, 71, 156, 81, 204, 231, 251, 154>>, 6746363}, {<<179, 161, 149, 108, 255, 30, 204, 128, 84, 184, 27, 12, 131, 185, 132, 124, 183, 19, 132, 184>>, 6746360}, {<<179, 205, 255, 27, 67, 149, 206, 89, 187, 102, 0, 25, 104, 71, 126, 196, 134, 254, 217, 216>>, 6746363}, {<<180, 100, 237, 82, 239, 90, 180, 128, 223, 32, 52, 64, 209, 153, 247, 250, 201, 52, 64, 157>>, 6746358}, {<<180, 190, 139, 185, 45, 65, 147, 87, 233, 253, 102, 230, 48, 129, 98, 124, 81, 219, 6, 31>>, 6746366}, {<<180, 208, 192, 159, 128, 164, 216, 169, 33, 207, 216, 99, 104, 71, 148, 221, 95, 109, 93, 237>>, 6746359}, {<<180, 243, 253, 13, 162, 216, 1, 174, 141, 2, 253, 209, 128, 34, 147, 225, 243, 158, 43, 150>>, 6746358}, {<<181, 68, 146, 22, 244, 128, 88, 53, 115, 223, 215, 5, 74, 102, 200, 6, 112, 237, 192, 1>>, 6746358}, {<<181, 211, 196, 184, 248, 63, 188, 236, 79, 4, 239, 158, 8, 59, 179, 67, 169, 84, 254, ...>>, 6746360}, {<<182, 58, 34, 13, 161, 16, 91, 178, 125, 218, 29, 207, 149, 60, 3, 18, 33, 201, ...>>, 6746359}, {<<182, 62, 99, 245, 237, 199, 236, 82, 40, 122, 98, 89, 241, 34, 202, 172, 206, ...>>, 6746362}, {<<182, 220, 15, 50, 38, 106, 100, 87, 50, 179, 161, 135, 77, 93, 149, 87, ...>>, 6746359}, {<<182, 249, 87, 153, 128, 174, 70, 246, 18, 23, 169, 145, 69, 100, 83, ...>>, 6746357}, {<<182, 249, 87, 153, 128, 174, 70, 246, 18, 23, 169, 145, 69, 100, ...>>, 6746358}, {<<182, 249, 87, 153, 128, 174, 70, 246, 18, 23, 169, 145, 69, ...>>, 6746359}, {<<182, 249, 87, 153, 128, 174, 70, 246, 18, 23, 169, 145, ...>>, 6746360}, {<<182, 249, 87, 153, 128, 174, 70, 246, 18, 23, 169, ...>>, 6746361}, {<<182, 249, 87, 153, 128, 174, 70, 246, 18, 23, ...>>, 6746362}, {<<182, 249, 87, 153, 128, 174, 70, 246, 18, ...>>, 6746363}, {<<182, 249, 87, 153, 128, 174, 70, 246, ...>>, 6746364}, {<<182, 249, 87, 153, 128, 174, 70, ...>>, 6746365}, {<<182, 249, 87, 153, 128, 174, ...>>, 6746366}, {<<182, 253, 201, 214, 227, ...>>, 6746358}, {<<183, 18, 184, 87, ...>>, 6746359}, {<<183, 18, 184, ...>>, 6746366}, {<<183, 41, ...>>, 6746360}, {<<183, ...>>, 6746363}, {<<...>>, ...}, {...}, ...], callback_module: Indexer.Fetcher.CoinBalance.Catchup, callback_module_state: [transport: EthereumJSONRPC.HTTP, transport_options: [http: EthereumJSONRPC.HTTP.HTTPoison, urls: ["http://x.x.x.x:8545"], trace_urls: ["http://localhost:8545"], eth_call_urls: ["http://localhost:8545"], fallback_urls: ["http://x.x.x.x:8545"], fallback_trace_urls: ["http://x.x.x.x:8545"], fallback_eth_call_urls: ["http://x.x.x.x:8545"], method_to_url: [eth_call: :eth_call, debug_traceTransaction: :trace, debug_traceBlockByNumber: :trace], http_options: [recv_timeout: 60000, timeout: 60000, hackney: [pool: :ethereum_jsonrpc]]], variant: EthereumJSONRPC.Geth]}]

[vbaranov]

Please enrich the public functions of this module with specification and documentation.

[Qwerty5Uiop]

Processed in 10f39c8

[vbaranov]

I observe this error after the changes with 2 vars are set: ETHEREUM_JSONRPC_HTTP_URL and ETHEREUM_JSONRPC_TRACE_URL:

2024-11-11T22:23:40.011 fetcher=coin_balance_catchup count=100 [error] Task #PID<0.2629.0> started from Indexer.Fetcher.CoinBalance.Catchup terminating
** (ArgumentError) errors were found at the given arguments:

  * 1st argument: not an already existing atom

    (erts 15.1) :erlang.binary_to_existing_atom("fallback_http://91.210.101.151:8545_urls", :utf8)
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/utility/common_helper.ex:40: EthereumJSONRPC.Utility.CommonHelper.url_type_to_urls/3
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/http.ex:243: EthereumJSONRPC.HTTP.url/2
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/http.ex:69: EthereumJSONRPC.HTTP.chunked_json_rpc/3
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/request_coordinator.ex:89: anonymous fn/4 in EthereumJSONRPC.RequestCoordinator.perform/4
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc/request_coordinator.ex:109: EthereumJSONRPC.RequestCoordinator.trace_request/2
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc.ex:532: EthereumJSONRPC.json_rpc/2
    (ethereum_jsonrpc 6.9.0) lib/ethereum_jsonrpc.ex:227: EthereumJSONRPC.fetch_balances/4
    (indexer 6.9.0) lib/indexer/fetcher/coin_balance/helper.ex:58: Indexer.Fetcher.CoinBalance.Helper.run/3
    (indexer 6.9.0) lib/indexer/fetcher/coin_balance/catchup.ex:65: Indexer.Fetcher.CoinBalance.Catchup.run/2
    (elixir 1.17.3) lib/task/supervised.ex:101: Task.Supervised.invoke_mfa/2
    (elixir 1.17.3) lib/task/supervised.ex:36: Task.Supervised.reply/4
Function: &Indexer.BufferedTask.log_run/1
    Args: [%{metadata: [fetcher: :coin_balance_catchup], batch: [{<<43, 189, 85, 172, 86, 27, 224, 48, 226, 213, 128, 248, 223, 64, 149, 18, 124, 68, 195, 253>>, 6746338}, {<<43, 189, 85, 172, 86, 27, 224, 48, 226, 213, 128, 248, 223, 64, 149, 18, 124, 68, 195, 253>>, 6746340}, {<<43, 219, 58, 219, 55, 6, 89, 233, 14, 196, 244, 166, 6, 148, 105, 113, 6, 106, 46, 95>>, 6746342}, {<<44, 21, 94, 208, 92, 148, 234, 15, 86, 44, 235, 13, 118, 249, 126, 23, 113, 218, 96, 39>>, 6746338}, {<<44, 79, 224, 113, 152, 171, 186, 221, 167, 158, 218, 46, 6, 67, 41, 215, 180, 14, 240, 145>>, 6746338}, {<<44, 79, 224, 113, 152, 171, 186, 221, 167, 158, 218, 46, 6, 67, 41, 215, 180, 14, 240, 145>>, 6746339}, {<<44, 79, 224, 113, 152, 171, 186, 221, 167, 158, 218, 46, 6, 67, 41, 215, 180, 14, 240, 145>>, 6746340}, {<<44, 79, 224, 113, 152, 171, 186, 221, 167, 158, 218, 46, 6, 67, 41, 215, 180, 14, 240, 145>>, 6746341}, {<<44, 79, 224, 113, 152, 171, 186, 221, 167, 158, 218, 46, 6, 67, 41, 215, 180, 14, 240, 145>>, 6746342}, {<<44, 79, 224, 113, 152, 171, 186, 221, 167, 158, 218, 46, 6, 67, 41, 215, 180, 14, 240, 145>>, 6746344}, {<<44, 79, 224, 113, 152, 171, 186, 221, 167, 158, 218, 46, 6, 67, 41, 215, 180, 14, 240, 145>>, 6746345}, {<<44, 79, 224, 113, 152, 171, 186, 221, 167, 158, 218, 46, 6, 67, 41, 215, 180, 14, 240, 145>>, 6746346}, {<<44, 148, 121, 194, 250, 99, 95, 120, 77, 181, 145, 57, 121, 210, 8, 250, 14, 173, 227, 196>>, 6746341}, {<<44, 148, 121, 194, 250, 99, 95, 120, 77, 181, 145, 57, 121, 210, 8, 250, 14, 173, 227, 196>>, 6746342}, {<<44, 148, 121, 194, 250, 99, 95, 120, 77, 181, 145, 57, 121, 210, 8, 250, 14, 173, 227, 196>>, 6746343}, {<<44, 218, 65, 100, 95, 45, 191, 251, 133, 42, 96, 94, 146, 177, 133, 80, 24, 1, 252, 40>>, 6746340}, {<<44, 218, 65, 100, 95, 45, 191, 251, 133, 42, 96, 94, 146, 177, 133, 80, 24, 1, 252, 40>>, 6746344}, {<<44, 224, 171, 47, 34, 161, 234, 63, 32, 153, 12, 148, 48, 22, 226, 7, 55, 201, 229, 180>>, 6746339}, {<<45, 93, 87, 99, 195, 38, 209, 196, 113, 27, 177, 17, 198, 232, 33, 27, 39, 142, 60, 153>>, 6746341}, {<<45, 93, 87, 99, 195, 38, 209, 196, 113, 27, 177, 17, 198, 232, 33, 27, 39, 142, 60, 153>>, 6746342}, {<<45, 112, 249, 168, 102, 222, 52, 192, 247, 56, 248, 203, 42, 241, 54, 27, 90, 241, 140, 170>>, 6746341}, {<<45, 112, 249, 168, 102, 222, 52, 192, 247, 56, 248, 203, 42, 241, 54, 27, 90, 241, 140, 170>>, 6746344}, {<<45, 183, 93, 132, 4, 20, 76, 213, 145, 136, 21, 164, 75, 138, 195, 244, 219, 42, 127, 175>>, 6746344}, {<<45, 242, 161, 16, 86, 104, 46, 180, 73, 33, 126, 37, 184, 35, 44, 145, 84, 31, 105, 180>>, 6746344}, {<<46, 59, 76, 211, 68, 109, 164, 173, 52, 15, 88, 19, 154, 9, 42, 100, 210, 202, 236, 35>>, 6746342}, {<<46, 82, 33, 176, 248, 85, 190, 78, 165, 206, 255, 251, 131, 17, 238, 208, 86, 59, 110, 135>>, 6746338}, {<<46, 82, 33, 176, 248, 85, 190, 78, 165, 206, 255, 251, 131, 17, 238, 208, 86, 59, 110, ...>>, 6746344}, {<<46, 88, 11, 174, 24, 46, 254, 93, 88, 79, 121, 90, 250, 136, 133, 214, 32, 92, ...>>, 6746339}, {<<46, 118, 120, 252, 9, 220, 164, 118, 19, 189, 255, 103, 13, 56, 117, 86, 212, ...>>, 6746342}, {<<46, 135, 196, 28, 155, 61, 147, 43, 109, 163, 200, 5, 186, 239, 183, 187, ...>>, 6746339}, {<<46, 152, 195, 133, 42, 221, 186, 139, 41, 45, 203, 98, 105, 30, 44, ...>>, 6746338}, {<<46, 152, 195, 133, 42, 221, 186, 139, 41, 45, 203, 98, 105, 30, ...>>, 6746342}, {<<46, 166, 53, 243, 251, 215, 95, 162, 158, 183, 185, 39, 114, ...>>, 6746338}, {<<46, 166, 53, 243, 251, 215, 95, 162, 158, 183, 185, 39, ...>>, 6746342}, {<<46, 166, 53, 243, 251, 215, 95, 162, 158, 183, 185, ...>>, 6746344}, {<<46, 223, 16, 132, 217, 158, 110, 152, 101, 89, ...>>, 6746339}, {<<47, 39, 173, 21, 181, 154, 120, 89, 121, ...>>, 6746344}, {<<47, 165, 33, 36, 184, 23, 1, 6, ...>>, 6746344}, {<<47, 168, 157, 163, 249, 48, 232, ...>>, 6746342}, {<<47, 173, 185, 208, 30, 88, ...>>, 6746344}, {<<47, 205, 176, 184, 138, ...>>, 6746339}, {<<47, 205, 176, 184, ...>>, 6746342}, {<<47, 233, 185, ...>>, 6746337}, {<<48, 96, ...>>, 6746342}, {<<48, ...>>, 6746338}, {<<...>>, ...}, {...}, ...]