BSC API: Increased number of response timeouts

[kaladinlight]

Overview

There has been a large number of 500's related to timeouts recently. I noticed this on bnbsmartchain specifically, but it appears to be somewhat prevalent across coinstacks.

I am able to make a lot of the same queries locally without any problem. This may be a kube networking issue? Need to investigate

References and additional details

::ffff:10.0.23.91 - GET /api/v1/account/0xef678d1ad8c897c71d0a6145390a474035db8f1f HTTP/1.1 500 41 - 10373.327 ms
ApiError [Internal Server Error]: timeout of 10000ms exceeded
    at handleError (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/utils.js:11:16)
    at Service.getAccount (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/service.js:89:43)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:533:9)
    at process.processTimers (node:internal/timers:507:7) {
  statusCode: 500
}
::ffff:10.0.158.60 - GET /api/v1/account/0x41cb654d1f47913acab158a8199191d160dabe4a HTTP/1.1 500 41 - 10003.075 ms
ApiError [Internal Server Error]: timeout of 10000ms exceeded
    at handleError (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/utils.js:11:16)
    at Service.getTxHistory (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/service.js:186:43)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:533:9)
    at process.processTimers (node:internal/timers:507:7) {
  statusCode: 500
}
::ffff:10.0.147.29 - GET /api/v1/account/0x5b00a277d7cc977f84e45e210c228440774cf943/txs?cursor=&pageSize=100 HTTP/1.1 500 41 - 10001.480 ms
ApiError [Internal Server Error]: timeout of 10000ms exceeded
    at handleError (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/utils.js:11:16)
    at Service.getTxHistory (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/service.js:186:43)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:533:9)
    at process.processTimers (node:internal/timers:507:7) {
  statusCode: 500
}
::ffff:10.0.81.12 - GET /api/v1/account/0x41cb654d1f47913acab158a8199191d160dabe4a/txs?cursor=&pageSize=100 HTTP/1.1 500 41 - 10001.623 ms
ApiError [Internal Server Error]: timeout of 10000ms exceeded
    at handleError (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/utils.js:11:16)
    at Service.getAccount (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/service.js:89:43)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:533:9)
    at process.processTimers (node:internal/timers:507:7) {
  statusCode: 500
}
::ffff:10.0.1.200 - GET /api/v1/account/0x5b00a277d7cc977f84e45e210c228440774cf943 HTTP/1.1 500 41 - 10002.057 ms
ApiError [Internal Server Error]: timeout of 10000ms exceeded
    at handleError (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/utils.js:11:16)
    at Service.getAccount (/app/node/coinstacks/bnbsmartchain/api/dist/common/api/src/evm/service.js:89:43)
    at runNextTicks (node:internal/process/task_queues:60:5)
    at listOnTimeout (node:internal/timers:533:9)
    at process.processTimers (node:internal/timers:507:7) {
  statusCode: 500
}
::ffff:10.0.147.29 - GET /api/v1/account/0x9f52a8a135d0c1010e0ddbeb089ae7df5da061c0 HTTP/1.1 500 41 - 10373.645 ms

Acceptance Criteria

• Determine root cause of timeouts
• No more timeouts

Need By Date

No response

Screenshots/Mockups

No response

Estimated effort

No response

[0xean]

getting intermittent reports of this happening across multiple endpoints (not just BSC)

[kaladinlight]

closed by #908