How to figure out when should express send 503 response code when server is real busy?

[mai1x9]

Hello there, when server is busy, express probably needs to send 503, how does one decide when server need to send 503. I have used middleware for returning 503 which is on top of every route and other middlewares. However on what metrics should I consider server is busy?

Currently, I am using toobusy-js npm library (https://www.npmjs.com/package/toobusy-js) and from the documentation, it says

 
// Set maximum lag to an aggressive value.
toobusy.maxLag(10);
 
// Set check interval to a faster value. This will catch more latency spikes
// but may cause the check to be too sensitive.
toobusy.interval(250);
 

What would be the best value for maxLag, or interval to be set. Under loadtesting with apache bench with concurrency 1000 and 20k requests most of the requests are hitting 503 response. I have set the values for maxLag=100 and interval=500 accordingly.

Other library options I have explored on,

• blocked https://www.npmjs.com/package/blocked
• event loop stats https://www.npmjs.com/package/event-loop-stats
• event loop lag https://www.npmjs.com/package/event-loop-lag

and calling them in a middleware. Is there any best metric to tell express to send 503?

Some of the options i could find out

• Based on event loop delay by using above packages. If so, what could be the best npm package that can be used which hands a concurrency of 1000 users without sending much 503 codes.
• Number of requests being queued by express. If so, how would I get queued requests count by express and send 503. Is it a good metric to consider?
• Limit the number of http connections
• Send 503 based on APM, like cpu% or mem%.

---

On side note, the application does not compute any cpu intensive tasks other than some basic json manipulation's, and have db operations/file operations.

[pacifiquem]

Easly you can count number of requests your server is handling at time and when they exceed maximum requests you send 503 status code .
here is described way to do it:

• Create a middleware that will run for each request .
• When your server receive a request increment the number of requests .
• Check if you've excedeed the maximum request your server can handle .
• When a desired time elapse turn request number to zero to start again in period of time .
• Request counting can be done : Every Minute, hourly, daily, monthly ... according to what you want .

That's how I handle that . you can contact me for more details about this .

[mai1x9]

@pacifiquem thanks for the suggestion. However i would like to point out few things:

• Let's say if some of the requests are taking time few ms upto 1-2 seconds, in such cases when we block based on number of requests, requests which are compute less and responds very fast within few ms would be delayed and return 503.
• One can say truly server is busy only when cpu its max 100%, so quick responses would ultimately be blocked if we count based on number of requests.
• Surely, global limit for number of connections can be limited to prevent from DOS attacks

Would like to hear from you on how the limit count for requests is set? Is it from rps point of view or requests latency or based on throughput. ?

[mai1x9]

@pacifiquem I would like to add one more point. Once request is processed we must decrease the counter, otherwise if we wait till time elapsed to reset to zero, there might be chances that server returns 503 even though it is not processing any requests.

Eg: let's say reset time is 60 seconds and max requests be 500. Within first 5 seconds i put 499 requests. Then at 30th second I keep 2 requests, the second request will respond with 503, even though only 1 request was processed or 0 requests (if response is very fast) at 30th second. Subsequently all following requests rom 30th second to time elapsed would return 503.

[pacifiquem]

Great @mai1x9 it looks like we are going to solve this,
By combining both Ideas now you can make and implement that middleware to do it for you .

[mai1x9]

@pacifiquem have you tried toobusy-js. Any thoughts on what could a max event loop lag that is acceptable for a server running in production.