-
Notifications
You must be signed in to change notification settings - Fork 4
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FR - Growing latencies on copy receipts #242
Comments
Just wanted to check if there is any update, or anything we can help with 🙏 |
Hi @mertcanyalhi, thanks a lot for these very detailed analysis, very appreciated! Indeed it looks like the scaling of the intermediate fix we had deployed for #192 did improve the situation, but still doesn't scale perfectly - although the rate of the performance deterioration became much slower, that's very obviously not enough. We've implemented a second fix that decouples the copy receipt performance from the number of processed receipts and should change the complexity from |
Hey @TSchmiedlechner, thanks for the information and your efforts 🙏 please let me know if we can provide any further input or help you in any way |
Summary
Although the recent change (#192) improved the performance of the signature requests for copy receipts, we're observing an increasing trend for copy receipt latencies, which started to cause performance issues in our integrations.
We have outlets with >200K queue items and a
CNumerator
of nearly 5K. We noticed that the latencies have direct correlation with the number of queue items, which can be seen in the table below.The table shows the correlation of copy receipt latency (red, left y-axis) and number of queue items (yellow, right y-axis) for multiple outlets (x-axis):
Impact
The impact can be seen in the sub-sections for each store. Each sub-section shows the queue size,
CNumerator
, and a latency heatmap for:TL;DR:
Store A
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
When we focus on the copy receipt latencies, it can be seen that the latency of the subsequent copy receipt requests that are made in a certain window is always lower. This can be the result of an index cache on the data layer. The table below shows copy receipt requests that are grouped in equal time windows:
Findings:
Store B
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
Findings:
Store I
Latency heatmap for all receipts:
Latency heatmap for copy receipts:
Latency heatmap for other receipts:
Findings:
Expectations
As SaaS and PaaS providers, our expectation is to have stable latencies regardless of the queue size. Growing queue size will cause delays or even interruptions in receipt printing, and as a result, degraded customer experience.
The text was updated successfully, but these errors were encountered: