You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When executing many requests (the number varies and depends on the total workload of the code submitted) I am getting a lot of exceptions thrown from Docker APIs.
I have classified those errors in three categories:
Inactive containers that cannot be removed (because they are not stopped)
The error I am getting is: Unhandled rejection Error: (HTTP code 409) unexpected - You cannot remove a running container XXX. Stop the container before attempting removal or force remove. and docker stats outputs
So it seems some containers are never stopped and thus they cannot be removed, however they do not occupy any resources. (JavaBox by the way is not affected, meaning that it answers back the expected result). I am not 100% sure if that bug has performance issues, though. Note: I am using container.stop(); and container.remove({f: true}) for each container. Option {f: true} made things better but not completely.
Containers that cannot be stopped and continue to run even after JavaBox replies back
The error in this case is Unhandled rejection Error: (HTTP code 500) server error - Cannot stop XXX: Cannot kill container XXX: rpc error: code = 14 desc = grpc: the connection is unavailable and the output of docker stats is something like:
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
7d9d77451f8e 9.60% 47.36MiB / 7.682GiB 0.60% 2.59kB / 0B 0B / 32.8kB 16
meaning that those containers are active and continue to execute (that mainly happens when submitted code requests contain infinite loops), sometimes taking up extreme amounts of CPU usage. Of course those slow down the whole system, but JavaBox still replies back with the appropriate results.
Connection error exceptions
The errors that have appeared so far are Unhandled rejection Error: (HTTP code 500) server error - endpoint with name YYYY already exists in network bridge, where YYYY is a weird name like wonderful_jones, Unhandled rejection Error: (HTTP code 500) server error - transport is closing, or Unhandled rejection Error: (HTTP code 500) server error - grpc: the connection is unavailable.
Those start to appear when the concurrent submissions are way more (than in the previous cases) and when those happen JavaBox does not reply back to the client (leaving the connection open).
The text was updated successfully, but these errors were encountered:
Also note: When running many multiple submissions and/or a lot of workload my system CPU usage tops at 100%, so maybe my OS is killing(?) or interfering in other ways with the containers causing the errors above.
When executing many requests (the number varies and depends on the total workload of the code submitted) I am getting a lot of exceptions thrown from Docker APIs.
I have classified those errors in three categories:
The error I am getting is:
Unhandled rejection Error: (HTTP code 409) unexpected - You cannot remove a running container XXX. Stop the container before attempting removal or force remove
. anddocker stats
outputsSo it seems some containers are never stopped and thus they cannot be removed, however they do not occupy any resources. (JavaBox by the way is not affected, meaning that it answers back the expected result). I am not 100% sure if that bug has performance issues, though.
Note: I am using
container.stop();
andcontainer.remove({f: true})
for each container. Option{f: true}
made things better but not completely.The error in this case is
Unhandled rejection Error: (HTTP code 500) server error - Cannot stop XXX: Cannot kill container XXX: rpc error: code = 14 desc = grpc: the connection is unavailable
and the output ofdocker stats
is something like:meaning that those containers are active and continue to execute (that mainly happens when submitted code requests contain infinite loops), sometimes taking up extreme amounts of CPU usage. Of course those slow down the whole system, but JavaBox still replies back with the appropriate results.
The errors that have appeared so far are
Unhandled rejection Error: (HTTP code 500) server error - endpoint with name YYYY already exists in network bridge
, where YYYY is a weird name like wonderful_jones,Unhandled rejection Error: (HTTP code 500) server error - transport is closing
, orUnhandled rejection Error: (HTTP code 500) server error - grpc: the connection is unavailable
.Those start to appear when the concurrent submissions are way more (than in the previous cases) and when those happen JavaBox does not reply back to the client (leaving the connection open).
The text was updated successfully, but these errors were encountered: