Memory usage steadily increases without recovery in Electric instance

[sullyo]

We're experiencing steadily increasing memory usage in our Electric instance that doesn't recover until restart.

Current behavior:

• Memory usage shows continuous growth over time
• No automatic recovery of unused memory
• Requires manual restarts to reset memory levels

Environment:

• Electric deployed on render via container
• 2gb memory, 1 cpu

Theory:

• Could be related to total requests. Saw memory exceed limit with more requests

Reproduction steps:

1. Run Electric instance
2. Monitor memory usage over time
3. Observe steady increase in memory consumption
4. Eventually requires restart to recover

Current workaround:
Manual restart of the instance clears the memory back to normal levels.

[balegas]

Hi, we've seen this happen before. We think it is related with how Elixir manages processes memory. We're gonna work on a fix ASAP.

[robacourt]

Please could you try something for us? We think garbage collection is causing the issue so please could you try running it manually:

On the machine with the electric docker image running run

docker exec -it IMAGE_NAME bin/entrypoint remote

(Use docker ps to get the IMAGE_NAME)

This will get you into an elixir shell, where you can run:

:erlang.garbage_collect()

It's worth running this 5 times to get the memory use right down.

Let us know if this helps, if it does we can automate the garbage collection to solve your issue.

[robacourt]

Another thing to try is to let electric use all 2GB of your container's RAM and see what happens. If our hypothesis is correct then hitting the container limit will cause a garbage collection and so electric should flatline at 2GB memory use and continue to operate normally.

If you'd like it to flatline at a lower value, the container's memory can be restricted with the --memory=1g docker run flag or with your cloud provider's settings.

[sullyo]

 :erlang.garbage_collect()
true
iex(electric@)2> :erlang.garbage_collect()
true
iex(electric@)3> :erlang.garbage_collect()
true
iex(electric@)4> :erlang.garbage_collect()
true
iex(electric@)5> :erlang.garbage_collect()
true
iex(electric@)6> :erlang.garbage_collect()
true
iex(electric@s7> 

Ran it a few times here

And not seeing any memory reducing here

[sullyo]

Another thing to try is to let electric use all 2GB of your container's RAM and see what happens. If our hypothesis is correct then hitting the container limit will cause a garbage collection and so electric should flatline at 2GB memory use and continue to operate normally.

If you'd like it to flatline at a lower value, the container's memory can be restricted with the --memory=1g docker run flag or with your cloud provider's settings.

Going to try something like 1.5g

[robacourt]

I've reproduced an issue that fits this description and have a PR ( #2240 ) out that fixes it. I'll let you know as soon as it's been approved and we have a release for you to try.

[robacourt]

@sullyo We've just released version 1.0.0-beta.4 which includes a potential fix for this issue. Please could you try it to see if it helps?

It hibernates shapes if they are not in use for 30 seconds, reducing memory consumption. This value is configurable with the ELECTRIC_SHAPE_HIBERNATE_AFTER environment variable, so to hibernate shapes after their idle for 10 seconds set ELECTRIC_SHAPE_HIBERNATE_AFTER to 10s

[sullyo]

@robacourt I'll update and give it a try!

[sullyo]

This seems to have resolved the issue. I updated our memory to 2gb and with the latest changes we're seeing at most 15% memory usage!