smash memory leak in v1.3.0

[johnalotoski]

A memory leak is observed with smash v1.3.0 whereby all RAM available is slowly consumed by the smash process until an OOM is forced, generally resulting in corrupt ledger state files and smash being unable to restart without manual intervention.

See JIRA ticket: SRE-149

[ksaric]

Noted, thanks @johnalotoski. I would wait until we deploy the new version or the one behind that one to start a serious investigation about this.

[ksaric]

Left SMASH running with the current changes overnight (for purposes of testing whether my current changes work), don't see an increase in memory usage, capped on 202 Mb, on the launchpad network.

[johnalotoski]

Thanks @ksaric. The leak might be related to load. We don't see the leak on low load deployments.

[mmahut]

I still see this on 1.4.0.

[ksaric]

@mmahut Do you see this on a SMASH instance that has a high amount of traffic? Or is this something you see gradually taking up more and more memory?

[mmahut]

@ksaric it is gradual. This smash instance has no http traffic to it at all, I'm just using the resulting database.