Enable npm Module Installation with Temporary Memory Enhancement

[MarianRaphael]

Epic

#626

Description

As a FlowFuse user,

I want to be able to temporarily enhance the memory allocation of my instance (automatically) during npm module installations,

So that I can successfully install necessary modules even if my default instance memory is insufficient, as long as the increased memory requirement is only during the installation process.

Which customers would this be availble to

Everyone - CE/Starter/Team/Enterprise

Acceptance Criteria

• detect when the available memory is insufficient for an npm module installation.
• temporarily increase the memory
• The temporary memory increase should be automatically reverted to the original limit upon completion of the installation process.

Have you provided an initial effort estimate for this issue?

I have provided an initial effort estimate

[hardillb]

This is going to be very difficult (if not impossible) to implement

1. The npm install is all handled in the Node-RED process, there is no place to hook this to trigger a memory increase
2. While kubernetes supports dynamic pod limit changes (see here, this is only available from Kubernetes v1.27 onwards
3. Docker does not allow for changing the memory constraints applied to a container
4. The amount of memory used is dependant on a number of different factors
   1. The node(s) being installed (the number of dependencies it has)
   2. The number of nodes already installed (de-duplicating dependencies)

Potential alternative approaches:

1. We may want to look at increasing the memory limits on the small stack on FF Cloud
2. Look to see if we can limit the memory used by npm (I have looked in the past and not found a solution)

[hardillb]

Having looked at the Node-RED code there may be some hooks in the install process, but will need to talk to Nick to see if these can be used

[Steve-Mcl]

@hardillb for kube, would a pre-emptive approach be a possible first iteration? e.g. upon start up, add xx% memory. Once running, reduce to pre-set value?

This is of course only viable for installs at fire up (i.e. does not take care of installations performed in the Node-RED editor palette)

[hardillb]

@Steve-Mcl the main problem is not the startup install, it's it failing to install nodes at run time.

Also as noted, with the currently used version of kubernetes we just can't change the memory limits.

[hardillb]

@ppawlowski mentioned that we may be able to change resource limits vs request for the Instance Pod on K8s.

Testing this in the staging environment with adding 128m extra to the limit value allows the installing of the openapi-red package, but it does take over 10mins on a Small instance.

This is not a great UX but is better than not installing at all.

I have opened node-red/node-red#4622 to try and improve the NR side of the UX things.

This extra memory should only be available to the npm or nr-launcher process (not NR) since we set the max old space size for the NR process in the launcher to be slightly less than the memory size (we may be able to increase this based on these findings)

(this is of course limited to K8s installs, not docker which has similar limitations)