Persistent Storage

[sammachin]

Description

Users will want to store data within their flows and retrieve it later.
Due to the ephemeral nature of the filesystem in our containers the file node are not suitable (see #318 )

Considerations of how to offer persistent storage within FlowForge:

• Dedicated storage, possibly an S3 bucket?
• Storage can be scoped to a team or to a project (could this be part of the stack perhaps?)
• Custom nodes that replace the file nodes
• Some sort of File Manager in the forge UI for users to upload/download and delete outside of Node-RED
• A way to track and report on usage of storage so that users can see how much they have used and admins can monitor & bill accordingly.

[hardillb]

Still think this is a bad idea.

Persistent context storage should be first and then access to "remote" file storage e.g. S3 buckets.

But you have reminded me to pick up that issue about removing the file nodes

[knolleary]

This needs putting into context (pardon the pun) with #212

Short term, the answer is 'use an appropriate set of database nodes with a 3rd party service'. That's an entirely natural Node-RED way of doing it.

Whether the FF platform provides a built-in storage service is a significant decision and would have a significant cost associated with it.

[sammachin]

For a user however there is a not insignificant overhead in setting up an account with another cloud storage provider eg AWS/S3, mongoDB etc and then managing those credentials and getting them into FlowForge.
This is where we have an opportunity to add value (and charge for that) by making this process as streamlined as possible.
If we create the storage for them and add it to their FlowForge Cloud bill along with pulling in the credentials this reduces friction

[knolleary]

I agree this is value to be had with this feature and it should get considered along with everything else.

I wanted to flag at an early stage that it comes with a significant cost which needs to factor into its prioritisation.

[sammachin]

Call with customer today expressed a strong interest in having this, keen for it to be integrated into FF rather than just a "BYO S3" for things like credential management and billing

[knolleary]

I'm taking this out of the 1.0 iteration as it is a very broad epic that can be addressed in lots of different ways.

In the short-term we have #998 planned for 1.1 which will come to address the core issue of having persistent storage options within a flow.

Being able to manage the files from the FF platform UI is out of scope of that issue - so something this issue could be refocussed on and kept on the backlog depending on feedback and demand.

[MarianRaphael]

Closing this epic here, because of: #3056 and #1779