OPA data and policy storage

[prudhvid]

Hello,
From the discussions it seems like OPA stores both policy and data in memory, if that is the case, when we do a create and update API to OPA using PUT /v1/data/{path:.+} or PATCH /v1/data/{path:.+}
Here are my questions regarding this?

1. Where does the data go to? If all of this is in-memory what happens if opa crashes and restarts?
2. Does opa internally store them in files? If yes, could we configure it to store at a given place?

The storage configuration https://www.openpolicyagent.org/docs/latest/misc-disk/ only talks about data that does not fit into memory. But then if opa crashes how does the services know which data to push and update?

[srenatus]

Hey there.

1. Where does the data go to? If all of this is in-memory what happens if opa crashes and restarts?

Since the memory will be in a pristine state when OPA restarts, that data will be gone.

2. Does opa internally store them in files? If yes, could we configure it to store at a given place?

Disk storage can be used for data persistency across restarts.

There are also other means to do that: Bundles. If your OPA service is configured to load bundles, it'll do so on every (re)start and periodically while it's running. Bundles can even be persisted on disk, so that if an OPA service restarts, it'll be able to use the previously fetched bundle right away.

⚠️ OPA should never, not even when using the disk storage backend, be used as the sole source of truth. Whatever data you put into OPA, you should be able to reassemble that data if need be. OPA does not come with backup or disaster recovery mechanism.

See the Bundles documentation for all the details.

[Vighi88]

Hello,

Is there a way we can use S3 bucket or other Storage instead of local disk storage ?

[srenatus]

You can use an s3 bucket for storing bundles. Using it as backing store for data more directly isn't possible.

[samlouiscohen]

@srenatus on the same note, I'm curious what this flow looks like in practice. If I have a small-to-medium amount of dynamic data used for rbac-style access control (think roles & role-subject-resource assignments) then it seems like "Option 4: Push Data" in the docs is the suggested pattern but I'm not super clear on what the replicator would look like. If the data is to be edited (for example, a new userX is given OWNER role over some resource) then is the expectation that the service would first write to the database (source of truth) and then sequentially write to the OPA in-memory store? Previously I was treating disk-storage (with badger) as the source of truth which I know is not recommended (but it did make every update to the data a single operation).

Are there suggestions on how to properly handle these two operations (write to database/write to in-mem store) going out of sync if there is a delay with the replicator? For example, maybe userA removes userB's access but then userB removes userA's access if this isn't reflected in the in-memory store quickly enough.

I appreciate any insight you could provide.