-
Notifications
You must be signed in to change notification settings - Fork 123
Data Storage Arch
The locker requires two primary use cases for data storage, historical archiving and live state.
The intent of the historical archive is to ensure that all collected data is available in a base simple format. This allows for reconstruction to any new required format and for old or recovered disks to have future use. The preferred format is JSON, but if this is technically not possible another format may be used, but it is stressed that it should be simple (near text) and visually inspectable in the future. The archive should contain all the changes and actions on data, not just the current state of the data, meaning deletes, updates, resyncs or anything else are individual records in the historical archive.
It's important to note that while the historical archive is a simple format this requirement does not preclude an index from being created into the data. As design grows, other needs for accessing the archive may arise and the use of an independent index of the data is able to enhance this process. A simple example would be using sqlite to store the ids, timestamps, and offsets into the archive in order to quickly retrieve specific sets of data
Collections need to maintain a historical archive of any extra data they maintain for aggregated data, but not the raw data itself.
Both Connectors and Collections should have their current state in an easily accessible and stored state. The live state is the current snapshot of the service, not the incremental changes. For example, your current contact list rather than all of the changes that have happened to your contact list in that service. This is needed first because services should be able to be stopped and restarted at any time, but also to support their exposed API as best as possible. Key points:
- Live state does not require an independent data store.
- There is no mandated platform data store, rather each data type should be carefully considered and use the data store that can best model the data and ease access to it.
- Data store direct access is limited to the service using it and core, if services share a common backend data store they should not be accessing the data of the other service directly.