Expose metadata about sync status of shapes & records

[KyleAMathews]

I'm sure this is on your roadmap but I haven't seen anything public so I thought I'd throw out my use case / thoughts.

I've been experimenting with how to specify the data that needs synced for specific routes in my app. For most apps I can think of building, I'd want to sync all data but it'd be ideal to only load initially the precise data needed for that route and then lazy load other data in the background.

Also I'd want to be able to distinguish between several states a shape can be in i.e. a shape never been synced, it's been synced in the past but while the app has been closed, is stale by 12 hours, and the shape is fully synced up with the backend. Then in my route loader function I could say either wait until fully synced or if the data isn't that stale, immediately render rather than waiting for the initial sync to finish (this has a big impact on the app loading time).

I could imagine a sync manager lib in userland that works similar to this:

      {
        path: `/video/:videoId`,
        element: <Video />,
        loader: async ({ params }) => {
          await syncManager.load([
            {
               shape: () => db.users.sync({ where: { id: params.user_id }}),
               isDone: (shapeMetatdata) => shapeMetadata.msSinceLastSynced < 12 * 60 * 60 * 1000 || shapeMetadata.state === `SYNCED`,
            },
            {
               shape: () => db.youtube_videos.sync({ where: { id: params.video_id }}),
               isDone: (shapeMetatdata) => shapeMetadata.msSinceLastSynced < 12 * 60 * 60 * 1000 || shapeMetadata.state === `SYNCED`,
            },
          ])
          return null
        },
      },

We'd want similar metadata exposed for individual records, especially for tracking individual records e.g. as Bm_Dorian wrote earlier on Discord:

I also have a question. Is it possible to get status if specific record was synced with BE ? Lets say situation: I start up app online, sync and go offline. While offline I insert some records, I would like to show to the user some status of that record that it's not yet on backend. I guess I can achieve that to just save last sync time and compare with time when record was created if still offline, but just wanted to know if there is something for that

or maybe just have boolean field that is when created offline has false value, but on write to backend backend just flips the boolean

[balegas]

That's spot on. We need to provide APIs for developer to handle shapes and transactions, etc. The status of a requested shape: "Has the shape that I requested synched already?". The status of the overall database: "Is data too stale that I don't want to keep diverging from server?".

We've been looking into a transaction manager that would move a transaction through multiple states: local, sent(/accepted), acked.
On top of that we can have more states to allow the developer to handle exception scenarios... like conflicts, and rejected transactions because of permissions, etc.... things get more and more interesting.

I haven't put so much thinking into the state of individual shape state beyond being requested and delivered. All updates come down through a single (causally-ordered) replication stream following commit order on the server -- shapes are as stale as late you are with the replication stream. However, it would be still interesting to know the last time data for a shape was delivered -- I have to think more :).

[KyleAMathews]

Cool! The transaction manager sounds very useful. Lots of apps will probably want a spinner going w/ full sync indication for sensitive stuff. I've also seen apps that list how many transactions still need to sync which is an interesting indicator when you're offline (and fun to see them going down when you get back online).

I can probably figure out the data I need for the sync manager in userland — I'll give that a go soonish and report back.

[balegas]

The transaction manager is something that is not very hard to implement. we have most of the metadata available already, but requires changes to the interface and we want it to serve write rejections and explicit conflict resolution. we haven’t rushed to implement it but at the same time is not very far :)

[KyleAMathews]

👋 we've been working the last month on a rebuild of the Electric server over at a temporary repo https://github.com/electric-sql/electric-next/

You can read more about why we made the decision at https://next.electric-sql.com/about

We're really excited about all the new possibilities the new server brings and we hope you'll check it out soon and give us your feedback.

We're now moving the temporary repo back here. As part of that migration we're closing all the old issues, PRs, and discussions. We really appreciate you taking the time to start this discussion! If it's still relevant in the new system, please reopen.