[feature] Centralized server

[johnnybubonic]

Hi! I'm the guy in IRC that said he wanted to POC a centralized synchronization server component for GTG (r00t^2).

I'm not opening this issue to request it since I said I'd work on it, but I did want a static, async form of discussion so I could get some feedback and other input as I work on it.

Before I start, would this be better suited as a plugin or would GTG want to adopt it proper into its core? If the latter, there's two ways it can go:

1. A minimal environment that can be run standalone. If this is preferred, I'd recommend making it an entirely different repository/project to keep the code separate for the server. The client will be a plugin. (This is the better option)
2. A locally running instance that, by default, only listens on localhost (and/or an IPC socket) that the client (GUI) then connects to, with the option of also opening a port that can listen on an external interface.

I'd recommend option 1, because otherwise I believe GTK libraries would need to be installed on a headless server.

Goals

The centralized server SHOULD:

• Provide a WSGI or CGI interface (so it can then be handled by e.g. Nginx and uWSGI, simplifying things like TLS transport — HTTPS, ACL, potentially authorization/authentication — see below, etc.)
  • HTTP REST opens up a plethora of possibilities for other clients/plugins as well.
  • Follow CRUD. Following "object" refers to a task, full task list, project, tag, etc. Assuming HTTP REST:

Request Method	Action
POST	Upload (non-destructively) an object to a database (or filesystem store?)
GET	Deliver an object stored in a database (or, depending on endpoint, a list of available objects
PUT	Update (by replacing) an object
PATCH	Update (by modifying an object
DELETE	Remote an object

Ideally, there MAY be a parameter to PUT (and/or PATCH) such as allow_create=1 that would create an object with the specified data if it does not yet exist, rather than failing the update operation.

The centralized server MAY:

• Use storage in a database or filesystem?
  • I'm tempted to just use one or the other for ease in writing and maintaining.
    • A database might be nice as would allow other outside clients access, potentially.
    • But then again, that's sort of the entire point of an API.
    • Though it would be much more easy to parse the data and it could then just generate an e.g. modified result to deliver from that.
• Have an authentication and realm system.
  • This may be something we'll want to leave to the web daemon (see above). i.e. Flask apps have access to the user, password, and realm passed in via the web daemon, but it'd be much easier to just handle a "realm" and let the web daemon handle authentication.
  • Why a realm? This would allow for shared task lists etc. among a group of different identities. This may be an over-complication, though.

Prereqs

First, I have a PR open to standardize the XML that GTG uses via XSDs (#431 #437 #438). Schemas are going to be VERY important if we're going to be merging/modifying data, as then we can be sure of data integrity and how that data should be handled/naturalized into python.

[johnnybubonic]

Thoughts? Comments, suggestions?

[diegogangl]

I'd recommend option 1, because otherwise I believe GTK libraries would need to be installed on a headless server.

I would go for option 1 too. I'm not sure option 2 is possible with Flatpak

Use storage in a database or filesystem?

If you want to do "real time" syncing you might have to use a database, otherwise you would have to coordinate when to write to the file.

[johnnybubonic]

If you want to do "real time" syncing you might have to use a database, otherwise you would have to coordinate when to write to the file.

and locking and/or inotify is a pain and would be a lot more intrusive to existing code too, yeah. good point. does everything else look good? am i missing anything obvious?

[diegogangl]

If you want to do "real time" syncing you might have to use a database, otherwise you would have to coordinate when to write to the file.

and locking and/or inotify is a pain and would be a lot more intrusive to existing code too, yeah. good point. does everything else look good? am i missing anything obvious?

Nothing I can think of. I can't really help you with the auth part since I don't have much experience there.
Also forgot to mention this should go into GTG as a plugin

[johnnybubonic]

Also forgot to mention this should go into GTG as a plugin

Yep, figured by the plugins label. That makes sense, sure. You mean the client component, yes? Once stabilized, I'd hope it'd be a core plugin as I'd imagine a lot of people would want to use it since it's been requested a lot.

The site says the plugin development guide may be outdated but it looks like it was last updated in April; I'm assuming the website rather than the doc is outdated?

The server component I'm envisioning as a separate @getting-things-gnome repo once I have something maintainable. I was thinking of calling them GTGSync (client plugin) and GTGSync-server respectively.

[diegogangl]

The site says the plugin development guide may be outdated but it looks like it was last updated in April; I'm assuming the website rather than the doc is outdated?

The docs are outdated, mostly about things like the toolbar (which we don't have anymore).

The server component I'm envisioning as a separate @getting-things-gnome repo once I have something maintainable. I was thinking of calling them GTGSync (client plugin) and GTGSync-server respectively.

Sounds good 👍

[primal-buddhist]

Very happy to help test this, document gigs etc.

Joplin does use data oriented markdown files on a shared storage and conflicts between phone and laptop are very rare. Wondering if any of their strategy is reusable.

[johnnybubonic]

@primal-buddhist Thanks! it'll be later than I expected (or wanted) to have anything testable since #279 being ratified and incorporated is kind of a definite blocker (so version comparisons can be made, etc. between clients' datasets), but watch this space! I'll comment when there's something ready for testing. Much appreciated!

I think @diegogangl's plan is to allow for Markdown in the tasks' <content> field (what the "body" of the task is), so that should at least be some parity.

[oyren]

Wouldn't it make more sense to use an existing synchronization application and try to implement gtg here. You would have more maintainers and clients (e.g. Android/IOS). The disadvantage would be that it would not be 100% GTG based. Am I still missing something here?
Caldav or EteSync would be an option, see #407.

[johnnybubonic]

@oyren I think they should be separate implementations. You'd lose fidelity (tags, subtask associations, etc.) by trying to wrangle GTG tasks-as-a-concept into either e.g. CalDAV or EteSync.

This doesn't need to be that complex, and there'd ideally be an actual Android/iOS client separate from the other projects. Heck, this could open that up as a possibility. But the formats supported by the above projects do not afford the associations that are key to what makes GTG stand out.

[oyren]

@johnnybubonic I also see the advantages of a GTG-tailored synchronization, but I have my doubts if the project is big enough (it just came back from the dead) to handle this task.
I hope so of course 😉 .
Another option would be to fork Etesync and adapt it to the needs of gtg, e.g. saving the work of writing authentication.

[johnnybubonic]

I don't think necessarily it's an issue of it being big enough. Right now I'm waiting for the new spec to be formalized (#279) so I can have proper XSDs to work with (#438) so 1.) data can be validated before it's merged, and 2.) the server can do format version comparisons. :)

[tkdchen]

Does the centralized server aim to be a standalone server running somewhere to allow GTG users to sync up their tasks? Or, it is just a web daemon running for personal task sync up purpose?

[johnnybubonic]

Does the centralized server aim to be a standalone server running somewhere to allow GTG users to sync up their tasks?

Sync (and by extension share), yes

Or, it is just a web daemon running for personal task sync up purpose?

It could certainly be used as that, as the above would certainly allow it. It wouldn't be a full web daemon, presumably/ideally; it'd be a flask app or something of the like. I'd like to leave things like authentication to a proper robust web daemon/reverse proxy (like Nginx) to keep things simple.

[tasn]

Lead EteSync dev here.

@oyren

Another option would be to fork Etesync and adapt it to the needs of gtg, e.g. saving the work of writing authentication.

What needs of GTG does EteSync not cover? We now also have Etebase which is just the generic encrypted store that powers EteSync, so I'm pretty sure it can at least work with that.
EteSync uses iCal (VTODO) internally, it's similar to CardDAV in that sense (just encrypted and much simpler), so I think GTG's needs can be covered. You also get the benefit of interoperating with the rest of the EteSync ecosystem including EteSync web, our DAV bridge, Tasks.org, Android and iOS apps and etc.

Btw, there's now an EteSync module for Evolution-data-server, have you considered maybe just supporting EDS's tasks instead of having a new backend? That should already provide GTG with so many backends for free...

I'll also mention this in #407

[oyren]

@tasn

Another option would be to fork Etesync and adapt it to the needs of gtg, e.g. saving the work of writing authentication.

What needs of GTG does EteSync not cover? We now also have Etebase which is just the generic encrypted store that powers EteSync, so I'm pretty sure it can at least work with that.

The sentence referred to @johnnybubonic answer to the suggestion from me to base the whole thing on Etesync.

I think they should be separate implementations. You'd lose fidelity (tags, subtask associations, etc.) by trying to wrangle GTG tasks-as-a-concept into either e.g. CalDAV or EteSync.

[tasn]

The sentence referred to @johnnybubonic answer to the suggestion from me to base the whole thing on Etesync.

Ahh sorry.

I think they should be separate implementations. You'd lose fidelity (tags, subtask associations, etc.) by trying to wrangle GTG tasks-as-a-concept into either e.g. CalDAV or EteSync.

iCal supports subtasks (RELATED-TO) and tags (CATEGORIES) and etc. iCal also supports extensions using custom props.

iCal is not great, don't get me wrong, but the reason why we went with it in EteSync is that you get portability with other implementations (like e.g. EDS, Tasks.org, KDE for free).