Can I use reader with a database other than SQLite?

[js-compilatrum]

Official doc states with configuration on local filesystem:

reader = make_reader('db.sqlite')

Is possible configure instead use file connect to external database, like for example SQLAlchemy provide when you define driver and necessary creds:

engine = create_engine("mysql+mysqldb://anthony:lemonserver@localhost/test")

Or I have to write it yourself?

[lemon24]

Currently, reader only supports SQLite.

That said, it is mostly possible to plug in alternative storage implementations. Update: Starting with reader 3.10, it is possible to use an alternate storage implementation via make_reader(_storage=...); the interface is documented in Internal API # Storage.

An SQLAlchemy-backed one would be especially useful, since it would make reader work on a variety of database engines; alas, I do not have enough time to support it at the moment.

If you are interested in writing your own, please let me know, so I can do the few necessary API changes; more details in #168 (comment)

[js-compilatrum]

As I can see it is simply override Search and Storage classes. Are you plan any changes in internal API for this classes or it is stable?

SQLite for smaller number of feed is really good choice. I only see two problems:

1. running on something like Raspberry Pi - a lot of reading / writing can kill SD card
2. save file outside folder on remote share (for example Samba) can affect overall performance and even create crashing / missing data.

[lemon24]

As I can see it is simply override Search and Storage classes.

More or less, yes.

Are you plan any changes in internal API for this classes or it is stable?

They are not guaranteed to be stable like the public API is, but they haven't changed in a long time, and when they have, it's mostly been additions. They are "mostly done", but because there's no other implementation (yet), I'm reluctant to lock in potential undiscovered mistakes. Long-term, the plan is to make them fully stable / part of the public API.

I can promise to document the changes in the changelog, and that I'll do my best to keep the internal API stable.

As mentioned in #168 (comment), I can start by extracting the required methods into protocols, so it's clear what's part of the (internal) API and what's not; I'll try to do this in the following month.

---

SQLite for smaller number of feed is really good choice.

I don't have definitive answers here, but I'll write some thoughts below in case it helps.

For reference, I use the web app on a t4g.nano AWS instance (the smallest one available), and with 180 feeds and 21k entries it still runs quite well (some example timings).

1. running on something like Raspberry Pi - a lot of reading / writing can kill SD card

I expect SQLite would have a similar amount of writes to any other database engine (assuming it's also backed by the SD card). SQLite implements locking by writing a block to disk (not just in memory, like a database server may), so that may result in a few more writes.

That said, reader does its best to write as little as possible, e.g. it won't update an entry if it didn't change.

2. save file outside folder on remote share (for example Samba) can affect overall performance and even create crashing / missing data.

NFS does work if fcntl() file locking works on that particular implementation, but it may indeed be slow due to the added network hop (likely slower than a database server).

For "live" backups, Litestream looks quite compelling, but I haven't had a chance to try it yet.

[lemon24]

I can start by extracting the required methods into protocols, so it's clear what's part of the (internal) API and what's not

Opened #325 to track this.