Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DietPi-Software | Lemmy: A link aggregator for the fediverse #6434

Draft
wants to merge 15 commits into
base: dev
Choose a base branch
from

Conversation

mtekman
Copy link
Contributor

@mtekman mtekman commented Jun 15, 2023

The Fediverse is really picking up steam since the latest Reddit drama and a lot of people are spawning their own Lemmy instances.

I’m impressed that Lemmy only needs 150 MB RAM and uses virtually no CPU, so it seems ideal for DietPi

This is a WIP based on the in the installation page, using the "From Scratch" approach, since Docker and Ansible are no-nos.

Surprisingly there are no Deb packages of existing builds out there, so compilation is taking ~40mins on my Rpi4

The install encompasses two builds:

  • Lemmy Backend
  • Lemmy UI

If we should split them into seperate things, lemme know!

@MichaIng MichaIng added this to the v8.19 milestone Jun 15, 2023
@MichaIng MichaIng self-requested a review June 15, 2023 16:18
@MichaIng MichaIng marked this pull request as draft June 15, 2023 16:18
@MichaIng MichaIng changed the title [WIP] Dietpi-Software | Lemmy DietPi-Software | Lemmy: A link aggregator for the fediverse Jun 15, 2023
@MichaIng
Copy link
Owner

Sounds like a great addition! Many thanks for working on it.

Comment on lines 4372 to 4374
G_EXEC eval "echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/dietpi-yarn.list"
G_AGUP
G_AGI yarn nodejs
Copy link
Owner

@MichaIng MichaIng Jun 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This shouldn't be needed as you defined our up-to-date Node.js implementation as dependency already:

Suggested change
G_EXEC eval "echo 'deb https://dl.yarnpkg.com/debian/ stable main' > /etc/apt/sources.list.d/dietpi-yarn.list"
G_AGUP
G_AGI yarn nodejs
G_EXEC_OUTPUT=1 G_EXEC npm i -g --no-audit yarn

It includes yarn.

EDIT: Ah nope it does not include yarn, but npm to install it.

mtekman added 5 commits June 16, 2023 23:35
Everything almost working except for not being able to connec to postgres
Pictrs raises some questions thoug
@mtekman
Copy link
Contributor Author

mtekman commented Jun 17, 2023

After much head-slamming, I finally got it to build and for lemmy to talk with the DB.

The whole thing is a stack of:

  • Postgres
  • Lemmy Server (talks to postgres)
  • Pict-rs (picture server, talks to lemmy server)
  • Lemmy UI (talks to lemmy server)

Most of the backend files are contained in the DB so I migrated the location of the DB to the dietpi_userdata/lemmy/postgres folder.

The picture server is optional, but I figured most users would want it. I pointed these to the dietpi_userdata/lemmy/pictrs folder.

Hopefully this will all enable users to migrate their files from one lemmy instance to another more easily, even if they delete the app.

I've also included an update script at /etc/bashrc.d/lemmy.sh to update Lemmy Server, Lemmy UI, and Pict-rs.... though it's really not clear where pict-rs is installed or if it's embedded into lemmy server....

TODO: Figure out Pict-rs install location. Or if it needs to be manually installed.


@MichaIng I'd appreciate if you could run your neatness magic on this, as it is sloppy as hell, and was a pain to write (I tried installing everything initially as the default root user, but that did not work).

@mtekman
Copy link
Contributor Author

mtekman commented Jun 21, 2023

I've also included an update script at /etc/bashrc.d/lemmy.sh to update Lemmy Server, Lemmy UI, and Pict-rs.... though it's really not clear where pict-rs is installed or if it's embedded into lemmy server....

TODO: Figure out Pict-rs install location. Or if it needs to be manually installed.

After a quick discussion in the above link, it seems the pict-rs install/update is not needed as a separate tool because it's embedded in the lemmy_server


# Postgres
## We need to tell postgres that we're doing password auth (md5) and not peer.
G_EXEC sed -i -r '/^local\s+all\s+all\s+peer/i local lemmy lemmy md5' /etc/postgresql/15/main/pg_hba.conf
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. The PostgreSQL version depends on the Debian version: https://packages.debian.org/postgresql
  2. However, it shouldn't be required to alter any config file for this. Something like this should create a user with password authentication:
if [[ $(sudo -u lemmy psql -tAc "SELECT 1 FROM pg_roles WHERE rolname='lemmy'") != 1 ]]
then
	G_EXEC sudo -u lemmy psql -c "CREATE ROLE synapse WITH LOGIN PASSWORD '$pass_database';"
else
	G_EXEC sudo -u lemmy psql -c "ALTER ROLE synapse WITH PASSWORD '$pass_database';"
fi

I think/hope MD5 is deprecated and not the default how PostgreSQL is hashing passwords. MD5 is okay for integrity checks only, not secure at all since decades 😉.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm! I tried this at first, but could not login to the database. I'll try it again just in case I overlooked something

Comment on lines +4545 to +4546
local db_files="$lemmy_userdata/postgres"
G_EXEC mkdir -p "$db_files"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just leave the database at the default PostgreSQL database directory. This is what what most admins will expect. SQLite is the only database engine where it is common to store databases within the client software data directory.

Comment on lines +4548 to +4550
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP DATABASE IF EXISTS lemmy;"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP TABLESPACE IF EXISTS lemmydata;"
G_EXEC_OUTPUT=1 G_EXEC sudo -iu postgres psql -c "DROP ROLE IF EXISTS lemmy;"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will drop all info on reinstalls, which is not intended, is it? In addition to above conditional database user creation, here is how we do in case of Synapse for the database itself:

if [[ $(sudo -u lemmy psql -tAc "SELECT 1 FROM pg_database WHERE datname='lemmy'") != 1 ]]
then
	G_EXEC sudo -u lemmy createdb --encoding=UTF8 --locale=C --template=template0 --owner=lemmy lemmy
fi

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep sorry, leftover from testing

dietpi/dietpi-software Outdated Show resolved Hide resolved
Comment on lines +4357 to +4358
G_EXEC mkdir -p "$lemm_home"
Create_User -d "$lemm_home" "$lemm_user"
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this user require a home directory in /home? For service users it is common to use the software data directory instead, i.e. /mnt/dietpi_userdata/lemmy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where I become unsure. I thought userdata was purely for data and configs generated by the user, but not the binaries themselves.

The lemm_server_path seems to install by default to /home/lemmy/.cargo/bin/, so I figured I would leave it there since it's not data. I did initially try installing to /usr/bin and then setting a sudoers file for lemmy to run it with no password, but I encountered some problems with that (something to do with cargo and rust environment issues).

What's the recommended course of action here?

Copy link
Owner

@MichaIng MichaIng Jun 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, yeah I see. I'd prefer /opt/lemmy for binaries. /home should at best only be used for actual login users, not service users.

In this case, I actually wonder whether Rust really needs to be/should be installed as lemmy user or better just as root and installed to /opt/lemmy? Since we install Rust for root in some other cases, it might reduce some overhead, at least some disk usage. And usually it is not needed and for security reasons even better if the service user has no write access to the binary/executable, but only to a dedicated working directory for configs and data.

I did initially try installing to /usr/bin and then setting a sudoers file for lemmy to run it with no password

Executables are usually word-executable, so every user can execute every command without sudo, or do I misunderstand?

dietpi/dietpi-software Outdated Show resolved Hide resolved
mtekman and others added 2 commits June 22, 2023 12:38
@mtekman
Copy link
Contributor Author

mtekman commented Jun 23, 2023

This PR will take some time, my box is down for reasons unclear to me, and Lemmy is very much a config queen to get running.
Once I get a good stable running config, I'll add more to this.

Thanks for your review so far!

@MichaIng
Copy link
Owner

No problem, take your time. I will also do some tests when I find time. Btw, we have a GitHub Actions workflow to test software installs: https://github.com/MichaIng/DietPi/actions/runs/5366395703

If I am not mistaken, you should be able to trigger them from your fork, when you enable GitHub Actions there. For the Trixie tests to work, the branch would need a rebase onto our current dev.

@StephanStS StephanStS modified the milestones: v8.19, v8.20 Jul 9, 2023
@MichaIng MichaIng modified the milestones: v8.20, v8.21 Jul 29, 2023
@MichaIng MichaIng modified the milestones: v8.21, v8.22 Aug 27, 2023
@StephanStS StephanStS modified the milestones: v8.22, v8.23 Sep 25, 2023
@MichaIng MichaIng modified the milestones: v8.23, v8.24 Oct 21, 2023
@MichaIng MichaIng modified the milestones: v8.24, v8.25 Nov 18, 2023
@MichaIng MichaIng modified the milestones: v8.25, v9.0 Dec 20, 2023
@MichaIng MichaIng modified the milestones: v9.0, v9.1 Jan 20, 2024
@StephanStS StephanStS modified the milestones: v9.1, v9.2 Feb 23, 2024
@StephanStS StephanStS modified the milestones: v9.2, v9.3 Mar 18, 2024
@MichaIng MichaIng removed this from the v9.3 milestone Apr 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants