Adding Combined modlog #5253

dessalines · 2024-12-10T00:15:41Z

Context: #2444

Must come after #5251

Notes:

This removes all the individual modlog fetches into a single combined one, with a type_ filter.
17 tables joined in total, with sometimes complicated joins required to fetch the derived data.
- For example, mod_remove_post doesn't have the modded_person_id directly on it, it needs to get it from the post.creator_id column. I've tried to keep the joins as organized and well-commented as possible.
I've added a ton of tests, filtering different slices of all the filter types: community, post, comment, mod/admin, modded_person, and type. Its verbose, but its something we didn't have before.
Some of the mod views had missing fields that seemed important (occasionally it was missing a community or the modded person). I've added these.
I discovered a few errors and inconsistencies that I cleaned up.
There are plenty of breaking changes here, which includes the types coming back for the v3 routes. I am not going to do backwards compatibility for these types. That will have to be handled by API libraries switching on v.0.19 vs v.0.20.

* add pagination cursor * store timestamp instead of id in cursor (partial) * Revert "store timestamp instead of id in cursor (partial)" This reverts commit 89359dd. * use paginated query builder

…into combined_tables_2

- Separating the profile fetch from its combined content fetch. - Starting to separate saved_only into its own combined view.

dessalines · 2025-01-06T01:25:36Z

crates/api/src/site/mod_log.rs

The API changes here let us remove all these view types.

dessalines · 2025-01-06T01:26:18Z

crates/api_common/src/site.rs

  pub type_: Option<ModlogActionType>,
  #[cfg_attr(feature = "full", ts(optional))]
-  pub other_person_id: Option<PersonId>,
+  pub modded_person_id: Option<PersonId>,


Changed the name of this everywhere I could, to be more consistent.

dessalines · 2025-01-06T01:27:11Z

crates/api_common/src/site.rs

@@ -168,23 +150,7 @@ pub struct GetModlog {
 /// The modlog fetch response.
 // TODO this should be redone as a list of tagged enums


dessalines · 2025-01-06T01:29:00Z

crates/api_common/src/utils.rs

+    let posts_mapped = &post_modlog.iter().filter_map(|p| {
+      if let ModlogCombinedView::ModRemovePost(v) = p {
+        Some(v)
+      } else {
+        None
+      }
+    });


These were necessary for tests. If anyone's wondering how to force the combined types into its specific one, you can use .filter_map like this above.

dessalines · 2025-01-06T01:29:44Z

crates/db_schema/replaceable_schema/triggers.sql

@@ -761,3 +761,78 @@ CALL r.create_person_saved_combined_trigger ('post');

 CALL r.create_person_saved_combined_trigger ('comment');

+-- modlog: (17 tables)


Replaceable schema changes below.

dessalines · 2025-01-06T01:31:51Z

crates/db_schema/src/impls/mod_log/admin.rs

@@ -19,25 +39,23 @@ use diesel_async::RunQueryDsl;
 impl Crud for AdminPurgePerson {
  type InsertForm = AdminPurgePersonForm;
  type UpdateForm = AdminPurgePersonForm;
-  type IdType = i32;
+  type IdType = AdminPurgePersonId;


The changes in all these db_schema files are just adding the type, and some cleaner imports. So no need to read through them all.

dessalines · 2025-01-06T01:34:13Z

crates/db_views_moderator/src/admin_allow_instance.rs

All these views are unecessary, as we should always used the combined view for them. Eliminates a lot of code 🙌

dessalines · 2025-01-06T01:35:11Z

crates/db_views_moderator/src/modlog_combined_view.rs

The main work of the PR is in this file. What's most important is that I've added a lot of tests to make sure all these are working.

dessalines · 2025-01-06T01:37:21Z

migrations/2024-12-08-165614_add_modlog_combined_table/up.sql

+-- Not doing a union all here, because there's way too many null columns
+INSERT INTO modlog_combined (published, admin_allow_instance_id)
+SELECT
+    when_,


lmk if you think this is worth it. Its a lot of work but I'll do it if you think I should.

dessalines · 2025-01-06T01:38:23Z

This is ready for review now.

cc @Nutomic @dullbananas

dessalines and others added 30 commits November 26, 2024 09:27

Combined tables try 2

39b3ec2

Finishing up combined report table.

1ff8ae0

Fix ts optionals.

2351c7a

Adding tests, triggers, and history updates for report_combined.

0ba961f

Adding profile.

d8dda44

Merge remote-tracking branch 'origin/main' into combined_tables_2

d68c0a6

Merge remote-tracking branch 'origin/main' into combined_tables_2

a9f28af

Merge remote-tracking branch 'origin/main' into combined_tables_2

3adadf9

Add cursor pagination to report_combined view (#5244)

588e1f6

* add pagination cursor * store timestamp instead of id in cursor (partial) * Revert "store timestamp instead of id in cursor (partial)" This reverts commit 89359dd. * use paginated query builder

Merge branch 'combined_tables_2' of https://github.com/LemmyNet/lemmy …

f25d346

…into combined_tables_2

Fixing migration and paged API.

7fdbb58

Using dullbananas trigger procedure

00beccd

Removing pointless list routes, reorganizing tests.

1776de2

Fixing column XOR check.

612c2e9

Merge remote-tracking branch 'origin/main' into combined_tables_2

5a8b9de

Forgot to remove list report actions.

0c854d2

Cleanup.

802a8a6

Merge remote-tracking branch 'origin/main' into combined_tables_2

4674821

Merge branch 'main' into combined_tables_2

91ca6d3

Use internal tagging.

4cac67c

Merge branch 'combined_tables_2' of https://github.com/LemmyNet/lemmy …

83988cd

…into combined_tables_2

Fixing api tests.

82a14fc

Adding a few indexes.

1f28407

Fixing migration name.

34a440a

Fixing unique constraints.

1a739a1

Merge remote-tracking branch 'origin/main' into combined_tables_2

e1affa8

Addressing PR comments.

921d532

Start working on profile combined

724856d

Adding views and replaceable schema.

1053df1

A few changes to profile view.

32b5411

- Separating the profile fetch from its combined content fetch. - Starting to separate saved_only into its own combined view.

dessalines added 23 commits December 12, 2024 09:39

Merge remote-tracking branch 'origin/main' into combined_tables_2

1e6b3fb

Addressing PR comments.

c72d5e8

Merge remote-tracking branch 'origin/main' into combined_modlog

942da64

Removing serialization

7c962d6

Merge remote-tracking branch 'origin/main' into combined_tables_2

ae9da4d

Merge branch 'combined_tables_2' into combined_profile

af543cc

Removing serialization

b4ce1d9

Merge remote-tracking branch 'origin/main' into combined_tables_2

154cea9

Merge branch 'combined_tables_2' into combined_profile

0b514c5

Merge remote-tracking branch 'origin/main' into combined_profile

a1792bb

Fixing duped trigger.

cfd302d

Remove saved_only test.

9a0e050

Remove pointless post_tags types.

776c500

Merge remote-tracking branch 'origin/main' into combined_profile

bd5210a

Merge remote-tracking branch 'origin/main' into combined_profile

8d856a0

Remove pointless index.

4bc4876

Merge remote-tracking branch 'origin/main' into combined_profile

918c5e8

Changing published to saved for person_saved_combined.

2244fd0

Merge branch 'combined_profile' into combined_modlog

b6d0ecf

Merge remote-tracking branch 'origin/main' into combined_profile

5cf0bb8

Merge branch 'combined_profile' into combined_modlog

b46fdab

Merge branch 'main' into combined_modlog

e5e11c1

Removing comment.

03e876b

dessalines commented Jan 6, 2025

View reviewed changes

dessalines marked this pull request as ready for review January 6, 2025 01:37

dessalines requested review from phiresky, dullbananas and SleeplessOne1917 as code owners January 6, 2025 01:37

dessalines requested a review from Nutomic January 6, 2025 01:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Combined modlog #5253

Adding Combined modlog #5253

dessalines commented Dec 10, 2024 •

edited

Loading

dessalines Jan 6, 2025

dessalines Jan 6, 2025

dessalines Jan 6, 2025

dessalines Jan 6, 2025

dessalines Jan 6, 2025

dessalines Jan 6, 2025

dessalines Jan 6, 2025

dessalines Jan 6, 2025

dessalines Jan 6, 2025

dessalines commented Jan 6, 2025

		@@ -168,23 +150,7 @@ pub struct GetModlog {
		/// The modlog fetch response.
		// TODO this should be redone as a list of tagged enums

		@@ -761,3 +761,78 @@ CALL r.create_person_saved_combined_trigger ('post');

		CALL r.create_person_saved_combined_trigger ('comment');

		-- modlog: (17 tables)

Adding Combined modlog #5253

Are you sure you want to change the base?

Adding Combined modlog #5253

Conversation

dessalines commented Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dessalines commented Jan 6, 2025

dessalines commented Dec 10, 2024 •

edited

Loading