WIP: Compute balances in database #247

FriesischScott · 2025-06-09T11:24:30Z

As discussed in #234 the goal of this PR is to drop the separate balance tables for users and groups and instead compute the balances from all expenses in the database.

I agree with @krokosik that we need to be certain we can handle the required loads for large instances with lots of expenses but I think postgresql will be up for the task.

This is a list of the required balances:

Friend balances
Overall group balances for one user
All balances for a specific group

here I've implemented the second and third and I think @krokosik has started working on the first.

Because prisma can't handle the necessary computations through it's javascript interface I've chosen to use typedSQL to write native SQL queries and still benifit from prisma's typing capabilities.

One minor downside is that typedSQL currently requires a live database connection to generate the types. However with prisma 7 the location of the generated client will move out of node_modules and we will have to set an output directory like this.

We could then commit the generated sql types in the sql subfolder of the prisma client and only generate the regular client during build. This should already be possible now (only in prisma 7 it's mandatory) and I'm working on setting it up.

This part no longer relies on the GroupBalances table

Uses a prisma typedSQL query

Only the `typedSQL` files are committed. The client is excluded from eslint.

prisma/seed.ts

prisma/schema.prisma

guenzd · 2025-06-09T15:47:19Z

I get this error when I try to run with existing data:

FriesischScott · 2025-06-09T15:48:47Z

I get this error when I try to run with existing data:

Yes, I haven't finished fixing a few type issues. You should be able to run it with pnpm dev but build is not yet working.

guenzd · 2025-06-09T16:06:28Z

I get this error when I try to run with existing data:

Yes, I haven't finished fixing a few type issues. You should be able to run it with pnpm dev but build is not yet working.

Then there are more runtime issues :/

FriesischScott · 2025-06-09T23:44:14Z

I get this error when I try to run with existing data:

Yes, I haven't finished fixing a few type issues. You should be able to run it with pnpm dev but build is not yet working.

Then there are more runtime issues :/

Can you try again now?

Turns out I wasn't up to date on the migrations and for some reason when the column is bigint the sum is returned as numeric. I'm now casting explicitly to bigint.

FriesischScott · 2025-06-09T23:45:40Z

Not sure why the linting fails when it doesn't for me but otherwise it should now build without errors.

FriesischScott · 2025-06-10T13:28:04Z

I've changed the seed.ts so you can use it check the performance yourself.

I think with roughly 80ms for 100k expenses we are well within the performance we require. I don't believe any group will reach that high.

Should be done with the eventual update to prisma 7.

FriesischScott · 2025-06-10T14:34:39Z

We have lift off 🥳. The path alias was confusing the linter so I've moved the prisma client back into node_modules.

1000 users in 101 groups with 10k expenses each

Type Hash for faster lookup with =

FriesischScott · 2025-06-18T12:32:06Z

I've updated the seed file to create 1000 users and 101 groups with 10k expenses each. The largest group has 30 members all the others have 10. It should give us a better understanding of the performance.

It does take quite a while to create all the expenses but it's a temporary change anyway. If we merge we can just revert the file completely.

krokosik · 2025-06-18T19:00:06Z

Fantastic work, thank you 🙌
I know I promised to take a swing at it this week, but unfortunately it's not likely to happen due to other obligations :/

FriesischScott · 2025-06-19T08:35:51Z

Fantastic work, thank you 🙌 I know I promised to take a swing at it this week, but unfortunately it's not likely to happen due to other obligations :/

No problem at all. Take your time. Better not to rush things :)

krokosik

Very sorry for taking so long with the review. I had to take care of other stuff as well as more pressing current issues. I requested some changes that would minimize this PR further as I believe such risky changes should be as easy to review as possible. I ran some initial tests on my machine and it is looking good, so I would like to proceed with this, but only after they have been resolved and the branch gets rebased onto main. Furthermore, if you could give me write access to your fork, I will implement friend balances as well :)

krokosik · 2025-07-07T15:48:04Z

prisma/seed.ts

The original seed script is useful for local dev work, while this one takes a long time to execute. Please move it to a separate file

My plan was to only use this as and intermediate script and revert to the original before we merge. If you think it's worth keeping around I'll move it to a separate script.

krokosik · 2025-07-07T15:49:15Z

prisma/sql/getAllBalancesForGroup.sql

Can you apply some SQL formatting to these queries?

krokosik · 2025-07-07T15:52:25Z

src/lib/simplify.test.ts

+const sortByIds = (a: getAllBalancesForGroup.Result, b: getAllBalancesForGroup.Result) => {
+  if (a.paidBy === b.paidBy) {
+    return a.borrowedBy - b.borrowedBy;
  }
-  return a.userId - b.userId;
+  return a.paidBy - b.paidBy;


I would prefer to keep this PR as minimal as possible. As such, please keep the property names identical and create a type GroupBalance = getAllBalancesForGroup.Result alias

Just to confirm, you want me to keep the existing firendId including the typo?

krokosik · 2025-07-07T15:54:40Z

src/server/api/routers/group.ts

+    const groupBalances = await ctx.db.$queryRawTyped(getGroupsWithBalances(ctx.session.user.id));
+
+    const _groups = groupBalances
+      .map((b) => {
+        return {
+          id: b.id,
+          name: b.name,
+        };
+      })
+      .filter((obj, index, self) => index === self.findIndex((t) => t.id === obj.id));
+
+    const groupsWithBalances = _groups.map((group) => {
+      const balancesForGroup: Record<string, bigint> = {};
+      groupBalances
+        .filter((b) => {
+          return b.id == group.id;
+        })
+        .forEach((b) => {
+          if (b.currency != null && b.balance != null) {
+            balancesForGroup[b.currency] = b.balance;
+          }
+        });


While we remain on the dev branch, I propose the following sanity check:
utilize both the legacy and new method and compare their results. When a mismatch is detected, throw an Error. We will remove it before creating a release.

krokosik · 2025-07-07T16:10:35Z

Oh and one more thing, this change would break the current splitwise import functionality, as it does not import expenses as of now. #207 needs to be implemented first and as such I don't expect to go ahead with this PR for 1.5 release

FriesischScott · 2025-07-09T18:21:22Z

Oh and one more thing, this change would break the current splitwise import functionality, as it does not import expenses as of now. #207 needs to be implemented first and as such I don't expect to go ahead with this PR for 1.5 release

No problem. More time to polish.

alexanderwassbjer · 2025-08-05T19:53:25Z

This is really good work! Calculating it based on the expenses rather then the balance table is amazing! 👏

FriesischScott added 10 commits June 5, 2025 23:52

Add typed query to get groups with balances

a9ed276

Return groups without expenses

beb89da

Order groups descending by newest

df6a7ad

Compute blances for group overview in db

ab4c7ee

This part no longer relies on the GroupBalances table

Compute balance list in db

56af112

Uses a prisma typedSQL query

Merge branch 'main' into sql-balances

de25193

Include group id in all balances

24d1b99

Fix linting

70e58c6

Move prisma client to src/prisma/client

a40ca8c

Only the `typedSQL` files are committed. The client is excluded from eslint.

Fix prettier warning

cee753c

krokosik reviewed Jun 9, 2025

View reviewed changes

prisma/seed.ts Outdated Show resolved Hide resolved

krokosik reviewed Jun 9, 2025

View reviewed changes

prisma/schema.prisma Outdated Show resolved Hide resolved

FriesischScott added 14 commits June 9, 2025 18:26

Use path alias to move prisma client

23fece2

gitignore prisma client files

b3002be

Fix sql imports

20fb843

Set prisma client output

44a3bdd

Run lint --fix

a36344d

Fix type issues

cd32056

Fix prettier warnings

c619e26

Add postgresql service container

646c2e0

Run migrations before generating sql types

7de67eb

Merge branch 'main' into sql-balances

bb996e9

Remove sql types

fa22c43

Fix workflow file

1ff2229

Fix bigint conversions

6ff9a98

Fix lint

0e5b29c

FriesischScott added 3 commits June 10, 2025 14:50

Seed db for stress testing

bec3f4d

Fix prettier warnings

af4c06d

Exclude seed.ts

2684a71

Don't set prisma output folder

4b02518

Should be done with the eventual update to prisma 7.

Exclude deleted expenses from group balance

c26dc4a

FriesischScott mentioned this pull request Jun 10, 2025

New group members can't be added to existing expenses. #216

Closed

FriesischScott added 3 commits June 18, 2025 13:34

Update seed

06eea78

1000 users in 101 groups with 10k expenses each

Add userid index to ExpenseParticipant

bca6c9d

Type Hash for faster lookup with =

Merge branch 'main' into sql-balances

4dc3d38

FriesischScott marked this pull request as ready for review June 18, 2025 12:29

Fix prettier warnings

1f330a5

krokosik requested changes Jul 7, 2025

View reviewed changes

krokosik changed the title ~~Compute balances in database~~ WIP: Compute balances in database Sep 5, 2025

krokosik marked this pull request as draft September 5, 2025 11:55

Uh oh!

WIP: Compute balances in database #247

Are you sure you want to change the base?

WIP: Compute balances in database #247

Uh oh!

Conversation

FriesischScott commented Jun 9, 2025

Uh oh!

Uh oh!

Uh oh!

guenzd commented Jun 9, 2025

Uh oh!

FriesischScott commented Jun 9, 2025

Uh oh!

guenzd commented Jun 9, 2025

Uh oh!

FriesischScott commented Jun 9, 2025

Uh oh!

FriesischScott commented Jun 9, 2025

Uh oh!

FriesischScott commented Jun 10, 2025

Uh oh!

FriesischScott commented Jun 10, 2025

Uh oh!

FriesischScott commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krokosik commented Jun 18, 2025

Uh oh!

FriesischScott commented Jun 19, 2025

Uh oh!

krokosik left a comment

Choose a reason for hiding this comment

Uh oh!

krokosik Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

FriesischScott Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

krokosik Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

krokosik Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

FriesischScott Jul 15, 2025

Choose a reason for hiding this comment

Uh oh!

krokosik Jul 7, 2025

Choose a reason for hiding this comment

Uh oh!

krokosik commented Jul 7, 2025

Uh oh!

FriesischScott commented Jul 9, 2025

Uh oh!

alexanderwassbjer commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

FriesischScott commented Jun 18, 2025 •

edited

Loading

alexanderwassbjer commented Aug 5, 2025 •

edited

Loading