🚀 Add bank account transaction integration

[alexanderwassbjer]

Implemented an integration that uses GoCardless api to fetch bank account transactions. Then when you want to add an expense you can either fill it manually or you can pick from your transactions and it will automatically be added as an expense. There is also a function to "multi add" multiple transactions at the same time.
If a transaction already has been added to a group you cant add it once again.
GoCardless page

Other changes:

• Added a clear field button to add expense page.
• Changed to on addExpense function redirect link.
• Added the new feature to start page.
• Added information about the new feature to readme.
• Made it easy to implement other ob api providers in the future.

Screenshots:

[alexanderwassbjer]

@KMKoushik :)

[alexanderwassbjer]

I have also added i18n support for this feature in another PR that is based on this PR: #89. If this gets merged I can open a PR with i18n support for this aswell.

https://github.com/alexanderwassbjer/split-pro/tree/feature/i18n-support

[KMKoushik]

this looks good overall with some comments. please lemme know what you think!

tysm for this PR really appreciate it. this is really awesome!

think it's best to merge in v2, test thoroughly and release it with other features! (i'll create a new branch)

[KMKoushik]

so it will work on one specific country alone?

[alexanderwassbjer]

No, it should work world wide. The bank provider picker in the settings page fetch all bank providers. It works with either one country or undefined which then returns all the bank providers.
But because the list can be long in that case, I added the env to have a option to only fetch some.

If you have another approach to this, feel free to tell me.
One way is to have a country picker first and then the providers but it feels a bit weird..

[KMKoushik]

we have default currency option, we can get the country from that currency!

[alexanderwassbjer]

I can't find the default currency? If you mean the currency that is set by the user when they try to add a expense it feels a bit weird to use that because it can change?

[alexanderwassbjer]

So for now, it only works with one country in the package I use. https://github.com/nordigen/nordigen-node/blob/9d535ca9d788115d4e199c749432733be3ea6cdd/lib/api/institutions.js#L23

That can be fixed, either in the package or that we have a country picker first.
But if we think of the user case, will it be necessary to have multiple countries at the same instance?

[alexanderwassbjer]

If we want to stick with the env and use it either with no value to fetch all bank providers or filter it based on the env. I have created a PR on the Nordigen-node package and hoping for a merge so we can use it.
nordigen/nordigen-node#63

[krokosik]

Can you submit this PR to my fork? :)

[KMKoushik]

@alexanderwassbjer changed the base to v2!

[alexanderwassbjer]

this looks good overall with some comments. please lemme know what you think!

tysm for this PR really appreciate it. this is really awesome!

think it's best to merge in v2, test thoroughly and release it with other features! (i'll create a new branch)

Awesome! Iam happy you liked it.
I will go over it again, check your comments and fix some other stuff.

It would be nice if it was released in v2!

[KMKoushik]

Awesome! Iam happy you liked it. I will go over it again, check your comments and fix some other stuff.

It would be nice if it was released in v2!

ofcourse this is going to be star feature!

[alexanderwassbjer]

@alexanderwassbjer
Some other improvement ideas:

• Encrypt the cache for the transactions.
• Add multiple bank accounts for different providers.

[krokosik]

1.5 stable is out :)

Regarding cron there are 2 options that I consider. One is the scheduler in pg-boss, which I would like to add for running background tasks in a separate node process. It does have cron, but I haven't read into the tradeoffs of using it versus pg-cron.

The cache policy should be configurable, one month seems like a lot of transactions to me.

[alexanderwassbjer]

1.5 stable is out :)

Regarding cron there are 2 options that I consider. One is the scheduler in pg-boss, which I would like to add for running background tasks in a separate node process. It does have cron, but I haven't read into the tradeoffs of using it versus pg-cron.

The cache policy should be configurable, one month seems like a lot of transactions to me.

Awesome with 1.5! Then we will try to merge this PR.

I haven’t read much about pg-boss, but I’m considering either that or pg-cron. My gut feeling is that having the cron jobs as a database extension with pg-cron seems more stable than handling them in code with pg-boss — though I might be wrong.

EDIT: The cache policy for manage is 1 day. But 1 month is the getTransactions from GoCardless interval (dateFrom, dateTo). Maybe that should be configurable?

[alexanderwassbjer]

@krokosik I have pushed some new code where all your comments are fixed and some other stuff.

Whats left to do is:

• Use your fork of the nordigen-node
• Implement some cron for pg to remove cache older then like 2 days.

I would like your thoughts on the cron stuff.

[krokosik]

Regarding cron, I will read more about it next week, but so far, it is unclear to me how well pg-cron integrates with Prisma. My initial lookup showed that Postgres extensions are supported as a preview feature, but I have no idea how well. I would also prefer to use pg-cron as a more battle tested and stable solution, but if it's a nightmare in Prisma (which is often the case with advanced DB features) we can skip that.

On the other hand we will use pg-boss anyway. Question is how extensively. If we could have both that would be great. If you could add pg-cron as a proof of concept that would certainly speed things up 🚀

[alexanderwassbjer]

Regarding cron, I will read more about it next week, but so far, it is unclear to me how well pg-cron integrates with Prisma. My initial lookup showed that Postgres extensions are supported as a preview feature, but I have no idea how well. I would also prefer to use pg-cron as a more battle tested and stable solution, but if it's a nightmare in Prisma (which is often the case with advanced DB features) we can skip that.

On the other hand we will use pg-boss anyway. Question is how extensively. If we could have both that would be great. If you could add pg-cron as a proof of concept that would certainly speed things up 🚀

Indeed! I’ve seen that this is still a preview feature, and at the moment Prisma only supports a limited set of extensions.
https://www.prisma.io/docs/postgres/database/postgres-extensions
Quote:

Another option would be to manually install pg_cron through a migration, but that approach isn’t really recommended either. And we would have to install the pg_cron inside the psql container in docker in order to make the migration to run.
prisma/prisma#12308

I will have a look at implementing pg-boss 👍

EDIT: I'am not sure how our self-hosters is running their instance of split-pro. But Netlify, Vercel for example don't keep workers alive, like pg-boss or node-cron in a separate node instance. Instead they recommend using their own way of managing cron jobs.
The self-hosters that run in docker on a local server, my self included would probably not have any issues with pg-boss being killed.

[alexanderwassbjer]

@krokosik I have added a proof of concept of the pg_cron. It works good without any problems. 3e873a0 I have some findings:

• Could not add pg_cron and schedule jobs in a prisma migration because it creates a shallow copy which do not have the pg_cron extension available.
• Created a cron setup file which can be called multiple times without duplicated cron jobs being added.
• Added a separate db:cron script which allows us to either run it after the db migration script or separate.
• Had to install pg_cron in the psql instance via a dockerfile which is used in the compose file.
• The pg_cron is logging all of its runs in a table called cron.job_run_details.

I have tested the implementation a bit and I think it's a good approach.

[krokosik]

Awesome work! Will review the changes this week.

[krokosik]

Oof bummer, gocardless does not support individual accounts in my country (Poland) and I don't have a business. We need to figure out a way to test it, are there some other providers I could try?

[alexanderwassbjer]

Oof bummer, gocardless does not support individual accounts in my country (Poland) and I don't have a business. We need to figure out a way to test it, are there some other providers I could try?

Ohno! But do you have like Revolut, because they are supported. Otherwise my plan is to maybe implement the plaid api aswell.

[krokosik]

The requirement to own a business is a requirement for a gocardless account itself. I didn't event get to adding accounts.

[alexanderwassbjer]

The requirement to own a business is a requirement for a gocardless account itself. I didn't event get to adding accounts.

Hmm what?
If you go to: https://developer.gocardless.com/bank-account-data/overview

Then it says:
"Before you start, please make sure you have acquired a user secret from GoCardless's Bank Account Data portal"

There is a link you can follow.

Which is free and no requirement of a business. That's the way it went and it works good.

[krokosik]

I can't even create an account, but in the developer page I noticed I can make a sandboxed account for testing, which seems perfect. I will be less hesitatnt to give fake info there if I don't have to connect my real bank account 🤞

[krokosik]

Aaaand the sandbox account does not let me into the Bank Account Data portal.

[krokosik]

Okay nevermind, they have a separate sandbox for bank account data! So I don't even need to make an account there <3

[alexanderwassbjer]

Okay nevermind, they have a separate sandbox for bank account data! So I don't even need to make an account there <3

Awesome! 🤩

[alexanderwassbjer]

@krokosik I just had to verify that it worked to create an account without a business.
https://bankaccountdata.gocardless.com/login
Then signin with a google account which will create an account.
It worked for me twice with no problem.
😊

[krokosik]

@alexanderwassbjer yeah, my bad. I didn't realize bank account data was a separate service. With that being said, I'm a bit swamped now with other responsibilities sadly.