`sqldb/v2` as separate package #10175

ViktorT-11 · 2025-08-26T15:01:12Z

This PR addresses Phase 1 of the release plan of comment #9762 (comment) in #9762.

This PR extracts the non-specific lnd specific logic of the sqldb package into a new sqldb/v2 package, and introduces the extra functionality that sqldb/v2 adds.

ellemouton

Did a first pass, looking really good!

sqldb/v2/migrations.go

sqldb/v2/interfaces.go

sqldb/v2/postgres_test.go

ellemouton · 2025-09-04T06:36:22Z

go.mod

+replace github.com/golang-migrate/migrate/v4 => github.com/lightninglabs/migrate/v4 v4.18.2-9023d66a-fork-pr-2
+


ok afaict, the big thing we added in the fork was the post migration call-backs logic.

Something id just like some clarity on before we push forward here is:
in our own rolled migration strategy in LND today that takes care of code migrations, we know that the migration version table will only be updated once the code migration is complete. Is the same true for the post-migration callback?

I discussed this with guggero previously.
Currently, no that is not true unfortunately, as the version table will be updated if the SQL migration succeeds, while the code migration fails. But I'm planning to address that asap 🔥!

The current plan is to submit a new PR to our migrate library fork. This PR will add a separate tracking table for code migrations. With this change, the logic will ensure that if the db is initialized and the persisted code migration version is lower than the SQL migration version, the code migration will be re-run prior to any SQL migration.

I haven't gotten to updating the migrate fork yet though. Perhaps we should hold of with merging this one until that change has been merged to the migrate fork, so that we don't need to push a new PR that updates the dependency in lnd though.

Does that sound like a good plan to you @ellemouton?

sounds good!

In the upcoming commits, we will introduce a new sqldb module, sqldb version 2. The intention of the new sqldb module, is to make it generalizable so that it contains no `lnd` specific code, to ensure that it can be reused in other projects. This commit adds the base of the new module, but does not include any implementation yet, as that will be done in the upcoming commits.

This commit moves all non lnd-specific code of sqldb/v1 to the new sqldb/v2 module. Note however, that without additional changes, this package still needs to reference lnd, as references to the lnd `sqlc` package is required without further changes. Those changes will be introduced in the upcoming commits, to fully decouple the new sqldb/v2 module from lnd.

This commit introduces a new struct named `MigrationStream`, which defines a structure for migrations SQL migrations. The `MigrationStream` struct contains the SQL migrations which will be applied, as well as corresponding post-migration code migrations which will be executed afterwards. The struct also contains fields which define how the execution of the migrations are tracked. Importantly, it is also possible to define multiple different `MigrationStream`s which are executed, to for example define one `prod` and one `dev` migration stream.

This commit updates the `sqldb/v2` package to utilize the new `MigrationStream` type for executing migrations, instead of passing `[]MigrationConfig`'s directly.

This commit updates the definition of the `BaseDB` struct to decouple it from lnd`s `sqlc` package. We also introduce new fields to the struct to make it possible to track the database type used at runtime.

Drop the GetSchemaVersion and SetSchemaVersion methods for the PostgresStore and SqliteStore structs in the sqldb/v2 package.

In order to make it possible to replace `tapd`'s internal `sqldb` package with the new generic `sqldb/v2` package, we need to make sure that all features and functionality that currently exist in the `tapd` package are also present in the new `sqldb/v2` package. This commit adds such additional missing features to the `sqldb/v2` package.

Previously, the test db helper files were suffixed with "_test", which would indicate that the files specifically contained tests. However, these files actually contain helper functions to be used in tests, and are not tests themselves. To better reflect their purpose, the files have been renamed to instead be prefixed with "test_".

ViktorT-11 · 2025-09-04T11:35:53Z

Thanks a lot for the review @ellemouton 🔥🎉! Did one separate push to address the latest feedback (and add the contents of #10193) + one push to rebase this PR on master.

I'm not re-requesting a review of you just yet until we have decided on #10175 (comment) @ellemouton.

ViktorT-11 · 2025-09-04T18:08:53Z

Will address failing CI checks on the next push after the next review round.

ViktorT-11 · 2025-09-05T10:09:02Z

Based the PR on the now updated sqldb-v2 branch 🎉!

GustavoStingelin

Great work here! I'm excited to see this moving forward and to use it in btcwallet.
I’ve opened a PR with some related work already, but this looks more polished and complete.

I’ve added a few comments as well.

GustavoStingelin · 2025-09-15T18:47:37Z

sqldb/v2/migrations.go

+	// MigrateTableName is the name of the table used by golang-migrate to
+	// track the current schema version.
+	MigrateTableName string


What do you think about renaming this to MigrationTableName? Or, to avoid repeating the word “migration”, maybe TrackingTableName would work better. To me, MigrateTableName sounds more like the table that is going to be migrated, rather than the one used for tracking migrations.

GustavoStingelin · 2025-09-15T18:52:50Z

sqldb/v2/migrations.go

+	// Schemas is the embedded file system containing the SQL migration
+	// files.
+	Schemas embed.FS
+
+	// SQLFileDirectory is the directory containing the SQL migration files.
+	SQLFileDirectory string


Here we’re using two different names to describe the DDL migrations. One is Schemas, which might be confusing since it sounds like the literal database.schema relation. The other is SQLFileDirectory, which I think works well. What do you think about renaming it to something like DDLFiles and DDLFilesDirectory (or something along those lines) to make the intent clearer?

Or SQLFiles and SQLFilesDirectory as well.

GustavoStingelin · 2025-09-15T19:13:36Z

sqldb/v2/no_sqlite.go

+// SqliteStore is a database store implementation that uses a sqlite backend.
+type SqliteStore struct {
+	cfg *SqliteConfig
+
+	*BaseDB
+}


Maybe it could be clearer that it's a fake implementation.

GustavoStingelin · 2025-09-15T19:16:25Z

sqldb/v2/paginate.go

+	// maxSQLiteBatchSize is the maximum number of items that can be
+	// included in a batch query IN clause for SQLite. This was determined
+	// using the TestSQLSliceQueries test.
+	maxSQLiteBatchSize = 32766
+
+	// maxPostgresBatchSize is the maximum number of items that can be
+	// included in a batch query IN clause for Postgres. This was determined
+	// using the TestSQLSliceQueries test.
+	maxPostgresBatchSize = 65535


I think this value applies to all query parameters used in the query, not only for those used in the IN statement, so if this limit is reached when we have other parameters aside the IN, it will return an error.

GustavoStingelin · 2025-09-15T19:43:18Z

sqldb/v2/postgres.go

+	// Ensure the migration tracker table exists before running migrations.
+	// This table tracks migration progress and ensures compatibility with
+	// SQLC query generation. If the table is already created by an SQLC
+	// migration, this operation becomes a no-op.
+	migrationTrackerSQL := `
+	CREATE TABLE IF NOT EXISTS migration_tracker (
+		version INTEGER UNIQUE NOT NULL,
+		migration_time TIMESTAMP NOT NULL
+	);`


Is this different from the MigrationStream table name?

GustavoStingelin · 2025-09-15T20:20:54Z

sqldb/v2/postgres_fixture.go

+// TearDown stops the underlying docker container.
+func (f *TestPgFixture) TearDown(t testing.TB) {
+	err := f.pool.Purge(f.resource)
+	require.NoError(t, err, "Could not purge resource")
+}


This could be passed as a test Cleanup function inside the fixture creation

t.Cleanup(func() { fixture.TearDown(t) })

GustavoStingelin · 2025-09-15T20:27:19Z

sqldb/v2/sqlerrors.go

+	// ErrRetriesExceeded is returned when a transaction is retried more
+	// than the max allowed valued without a success.
+	ErrRetriesExceeded = errors.New("db tx retries exceeded")


Suggested change

// ErrRetriesExceeded is returned when a transaction is retried more

// than the max allowed valued without a success.

ErrRetriesExceeded = errors.New("db tx retries exceeded")

// ErrTxRetriesExceeded is returned when a transaction is retried more

// than the max allowed valued without a success.

ErrTxRetriesExceeded = errors.New("db tx retries exceeded")

Maybe ErrTxRetriesExceeded to avoid confusion with other types of retry.

GustavoStingelin · 2025-09-15T20:29:07Z

sqldb/v2/sqlerrors.go

+	// postgresErrMsgs are strings that signify retriable errors resulting
+	// from serialization failures.
+	postgresErrMsgs = []string{
+		"could not serialize access",
+		"current transaction is aborted",
+		"not enough elements in RWConflictPool",
+		"deadlock detected",
+		"commit unexpectedly resulted in rollback",
+	}


Maybe postgresRetriableErrMsgs to be clearer.

Suggested change

// postgresErrMsgs are strings that signify retriable errors resulting

// from serialization failures.

postgresErrMsgs = []string{

"could not serialize access",

"current transaction is aborted",

"not enough elements in RWConflictPool",

"deadlock detected",

"commit unexpectedly resulted in rollback",

}

// postgresRetriableErrMsgs are strings that signify retriable errors resulting

// from serialization failures.

postgresRetriableErrMsgs = []string{

"could not serialize access",

"current transaction is aborted",

"not enough elements in RWConflictPool",

"deadlock detected",

"commit unexpectedly resulted in rollback",

}

GustavoStingelin · 2025-09-15T20:33:40Z

sqldb/v2/sqlite.go

+// NewTestSqliteDB is a helper function that creates an SQLite database for
+// testing.
+func NewTestSqliteDB(t testing.TB, streams []MigrationStream) *SqliteStore {


I think we should separate the test helpers in another file from the functions used in "production".

GustavoStingelin · 2025-09-15T20:45:32Z

sqldb/v2/test_postgres.go

+	t.Cleanup(func() {
+		pgFixture.TearDown(t)
+	})
+


Maybe we can move this to DB creation, like done in SQLite.

GustavoStingelin · 2025-10-08T02:09:01Z

Anything that I can do to help put energy into this?

ViktorT-11 · 2025-10-08T08:30:49Z

Hey @GustavoStingelin! Thank you so much for the review above 🙏!

Anything that I can do to help put energy into this?

Apologies, I should probably have informed you this in this PR earlier:
We're waiting for this PR in the dependent migrate fork to be merged before I'm going to start pushing for this PR again, including addressing your great feedback:
lightninglabs/migrate#3

As that PR will require updates of the implementation of this PR, it makes most sense that it is finalised and merged before changing this dependent PR.

In terms of areas where you can help out: Continuing to review this PR once the above is merged and I've addressed your feedback here would be very helpful. Additionally, if you want to, reviewing the above migrate PR would also be very helpful 🙏.

mohamedawnallah · 2025-11-03T17:21:43Z

sqldb/v2/postgres_fixture.go

+	testPgUser   = "test"
+	testPgPass   = "test"
+	testPgDBName = "test"
+	PostgresTag  = "15"


To what extent is making the Postgres DB version 15 intentional? PostgreSQL 15 was released 4 years ago, and between that time and now (PostgreSQL 18), I heuristically bet there are significant performance optimizations that have been added to PostgreSQL. I think we could target a more recent version, like the pre-latest release (PostgreSQL 17 seems like a sweet spot)

Related:
https://endoflife.date/postgresql

mohamedawnallah · 2025-11-03T17:24:47Z

sqldb/v2/postgres_fixture.go

+	testPgUser   = "test"
+	testPgPass   = "test"
+	testPgDBName = "test"
+	PostgresTag  = "15"


Relatedly, it looks like the PostgresTag version is not configurable. I understand the intent may be to enforce a specific version for all consumers, but this may not always be feasible now or in the future. Each project may have evolving dynamics that require a specific PostgreSQL version. I think it's safe to make it configurable with a default value that can be overridden when needed

mohamedawnallah · 2025-11-03T17:28:53Z

sqldb/v2/postgres_fixture.go

+func NewTestPgFixture(t testing.TB, expiry time.Duration) *TestPgFixture {
+	// Use a sensible default on Windows (tcp/http) and linux/osx (socket)
+	// by specifying an empty endpoint.
+	pool, err := dockertest.NewPool("")


Since we are gonna use docker container anyway here, Why not using testcontainers which is out of the box, more mature, and maintainable solution. Seeing https://testcontainers.com/modules/postgresql/?language=go

Credits:
Originally inspired by @GustavoStingelin's PR: btcsuite/btcwallet#1027

mohamedawnallah · 2025-11-03T17:32:18Z

sqldb/v2/postgres_fixture.go

+			fmt.Sprintf("POSTGRES_PASSWORD=%v", testPgPass),
+			fmt.Sprintf("POSTGRES_DB=%v", testPgDBName),
+			"listen_addresses='*'",
+		},


Those environment variables need to be configurable. It is up to the API consumer to pass the intended parameters. We can have some sort of base arguments and additional arguments.

If no additional args provided, the base args would be used

If additional args provided and no overlapping with base args, the base args would be augmented

If additional args provided and there is overlapping with base args, the base args would be overriden by the corresponding values of the additional args and augmented with the new ones if any

mohamedawnallah · 2025-11-03T17:32:30Z

sqldb/v2/postgres_fixture.go

+		Cmd: []string{
+			"postgres",
+			"-c", "log_statement=all",
+			"-c", "log_destination=stderr",
+			"-c", "max_connections=5000",
+		},


Related to the above comment

mohamedawnallah · 2025-11-03T17:33:29Z

sqldb/v2/postgres_fixture.go

+	pool     *dockertest.Pool
+	resource *dockertest.Resource
+	host     string
+	port     int


How are we making sure here there is no port collision in case of running multiple PostgreSQL DB containers?

mohamedawnallah · 2025-11-03T17:36:49Z

sqldb/v2/postgres_fixture.go

+
+	// Tell docker to hard kill the container in "expiry" seconds.
+	require.NoError(t, resource.Expire(uint(expiry.Seconds())))
+


Relatedly, That all docker containers "management" lifecycle part here and there across this PR is managed out of the box by testcontainers

mohamedawnallah · 2025-11-03T17:41:17Z

sqldb/v2/sqlerrors.go

+
+// MapSQLError attempts to interpret a given error as a database agnostic SQL
+// error.
+func MapSQLError(err error) error {


Why do we need to agnostically map that SQL error? How would doing that be helpful? It looks like at the application layer there is a state to signal whether we are dealing with SQLite or PostgreSQL. We could have separate functions for this: one for MapPostgreSQLError and another for MapSQLiteError. That way the errors would be extendable in the future, given that there is almost a major release version for PostgreSQL every year (hence, heuristically, new errors)

mohamedawnallah · 2025-11-03T17:58:40Z

sqldb/v2/postgres.go

+		"INTEGER PRIMARY KEY": "BIGSERIAL PRIMARY KEY",
+		"TIMESTAMP":           "TIMESTAMP WITHOUT TIME ZONE",
+		"UNHEX":               "DECODE",
+	}


How is having an intermediary schema patching approach helpful and reliable over time?

Why not make the schema creation process redundant for each target DB (e.g., one for PostgreSQL and one for SQLite)? That way we can have flexibility and reliability in how we define schemas according to the target DB.

To boil down my arguments for why this intermediary schema patching design approach is unreliable long-term and even with some reasonable time in short-term:

PostgreSQL has frequent version releases (approximately one per year) with EOL releases every ~5 years. These uncontrolled changes naturally require updates to the replacement mappings, which may necessitate more migrations or make the process increasingly complex.

There was an internal issue discussed in LL where an explicit AUTO_INCREMENT was required for intended behavior (which, as observed, required a migration in the next release). Issues like this could be completely avoided by not using patching and instead following a "Write Everything Twice" approach for schema creation. Note: this wouldn't apply to queries, since queries are easier to write in a universally-agnostic manner than schema definitions.

Future database support concerns: What if there's a plan to support another database? Would the schema patching map become a multi-type hierarchy? We'd eventually end up solving a simple problem with a complex solution.

For example: What's the problem with having the entire graph schema creation written twice—once for SQLite and once for PostgreSQL?

Seeing: https://github.com/lightningnetwork/lnd/blob/master/sqldb/sqlc/migrations/000008_graph.up.sql

cc: @yyforyongyu for input need – since the btcwallet SQL project will eventually depend on this sqldb package, having this intermediary patching approach could potentially impact the reliability and criticality of the btcwallet sqlization project given the vision for btcwallet sqlization to stand for a decade, with that patching, and time constraint those parameters don't look like align with that vision

lightninglabs-deploy · 2025-11-17T19:45:20Z

@ziggie1984: review reminder
@ViktorT-11, remember to re-request review from reviewers when ready

ViktorT-11 changed the title ~~2025 08 sqldb v2 separate package~~ sqldb/v2 as separate package Aug 26, 2025

ViktorT-11 changed the base branch from sqldb-v2 to master August 26, 2025 21:43

ViktorT-11 force-pushed the 2025-08-sqldb-v2-separate-package branch 3 times, most recently from 6b5e428 to 6899a39 Compare September 2, 2025 23:30

ViktorT-11 mentioned this pull request Sep 3, 2025

[sql-43] Add support for sqldb/v2 in litd lightninglabs/lightning-terminal#1085

Draft

ellemouton reviewed Sep 4, 2025

View reviewed changes

ViktorT-11 force-pushed the 2025-08-sqldb-v2-separate-package branch from 6899a39 to 7ca3000 Compare September 4, 2025 11:26

ViktorT-11 added 9 commits September 4, 2025 13:28

sqldb/v2: Use MigrationStream for migrations

9c1df3d

This commit updates the `sqldb/v2` package to utilize the new `MigrationStream` type for executing migrations, instead of passing `[]MigrationConfig`'s directly.

sqldb/v2: introduce sqldb/v2 BaseDB

a1d3359

This commit updates the definition of the `BaseDB` struct to decouple it from lnd`s `sqlc` package. We also introduce new fields to the struct to make it possible to track the database type used at runtime.

sqldb/v2: drop GetSchemaVersion & SetSchemaVersion

fafd451

Drop the GetSchemaVersion and SetSchemaVersion methods for the PostgresStore and SqliteStore structs in the sqldb/v2 package.

lnd: add sqldb/v2 dependency to lnd

233f1fb

ViktorT-11 force-pushed the 2025-08-sqldb-v2-separate-package branch from 7ca3000 to 233f1fb Compare September 4, 2025 11:32

ViktorT-11 requested a review from ziggie1984 September 4, 2025 11:36

ViktorT-11 changed the base branch from master to sqldb-v2 September 5, 2025 10:08

ViktorT-11 marked this pull request as ready for review September 5, 2025 10:09

GustavoStingelin suggested changes Sep 15, 2025

View reviewed changes

GustavoStingelin mentioned this pull request Sep 15, 2025

multi: add database test helpers btcsuite/btcwallet#1027

Draft

saubyk added this to the v0.21.0 milestone Sep 16, 2025

mohamedawnallah mentioned this pull request Oct 9, 2025

Add blocks table structure to unblock address manager sqlization btcsuite/btcwallet#1101

Merged

7 tasks

mohamedawnallah suggested changes Nov 3, 2025

View reviewed changes

		replace github.com/golang-migrate/migrate/v4 => github.com/lightninglabs/migrate/v4 v4.18.2-9023d66a-fork-pr-2


		// Tell docker to hard kill the container in "expiry" seconds.
		require.NoError(t, resource.Expire(uint(expiry.Seconds())))

sqldb/v2 as separate package #10175

Are you sure you want to change the base?

sqldb/v2 as separate package #10175

Uh oh!

Conversation

ViktorT-11 commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ellemouton left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ViktorT-11 Sep 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ViktorT-11 commented Sep 4, 2025

Uh oh!

ViktorT-11 commented Sep 4, 2025

Uh oh!

ViktorT-11 commented Sep 5, 2025

Uh oh!

GustavoStingelin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GustavoStingelin commented Oct 8, 2025

Uh oh!

ViktorT-11 commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohamedawnallah Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohamedawnallah Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohamedawnallah Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

`sqldb/v2` as separate package #10175

`sqldb/v2` as separate package #10175

ViktorT-11 commented Aug 26, 2025 •

edited

Loading

ViktorT-11 Sep 4, 2025 •

edited

Loading

mohamedawnallah Nov 3, 2025 •

edited

Loading

mohamedawnallah Nov 3, 2025 •

edited

Loading

mohamedawnallah Nov 3, 2025 •

edited

Loading