Skip to content

Migrate psycopg2 -> psycopg 3 - #903

Merged
anderdc merged 3 commits into
entrius:testfrom
seroperson:psycopg
May 6, 2026
Merged

Migrate psycopg2 -> psycopg 3#903
anderdc merged 3 commits into
entrius:testfrom
seroperson:psycopg

Conversation

@seroperson

Copy link
Copy Markdown
Contributor

Closes #902

Summary

  1. Replace psycopg2-binary with psycopg[binary]>=3.2
  2. Drop execute_values (unsupported in v3) and switch each bulk upsert to cursor.executemany, with the SQL templates expanded from VALUES %s to explicit VALUES (%s, %s, ...) row tuples
  3. Wrap all per-miner writes inside db_connection.pipeline() to batch queries into one round-trip while preserving the existing transaction boundary. From documentation:

"The pipeline mode allows PostgreSQL client applications to send a query without having to read the result of the previously sent query... multiple queries/results can be sent/received in a single network roundtrip."

  1. Set connection.prepare_threshold = 0 so our queries are auto-prepared, instead of old behavior, when it was undefined. From documentation:

"A query is prepared automatically after it is executed more than prepare_threshold times on a connection..."

  1. No schema change, no public API change on Repository or DatabaseStorage
  2. Use dbname arg instead of database, like in canonical usage example: https://www.psycopg.org/psycopg3/docs/basic/usage.html#main-objects-in-psycopg-3; Keyword list: https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS

Test plan

I haven't added new tests as currently there are no any tests which cover the real database connection, either any migrations to spin-up everything locally. I did some synthetic manual testing and everything seems good.

I can try to implement necessary code here to test it against a real connection and do real tests if you want.

@xiao-xiao-mao xiao-xiao-mao Bot added the refactor Code restructuring without behavior change label Apr 30, 2026

@anderdc anderdc left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Fix conflicts.
  2. Pin psycopg[binary] to a specific version (e.g. ==3.3.3) — rest of pyproject.toml uses exact pins.

@anderdc
anderdc merged commit 5530c42 into entrius:test May 6, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

refactor Code restructuring without behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate psycopg2 -> psycopg 3

2 participants