DatabaseConnector 7

[schuemie]

When DatabaseConnector was first created in April 2014 DBI was still in its infancy, with no drivers for most platforms, and dbplyr did not exist yet. In the 10 years since then, the landscape has changed dramatically, also forcing us to re-evaluate the role of DatatabaseConnector.

I believe there still is a place for DatabaseConnector. DBI is a rather loose interface, leaving many per-platform idiosyncrasies. I propose DatabaseConnector is the place where those idiosyncrasies are resolved, so that anybody wanting to write analytics packages that run on all platforms we support in OHDSI, they only need to know about DatabaseConnector, and not have to worry about each individual platform. In other words, if DatabaseConnector does its job, no other OHDS package should have code that looks like if dbms == 'postgresql' do this else do this. All that should be handled by DatabaseConnector. In that sense, I think of DatabaseConnector as the 'last mile' solver: it standardizes what DBI doesn't. Hopefully in the future we won't need DatabaseConnector anymore, and we should work towards that.

Translation mechanisms

Each database platform speaks its own SQL dialect. We initially developed SqlRender so OHDSI packages would write SQL once (in what we now call OhdsiSql), and we can translate that to all other dialects. Since then, dbplyr has been developed, which allows constructing platform-specific SQL by chaining R commands. Each has its pros and cons:

SqlRender pros:

• Easier for people who know SQL
• Allows date operations (e.g. DATE_ADD(), DATE_DIFF(), YEAR())
• Closer link between SQL you write and SQL that gets executed
• Allows translation of SQL not generated in R (e.g. allows translation of Circe cohort SQL).

SqlRender cons:

• Harder to debug in the context of R code
• SQL itself is not very structured (e.g. can't define functions in a cross-platform compatible way), leading to long code chunks.
• Requires a set of translation rules per dialect
• Not included in unit test coverage statistcs

dbplyr pros:

• Easier for people who know R (especially dplyr)
• Easier to write and debug in the context of R code
• Easier to structure, e.g. move parts into functions
• No mixing of languages in a package

dbplyr cons:

• No consistent support for date operations
• More separation between code you write and SQL that gets executed. (SQL could be highly suboptimal)
• Doesn't help if you already have SQL (E.g. Circe cohort definitions)

I therefore think for the foreseeable future we want to support both SqlRender and dbplyr.

Database connections

Currently most connections are supported via JDBC drivers, although SQLite and DuckDB now use their respective DBI drivers. I think we should be agnostic to the type of driver used, and perhaps we can switch some of the JDBC drivers to DBI. I do still think that

1. DatabaseConnector should wrap the other drivers. That will allow DatabaseConnector to shield the user from database-specific idiosyncrasies.
2. DatabaseConnector should specify the exact driver to use for each platform. For example, there are multiple DBI drivers and a JDBC driver for Postgres, but we should pick only one we support. If we don't do that, we won't be able to ensure DatabaseConnector behaves in a consistent manner, because every driver (like every platform) has its own weird things (sometimes even changing the SQL dialect).

Proposed high-level changes

For DatabaseConnector 7 I suggest we lean more into the role of last-mile solver. DatabaseConnector should get leaner, offloading as much as possible to other packages (e.g. dbplyr support should be done entirely by back-ends not in DatabaseConnector). I also suggest we more closely align with DBI, and some of its unwritten conventions. (e.g. DatabaseConnector uses databaseSchema, whereas DBI unofficially uses catalog and schema).

I'm unsure what to do with renderTranslateQuerySql() and renderTranslateExecuteSql(). I use these functions all the time, because they (a) translate on the fly, (b) throw nice shareable error reports, and (c) have progress bars. However, they are not part of the DBI specification.

[ablack3]

Very nice!

I'll add a few links and ideas.

Consider supporting Boolean datatypes which are available on the following dbms

This is nice because users sometimes want to move data from R to the database and other drivers will preserve the datatype as boolean. So if a user changes drivers their code will not need to change.

• Oracle >=23: https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlrf/Data-Types.html#GUID-285FFCA8-390D-4FA9-9A51-47B84EF5F83A
• BigQuery: https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#boolean_type
• Postgres: https://www.postgresql.org/docs/current/datatype-boolean.html
• Redshift: https://docs.aws.amazon.com/redshift/latest/dg/r_Boolean_type.html
• Snowflake: https://docs.snowflake.com/en/sql-reference/data-types-logical
• Databricks/Spark: https://docs.databricks.com/en/sql/language-manual/data-types/boolean-type.html
• duckdb: https://duckdb.org/docs/sql/data_types/boolean.html
• Sql Server: https://learn.microsoft.com/en-us/sql/t-sql/language-elements/comparison-operators-transact-sql?view=sql-server-ver16#boolean-data-type

SQLite does not have a bool type but apparently will store the keyword TRUE as 1 and FALSE as 0. It also does not have date types. duckdb is now past v1 and might be better than sqlite for many of our use cases.

Hive, netezza, impala, and pdw, (and possibly synapse?) aren't really supported since we don't have test servers so it seems like all OHDSI databases except sqlite have boolean datatypes.

DBITest

We could try to improve DatabaseConnectors DBI support by implementing more tests from the DBITest package. https://dbitest.r-dbi.org/

dbplyr integration

This PR might work with dbplyr. Since DatabaseConnector supports so many dialects I guess we need to set the class of the connection type dynamically when the connection is created.

I think we should keep renderTranslateQuerySql() and renderTranslateExecuteSql() simply because they are used a lot in ohdsi code. To make the package leaner we could make the dependency on SqlRender optional (Suggests rather than Depends).

Possible issues to fix:

I got this when I add the PqConnection class to a databaseconnector connection

[schuemie]

@ablack3 and I discussed the following:

Plan for DatabaseConnector version 7:

1. Swap out dbplyr backends. The current implementation uses the SQL Server backend, after which we use SqlRender to translate to the various SQL dialects. The new implementation will use each DBMS's respective backend instead, without using SqlRender. This probably requires the DatabaseConnector connections to inherit from the ODBC classes.
2. Make sure all top-level functions (e.g. executeSql, connect) talk DBI to lower-level functions. Currently these functions have tight integration with some of the driver implementations, which makes it harder to switch out drivers when needed.
3. Adopt DBI's approach for catalog & schema, deprecate databaseSchema. These are two solutions to the same problem, but the DBI approach keeps the two separate, for example making it easier to add quotes.
4. Make sure all drivers in DatabaseConnector (JDBC + wrapped DBI drivers) conform to DBI standard. No automatic translation.
5. Add support for logical data types. As described in Adam's post. SQLite doesn't support logical types, so we will stick to the current behavior there (convert to INT and throw a warning). We suspect we will phase out SQLite in favor of DuckDB in OHDSI in the future.

At a higher level, the idea is to define some standard we'll call DBI+, which is DBI plus additional constraints currently not specified in DBI, for example:

• Support for temp tables, even on platforms that don't have them
• Bigint handling
• DATETIME timezone handling (when timezone not specified on server, use local timezone, server timezone, or UTC?)

DatabaseConnector will enforce DBI+ for its drivers, truly allowing the same code to be used for all platforms. At some point, we may convince Posit to adopt DBI+ themselves, at which point DatabaseConnector will become mostly obsolete.

The higher-level connect() function will keep its two important features:

1. Allow connectionDetails input. This is necessary for more complex use cases where multiple connections need to be spawned (in multiple threads).
2. Select a single driver per platform. This will mean that packages that enforce the use of connect() (e.g. all packages in HADES) have only one driver per platform they need to test against.

Figure 1 General architecture overview. The higher-level functions will interact with the drivers through DBI+. The drivers will implement DBI+. If a DBI driver doesn't already implement DBI+, it will need to be wrapped. Otherwise, we can use it as is.

@ablack3 : could you check if I represented what we discussed correctly?

[ablack3]

Yes @schuemie this is what we discussed and seems like a good plan to me. I actually think we could use odbc, RPostgres, and bigrquery DBI driver packages possibly removing the need to use JDBC at all. I do like having the option to use JDBC or odbc with DatabaseConnector since I have found that one or the other works better in certain environments (sometimes there are weird technical issues so having a backup is nice). I think that we agreed that DatabaseConnector 7 could support both the current JDBC drivers that are used by the package as well as add wrappers around other DBI packages. However in Hades we would use one set of drivers that are well tested. I will start working on "1. Swap out dbplyr backends" since this is important for the use of DatabaseConnector with some of the tools developed in Darwin. For this I will define a new class for each connection type "DatabaseConnectorConnectionPostgres" for example that inherits from both "DatabaseConnectorConnection" and "PqConnection" so that the dbplyr translation environment will be handled correctly. https://github.com/tidyverse/dbplyr/blob/860bd6adf988a2b039485030d234eba05ee46a3c/R/backend-postgres.R#L79

Also I'll note that with the help of @edward-burn we are close to having all the date manipulation functionality we need in dplyr. So one can write dplyr code that will work on dataframes or any ohdsi dbms that does date manipulations correctly.
These are done using functions in the clock package. https://github.com/tidyverse/dbplyr/blob/860bd6adf988a2b039485030d234eba05ee46a3c/R/backend-postgres.R#L239