Skip to content

Commit 13d4cad

Browse files
raulbraulb
authored
refactor: remove key field and update table field (#135)
* refactor: use tables in plural * update readme * bump conduit-commons and connector-protocol * update readme * use old field name * update readme * remove keys from configuration * fix lint * add support to table * rename connection pool var * fix test * fix readme * remove comment
1 parent 4466f41 commit 13d4cad

File tree

7 files changed

+101
-110
lines changed

7 files changed

+101
-110
lines changed

‎README.md‎

Lines changed: 19 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,15 @@
33
# Source
44

55
The Postgres Source Connector connects to a database with the provided `url` and starts creating records for each change
6-
detected in a table.
6+
detected in the provided tables.
77

8-
Upon starting, the source takes a snapshot of a given table in the database, then switches into CDC mode. In CDC mode,
8+
Upon starting, the source takes a snapshot of the provided tables in the database, then switches into CDC mode. In CDC mode,
99
the plugin reads from a buffer of CDC events.
1010

1111
## Snapshot Capture
1212

13-
When the connector first starts, snapshot mode is enabled. The connector acquires a read-only lock on the table, and
14-
then reads all rows of the table into Conduit. Once all rows in that initial snapshot are read the connector releases
13+
When the connector first starts, snapshot mode is enabled. The connector acquires a read-only lock on the tables, and
14+
then reads all rows of the tables into Conduit. Once all rows in that initial snapshot are read the connector releases
1515
its lock and switches into CDC mode.
1616

1717
This behavior is enabled by default, but can be turned off by adding `"snapshotMode":"never"` to the Source
@@ -37,8 +37,7 @@ Example configuration for CDC features:
3737
```json
3838
{
3939
"url": "url",
40-
"key": "key",
41-
"table": "records",
40+
"tables": "records",
4241
"cdcMode": "logrepl",
4342
"logrepl.publicationName": "meroxademo",
4443
"logrepl.slotName": "meroxademo"
@@ -47,20 +46,20 @@ Example configuration for CDC features:
4746

4847
## Key Handling
4948

50-
If no `key` field is provided, then the connector will attempt to look up the primary key column of the table. If that
51-
can't be determined it will fail.
49+
The connector will automatically look up the primary key column for the specified tables. If that can't be determined,
50+
the connector will return an error.
5251

5352
## Configuration Options
5453

55-
| name | description | required | default |
56-
|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------|
57-
| `url` | Connection string for the Postgres database. | true | |
58-
| `table` | List of table names to read from, separated by comma. Using `*` will read from all public tables. | true | |
59-
| `key` | List of Key column names per table, separated by comma. Example:`"table1:key1,table2:key2"`, if not supplied, the table(s) primary keys will be used as the `'Key'` field for the records. | false | |
60-
| `snapshotMode` | Whether or not the plugin will take a snapshot of the entire table before starting cdc mode (allowed values: `initial` or `never`). | false | `initial` |
61-
| `cdcMode` | Determines the CDC mode (allowed values: `auto`, `logrepl` or `long_polling`). | false | `auto` |
62-
| `logrepl.publicationName` | Name of the publication to listen for WAL events. | false | `conduitpub` |
63-
| `logrepl.slotName` | Name of the slot opened for replication events. | false | `conduitslot` |
54+
| name | description | required | default |
55+
|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------|----------|---------------|
56+
| `url` | Connection string for the Postgres database. | true | |
57+
| `tables` | List of table names to read from, separated by comma. Example: `"employees,offices,payments"`. Using `*` will read from all public tables. | true | |
58+
| `snapshotMode` | Whether or not the plugin will take a snapshot of the entire table before starting cdc mode (allowed values: `initial` or `never`). | false | `initial` |
59+
| `cdcMode` | Determines the CDC mode (allowed values: `auto`, `logrepl` or `long_polling`). | false | `auto` |
60+
| `logrepl.publicationName` | Name of the publication to listen for WAL events. | false | `conduitpub` |
61+
| `logrepl.slotName` | Name of the slot opened for replication events. | false | `conduitslot` |
62+
| ~~`table`~~ | List of table names to read from, separated by comma. **Deprecated: use `tables` instead.** | false | |
6463

6564
# Destination
6665

@@ -77,11 +76,10 @@ If there is no key, the record will be simply appended.
7776

7877
## Configuration Options
7978

80-
| name | description | required | default |
81-
|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------------------------------------------|
82-
| `url` | Connection string for the Postgres database. | true | |
79+
| name | description | required | default |
80+
|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|----------------------------------------------|
81+
| `url` | Connection string for the Postgres database. | true | |
8382
| `table` | Table name. It can contain a Go template that will be executed for each record to determine the table. By default, the table is the value of the `opencdc.collection` metadata field. | false | `{{ index .Metadata "opencdc.collection" }}` |
84-
| `key` | Column name used to detect if the target table already contains the record. | false | |
8583

8684
# Testing
8785

‎go.mod‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@ go 1.22
55
require (
66
github.com/Masterminds/sprig/v3 v3.2.3
77
github.com/Masterminds/squirrel v1.5.4
8-
github.com/conduitio/conduit-commons v0.1.2-0.20240405195636-cb5e072472b0
9-
github.com/conduitio/conduit-connector-sdk v0.8.1-0.20240408123504-cec49fc57887
8+
github.com/conduitio/conduit-commons v0.2.0
9+
github.com/conduitio/conduit-connector-sdk v0.9.1
1010
github.com/daixiang0/gci v0.13.4
1111
github.com/golangci/golangci-lint v1.57.2
1212
github.com/google/go-cmp v0.6.0
@@ -54,7 +54,7 @@ require (
5454
github.com/charithe/durationcheck v0.0.10 // indirect
5555
github.com/chavacava/garif v0.1.0 // indirect
5656
github.com/ckaznocha/intrange v0.1.1 // indirect
57-
github.com/conduitio/conduit-connector-protocol v0.5.1-0.20240408121719-ffe7a46af296 // indirect
57+
github.com/conduitio/conduit-connector-protocol v0.6.0 // indirect
5858
github.com/curioswitch/go-reassign v0.2.0 // indirect
5959
github.com/davecgh/go-spew v1.1.1 // indirect
6060
github.com/denis-tingaikin/go-header v0.5.0 // indirect
@@ -203,16 +203,16 @@ require (
203203
go.uber.org/multierr v1.11.0 // indirect
204204
go.uber.org/zap v1.27.0 // indirect
205205
golang.org/x/crypto v0.22.0 // indirect
206-
golang.org/x/exp v0.0.0-20240325151524-a685a6edb6d8 // indirect
206+
golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f // indirect
207207
golang.org/x/exp/typeparams v0.0.0-20240314144324-c7f7c6466f7f // indirect
208208
golang.org/x/mod v0.17.0 // indirect
209209
golang.org/x/net v0.24.0 // indirect
210210
golang.org/x/sync v0.7.0 // indirect
211211
golang.org/x/sys v0.19.0 // indirect
212212
golang.org/x/text v0.14.0 // indirect
213213
golang.org/x/time v0.5.0 // indirect
214-
google.golang.org/genproto/googleapis/rpc v0.0.0-20240325203815-454cdb8f5daa // indirect
215-
google.golang.org/grpc v1.63.0 // indirect
214+
google.golang.org/genproto/googleapis/rpc v0.0.0-20240415180920-8c6c420018be // indirect
215+
google.golang.org/grpc v1.63.2 // indirect
216216
google.golang.org/protobuf v1.33.0 // indirect
217217
gopkg.in/ini.v1 v1.67.0 // indirect
218218
gopkg.in/yaml.v2 v2.4.0 // indirect

‎go.sum‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -126,12 +126,12 @@ github.com/ckaznocha/intrange v0.1.1 h1:gHe4LfqCspWkh8KpJFs20fJz3XRHFBFUV9yI7Itu
126126
github.com/ckaznocha/intrange v0.1.1/go.mod h1:RWffCw/vKBwHeOEwWdCikAtY0q4gGt8VhJZEEA5n+RE=
127127
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
128128
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
129-
github.com/conduitio/conduit-commons v0.1.2-0.20240405195636-cb5e072472b0 h1:LCnxTOn9L/vJCX6TThdJjpLfh5Fr9uGP9/5AkZzdb7w=
130-
github.com/conduitio/conduit-commons v0.1.2-0.20240405195636-cb5e072472b0/go.mod h1:shChx2O5D22aUnw6L5biPiV2Tm0CCL+UwKh758lSEfE=
131-
github.com/conduitio/conduit-connector-protocol v0.5.1-0.20240408121719-ffe7a46af296 h1:Qhl0Icbhh/32MuVfv+YReQMtH6qeKq+tMrxL2XC1U34=
132-
github.com/conduitio/conduit-connector-protocol v0.5.1-0.20240408121719-ffe7a46af296/go.mod h1:C1nWLpGCmK2bTjC5CGPAmjRhgBccx6/aS7XDDHKdZYA=
133-
github.com/conduitio/conduit-connector-sdk v0.8.1-0.20240408123504-cec49fc57887 h1:IZMXstp4NK+SRw/SNg6RsPD4FE2Lzw4nKGI0ijaMvrU=
134-
github.com/conduitio/conduit-connector-sdk v0.8.1-0.20240408123504-cec49fc57887/go.mod h1:U48f9UoQKEaPMX7beOXR6x+uRl4XbG7A2KnH7MqZrLs=
129+
github.com/conduitio/conduit-commons v0.2.0 h1:TMpVGXi0Wski537qLAyQWdGjuGHEhaZxOS5L90pZJSQ=
130+
github.com/conduitio/conduit-commons v0.2.0/go.mod h1:i7Q2jm7FBSi2zj1/4MCsFD1hIKAbvamlNtSQfkhUTiY=
131+
github.com/conduitio/conduit-connector-protocol v0.6.0 h1:2gMOCOpa+c97CHIpZv7Niu3V4o5UgRr6fzj9kzfRV7o=
132+
github.com/conduitio/conduit-connector-protocol v0.6.0/go.mod h1:3mo59xYX9etFoR3n82R7J50La1iWK+Vm63H8z2wo4QM=
133+
github.com/conduitio/conduit-connector-sdk v0.9.1 h1:DiMUn7udnjWvyaDsyeTZFHeYTEIdqUU6dqPunEEE3Kw=
134+
github.com/conduitio/conduit-connector-sdk v0.9.1/go.mod h1:cNoofumgDlsaThkxkNYg7zab4AkmRZt1V711aO7guGU=
135135
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
136136
github.com/cpuguy83/go-md2man/v2 v2.0.3/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
137137
github.com/curioswitch/go-reassign v0.2.0 h1:G9UZyOcpk/d7Gd6mqYgd8XYWFMw/znxwGDUstnC9DIo=
@@ -280,8 +280,8 @@ github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hf
280280
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
281281
github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
282282
github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
283-
github.com/google/pprof v0.0.0-20240327155427-868f304927ed h1:n8QtJTrwsv3P7dNxPaMeNkMcxvUpqocsHLr8iDLGlQI=
284-
github.com/google/pprof v0.0.0-20240327155427-868f304927ed/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
283+
github.com/google/pprof v0.0.0-20240422182052-72c8669ad3e7 h1:3q13T5NW3mlTJZM6B5UAsf2N5NYFbYWIyI3W8DlvBDU=
284+
github.com/google/pprof v0.0.0-20240422182052-72c8669ad3e7/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
285285
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
286286
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
287287
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
@@ -666,8 +666,8 @@ golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u0
666666
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
667667
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
668668
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
669-
golang.org/x/exp v0.0.0-20240325151524-a685a6edb6d8 h1:aAcj0Da7eBAtrTp03QXWvm88pSyOt+UgdZw2BFZ+lEw=
670-
golang.org/x/exp v0.0.0-20240325151524-a685a6edb6d8/go.mod h1:CQ1k9gNrJ50XIzaKCRR2hssIjF07kZFEiieALBM/ARQ=
669+
golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f h1:99ci1mjWVBWwJiEKYY6jWa4d2nTQVIEhZIptnrVb1XY=
670+
golang.org/x/exp v0.0.0-20240416160154-fe59bbe5cc7f/go.mod h1:/lliqkxwWAhPjf5oSOIJup2XcqJaw8RGS6k3TGEc7GI=
671671
golang.org/x/exp/typeparams v0.0.0-20220428152302-39d4317da171/go.mod h1:AbB0pIl9nAr9wVwH+Z2ZpaocVmF5I4GyWCDIsVjR0bk=
672672
golang.org/x/exp/typeparams v0.0.0-20230203172020-98cc5a0785f9/go.mod h1:AbB0pIl9nAr9wVwH+Z2ZpaocVmF5I4GyWCDIsVjR0bk=
673673
golang.org/x/exp/typeparams v0.0.0-20240314144324-c7f7c6466f7f h1:phY1HzDcf18Aq9A8KkmRtY9WvOFIxN8wgfvy6Zm1DV8=
@@ -965,8 +965,8 @@ google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7Fc
965965
google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
966966
google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
967967
google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
968-
google.golang.org/genproto/googleapis/rpc v0.0.0-20240325203815-454cdb8f5daa h1:RBgMaUMP+6soRkik4VoN8ojR2nex2TqZwjSSogic+eo=
969-
google.golang.org/genproto/googleapis/rpc v0.0.0-20240325203815-454cdb8f5daa/go.mod h1:WtryC6hu0hhx87FDGxWCDptyssuo68sk10vYjF+T9fY=
968+
google.golang.org/genproto/googleapis/rpc v0.0.0-20240415180920-8c6c420018be h1:LG9vZxsWGOmUKieR8wPAUR3u3MpnYFQZROPIMaXh7/A=
969+
google.golang.org/genproto/googleapis/rpc v0.0.0-20240415180920-8c6c420018be/go.mod h1:WtryC6hu0hhx87FDGxWCDptyssuo68sk10vYjF+T9fY=
970970
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
971971
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
972972
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
@@ -979,8 +979,8 @@ google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKa
979979
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
980980
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
981981
google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
982-
google.golang.org/grpc v1.63.0 h1:WjKe+dnvABXyPJMD7KDNLxtoGk5tgk+YFWN6cBWjZE8=
983-
google.golang.org/grpc v1.63.0/go.mod h1:WAX/8DgncnokcFUldAxq7GeB5DXHDbMF+lLvDomNkRA=
982+
google.golang.org/grpc v1.63.2 h1:MUeiw1B2maTVZthpU5xvASfTh3LDbxHd6IJ6QQVU+xM=
983+
google.golang.org/grpc v1.63.2/go.mod h1:WAX/8DgncnokcFUldAxq7GeB5DXHDbMF+lLvDomNkRA=
984984
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
985985
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
986986
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=

0 commit comments

Comments
 (0)