You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This tutorial walks you through the process of creating a self-managed connector for a PostgreSQL data source. You’ll be using the [self-managed connector](/reference/search-connectors/self-managed-connectors.md) workflow in the Kibana UI. This means you’ll be deploying the connector on your own infrastructure. Refer to the [Elastic PostgreSQL connector reference](/reference/search-connectors/es-connectors-postgresql.md) for more information about this connector.
12
-
12
+
# Set up a self-managed connector to search a PostgreSQL database
13
+
14
+
This tutorial walks you through the process of creating a self-managed connector for a PostgreSQL data source.
15
+
<!--
16
+
TBD: Moot point now?
17
+
You’ll be using the [self-managed connector](/reference/search-connectors/self-managed-connectors.md) workflow in the Kibana UI. This means you’ll be deploying the connector on your own infrastructure.
18
+
-->
19
+
<!--
20
+
TBD: Too soon to refer elsewhere?
21
+
Refer to the [Elastic PostgreSQL connector reference](/reference/search-connectors/es-connectors-postgresql.md) for more information about this connector.
22
+
-->
13
23
In this exercise, you’ll be working in both the terminal (or your IDE) and the Kibana UI.
14
24
15
25
If you want to deploy a self-managed connector for another data source, use this tutorial as a blueprint. Refer to the list of available [connectors](/reference/search-connectors/index.md).
16
26
17
27
::::{tip}
18
28
Want to get started quickly testing a self-managed connector using Docker Compose? Refer to this [guide](https://github.com/elastic/connectors/tree/main/scripts/stack#readme) in the `elastic/connectors` repo for more information.
29
+
<!--
30
+
TBD: Looks like this is only for the Stack scenario, not serverless?
First, ensure you satisfy the [prerequisites](/reference/search-connectors/self-managed-connectors.md#es-build-connector-prerequisites) for self-managed connectors.
27
39
40
+
<!--
41
+
TBD: What privileges are required to complete these steps?
42
+
- If you're using {{serverless-full}}, you must have a `developer` or `admin` predefined role or an equivalent custom role to add the connector.
43
+
- If you're using {{ech}} or a self-managed cluster, start {{es}} and {{kib}}. To add the connector, you must have ...
To complete this tutorial, you’ll need to complete the following steps:
55
-
56
-
1.[Create an Elasticsearch index](#es-postgresql-connector-client-tutorial-create-index)
57
-
2.[Set up the connector](#es-postgresql-connector-client-tutorial-setup-connector)
58
-
3.[Run the `connectors` connector service](#es-postgresql-connector-client-tutorial-run-connector-service)
59
-
4.[Sync your PostgreSQL data source](#es-postgresql-connector-client-tutorial-sync-data-source)
71
+
::::{step} Create an Elasticsearch index
72
+
Elastic connectors enable you to create searchable, read-only replicas of your data sources in Elasticsearch.
73
+
The first step in setting up your self-managed connector is to create an index.
60
74
75
+
In the Kibana UI, navigate to **{{index-manage-app}}** or use the [global search field](docs-content://explore-analyze/find-and-organize/find-apps-and-objects.md).
61
76
62
-
## Create an Elasticsearch index [es-postgresql-connector-client-tutorial-create-index]
63
-
64
-
Elastic connectors enable you to create searchable, read-only replicas of your data sources in Elasticsearch. The first step in setting up your self-managed connector is to create an index.
65
-
66
-
In the [Kibana^](docs-content://get-started/the-stack.md) UI, navigate to **Search > Content > Elasticsearch indices** from the main menu, or use the [global search field](docs-content://explore-analyze/query-filter/filtering.md#_finding_your_apps_and_objects).
67
-
68
-
Create a new connector index:
77
+
{applies_to}`serverless: ga` Follow the index creation workflow then optionally define field mappings.
78
+
For example, to add semantic search capabilities, you could add an extra field that stores your vectors for semantic search.
69
79
80
+
{applies_to}`stack: ga` Create a new connector index:
81
+
<!--
82
+
TBD: This steps don't apply to serverless?
83
+
-->
70
84
1. Under **Select an ingestion method** choose **Connector**.
71
85
2. Choose **PostgreSQL** from the list of connectors.
72
86
3. Name your index and optionally change the language analyzer to match the human language of your data source. (The index name you provide is automatically prefixed with `search-`.)
Before you can configure the connector, you need to gather some details about your Elastic deployment:
81
-
82
-
***Elasticsearch endpoint**.
83
-
84
-
* If you’re an Elastic Cloud user, find your deployment’s Elasticsearch endpoint in the Cloud UI under **Cloud > Deployments > <your-deployment> > Elasticsearch**.
85
-
* If you’re running your Elastic deployment and the connector service in Docker, the default Elasticsearch endpoint is `http://host.docker.internal:9200`.
86
-
87
-
***API key.** You’ll need this key to configure the connector. Use an existing key or create a new one.
88
-
***Connector ID**. Your unique connector ID is automatically generated when you create the connector. Find this in the Kibana UI.
89
-
90
90
::::
91
91
92
+
::::{step} Configure the connector
93
+
Once you’ve created an index, you can set up the connector.
94
+
You will be guided through this process in the UI.
92
95
96
+
In the Kibana UI, navigate to **{{connectors-app}}** or use the global search field.
97
+
If you're using {{es-serverless}}, click **{icon}`plus_in_circle` Self-managed connector**.
93
98
94
-
## Set up the connector [es-postgresql-connector-client-tutorial-setup-connector]
95
-
96
-
Once you’ve created an index, you can set up the connector. You will be guided through this process in the UI.
97
-
98
-
1.**Edit the name and description for the connector.** This will help your team identify the connector.
99
-
2.**Clone and edit the connector service code.** For this example, we’ll use the [Python framework](https://github.com/elastic/connectors/tree/main). Follow these steps:
100
-
101
-
* Clone or fork that repository locally with the following command: `git clone https://github.com/elastic/connectors`.
102
-
* Open the `config.yml` configuration file in your editor of choice.
103
-
* Replace the values for `host`, `api_key`, and `connector_id` with the values you gathered [earlier](#es-postgresql-connector-client-tutorial-gather-elastic-details). Use the `service_type` value `postgresql` for this connector.
104
-
105
-
::::{dropdown} Expand to see an example config.yml file
106
-
Replace the values for `host`, `api_key`, and `connector_id` with your own values. Use the `service_type` value `postgresql` for this connector.
107
-
108
-
```yaml
109
-
elasticsearch:
110
-
host: <https://<my-elastic-deployment.es.us-west2.gcp.elastic-cloud.com>> # Your Elasticsearch endpoint
111
-
api_key: '<YOUR-API-KEY>'# Your top-level Elasticsearch API key
112
-
...
113
-
connectors:
114
-
-
115
-
connector_id: "<YOUR-CONNECTOR-ID>"
116
-
api_key: "'<YOUR-API-KEY>"# Your scoped connector index API key (optional). If not provided, the top-level API key is used.
117
-
service_type: "postgresql"
118
-
119
-
99
+
1. Edit the name and description for the connector. This will help your team identify the connector.
100
+
1. {applies_to}`serverless: ga` Choose PostgreSQL from the list of connector types.
101
+
1. {applies_to}`serverless: ga` Copy the configuration details. Keep in mind the `api_key` for both `connector` and `elasticsearch` variables is the same.
102
+
1. {applies_to}`stack: ga` Gather configuration details. Before you can proceed to the next step, you need to gather some details about your Elastic deployment:
120
103
121
-
# Self-managed connector settings
122
-
connector_id: '<YOUR-CONNECTOR-ID>'# Your connector ID
123
-
service_type: 'postgresql'# The service type for your connector
## Run the connector service [es-postgresql-connector-client-tutorial-run-connector-service]
145
-
146
-
Now that you’ve configured the connector code, you can run the connector service.
147
-
148
-
In your terminal or IDE:
149
-
150
-
1. `cd` into the root of your `connectors` clone/fork.
151
-
2. Run the following command: `make run`.
104
+
* Elasticsearch endpoint:
105
+
* If you’re an Elastic Cloud user, find your deployment’s Elasticsearch endpoint in the Cloud UI under **Cloud > Deployments > <your-deployment> > Elasticsearch**.
106
+
* If you’re running your Elastic deployment and the connector service in Docker, the default Elasticsearch endpoint is `http://host.docker.internal:9200`.
107
+
* API key: You’ll need this key to configure the connector. Use an existing key or create a new one.
108
+
* Connector ID: Your unique connector ID is automatically generated when you create the connector. Find this in the Kibana UI.
109
+
::::
110
+
::::{step} Link your connector to Elasticsearch
111
+
You need to run the connector code on your own infrastructure and link it to {{es}}.
112
+
You have two options: [Run with Docker](/search-connectors/es-connectors-run-from-docker.md) and [Run from source](/search-connectors/es-connectors-run-from-source.md).
113
+
For this example, we’ll use the latter method:
114
+
115
+
1. Clone or fork that repository locally with the following command: `git clone https://github.com/elastic/connectors`.
116
+
1. Open the `config.yml` configuration file in your editor of choice.
117
+
1. Replace the values for `host`, `api_key`, and `connector_id` with the values you gathered earlier. Use the `service_type` value `postgresql` for this connector.
118
+
119
+
:::{dropdown} Expand to see an example config.yml file
120
+
Replace the values for `host`, `api_key`, and `connector_id` with your own values. Use the `service_type` value `postgresql` for this connector.
121
+
122
+
```yaml
123
+
elasticsearch:
124
+
host: <https://<my-elastic-deployment.es.us-west2.gcp.elastic-cloud.com>> # Your Elasticsearch endpoint
125
+
api_key: '<YOUR-API-KEY>'# Your top-level Elasticsearch API key
126
+
...
127
+
connectors:
128
+
-
129
+
connector_id: "<YOUR-CONNECTOR-ID>"
130
+
api_key: "'<YOUR-API-KEY>"# Your scoped connector index API key (optional). If not provided, the top-level API key is used.
131
+
service_type: "postgresql"
132
+
133
+
134
+
135
+
# Self-managed connector settings
136
+
connector_id: '<YOUR-CONNECTOR-ID>'# Your connector ID
137
+
service_type: 'postgresql'# The service type for your connector
1. Now that you’ve configured the connector code, you can run the connector service. In your terminal or IDE:
156
+
157
+
1. `cd` into the root of your `connectors` clone/fork.
158
+
1. Run the following command: `make run`.
152
159
153
160
The connector service should now be running. The UI will let you know that the connector has successfully connected to Elasticsearch.
154
161
155
-
Here we’re working locally. In production setups, you’ll deploy the connector service to your own infrastructure. If you prefer to use Docker, refer to the [repo docs](https://github.com/elastic/connectors/tree/main/docs/DOCKER.md) for instructions.
156
-
157
-
158
-
## Sync your PostgreSQL data source [es-postgresql-connector-client-tutorial-sync-data-source]
159
-
160
-
161
-
### Enter your PostgreSQL data source details [es-postgresql-connector-client-tutorial-sync-data-source-details]
162
-
162
+
:::{tip}
163
+
Here we’re working locally. In production setups, you’ll deploy the connector service to your own infrastructure.
164
+
:::
165
+
::::
166
+
::::{step} Enter your data source details
163
167
Once you’ve configured the connector, you can use it to index your data source.
164
168
165
-
You can now enter your PostgreSQL instance details in the Kibana UI.
166
-
167
-
Enter the following information:
169
+
Enter the following PostgreSQL instance details in the Kibana UI:
168
170
169
-
* **Host**. Server host address for your PostgreSQL instance.
170
-
* **Port**. Port number for your PostgreSQL instance.
171
-
* **Username**. Username of the PostgreSQL account.
172
-
* **Password**. Password for that user.
173
-
* **Database**. Name of the PostgreSQL database.
174
-
* **Comma-separated list of tables**. `*` will fetch data from all tables in the configured database.
171
+
* **Host**: The server host address for your PostgreSQL instance.
172
+
* **Port**: The port number for your PostgreSQL instance.
173
+
* **Username**: The username of the PostgreSQL account.
174
+
* **Password**: The password for that user.
175
+
* **Database**: The name of the PostgreSQL database.
176
+
* **Comma-separated list of tables**: `*` will fetch data from all tables in the configured database.
175
177
176
-
Once you’ve entered all these details, select **Save configuration**.
177
-
178
-
179
-
### Launch a sync [es-postgresql-connector-client-tutorial-sync-data-source-launch-sync]
178
+
After you’ve entered all these details, select **Save configuration**.
179
+
::::
180
+
::::{step} Link your index
181
+
{applies_to}`serverless: ga` Select the index to which the data must be synchronized.
182
+
In this case, use the index we created in an earlier step.
183
+
Click **Sync**.
184
+
The connector will traverse the database and synchronize documents to your index.
185
+
<!--
186
+
TBD: Confirm whether the status at this stage should be "configured" or "sync complete"
187
+
In the main **{{connectors-app}}** page, you can see the connector’s status.
188
+
It should now indicate that the connector is configured and that the sync is complete.
189
+
-->
190
+
::::
191
+
:::::
180
192
181
-
If you navigate to the **Overview** tab in the Kibana UI, you can see the connector’s *ingestion status*. This should now have changed to **Configured**.
193
+
## Sync your data source [es-postgresql-connector-client-tutorial-sync-data-source]
182
194
183
-
It’s time to launch a sync by selecting the **Sync** button.
195
+
In the **{{connectors-app}}** page, you can launch a sync on-demand or on a schedule.
184
196
185
-
If you navigate to the terminal window where you’re running the connector service, you should see output like the following:
197
+
If you navigate to the terminal window where you’re running the connector service, after a sync occurs you should see output like the following:
@@ -193,14 +205,21 @@ If you navigate to the terminal window where you’re running the connector serv
193
205
(27 seconds)
194
206
```
195
207
196
-
This confirms the connector has fetched records from your PostgreSQL table(s) and transformed them into documents in your Elasticsearch index.
208
+
This confirms the connector has fetched records from your PostgreSQL tables and transformed them into documents in your Elasticsearch index.
209
+
210
+
If you verify your Elasticsearch documents and you’re happy with the results, set a recurring sync schedule.
211
+
This will ensure your searchable data in {{es}} is always up to date with changes to your PostgreSQL data source.
197
212
198
-
Verify your Elasticsearch documents in the **Documents** tab in the Kibana UI.
213
+
For example, in the **{{connectors-app}}** page, click on the connector, and then click **Scheduling**.
214
+
Select every hour and click **Save**.
215
+
The content will be synchronized at the top of every hour, as long as the connector is up and running.
199
216
200
-
If you’re happy with the results, set a recurring sync schedule in the **Scheduling** tab. This will ensure your *searchable* data in Elasticsearch is always up to date with changes to your PostgreSQL data source.
217
+
## Next steps
201
218
219
+
You just learned how to synchronize data from an external database to {{es}}.
220
+
For an overview of how to start searching and analyzing your data in Kibana, go to [Explore and analyze](docs-content://explore-analyze/index.md).
202
221
203
-
## Learn more [es-postgresql-connector-client-tutorial-learn-more]
222
+
Learn more:
204
223
205
224
* [Overview of self-managed connectors and frameworks](/reference/search-connectors/self-managed-connectors.md)
Copy file name to clipboardExpand all lines: docs/reference/search-connectors/self-managed-connectors.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,7 +18,8 @@ Self-managed [Elastic connectors](/reference/search-connectors/index.md) are run
18
18
## Availability and Elastic prerequisites [es-build-connector-prerequisites]
19
19
20
20
::::{note}
21
-
Self-managed connectors currently don’t support Windows. Use this [compatibility matrix](https://www.elastic.co/support/matrix#matrix_os) to check which operating systems are supported by self-managed connectors. Find this information under **self-managed connectors** on that page.
21
+
Self-managed connectors currently don’t support Windows. Use this [compatibility matrix](https://www.elastic.co/support/matrix#matrix_os) to check which operating systems are supported by self-managed connectors.
22
+
% Find this information under **self-managed connectors** on that page.
22
23
23
24
::::
24
25
@@ -28,7 +29,7 @@ Your Elastic deployment must include the following Elastic services:
28
29
***Elasticsearch**
29
30
***Kibana**
30
31
31
-
(A new Elastic Cloud deployment includes these services by default.)
32
+
A new {{ech}} deployment or {{es-serverless}} project includes these services by default.
32
33
33
34
To run self-managed connectors, your self-deployed connector service version must match your Elasticsearch version. For example, if you’re running Elasticsearch 8.10.1, your connector service should be version 8.10.1.x. Elastic does not support deployments running mismatched versions (except during upgrades).
0 commit comments