Skip to content

Commit cfb585c

Browse files
Anush008Anush008
committed
docs: Confluent integration (#1008)
1 parent c62a1b0 commit cfb585c

File tree

8 files changed

+284
-0
lines changed

8 files changed

+284
-0
lines changed

‎qdrant-landing/content/documentation/frameworks/_index.md

+1
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ weight: 33
1212
| [Bubble](./bubble) | Development platform for application development with a no-code interface |
1313
| [Canopy](./canopy/) | Framework from Pinecone for building RAG applications using LLMs and knowledge bases. |
1414
| [Cheshire Cat](./cheshire-cat/) | Framework to create personalized AI assistants using custom data. |
15+
| [Confluent](./confluent/) | Fully-managed data streaming platform with a cloud-native Apache Kafka engine. |
1516
| [DLT](./dlt/) | Python library to simplify data loading processes between several sources and destinations. |
1617
| [DocArray](./docarray/) | Python library for managing data in multi-modal AI applications. |
1718
| [DocsGPT](./docsgpt/) | Tool for ingesting documentation sources and enabling conversations and queries. |

‎qdrant-landing/content/documentation/frameworks/confluent.md

+283
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,283 @@
1+
---
2+
title: Confluent
3+
weight: 3700
4+
---
5+
6+
![Confluent Logo](/documentation/frameworks/confluent/confluent-logo.png)
7+
8+
[Confluent Cloud](https://www.confluent.io/confluent-cloud/?utm_campaign=tm.pmm_cd.cwc_partner_Qdrant_generic&utm_source=Qdrant&utm_medium=partnerref) is a fully-managed data streaming platform, available on AWS, GCP, and Azure, with a cloud-native Apache Kafka engine for elastic scaling, enterprise-grade security, stream processing, and governance.
9+
10+
With our [Qdrant-Kafka Sink Connector](https://github.com/qdrant/qdrant-kafka), Qdrant is part of the [Connect with Confluent](https://www.confluent.io/partners/connect/) technology partner program. It brings fully managed data streams directly to organizations through the Confluent Cloud platform. Making it easier for organizations to stream any data to Qdrant with a fully managed Apache Kafka service.
11+
12+
## Usage
13+
14+
### Pre-requisites
15+
16+
- A Confluent Cloud account. You can begin with a [free trial](https://www.confluent.io/confluent-cloud/tryfree/?utm_campaign=tm.pmm_cd.cwc_partner_qdrant_tryfree&utm_source=qdrant&utm_medium=partnerref) with credits for the first 30 days.
17+
- Qdrant instance to connect to. You can get a free cloud instance at [cloud.qdrant.io](https://cloud.qdrant.io/).
18+
19+
### Installation
20+
21+
1) Download the latest connector zip file from [Confluent Hub](https://github.com/qdrant/landing_page).
22+
23+
2) Configure an environment and cluster on Confluent and create a topic to produce messages for.
24+
25+
3) Navigate to the `Connectors` section of the Confluent cluster and click `Add Plugin`. Upload the zip file with the following info.
26+
27+
![Qdrant Connector Install](/documentation/frameworks/confluent/install.png)
28+
29+
4) Once installed, navigate to the connector and set the following configuration values.
30+
31+
![Qdrant Connector Config](/documentation/frameworks/confluent/config.png)
32+
33+
Replace the placeholder values with your credentials.
34+
35+
5) Add the Qdrant instance host to the allowed networking endpoints.
36+
37+
![Qdrant Connector Endpoint](/documentation/frameworks/confluent/endpoint.png)
38+
39+
7) Start the connector.
40+
41+
## Producing Messages
42+
43+
You can now produce messages for the configured topic, and they'll be written into the configured Qdrant instance.
44+
45+
![Qdrant Connector Message](/documentation/frameworks/confluent/message.png)
46+
47+
## Message Formats
48+
49+
The connector supports messages in the following formats.
50+
51+
_Click each to expand._
52+
53+
<details>
54+
<summary><b>Unnamed/Default vector</b></summary>
55+
56+
Reference: [Creating a collection with a default vector](https://qdrant.tech/documentation/concepts/collections/#create-a-collection).
57+
58+
```json
59+
{
60+
"collection_name": "{collection_name}",
61+
"id": 1,
62+
"vector": [
63+
0.1,
64+
0.2,
65+
0.3,
66+
0.4,
67+
0.5,
68+
0.6,
69+
0.7,
70+
0.8
71+
],
72+
"payload": {
73+
"name": "kafka",
74+
"description": "Kafka is a distributed streaming platform",
75+
"url": "https://kafka.apache.org/"
76+
}
77+
}
78+
```
79+
80+
</details>
81+
82+
<details>
83+
<summary><b>Named multiple vectors</b></summary>
84+
85+
Reference: [Creating a collection with multiple vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
86+
87+
```json
88+
{
89+
"collection_name": "{collection_name}",
90+
"id": 1,
91+
"vector": {
92+
"some-dense": [
93+
0.1,
94+
0.2,
95+
0.3,
96+
0.4,
97+
0.5,
98+
0.6,
99+
0.7,
100+
0.8
101+
],
102+
"some-other-dense": [
103+
0.1,
104+
0.2,
105+
0.3,
106+
0.4,
107+
0.5,
108+
0.6,
109+
0.7,
110+
0.8
111+
]
112+
},
113+
"payload": {
114+
"name": "kafka",
115+
"description": "Kafka is a distributed streaming platform",
116+
"url": "https://kafka.apache.org/"
117+
}
118+
}
119+
```
120+
121+
</details>
122+
123+
<details>
124+
<summary><b>Sparse vectors</b></summary>
125+
126+
Reference: [Creating a collection with sparse vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-sparse-vectors).
127+
128+
```json
129+
{
130+
"collection_name": "{collection_name}",
131+
"id": 1,
132+
"vector": {
133+
"some-sparse": {
134+
"indices": [
135+
0,
136+
1,
137+
2,
138+
3,
139+
4,
140+
5,
141+
6,
142+
7,
143+
8,
144+
9
145+
],
146+
"values": [
147+
0.1,
148+
0.2,
149+
0.3,
150+
0.4,
151+
0.5,
152+
0.6,
153+
0.7,
154+
0.8,
155+
0.9,
156+
1.0
157+
]
158+
}
159+
},
160+
"payload": {
161+
"name": "kafka",
162+
"description": "Kafka is a distributed streaming platform",
163+
"url": "https://kafka.apache.org/"
164+
}
165+
}
166+
```
167+
168+
</details>
169+
170+
<details>
171+
<summary><b>Multi-vectors</b></summary>
172+
173+
Reference:
174+
175+
- [Multi-vectors](https://qdrant.tech/documentation/concepts/vectors/#multivectors)
176+
177+
```json
178+
{
179+
"collection_name": "{collection_name}",
180+
"id": 1,
181+
"vector": {
182+
"some-multi": [
183+
[
184+
0.1,
185+
0.2,
186+
0.3,
187+
0.4,
188+
0.5,
189+
0.6,
190+
0.7,
191+
0.8,
192+
0.9,
193+
1.0
194+
],
195+
[
196+
1.0,
197+
0.9,
198+
0.8,
199+
0.5,
200+
0.4,
201+
0.8,
202+
0.6,
203+
0.4,
204+
0.2,
205+
0.1
206+
]
207+
]
208+
},
209+
"payload": {
210+
"name": "kafka",
211+
"description": "Kafka is a distributed streaming platform",
212+
"url": "https://kafka.apache.org/"
213+
}
214+
}
215+
```
216+
217+
</details>
218+
219+
<details>
220+
<summary><b>Combination of named dense and sparse vectors</b></summary>
221+
222+
Reference:
223+
224+
- [Creating a collection with multiple vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-multiple-vectors).
225+
226+
- [Creating a collection with sparse vectors](https://qdrant.tech/documentation/concepts/collections/#collection-with-sparse-vectors).
227+
228+
```json
229+
{
230+
"collection_name": "{collection_name}",
231+
"id": "a10435b5-2a58-427a-a3a0-a5d845b147b7",
232+
"vector": {
233+
"some-other-dense": [
234+
0.1,
235+
0.2,
236+
0.3,
237+
0.4,
238+
0.5,
239+
0.6,
240+
0.7,
241+
0.8
242+
],
243+
"some-sparse": {
244+
"indices": [
245+
0,
246+
1,
247+
2,
248+
3,
249+
4,
250+
5,
251+
6,
252+
7,
253+
8,
254+
9
255+
],
256+
"values": [
257+
0.1,
258+
0.2,
259+
0.3,
260+
0.4,
261+
0.5,
262+
0.6,
263+
0.7,
264+
0.8,
265+
0.9,
266+
1.0
267+
]
268+
}
269+
},
270+
"payload": {
271+
"name": "kafka",
272+
"description": "Kafka is a distributed streaming platform",
273+
"url": "https://kafka.apache.org/"
274+
}
275+
}
276+
```
277+
278+
</details>
279+
280+
## Further Reading
281+
282+
- [Kafka Connect Docs](https://docs.confluent.io/platform/current/connect/index.html)
283+
- [Confluent Connectors Docs](https://docs.confluent.io/cloud/current/connectors/bring-your-connector/custom-connector-qs.html)

‎qdrant-landing/static/documentation/frameworks/confluent-social-preview.png

510 KB
Loading

‎qdrant-landing/static/documentation/frameworks/confluent/config.png

136 KB
Loading

‎qdrant-landing/static/documentation/frameworks/confluent/confluent-logo.png

31.4 KB
Loading

‎qdrant-landing/static/documentation/frameworks/confluent/endpoint.png

93.8 KB
Loading

‎qdrant-landing/static/documentation/frameworks/confluent/install.png

163 KB
Loading

‎qdrant-landing/static/documentation/frameworks/confluent/message.png

262 KB
Loading

0 commit comments

Comments
 (0)