Skip to content

Commit 92ec745

Browse files
committed
Add questdb and topics
1 parent bbc2ebe commit 92ec745

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

278 files changed

+61
-370
lines changed

HDSD.txt

+13-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,16 @@
11
Bước 1: Chạy kafka server và zookeeper server.
22
Bước 2: Submit Topic bằng lệnh "bin/kafka-topics.sh --create --topic result_BTC --bootstrap-server localhost:9092".
3-
Bước 2: Chạy file streamer_BTC.py bằng lệnh "spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.1 stream_BTC.py".
4-
Bước 3: Chạy file producer.py bằng lệnh "python3 producer.py
5-
Bước 4: Chạy consumer bằng lệnh "bin/kafka-console-consumer.sh --topic result_BTC --bootstrap-server localhost:9092"
3+
Bước 3: Chạy file streamer_BTC.py bằng lệnh "spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.1 stream_BTC.py".
4+
Bước 4: Chạy file producer.py bằng lệnh "python3 producer.py
5+
Bước 5: Chạy consumer bằng lệnh "bin/kafka-console-consumer.sh --topic result_BTC --bootstrap-server localhost:9092"
66

7-
NOTE: Ấn Ctrl+C ở tất cả các terminal để dừng.
7+
NOTE: Ấn Ctrl+C ở tất cả các terminal để dừng.
8+
9+
10+
11+
bin/zookeeper-server-start.sh config/zookeeper.properties
12+
bin/kafka-server-start.sh config/server.properties
13+
bin/kafka-topics.sh --create --topic topic_BTC --bootstrap-server localhost:9092
14+
bin/kafka-console-consumer.sh --topic result --bootstrap-server localhost:9092
15+
bin/connect-standalone.sh config/connect-standalone.properties config/questdb-connector.properties
16+
spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.3.1 test.py

consumer.py

+4-4
Original file line numberDiff line numberDiff line change
@@ -8,14 +8,14 @@
88
# initialize Kafka consumers and producer
99
print('Starting Apache Kafka consumers and producer')
1010
consumer_1 = initConsumer(config['topic_1'])
11-
consumer_2 = initConsumer(config['topic_2'])
12-
consumer_3 = initConsumer(config['topic_3'])
11+
# consumer_2 = initConsumer(config['topic_2'])
12+
# consumer_3 = initConsumer(config['topic_3'])
1313
producer = initProducer()
1414

1515
# intialize local dataframe
1616
data_1 = pd.DataFrame(columns=['time', 'value'])
17-
data_2 = pd.DataFrame(columns=['time', 'value'])
18-
data_3 = pd.DataFrame(columns=['time', 'value'])
17+
# data_2 = pd.DataFrame(columns=['time', 'value'])
18+
# data_3 = pd.DataFrame(columns=['time', 'value'])
1919

2020
while True:
2121
# consume data from Kafka
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.
-12 Bytes
Binary file not shown.

kafka-ml-test/checkpoint/commits/0

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/1

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/10

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/11

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/12

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/13

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/14

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/15

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/16

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/17

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/18

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/19

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/2

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/20

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/21

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/22

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/23

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/24

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/25

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/26

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/27

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/28

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/29

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/3

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/30

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/31

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/32

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/33

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/34

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/35

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/36

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/37

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/38

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/39

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/4

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/40

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/41

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/42

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/43

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/44

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/45

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/46

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/47

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/48

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/49

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/5

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/50

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/51

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/52

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/53

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/54

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/55

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/56

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/57

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/58

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/59

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/6

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/60

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/61

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/62

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/63

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/64

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/65

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/66

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/7

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/8

-2
This file was deleted.

kafka-ml-test/checkpoint/commits/9

-2
This file was deleted.

kafka-ml-test/checkpoint/metadata

-1
This file was deleted.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.
-16 Bytes
Binary file not shown.

kafka-ml-test/checkpoint/offsets/0

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/1

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/10

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/11

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/12

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/13

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/14

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/15

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/16

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/17

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/18

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/19

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/2

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/20

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/21

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/22

-3
This file was deleted.

kafka-ml-test/checkpoint/offsets/23

-3
This file was deleted.

0 commit comments

Comments
 (0)