Skip to content

Commit

Permalink
Modified notebook with feedback (#125)
Browse files Browse the repository at this point in the history 
* Modified notebook with feedback

* Added Kafka Notebook

* Removed additional sql

* Update notebook.ipynb

---------

Co-authored-by: chetan thote <[email protected]>
Co-authored-by: Kevin D Smith <[email protected]>
  • Loading branch information
@chetanthote @kesmit13
3 people authored Nov 26, 2024
1 parent b011d93 commit 2d94c22
Show file tree
Hide file tree
Showing 3 changed files with 425 additions and 76 deletions.
156 changes: 80 additions & 76 deletions notebooks/load-csv-data-s3-placeholder/notebook.ipynb
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"cells": [
{
"id": "e82b93c8",
"id": "c55dedcc",
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -17,7 +17,7 @@
]
},
{
"id": "d657c3f4",
"id": "e8bf6bf1",
"cell_type": "markdown",
"metadata": {},
"source": [
Expand All @@ -33,61 +33,91 @@
{
"attachments": {},
"cell_type": "markdown",
"id": "4b3c156d",
"metadata": {},
"source": [
"<div class=\"alert alert-block alert-warning\">\n",
" <b class=\"fa fa-solid fa-exclamation-circle\"></b>\n",
" <div>\n",
" <p><b>Input Credentials</b></p>\n",
" <p>Define the <b>URL</b>, <b>REGION</b>, <b>ACCESS_KEY</b>, and <b>SECRET_ACCESS_KEY</b> variables below for integration, replacing the placeholder values with your own.</p>\n",
" </div>\n",
"</div>"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "80939f42",
"metadata": {},
"outputs": [],
"source": [
"URL = 's3://your-bucket-name/your-data-file.csv'\n",
"REGION = 'your-region'\n",
"ACCESS_KEY = 'access_key_id'\n",
"SECRET_ACCESS_KEY = 'access_secret_key'"
]
},
{
"cell_type": "markdown",
"id": "64fdd646",
"metadata": {},
"source": [
"This notebook demonstrates how to create a sample table in SingleStore, set up a pipeline to import data from an Amazon S3 bucket, and run queries on the imported data. It is designed for users who want to integrate S3 data with SingleStore and explore the capabilities of pipelines for efficient data ingestion."
],
"id": "e9e6acea"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "85c97bbb",
"metadata": {},
"source": [
"<h3>Demo Flow</h3>"
],
"id": "4933b61c"
"<h3>Pipeline Flow Illustration</h3>"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "7b97f983",
"metadata": {},
"source": [
"<img src=https://singlestoreloaddata.s3.ap-south-1.amazonaws.com/images/LoadDataCSV.png width=\"100%\" hight=\"50%\"/>"
],
"id": "ef90c6c9"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "d1ea9458",
"metadata": {},
"source": [
"## Sample Table in SingleStore\n",
"## Creating Table in SingleStore\n",
"\n",
"Start by creating a table that will hold the data imported from S3."
],
"id": "8a85c544"
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 2,
"id": "66eb1b49",
"metadata": {},
"outputs": [],
"source": [
"%%sql\n",
"CREATE TABLE IF NOT EXISTS sample_table (\n",
"/* Feel free to change table name and schema */\n",
"\n",
"CREATE TABLE IF NOT EXISTS my_table (\n",
" id INT,\n",
" name VARCHAR(255),\n",
" age INT,\n",
" address TEXT,\n",
" created_at TIMESTAMP\n",
");"
],
"id": "2ca9281c"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "3704a19a",
"metadata": {},
"source": [
"## Create a Pipeline to Import Data from S3\n",
Expand All @@ -98,193 +128,167 @@
"You have access to the S3 bucket.\n",
"Proper IAM roles or access keys are configured in SingleStore.\n",
"The CSV file has a structure that matches the table schema.</i>"
],
"id": "a88192c9"
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set Up Variables\n",
"\n",
"Define the <b>URL</b>, <b>REGION</b>, <b>ACCESS_KEY</b>, and <b>SECRET_ACCESS_KEY</b> variables for integration, replacing the placeholder values with your own."
],
"id": "87d2c776"
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"URL = 's3://your-bucket-name/your-data-file.csv'\n",
"REGION = 'your-region'\n",
"ACCESS_KEY = 'access_key_id'\n",
"SECRET_ACCESS_KEY = 'access_secret_key'"
],
"id": "78c44e19"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "83a75641",
"metadata": {},
"source": [
"Using these identifiers and keys, execute the following statement."
],
"id": "eb0c3643"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "e88495e2",
"metadata": {},
"outputs": [],
"source": [
"%%sql\n",
"\n",
"CREATE PIPELINE s3_import_pipeline\n",
"AS LOAD DATA S3 '{{URL}}'\n",
"CONFIG '{\\\"REGION\\\":\\\"{{REGION}}\\\"}'\n",
"CREDENTIALS '{\\\"AWS_ACCESS_KEY_ID\\\": \\\"{{ACCESS_KEY}}\\\",\n",
" \\\"AWS_SECRET_ACCESS_KEY\\\": \\\"{{SECRET_ACCESS_KEY}}\\\"}'\n",
"INTO TABLE sample_table\n",
"INTO TABLE my_table\n",
"FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\\\"'\n",
"LINES TERMINATED BY '\\n'\n",
"IGNORE 1 lines;"
],
"id": "6efe3112"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "08781117",
"metadata": {},
"source": [
"## Start the Pipeline\n",
"\n",
"To start the pipeline and begin importing the data from the S3 bucket:"
],
"id": "5902a86a"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "adb89dc1",
"metadata": {},
"outputs": [],
"source": [
"%%sql\n",
"START PIPELINE s3_import_pipeline;"
],
"id": "bb436fc2"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "15275805",
"metadata": {},
"source": [
"## Select Data from the Table\n",
"\n",
"Once the data has been imported, you can run a query to select it:"
],
"id": "c82c8439"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "f2855248",
"metadata": {},
"outputs": [],
"source": [
"%%sql\n",
"SELECT * FROM sample_table LIMIT 10;"
],
"id": "bb740975"
"SELECT * FROM my_table LIMIT 10;"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ea34b089",
"metadata": {},
"source": [
"### Check if all data of the data is loaded"
],
"id": "df1cdb14"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "4e9023e8",
"metadata": {},
"outputs": [],
"source": [
"%%sql\n",
"SELECT count(*) FROM sample_table"
],
"id": "dd98c9a3"
"SELECT count(*) FROM my_table"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "236cb111",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"We have shown how to insert data from a Amazon S3 using `Pipelines` to SingleStoreDB. These techniques should enable you to\n",
"integrate your Amazon S3 with SingleStoreDB."
],
"id": "892e7f8d"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "6d195418",
"metadata": {},
"source": [
"## Clean up\n",
"\n",
"Remove the '#' to uncomment and execute the queries below to clean up the pipeline and table created."
],
"id": "3c053a57"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "8eafa43e",
"metadata": {},
"source": [
"#### Drop Pipeline"
],
"id": "8874a110"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "451e80c9",
"metadata": {},
"outputs": [],
"source": [
"%%sql\n",
"#STOP PIPELINE s3_import_pipeline;\n",
"\n",
"#DROP PIPELINE s3_import_pipeline;"
],
"id": "043861f7"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "ae825668",
"metadata": {},
"source": [
"#### Drop Data"
],
"id": "445c6369"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "3c4b631d",
"metadata": {},
"outputs": [],
"source": [
"%%sql\n",
"#DROP TABLE sample_table;"
],
"id": "f8b697e5"
"#DROP TABLE my_table;"
]
},
{
"id": "39231766",
"id": "89365517",
"cell_type": "markdown",
"metadata": {},
"source": [
Expand Down
11 changes: 11 additions & 0 deletions notebooks/load-kafka-template/meta.toml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
[meta]
authors=["chetan-thote"]
title="Importing Data from Kafka into SingleStore using Pipelines"
description="""\
This notebook demonstrates how to create a sample table in SingleStore, set up a pipeline to import data from Kafka topic."""
difficulty="beginner"
tags=["starter", "loaddata", "kafka"]
lesson_areas=["Ingest"]
icon="database"
destinations=["spaces"]
minimum_tier="free-shared"
Loading

0 comments on commit 2d94c22

Please sign in to comment.