This project is demonstrating how to connect to a MSSQL database, extract a table, convert the data in AVRO format and store the resulting file in a local folder.
- Java 8
- Maven (optional, docker can be used instead, see below)
- Docker and docker-compose (for end-to-end test)
A docker-compose.yml
file is provided to do an end-to-end test.
The docker container will start MSSQL, create a table and load
some test data in it.
The docker container configuration is taken from https://cardano.github.io/blog/2017/11/15/mssql-docker-container
Start the MSSQL container using docker-compose
:
docker-compose up -d mssql
Wait a few seconds, and your container is running, listening on port 1433 with some pre-loaded data.
Please refer to mssql-docker documentation if you need more details.
The project is Maven based, running
mvn clean package
Note: if you don't have maven
installed, you can build inside a docker container like this:
docker run -it -v "$(pwd)":/usr/src/mymaven -w /usr/src/mymaven maven:3-jdk-8 mvn clean install
Once built, the project produces a runnable jar
with all the required dependencies
in target
folder, namely target/ingestion_example-1.0-SNAPSHOT-jar-with-dependencies.jar
Once built, you can run it quickly using the fat jar
:
java -jar target/ingestion_example-1.0-SNAPSHOT-jar-with-dependencies.jar
If MSSQL container is running and the folder /tmp/output
doesn't exist yet,
the application will run properly and create a new folder /tmp/output
with
the following layout:
ls -1 /tmp/output/
_SUCCESS
part-00000-e26a5368-12c8-44d5-9fe7-2366e8d91676.avro
The file part-00000-e26a5368-12c8-44d5-9fe7-2366e8d91676.avro
is your new AVRO file which
can be shared with other teams.