Getting Started with Developing HDFS Toolkit

Getting Started at the Command-Line

To get started with developing the HDFS toolkit.

Before you start with making the streamsx.hdfs toolkit, make sure that:

• The latest version of IBM Streams installed on your server.
• The git is installed on your stream server and you have access to the github repositories on https://github.com/IBMStreams/
• The apache tools maven and ant are installed on your server

Here is a short installation guide for maven and ant:

1- Install maven. Once maven is installed, export M2_HOME="maven_Install_Location"

su -
mkdir -p /opt/apache
cd /opt/apache
wget mirror.checkdomain.de/apache/maven/maven-3/3.6.1/binaries/apache-maven-3.6.1-bin.tar.gz
tar -xzvf apache-maven-3.6.1-bin.tar.gz
su - streamsadmin
echo "export M2_HOME=/opt/apache/apache-maven-3.6.1" >> ~/.bashrc

2- Install ant. Once ant is installed, export ANT_HOME="ant_Install_Location"

su -
mkdir -p /opt/apache
cd /opt/apache
wget  apache.mirror.digionline.de//ant/binaries/apache-ant-1.10.6-bin.tar.gz
tar -xzvf apache-ant-1.10.6-bin.tar.gz
su - streamsadmin
echo "export ANT_HOME=/opt/apache/apache-ant-1.10.6" >> ~/.bashrc

3- Set the environment variable PATH

echo "export JAVA_HOME=$STREAMS_INSTALL/java" >> ~/.bashrc
echo "export PATH=$JAVA_HOME/bin:ANT_HOME/bin:M2_HOME/bin:$PATH" >> ~/.bashrc
source ~/.bashrc

4- Clone the repository to your local file system.

mkdir ~/workspace
cd ~/workspace
git clone https://github.com/IBMStreams/streamsx.hdfs.git

5- Go to the root of the toolkit (streamsx.hdfs) and type ant

cd streamsx.hdfs
ant

This will invoke the Ant Build. build.xml will in turn invoke Maven to download all dependencies required. build.xml will proceed to build the java files and call spl-make-toolkit.

Getting Started with Streams Studio

Note: Make sure Ant and Maven are installed in your remote host.

To import this toolkit project into Streams Studio:

1. Install "Egit" into Streams Studio (https://www.eclipse.org/egit/)
2. In the Git Repositories View, click on the "Clone a Git Repository" button
3. After the repository is cloned, follow the dialog to import existing project into your workspace.
4. You will not have the dependencies needed for Studio to build the project correctly. Go to the command line and to the root of the toolkit project.
5. Type "ant maven-deps" - to kick off a build at the command line for the first time. This will get all the dependencies downloaded.
6. Refresh your project in Streams Studio to get all the jar files included in your workspace
7. Build the com.ibm.streamsx.hdfs toolkit project in Studio - The build should be successful at this point.

Getting Started with Streams Studio on Windows

Note: Make sure Ant and Maven are installed in your remote host.

You can set up remote development on Windows.

To import this toolkit project into Streams Studio on Windows:

1. Install "Egit" into Streams Studio (https://www.eclipse.org/egit/)
2. In the Git Repositories View, click on the "Clone a Git Repitory" button
3. After the repository is cloned, follow the dialog to import existing project into your workspace.
4. Select the toolkit project, Right click -> Copy project to remote host - This will copy the project to the remote Linux host where actual compilation needs to happen.
5. You will not have the dependencies needed for Studio to build the project correctly at this point. Open a command prompt to the remote Linux host.
6. Go to the root directory of the toolkit project
7. Type "ant maven-deps" - to kick off the ant build that pulls in all the necessary dependencies.
8. In Streams Studio on Windows, select the "opt" folder under the toolkit project. Right click -> Remote Reconciler -> Pull from (remote host) - This action will pull all the jar files from your remote Linux host to your Windows machine.
9. Build the com.ibm.streamsx.hdfs toolkit project in Studio - The build should be successful at this point.