storm jar uber-dataingest-1.0.0.jar com.xavient.storm.dataingest.DataIngestionTopology

[ppramanick]

Hi team/Sir,

I am deploying the jar as below from the path as
/usr/hdp/2.6.4.0-91/storm/lib
storm jar uber-dataingest-1.0.0.jar com.xavient.storm.dataingest.DataIngestionTopology
and getting error as storm.dependency.jars= -Dstorm.dependency.artifacts={} com.xavient.storm.dataingest.DataIngestionTopology
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/topology/IRichSpout
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: backtype.storm.topology.IRichSpout
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more

My sandbox version is
Sandbox information:
Created on: 01_02_2018_10_47_41
Hadoop stack version: Hadoop 2.7.3.2.6.4.0-91
Ambari Version: 2.6.1.0-143
Ambari Hash: 2989989d67edacff7e9db702b4cf0c080556dddc
Ambari build: Release : 143
Java version: 1.8.0_161
OS Version: CentOS release 6.9 (Final)

The storm version is as below :-
1.1.0

Could Anyone please help me here ,

Thanks
Prosenjit

[mohnkhan]

Greetings.... will look into it on Tuesday..

[ppramanick]

Hi ,
Thanks for writing me back ,However I have resolved the issue and I have one question ,after I submit the jar then I need to do Open the UI for the application by visiting the URL "http://tomcat-server:port/DataIngestGUI/UI.jsp",I am not finding the UI.jsp file ,could you please help me in this ,

One question If I directly put the csv's JSON's and XML directly into the KAFKA_TOPIC, will the storm spout process it to the next bolt,Kindly let me know.

Thanks
Prosenjit

[ppramanick]

Hi ,

I have resolved the issue however when I pass messages to Kafka_topic like
bin/kafka-console-producer.sh --broker-list sandbox-hdp.hortonworks.com:6667 --topic kafka_topic
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N
{"id":"XIND10000","author":"Jack White","title":"Hadoop Made Easy","genre":"Programming","price":"50000","publish_date":"2001-09-10","description":"ebook"}
{"id":"XIND10002","author":"Steven Berg","title":"Apache Source","genre":"Programming","price":"45000","publish_date":"2001-10-10","description":"ebook"}

and I receive the messages in consumer as well
but when I see them in storm topology ,I get the below error
ava.lang.NullPointerException: topic cannot be null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:226) at org.apache.storm.kafka.DynamicBrokersReader.(DynamicBrokersReader.java:55) at org.apache.storm.kafka.trident.ZkBrokerReader.(ZkBrokerReader.java:43) at org.apache.storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:58) at org.apache.storm.kafka.KafkaSpout.open(KafkaSpout.java:77) at org.apache.storm.daemon.executor$fn__10182$fn__10197.invoke(executor.clj:603) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:482) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748)

Could you please help me with this ,waiting for your kind reply ,if you want we can have a session remote as well.

Thanks
Prosenjit

[ppramanick]

Hi ,

As agreed can we please look into it , had resolved the above error
but getting the below error java.lang.ExceptionInInitializerError at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:85) at kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:26) at kafka.consumer.FetchRequestAndResponseMetrics.<init>(FetchRequestAndResponseStats.scala:35) at kafka.consumer.FetchRequestAndResponseStats.<init>(FetchRequestAndResponseStats.scala:47) at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:60) at kafka.consumer.FetchRequestAndResponseStatsRegistry$$anonfun$2.apply(FetchRequestAndResponseStats.scala:60) at kafka.utils.Pool.getAndMaybePut(Pool.scala:59) at kafka.consumer.FetchRequestAndResponseStatsRegistry$.getFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:64) at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:44) at kafka.javaapi.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:34) at org.apache.storm.kafka.DynamicPartitionConnections.register(DynamicPartitionConnections.java:60) at org.apache.storm.kafka.PartitionManager.<init>(PartitionManager.java:70) at org.apache.storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:98) at org.apache.storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:69) at org.apache.storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:129) at org.apache.storm.daemon.executor$fn__10182$fn__10197$fn__10228.invoke(executor.clj:647) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) at java.lang.Runtime.addShutdownHook(Runtime.java:211) at com.yammer.metrics.Metrics.<clinit>(Metrics.java:21) ... 19 more

Can you please help today is tuesday.

Thanks
Prosenjit

[ppramanick]

Hi ,

Can someone please look into the error .

Thanks
Prosenjit

[mohnkhan]

@BigData-Engineer FYI.,

[lazy-eval]

This seems to be versioning issue. As you are using newer version of HDP please change the versions of storm, Kafka etc in pom file. Rebuild and run.

[ppramanick]

@mohnkhan @BigData-Developer @rskumar @nsabharwal I am changed the POM.xml as per the version I am using in Ambari ,please find the pom.xml and the screenshot with the component versions attached .

I am attaching both kindly let me know where I am going wrong.

KIndly help me here .

Thanks
Prosenjit
Components Version as defined in Ambari Services.docx
pom.txt