【原创】大叔问题定位分享（9）oozie提交spark任务报 java.lang.NoClassDefFoundError: org/apache/kafka/clients/producer/Kaf

时间 2019-11-21

标签原创问题定位分享 oozie 提交 spark 任务 java.lang.noclassdeffounderror java lang noclassdeffounderror org apache kafka clients producer kaf 栏目系统网络繁體版

原文原文链接

oozie中支持不少的action类型，好比spark、hive，对应的标签为：tcp

<spark xmlns="uri:oozie:spark-action:0.1"> oop

...spa

oozie中sharelib用于存放每一个action类型须要的依赖，能够查看当前全部的action类型以及每一个action类型的依赖xml

oozie admin -shareliblist
[Available ShareLib]
hive
sparkbak
distcp
mapreduce-streaming
spark
oozie
hcatalog
hive2
sqoop
pigget

oozie admin -shareliblist sparkkafka

...spark

上述报错是因为oozie中spark依赖里的kafka基础依赖缺乏kafka-clients-0.8.2.2.jar，将jar上传到sharelib目录，而后手动更新便可io

oozie admin -sharelibupdatestream

更新以后可见：基础

oozie admin -shareliblist spark|grep kafka
        hdfs://hdfs_name/oozie/share/lib_20180801121138/spark/kafka-clients-0.8.2.2.jar
        hdfs://hdfs_name/oozie/share/lib_20180801121138/spark/kafka_2.11-0.8.2.1.jar
        hdfs://hdfs_name/oozie/share/lib_20180801121138/spark/spark-streaming-kafka-0-8_2.11-2.1.0.jar

添加成功