Exception in thread “streaming-start” java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer

时间 2020-07-17 标签 exception thread streaming start java.lang.nosuchmethoderror java lang nosuchmethoderror org.apache.kafka.clients.consumer org apache kafka clients consumer

Exception in thread “streaming-start” java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)V

Ask Questionhtml

When I submit the spark application, getting the below error:java

Exception in thread "streaming-start" java.lang.NoSuchMethodError:apache

org.apache.kafka.clients.consumer.KafkaConsumer.subscribe(Ljava/util/Collection;)Vapp

Went through the below URL: http://apache-spark-developers-list.1001551.n3.nabble.com/testing-the-kafka-0-10-connector-td17956.htmlthis

Set the below environment variable: SPARK_KAFKA_VERSION=0.10spa

Added the below dependency:code

<dependency>
  <groupId>org.apache.spark</groupId>
  <artifactId>spark-streaming-kafka-0-10_2.11</artifactId>
  <version>2.1.0.cloudera1</version>
</dependency>

Added the below jar: spark-streaming-kafka-0-10-assembly_2.11.jarhtm

Versions I'm using:ci

Spark-2.0.2
Scala-2.11
Kafka-0.10

A piece of code:get

val CLASS_NAME = this.getClass
val log = org.apache.log4j.LogManager.getLogger(CLASS_NAME)

Your help is highly appreciated.

解决办法：export SPARK_KAFKA_VERSION=0.10

转自：https://stackoverflow.com/questions/45840824/exception-in-thread-streaming-start-java-lang-nosuchmethoderror-org-apache-ka