【原创】大叔经验分享（5）oozie提交spark任务如何添加依赖

spark任务添加依赖的方式：

1 若是是local方式运行，能够经过--jars来添加依赖；

2 若是是yarn方式运行，能够经过spark.yarn.jars来添加依赖；

这两种方式在oozie上都行不通，首先oozie上没办法也不该该经过local运行，其次经过spark.yarn.jars方式配置你会发现根本不会生效，来看为何

查看LauncherMapper的日志

 

Spark Version 2.1.1

Spark Action Main class        : org.apache.spark.deploy.SparkSubmit

 

Oozie Spark action configuration

=================================================================

...

                    --conf

                    spark.yarn.jars=hdfs://hdfs_name/jarpath/*.jar

                    --conf

                    spark.yarn.jars=hdfs://hdfs_name/oozie/share/lib_20180801121138/spark/spark-yarn_2.11-2.1.1.jar

 

可见oozie会本身添加一个新的spark.yarn.jars配置，若是提供两个相同的key，spark会如何处理

 

org.apache.spark.deploy.SparkSubmit

    val appArgs = new SparkSubmitArguments(args)

 

org.apache.spark.launcher.SparkSubmitOptionParser

        if (!handle(name, value)) {

 

org.apache.spark.deploy.SparkSubmitArguments

  override protected def handle(opt: String, value: String): Boolean = {

  ...

      case CONF =>

        value.split("=", 2).toSeq match {

          case Seq(k, v) => sparkProperties(k) = v

          case _ => SparkSubmit.printErrorAndExit(s"Spark config without '=': $value")

        }

 

可见会直接覆盖，使用最后一个配置，即oozie的配置，而不是应用本身提供的配置，这样就须要应用本身将特殊依赖打包到应用jar中，具体使用maven的maven-assembly-plugin，配置其中的<dependencySets><dependencySet><includes><include>，详细配置以下：

 

<assembly xmlns="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0"

          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

          xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd">

    <!-- TODO: a jarjar format would be better -->

    <id>jar-with-dependencies</id>

    <formats>

        <format>jar</format>

    </formats>

    <includeBaseDirectory>false</includeBaseDirectory>

    <dependencySets>

        <dependencySet>

            <outputDirectory>/</outputDirectory>

            <useProjectArtifact>true</useProjectArtifact>

            <unpack>true</unpack>

            <scope>runtime</scope>

            <includes>

                <include>redis.clients:jedis</include>

                <include>org.apache.commons:commons-pool2</include>

            </includes>

        </dependencySet>

    </dependencySets>

</assembly>

 

这里只是将默认提供的jar-with-dependencies.xml内容拷贝出来添加includes配置；