从零教你如何获取hadoop2.4源码并使用eclipse关联hadoop2.4源码

时间  2019-12-10
标签 如何 获取 hadoop2.4 hadoop 源码 并使 eclipse 关联 栏目 Hadoop 繁體版
原文   原文链接

从零教你如何获取hadoop2.4源码并使用eclipse关联hadoop2.4源码
http://www.aboutyun.com/thread-8211-1-1.html
(出处: about云开发)php

 

问题导读：
1.如何经过官网src包，获取hadoop的所有代码
2.经过什么样的操做，能够查看hadoop某个函数或则类的实现？
3.maven的做用是什么？






咱们若是想搞开发，研究源码对咱们的帮助很大。不明白原理就如同黑盒子，遇到问题，咱们也摸不着思路。因此这里交给你们
一.如何获取源码
二.如何关联源码

一.如何获取源码

1.下载hadoop的maven程序包

（1）官网下载
这里咱们先从官网上下载maven包hadoop-2.4.0-src.tar.gz。
官网下载地址

对于不知道怎么去官网下载，能够查看：新手指导：hadoop官网介绍及如何下载hadoop（2.4）各个版本与查看hadoop API介绍

（2）网盘下载
也能够从网盘下载：
http://pan.baidu.com/s/1kToPuGB

2.经过maven获取源码
获取源码的方式有两种，一种是经过命令行的方式，一种是经过eclipse。这里主要讲经过命令的方式

经过命令的方式获取源码：
1.解压包

 


解压包的时候遇到了下面问题。不过不用管，咱们继续往下走html

1        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\applicationhistoryservice\ApplicationHistoryClientService$ApplicationHSClientProtocolHandler.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
2        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\classes\org\apache\hadoop\yarn\server\applicationhistoryservice\timeline\LeveldbTimelineStore$LockMap$CountingReentrantLock.class：系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
3        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-applicationhistoryservice\target\test-classes\org\apache\hadoop\yarn\server\applicationhistoryservice\webapp\TestAHSWebApp$MockApplicationHistoryManagerImpl.class：系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
4        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\monitor\capacity\TestProportionalCapacityPreemptionPolicy$IsPreemptionRequestFor.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
5        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestFSRMStateStore$TestFSRMStateStoreTester$TestFileSystemRMStore.class：系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
6        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStore$TestZKRMStateStoreTester$TestZKRMStateStoreInternal.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
7        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStoreZKClientConnections$TestZKClient$TestForwardingWatcher.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
8        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\recovery\TestZKRMStateStoreZKClientConnections$TestZKClient$TestZKRMStateStore.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip
9        : 没法建立文件：D:\hadoop2\hadoop-2.4.0-src\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-server\hadoop-yarn-server-resourcemanager\target\test-classes\org\apache\hadoop\yarn\server\resourcemanager\rmapp\attempt\TestRMAppAttemptTransitions$TestApplicationAttemptEventDispatcher.class：
路径和文件名总长度不能超过260个字符
系统找不到指定的路径。        D:\hadoop2\hadoop-2.4.0-src.zip



2.经过maven获取源码

这里须要说明的是，在使用maven的时候，须要先安装jdk，protoc ，若是没有安装能够参考win7如何安装maven、安装protoc

（1）进入hadoop-2.4.0-src\hadoop-maven-plugins，运行mvn installjava

  1. D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins>mvn install
复制代码



显示以下信息node

  2. [INFO] Scanning for projects...
  3. [WARNING]
  4. [WARNING] Some problems were encountered while building the effective model for
  5. org.apache.hadoop:hadoop-maven-plugins:maven-plugin:2.4.0
  6. [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but found d
  7. uplicate declaration of plugin org.apache.maven.plugins:maven-enforcer-plugin @
  8. org.apache.hadoop:hadoop-project:2.4.0, D:\hadoop2\hadoop-2.4.0-src\hadoop-proje
  9. ct\pom.xml, line 1015, column 15
  10. [WARNING]
  11. [WARNING] It is highly recommended to fix these problems because they threaten t
  12. he stability of your build.
  13. [WARNING]
  14. [WARNING] For this reason, future Maven versions might no longer support buildin
  15. g such malformed projects.
  16. [WARNING]
  17. [INFO]
  18. [INFO] Using the builder org.apache.maven.lifecycle.internal.builder.singlethrea
  19. ded.SingleThreadedBuilder with a thread count of 1
  20. [INFO]
  21. [INFO] ------------------------------------------------------------------------
  22. [INFO] Building Apache Hadoop Maven Plugins 2.4.0
  23. [INFO] ------------------------------------------------------------------------
  24. [INFO]
  25. [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-maven-plugins
  26. ---
  27. [INFO] Executing tasks
  29. main:
  30. [INFO] Executed tasks
  31. [INFO]
  32. [INFO] --- maven-plugin-plugin:3.0:descriptor (default-descriptor) @ hadoop-mave
  33. n-plugins ---
  34. [INFO] Using 'UTF-8' encoding to read mojo metadata.
  35. [INFO] Applying mojo extractor for language: java-annotations
  36. [INFO] Mojo extractor for language: java-annotations found 2 mojo descriptors.
  37. [INFO] Applying mojo extractor for language: java
  38. [INFO] Mojo extractor for language: java found 0 mojo descriptors.
  39. [INFO] Applying mojo extractor for language: bsh
  40. [INFO] Mojo extractor for language: bsh found 0 mojo descriptors.
  41. [INFO]
  42. [INFO] --- maven-resources-plugin:2.2:resources (default-resources) @ hadoop-mav
  43. en-plugins ---
  44. [INFO] Using default encoding to copy filtered resources.
  45. [INFO]
  46. [INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-maven-
  47. plugins ---
  48. [INFO] Nothing to compile - all classes are up to date
  49. [INFO]
  50. [INFO] --- maven-plugin-plugin:3.0:descriptor (mojo-descriptor) @ hadoop-maven-p
  51. lugins ---
  52. [INFO] Using 'UTF-8' encoding to read mojo metadata.
  53. [INFO] Applying mojo extractor for language: java-annotations
  54. [INFO] Mojo extractor for language: java-annotations found 2 mojo descriptors.
  55. [INFO] Applying mojo extractor for language: java
  56. [INFO] Mojo extractor for language: java found 0 mojo descriptors.
  57. [INFO] Applying mojo extractor for language: bsh
  58. [INFO] Mojo extractor for language: bsh found 0 mojo descriptors.
  59. [INFO]
  60. [INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ ha
  61. doop-maven-plugins ---
  62. [INFO] Using default encoding to copy filtered resources.
  63. [INFO]
  64. [INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoo
  65. p-maven-plugins ---
  66. [INFO] No sources to compile
  67. [INFO]
  68. [INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hadoop-maven-plugins
  69. ---
  70. [INFO] No tests to run.
  71. [INFO]
  72. [INFO] --- maven-jar-plugin:2.3.1:jar (default-jar) @ hadoop-maven-plugins ---
  73. [INFO] Building jar: D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\had
  74. oop-maven-plugins-2.4.0.jar
  75. [INFO]
  76. [INFO] --- maven-plugin-plugin:3.0:addPluginArtifactMetadata (default-addPluginA
  77. rtifactMetadata) @ hadoop-maven-plugins ---
  78. [INFO]
  79. [INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ hadoop-
  80. maven-plugins ---
  81. [INFO]
  82. [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ hadoop-maven-p
  83. lugins ---
  84. [INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\target\hadoop
  85. -maven-plugins-2.4.0.jar to C:\Users\hyj\.m2\repository\org\apache\hadoop\hadoop
  86. -maven-plugins\2.4.0\hadoop-maven-plugins-2.4.0.jar
  87. [INFO] Installing D:\hadoop2\hadoop-2.4.0-src\hadoop-maven-plugins\pom.xml to C:
  88. \Users\hyj\.m2\repository\org\apache\hadoop\hadoop-maven-plugins\2.4.0\hadoop-ma
  89. ven-plugins-2.4.0.pom
  90. [INFO] ------------------------------------------------------------------------
  91. [INFO] BUILD SUCCESS
  92. [INFO] ------------------------------------------------------------------------
  93. [INFO] Total time: 4.891 s
  94. [INFO] Finished at: 2014-06-23T14:47:33+08:00
  95. [INFO] Final Memory: 21M/347M
  96. [INFO] ------------------------------------------------------------------------
复制代码


部分截图以下：

 


 



（2）运行

web

  1. mvn eclipse:eclipse -DskipTests
复制代码

这时候注意，咱们进入的是hadoop_home,我这里是D:\hadoop2\hadoop-2.4.0-src

部分信息以下shell

  1. [INFO]
  2. [INFO] ------------------------------------------------------------------------
  3. [INFO] Reactor Summary:
  4. [INFO]
  5. [INFO] Apache Hadoop Main ................................ SUCCESS [  0.684 s]
  6. [INFO] Apache Hadoop Project POM ......................... SUCCESS [  0.720 s]
  7. [INFO] Apache Hadoop Annotations ......................... SUCCESS [  0.276 s]
  8. [INFO] Apache Hadoop Project Dist POM .................... SUCCESS [  0.179 s]
  9. [INFO] Apache Hadoop Assemblies .......................... SUCCESS [  0.121 s]
  10. [INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [  1.680 s]
  11. [INFO] Apache Hadoop MiniKDC ............................. SUCCESS [  1.802 s]
  12. [INFO] Apache Hadoop Auth ................................ SUCCESS [  1.024 s]
  13. [INFO] Apache Hadoop Auth Examples ....................... SUCCESS [  0.160 s]
  14. [INFO] Apache Hadoop Common .............................. SUCCESS [  1.061 s]
  15. [INFO] Apache Hadoop NFS ................................. SUCCESS [  0.489 s]
  16. [INFO] Apache Hadoop Common Project ...................... SUCCESS [  0.056 s]
  17. [INFO] Apache Hadoop HDFS ................................ SUCCESS [  2.770 s]
  18. [INFO] Apache Hadoop HttpFS .............................. SUCCESS [  0.965 s]
  19. [INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [  0.629 s]
  20. [INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [  0.284 s]
  21. [INFO] Apache Hadoop HDFS Project ........................ SUCCESS [  0.061 s]
  22. [INFO] hadoop-yarn ....................................... SUCCESS [  0.052 s]
  23. [INFO] hadoop-yarn-api ................................... SUCCESS [  0.842 s]
  24. [INFO] hadoop-yarn-common ................................ SUCCESS [  0.322 s]
  25. [INFO] hadoop-yarn-server ................................ SUCCESS [  0.065 s]
  26. [INFO] hadoop-yarn-server-common ......................... SUCCESS [  0.972 s]
  27. [INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [  0.580 s]
  28. [INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [  0.379 s]
  29. [INFO] hadoop-yarn-server-applicationhistoryservice ...... SUCCESS [  0.281 s]
  30. [INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [  0.378 s]
  31. [INFO] hadoop-yarn-server-tests .......................... SUCCESS [  0.534 s]
  32. [INFO] hadoop-yarn-client ................................ SUCCESS [  0.307 s]
  33. [INFO] hadoop-yarn-applications .......................... SUCCESS [  0.050 s]
  34. [INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [  0.202 s]
  35. [INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [  0.194 s]
  36. [INFO] hadoop-yarn-site .................................. SUCCESS [  0.057 s]
  37. [INFO] hadoop-yarn-project ............................... SUCCESS [  0.066 s]
  38. [INFO] hadoop-mapreduce-client ........................... SUCCESS [  0.091 s]
  39. [INFO] hadoop-mapreduce-client-core ...................... SUCCESS [  1.321 s]
  40. [INFO] hadoop-mapreduce-client-common .................... SUCCESS [  0.786 s]
  41. [INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [  0.456 s]
  42. [INFO] hadoop-mapreduce-client-app ....................... SUCCESS [  0.508 s]
  43. [INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [  0.834 s]
  44. [INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [  0.541 s]
  45. [INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [  0.284 s]
  46. [INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [  0.851 s]
  47. [INFO] hadoop-mapreduce .................................. SUCCESS [  0.099 s]
  48. [INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [  0.742 s]
  49. [INFO] Apache Hadoop Distributed Copy .................... SUCCESS [  0.335 s]
  50. [INFO] Apache Hadoop Archives ............................ SUCCESS [  0.397 s]
  51. [INFO] Apache Hadoop Rumen ............................... SUCCESS [  0.371 s]
  52. [INFO] Apache Hadoop Gridmix ............................. SUCCESS [  0.230 s]
  53. [INFO] Apache Hadoop Data Join ........................... SUCCESS [  0.184 s]
  54. [INFO] Apache Hadoop Extras .............................. SUCCESS [  0.217 s]
  55. [INFO] Apache Hadoop Pipes ............................... SUCCESS [  0.048 s]
  56. [INFO] Apache Hadoop OpenStack support ................... SUCCESS [  0.244 s]
  57. [INFO] Apache Hadoop Client .............................. SUCCESS [  0.590 s]
  58. [INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [  0.230 s]
  59. [INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [  0.650 s]
  60. [INFO] Apache Hadoop Tools Dist .......................... SUCCESS [  0.334 s]
  61. [INFO] Apache Hadoop Tools ............................... SUCCESS [  0.042 s]
  62. [INFO] Apache Hadoop Distribution ........................ SUCCESS [  0.144 s]
  63. [INFO] ------------------------------------------------------------------------
  64. [INFO] BUILD SUCCESS
  65. [INFO] ------------------------------------------------------------------------
  66. [INFO] Total time: 31.234 s
  67. [INFO] Finished at: 2014-06-23T14:55:08+08:00
  68. [INFO] Final Memory: 84M/759M
  69. [INFO] ------------------------------------------------------------------------
复制代码

这时候，咱们已经把源码给下载下来了。这时候，咱们会看到文件会明显增大。


<ignore_js_op> 



3.关联eclipse源码

加入咱们如下程序
<ignore_js_op> hadoop2.2mapreduce例子.rar (1.14 MB, 下载次数: 7, 售价: 1 云币) 
以下图示，对他们进行了打包
<ignore_js_op> 

这两个文件， MaxTemperature.zip为mapreduce例子，mockito-core-1.8.5.jar为mapreduce例子所引用的包
（这里须要说明的是，mapreduce为2.2，可是不影响关联源码，只是交给你们该如何关联源码）
咱们解压以后，导入eclipse
（对于导入项目不熟悉，参考零基础教你如何导入eclipse项目）

<ignore_js_op> 


咱们导入以后，看到不少的红线，这些其实都是没有引用包，下面咱们开始解决这些语法问题。
1、解决导入jar包
（1）引入mockito-core-1.8.5.jar

（2）hadoop2.4编译包中的jar文件，这些文件的位置以下：

hadoop_home中share\hadoop文件夹下，具体我这的位置D:\hadoop2\hadoop-2.4.0\share\hadoop
找到里面的jar包，举例以下：lib文件中的jar包，以及下面的jar包都添加到buildpath中。
若是对于引用包，不知道该如何添加这些jar包，参考hadoop开发方式总结及操做指导。
（注意的是，咱们这里是引入的是编译包，编译的下载hadoop--642.4.0.tar.gz
连接: http://pan.baidu.com/s/1c0vPjG0 密码:xj6l）

更多包下载能够参考hadoop家族、strom、spark、Linux、flume等jar包、安装包汇总下载

apache



 


2、关联源码
1.咱们导入jar包以后，就没有错误了，以下图所示

 

2.找不到源码

当咱们想看一个类或则函数怎么实现的时候，经过Open Call Hierarchy，却找不到源文件。

 



 


3.Attach Source

 

上面三处，咱们按照顺序添加便可，咱们选定压缩包以后，单击肯定，ok了，咱们的工做已经完毕。

注意：对于hadoop-2.2.0-src.zip则是咱们上面经过maven下载的源码，而后压缩的文件，记得必定是压缩文件zip的形式


4.验证关联后查看源码

咱们再次执行上面操做，经过Open Call Hierarchy




看到下面内容



 


而后咱们双击上图主类，即红字部分，咱们看到下面内容：

> api

 




问题：
细心的同窗，这里面咱们产生一个问题，由于咱们看到的是.class文件，而不是.java文件。那么他会不会和咱们所看到的.java文件不同那。
实际上是同样的，感兴趣的同窗，能够验证一下。


下一篇：
如何经过eclipse查看、阅读hadoop2.4源码app

相关文章
相关标签/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公众号
   欢迎关注本站公众号,获取更多信息
联系我们 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程