基于Hadoop1.2.1彻底分布式集群的部署
1、准备工做
- 同一个局域网中的三台Linux虚拟机，发行版本均使用64位CentOS6.3,主机是 Windows 10 64位操做系统；经过 vmware workstation 实现三台虚机，这样就造成了一个以物理机为DNS服务器的局域网，物理机和3台虚机都有一个局域网IP，从而实现相互之间的通讯。
- Hadoop是一个用java语言编程的开源软件，因此须要安装JDK，小编采用的JDK的版本是最新的jdk1.8；此外，Hadoop集群部分功能是用perl语言实现的，所以，还需安装perl环境；集群之间的通信是经过SSH实现的，这就不须要小编介绍啦！你懂的......
- 安装Linux后必定要确讣iptables,selinux等防火墙戒访问控制机制已经关闭，不然实
验极可能受影响. - 本次试验搭建的环境为：
namenode : 192.168.115.130 matserhtml
datanode1：192.168.115.131 slavejava
datanode2：192.168.115.132 slavenode
2、配置hosts
在全部节点上（namenode、datanode一、datanode2）的终端上经过 vi /etc/hosts上添加以下代码：linux
192.168.115.130 namenode 192.168.115.131 datanode1 192.168.115.132 datanode2
查看可否ping通，若出现下图，标识配置hosts成功：c++
3、JDK的安装
- 终端使用wget下载 jdk-8u111-linux-x64.tar.gz
wge http://download.oracle.com/otn-pub/java/jdk/8u111-b14/jdk-8u111-linux-x64.tar.gz
- 解压 tar -xzvf jdk-8u111-linux-x64.tar.gz
[root@namenode tpf]# tar -xzvf jdk-8u111-linux-x64.tar.gz
- 配置环境变量 vi /etc/profile ，在文件的结尾处添加以下代码，保存退出。
#JAVA JAVA_HOME=/usr/local/jdk1.8.0_111 PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export JAVA_HOME PATH CLASSPATH
- 执行source /etc/profile，使修改的文件生效。
- 查看文件是否生效 echo $PATH
[root@namenode Desktop]# echo $PATH /usr/local/jdk1.8.0_111/bin:/usr/local/jdk1.8.0_111/jre/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/tpf/bin
- 依次输入以下三个命令，出现下图状况，表示JDK安装成功；
[root@namenode Desktop]# java -version java version "1.8.0_111" Java(TM) SE Runtime Environment (build 1.8.0_111-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
- 常规来讲，若是出现上图，JDK的安装多是成功的，但要想确保安装准确无误，须要进一步验证（见下图）：
[root@namenode Desktop]# java
Usage: java [-options] class [args...]
(to execute a class)
or java [-options] -jar jarfile [args...]
(to execute a jar file)
where options include:
-d32 use a 32-bit data model if available
-d64 use a 64-bit data model if available
-server to select the "server" VM
The default VM is server.
-cp <class search path of directories and zip/jar files>
-classpath <class search path of directories and zip/jar files>
A : separated list of directories, JAR archives,
and ZIP archives to search for class files.
-D<name>=<value>
set a system property
-verbose:[class|gc|jni]
[root@namenode Desktop]# javac
Usage: javac <options> <source files>
where possible options include:
-g Generate all debugging info
-g:none Generate no debugging info
-g:{lines,vars,source} Generate only some debugging info
-nowarn Generate no warnings
-verbose Output messages about what the compiler is doing
-deprecation Output source locations where deprecated APIs are used
-classpath <path> Specify where to find user class files and annotation processors
-cp <path> Specify where to find user class files and annotation processors
-sourcepath <path> Specify where to find input source files
-bootclasspath <path> Override location of bootstrap class files
-extdirs <dirs> Override location of installed extensions
-endorseddirs <dirs> Override location of endorsed standards path
-proc:{none,only} Control whether annotation processing and/or compilation is done.
-processor <class1>[,<class2>,<class3>...] Names of the annotation processors to r
若是出现上述图形的状况，小编恭喜你JDK安装成功啦！web
4、SSH互信
- 在虚机192.168.115.130/namenode节点上root权限下运行命令 ssh-keygen -t rsa
[root@namenode tpf]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 2d:ed:ec:31:00:eb:18:23:59:83:8b:1f:df:ee:73:9b root@namenode The key's randomart image is: +--[ RSA 2048]----+ | | | . | | . o . | | . + . o o | |. = o . S o | | . + * = | | . o o = | | .. .o o | | .ooE.. | +-----------------+
- 查看密钥
[root@namenode tpf]# cd /root/ [root@namenode ~]# ls anaconda-ks.cfg Desktop Documents Downloads install.log install.log.syslog Music Pictures Public Templates Videos [root@namenode ~]# ls -a . .bash_history .bashrc .dbus Downloads .gconfd .gnupg .gvfs install.log.syslog .nautilus Public .recently-used.xbel .tcshrc .xauthCtsxNO .. .bash_logout .config Desktop .esd_auth .gnome2 .gstreamer-0.10 .ICEauthority .local .oracle_jre_usage .pulse .spice-vdagent Templates anaconda-ks.cfg .bash_profile .cshrc Documents .gconf .gnote .gtk-bookmarks install.log Music Pictures .pulse-cookie .ssh Videos [root@namenode ~]# cd .ssh/ [root@namenode .ssh]# ls id_rsa id_rsa.pub known_hosts
-
查看文件的属性，确保下图所示的文件属性shell
[root@namenode .ssh]# ls -l total 12 -rw-------. 1 root root 1675 Nov 25 23:03 id_rsa -rw-r--r--. 1 root root 395 Nov 25 23:03 id_rsa.pub -rw-r--r--. 1 root root 2392 Nov 25 11:10 known_hosts
- 转到虚机192.168.115.131 slave/datanode1节点上以root权限重复上述步骤生成密钥
[root@datanode2 ~]# cd /home/datanode2/ [root@datanode2 datanode2]# pwd /home/datanode2 [root@datanode2 datanode2]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: a9:da:e2:80:3a:d0:a9:1b:65:c0:c7:e6:10:61:a7:65 root@datanode2 The key's randomart image is: +--[ RSA 2048]----+ | +.E | |o B | |.+ + | | .= . | | .oo S | |.+o . | |+.. . | |+. ..o | |oo .o.. | +-----------------+ [root@datanode2 datanode2]# cd /root/ [root@datanode2 ~]# ls anaconda-ks.cfg install.log install.log.syslog [root@datanode2 ~]# ls -a . .. anaconda-ks.cfg .bash_history .bash_logout .bash_profile .bashrc .cshrc install.log install.log.syslog .oracle_jre_usage .ssh .tcshrc .xauth9yJBa2 [root@datanode2 ~]# cd .ssh [root@datanode2 .ssh]# ls id_rsa id_rsa.pub known_hosts [root@datanode2 .ssh]# ls -l total 12 -rw-------. 1 root root 1671 Nov 25 23:28 id_rsa -rw-r--r--. 1 root root 396 Nov 25 23:28 id_rsa.pub -rw-r--r--. 1 root root 813 Nov 25 22:50 known_hosts
-
转到虚机192.168.115.132 slave/datanode2节点上以root权限重复上述步骤生成密钥；apache
[datanode2@datanode2 ~]$ pwd /home/datanode2 [datanode2@datanode2 ~]$ ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/home/datanode2/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /home/datanode2/.ssh/id_rsa. Your public key has been saved in /home/datanode2/.ssh/id_rsa.pub. The key fingerprint is: b3:10:5c:18:5c:40:4c:4a:d0:00:26:0b:ec:b3:34:ed datanode2@datanode2 The key's randomart image is: +--[ RSA 2048]----+ |=oo+.===o | |+o ..o+. | |o . . o | | = . . | |. = . S | | . E . o | | . | | | | | +-----------------+ [datanode2@datanode2 ~]$ cd /root/ bash: cd: /root/: Permission denied [datanode2@datanode2 ~]$ ls -a . .bash_profile .dbus Downloads .gconfd .gstreamer-0.10 .icons .nautilus .pulse .ssh Videos .. .bashrc Desktop .esd_auth .gnome2 .gtk-bookmarks .local .oracle_jre_usage .pulse-cookie Templates .viminfo .bash_history .cache .dmrc .fontconfig .gnote .gvfs .mozilla Pictures .recently-used.xbel .themes .xsession-errors .bash_logout .config Documents .gconf .gnupg .ICEauthority Music Public .spice-vdagent .thumbnails .xsession-errors.old [datanode2@datanode2 ~]$ cd .ssh [datanode2@datanode2 .ssh]$ ls id_rsa id_rsa.pub [datanode2@datanode2 .ssh]$ ls -l total 8 -rw-------. 1 datanode2 datanode2 1675 Nov 25 23:35 id_rsa -rw-r--r--. 1 datanode2 datanode2 401 Nov 25 23:35 id_rsa.pub
-
将节点1（192.168.115.130/namenode）的id_rsa.pub文件分别拷贝到节点2和节点3的/root/.ssh目录下，并重命名为authorized_keys；编程
[root@namenode .ssh]# scp ./id_rsa.pub root@datanode1:/root/.ssh/authorized_keys root@datanode1's password: id_rsa.pub 100% 395 0.4KB/s 00:00 [root@namenode .ssh]# scp ./id_rsa.pub root@datanode2:/root/.ssh/authorized_keys root@datanode2's password: id_rsa.pub 100% 395 0.4KB/s 00:00 [root@namenode .ssh]# scp ./id_rsa.pub root@datanode2:/root/.ssh/authorized_keys id_rsa.pub 100% 395 0.4KB/s 00:00
-
分别在节点2 和节点3 的对于目录下查看拷贝文件的属性；bootstrap
-
验证是否生效，实现免密码登陆；
以上仅仅只是实现了从namenode节点SSH免密码登陆到datanode1节点和datanode2节点，由于datanode1节点和datanode2节点只保留了namenode节点的公钥，且上述步骤并不能使namenode节点免密码SSH登陆到自身节点，见下图：
若是咱们想要在3个节点间实现任意节点的免密码SSH登陆而且能够免密码SSH登陆到自身节点，首先咱们须要造在namenode节点的/root/.ssh目录下建立authorized_keys文件，将其余节点的公钥拷贝到的authorized_keys文件中，从而实现任意节点间的免密码SSH登陆。
- 以namenode节点为例实现自身的SSH免密码登陆；
[root@namenode ~]# scp /root/.ssh/id_rsa.pub root@namenode:/root/.ssh/authorized_keys root@namenode's password: id_rsa.pub 100% 395 0.4KB/s 00:00 [root@namenode ~]# cd /root/ [root@namenode ~]# ls anaconda-ks.cfg Desktop Documents Downloads install.log install.log.syslog Music Pictures Public Templates Videos [root@namenode ~]# cd .ssh [root@namenode .ssh]# ls authorized_keys id_rsa id_rsa.pub known_hosts [root@namenode .ssh]# ls -l total 16 -rw-r--r--. 1 root root 395 Nov 26 00:24 authorized_keys -rw-------. 1 root root 1675 Nov 25 23:03 id_rsa -rw-r--r--. 1 root root 395 Nov 25 23:03 id_rsa.pub -rw-r--r--. 1 root root 2392 Nov 25 11:10 known_hosts [root@namenode .ssh]# ssh namenode Last login: Sat Nov 26 00:22:57 2016 from datanode2
-
使用vi authorized_keys命令编辑每一个节点的authorized_keys文件，将缺乏的节点公钥拷贝到该文件里，保存并退出；
-
验证任意节点的免密码自登录及任意节点间的免密码SSH登陆；
[root@namenode .ssh]# ssh namenode Last login: Sat Nov 26 00:24:59 2016 from namenode [root@namenode ~]# ssh datanode1 Last login: Sat Nov 26 00:34:16 2016 from namenode [root@datanode1 ~]# ssh datanode2 Last login: Fri Nov 25 23:58:37 2016 from namenode [root@datanode2 ~]# ssh datanode1 Last login: Sat Nov 26 00:48:14 2016 from namenode [root@datanode1 ~]# ssh namenode Last login: Sat Nov 26 00:48:02 2016 from namenode
经过以上步骤，便可实现了本集群中任意一台机器免密码SSH登陆到另一台机器；接下来，小编带你进入Hadoop的正式安装......
5、Hadoop1.2.1的安装
hadoop的安装通常是先在namenode节点上修改相关文件，而后将配置好的namenode节点上的hadoop文件夹向各节点复制；小编先是配置虚机192.168.115.130/namenode节点上的hadoop,而后向虚机92.168.115.131/datanode1节点和192.168.115.132/datanode2节点复制修改好的hadoop文件。
- 确保iptables,selinux等防火墙或访问控制机制已关闭
[root@namenode Desktop]# service iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ]
- 下载并解压到文件夹hadoop-1.2.1
- cd /usr/local/hadoop-1.2.1/conf/ 转到hadoop配置目录下 进行文件的配置
[tpf@namenode hadoop-1.2.1]$ cd conf [tpf@namenode conf]$ ls capacity-scheduler.xml fair-scheduler.xml hadoop-policy.xml mapred-queue-acls.xml slaves taskcontroller.cfg configuration.xsl hadoop-env.sh hdfs-site.xml mapred-site.xml ssl-client.xml.example task-log4j.properties core-site.xml hadoop-metrics2.properties log4j.properties masters ssl-server.xml.example
- 修改hadoop-env.sh
# The java implementation to use. Required. export JAVA_HOME=/usr/local/jdk1.8.0_111
- 修改core-site.xml 写入以下信息，要注意，本身要在/usr/local/hadoop-1.2.1/ 目录下手动创建tmp目录，如没有配置hadoop.tmp.dir参数，此时系统默认的临时目录为：/tmp/hadoop-dfs。而这个目录在每次重启后都会被删掉，必须从新执行format才行，不然会出错，hdfs后面的ip号就是名称节点的ip,即namenode的ip,端口号默认9000,不用改动 ；
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://namenode:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop-1.2.1/tmp</value>
</property>
</configuration>
- 修改hdfs-site.xml
设置数据备份数目，这里咱们有有一个namenode，两个datanode，因此备份数目设置为2
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
- 修改mapred-site.xml 设置map-reduce的做业跟踪器（jobtracker）所在节点，这里咱们一样将其放置在namenode(192.168.115.130)节点上，端口号9001
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>http://namenode:9001</value>
</property>
</configuration>
- 修改master 指定namenode所在节点；
namenode
- 修改slave 指定集群的datanode节点 ；
datanode1 datanode2
- 向各节点复制hadoop
[root@namenode local]# scp -r /usr/local/hadoop-1.2.1/ datanode1:/usr/local/ [root@namenode local]# scp -r /usr/local/hadoop-1.2.1/ datanode2:/usr/local/
- 执行文件系统格式化：bin/hadoop namenode -format,若是没有任何warning,error,fatal等而且最后出现，format successfully字样，则格式化成；
[root@namenode hadoop-1.2.1]# bin/hadoop namenode -format 16/11/26 04:30:08 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = namenode/192.168.115.130 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.2.1 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.2 -r 1503152; compiled by 'mattf' on Mon Jul 22 15:23:09 PDT 2013 STARTUP_MSG: java = 1.8.0_111 ************************************************************/ 16/11/26 04:30:09 INFO util.GSet: Computing capacity for map BlocksMap 16/11/26 04:30:09 INFO util.GSet: VM type = 64-bit 16/11/26 04:30:09 INFO util.GSet: 2.0% max memory = 1013645312 16/11/26 04:30:09 INFO util.GSet: capacity = 2^21 = 2097152 entries 16/11/26 04:30:09 INFO util.GSet: recommended=2097152, actual=2097152 16/11/26 04:30:09 INFO namenode.FSNamesystem: fsOwner=root 16/11/26 04:30:09 INFO namenode.FSNamesystem: supergroup=supergroup 16/11/26 04:30:09 INFO namenode.FSNamesystem: isPermissionEnabled=true 16/11/26 04:30:10 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100 16/11/26 04:30:10 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s) 16/11/26 04:30:10 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0 16/11/26 04:30:10 INFO namenode.NameNode: Caching file names occuring more than 10 times 16/11/26 04:30:11 INFO common.Storage: Image file /usr/local/hadoop-1.2.1/tmp/dfs/name/current/fsimage of size 110 bytes saved in 0 seconds. 16/11/26 04:30:11 INFO namenode.FSEditLog: closing edit log: position=4, editlog=/usr/local/hadoop-1.2.1/tmp/dfs/name/current/edits 16/11/26 04:30:11 INFO namenode.FSEditLog: close success: truncate to 4, editlog=/usr/local/hadoop-1.2.1/tmp/dfs/name/current/edits 16/11/26 04:30:11 INFO common.Storage: Storage directory /usr/local/hadoop-1.2.1/tmp/dfs/name has been successfully formatted. 16/11/26 04:30:11 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at namenode/192.168.115.130 ************************************************************/
- 启动hadoop
[root@namenode hadoop-1.2.1]# bin/start-all.sh starting namenode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-tpf-namenode-namenode.out datanode2: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-datanode2.out datanode1: starting datanode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-datanode-datanode1.out namenode: starting secondarynamenode, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-secondarynamenode-namenode.out starting jobtracker, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-tpf-jobtracker-namenode.out datanode1: starting tasktracker, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-datanode1.out datanode2: starting tasktracker, logging to /usr/local/hadoop-1.2.1/libexec/../logs/hadoop-root-tasktracker-datanode2.out
namenode节点的启动日志：
[root@namenode hadoop-1.2.1]# cd logs/ [root@namenode logs]# ls hadoop-root-secondarynamenode-namenode.log hadoop-tpf-jobtracker-namenode.log hadoop-tpf-namenode-namenode.log history hadoop-root-secondarynamenode-namenode.out hadoop-tpf-jobtracker-namenode.out hadoop-tpf-namenode-namenode.out
datanode1节点的启动日志：
[root@datanode1 logs]# ls hadoop-root-datanode-datanode1.log hadoop-root-datanode-datanode1.out hadoop-root-tasktracker-datanode1.log hadoop-root-tasktracker-datanode1.out
datanode2节点的启动日志：
[root@datanode2 hadoop-1.2.1]# cd logs/ [root@datanode2 logs]# ls hadoop-root-datanode-datanode2.log hadoop-root-datanode-datanode2.out hadoop-root-tasktracker-datanode2.log hadoop-root-tasktracker-datanode2.out
- 用jps检验各后台进程是否成功启动
[root@namenode hadoop-1.2.1]# jps 55553 SecondaryNameNode 55411 NameNode 55637 JobTracker 55737 Jps
[root@datanode1 hadoop-1.2.1]# jps 39219 Jps 39155 TaskTracker 39077 DataNode
[root@datanode2 hadoop-1.2.1]# jps 6641 DataNode 6814 Jps 6719 TaskTracker
第一次启动的时候，datanode1节点老是启动不了；最后找下来发现是因为datanode1的防火墙没关闭，阻止了相应的端口号。
[root@namenode hadoop-1.2.1]# bin/stop-all.sh stopping jobtracker datanode2: stopping tasktracker datanode1: no tasktracker to stop stopping namenode datanode2: stopping datanode datanode1: no datanode to stop namenode: stopping secondarynamenode [root@datanode1 logs]# jps 38824 Jps
6、集群测试
[root@namenode hadoop-1.2.1]# bin/hadoop fs -put /usr/local/input/ in [root@namenode hadoop-1.2.1]# bin/hadoop fs -cat /usr/local/input/ in cat: File does not exist: /usr/local/input cat: File does not exist: /user/root/in [root@namenode hadoop-1.2.1]# bin/hadoop fs -cat /usr/local/input/ ./in/* cat: File does not exist: /usr/local/input hello world hello hadoop [root@namenode hadoop-1.2.1]# bin/hadoop fs -cat /usr/local/input/ ./in cat: File does not exist: /usr/local/input cat: File does not exist: /user/root/in [root@namenode hadoop-1.2.1]# bin/hadoop fs -ls ./in Found 2 items -rw-r--r-- 2 root supergroup 12 2016-11-26 07:18 /user/root/in/text1.txt -rw-r--r-- 2 root supergroup 13 2016-11-26 07:18 /user/root/in/text2.txt [root@namenode hadoop-1.2.1]# bin/hadoop fs -cat ./in/text1.txt hello world [root@namenode hadoop-1.2.1]# bin/hadoop fs -cat ./in/text2.txt hello hadoop [root@namenode hadoop-1.2.1]# ls bin c++ conf docs hadoop-client-1.2.1.jar hadoop-examples-1.2.1.jar hadoop-test-1.2.1.jar ivy lib LICENSE.txt NOTICE.txt sbin src webapps build.xml CHANGES.txt contrib hadoop-ant-1.2.1.jar hadoop-core-1.2.1.jar hadoop-minicluster-1.2.1.jar hadoop-tools-1.2.1.jar ivy.xml libexec logs README.txt share tmp [root@namenode hadoop-1.2.1]# bin/hadoop jar hadoop-examples-1.2.1.jar wordcount in out 16/11/26 07:24:35 INFO input.FileInputFormat: Total input paths to process : 2 16/11/26 07:24:35 INFO util.NativeCodeLoader: Loaded the native-hadoop library 16/11/26 07:24:35 WARN snappy.LoadSnappy: Snappy native library not loaded 16/11/26 07:24:36 INFO mapred.JobClient: Running job: job_201611260717_0001 16/11/26 07:24:37 INFO mapred.JobClient: map 0% reduce 0% 16/11/26 07:24:49 INFO mapred.JobClient: map 50% reduce 0% 16/11/26 07:24:52 INFO mapred.JobClient: map 100% reduce 0% 16/11/26 07:25:03 INFO mapred.JobClient: map 100% reduce 100% 16/11/26 07:25:06 INFO mapred.JobClient: Job complete: job_201611260717_0001 16/11/26 07:25:06 INFO mapred.JobClient: Counters: 29 16/11/26 07:25:06 INFO mapred.JobClient: Map-Reduce Framework 16/11/26 07:25:06 INFO mapred.JobClient: Spilled Records=8 16/11/26 07:25:06 INFO mapred.JobClient: Map output materialized bytes=61 16/11/26 07:25:06 INFO mapred.JobClient: Reduce input records=4 16/11/26 07:25:06 INFO mapred.JobClient: Virtual memory (bytes) snapshot=5800710144 16/11/26 07:25:06 INFO mapred.JobClient: Map input records=2 16/11/26 07:25:06 INFO mapred.JobClient: SPLIT_RAW_BYTES=216 16/11/26 07:25:06 INFO mapred.JobClient: Map output bytes=41 16/11/26 07:25:06 INFO mapred.JobClient: Reduce shuffle bytes=61 16/11/26 07:25:06 INFO mapred.JobClient: Physical memory (bytes) snapshot=418377728 16/11/26 07:25:06 INFO mapred.JobClient: Reduce input groups=3 16/11/26 07:25:06 INFO mapred.JobClient: Combine output records=4 16/11/26 07:25:06 INFO mapred.JobClient: Reduce output records=3 16/11/26 07:25:06 INFO mapred.JobClient: Map output records=4 16/11/26 07:25:06 INFO mapred.JobClient: Combine input records=4 16/11/26 07:25:06 INFO mapred.JobClient: CPU time spent (ms)=3400 16/11/26 07:25:06 INFO mapred.JobClient: Total committed heap usage (bytes)=337780736 16/11/26 07:25:06 INFO mapred.JobClient: File Input Format Counters 16/11/26 07:25:06 INFO mapred.JobClient: Bytes Read=25 16/11/26 07:25:06 INFO mapred.JobClient: FileSystemCounters 16/11/26 07:25:06 INFO mapred.JobClient: HDFS_BYTES_READ=241 16/11/26 07:25:06 INFO mapred.JobClient: FILE_BYTES_WRITTEN=173342 16/11/26 07:25:06 INFO mapred.JobClient: FILE_BYTES_READ=55 16/11/26 07:25:06 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=25 16/11/26 07:25:06 INFO mapred.JobClient: Job Counters 16/11/26 07:25:06 INFO mapred.JobClient: Launched map tasks=2 16/11/26 07:25:06 INFO mapred.JobClient: Launched reduce tasks=1 16/11/26 07:25:06 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=12312 16/11/26 07:25:06 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 16/11/26 07:25:06 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=20972 16/11/26 07:25:06 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 16/11/26 07:25:06 INFO mapred.JobClient: Data-local map tasks=2 16/11/26 07:25:06 INFO mapred.JobClient: File Output Format Counters 16/11/26 07:25:06 INFO mapred.JobClient: Bytes Written=25 [root@namenode hadoop-1.2.1]# bin/hadoop fs -ls Found 2 items drwxr-xr-x - root supergroup 0 2016-11-26 07:18 /user/root/in drwxr-xr-x - root supergroup 0 2016-11-26 07:25 /user/root/out [root@namenode hadoop-1.2.1]# bin/hadoop fs -ls ./out Found 3 items -rw-r--r-- 2 root supergroup 0 2016-11-26 07:25 /user/root/out/_SUCCESS drwxr-xr-x - root supergroup 0 2016-11-26 07:24 /user/root/out/_logs -rw-r--r-- 2 root supergroup 25 2016-11-26 07:24 /user/root/out/part-r-00000 [root@namenode hadoop-1.2.1]# bin/hadoop fs -cat ./out/part-r-00000 hadoop 1 hello 2 world 1
7、注意事项
- 确保关闭防火墙
#关闭: service iptables stop #查看: service iptables status #重启不启动: chkconfig iptables off #重启启动: chkconfig iptables on
[INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main ................................. SUCCESS [ 01:05 h] [INFO] Apache Hadoop Project POM .......................... SUCCESS [05:58 min] [INFO] Apache Hadoop Annotations .......................... SUCCESS [03:01 min] [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.598 s] [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [02:07 min] [INFO] Apache Hadoop Maven Plugins ........................ FAILURE [02:08 min] [INFO] Apache Hadoop MiniKDC .............................. SKIPPED [INFO] Apache Hadoop Auth ................................. SKIPPED [INFO] Apache Hadoop Auth Examples ........................ SKIPPED [INFO] Apache Hadoop Common ............................... SKIPPED [INFO] Apache Hadoop NFS .................................. SKIPPED [INFO] Apache Hadoop Common Project ....................... SKIPPED [INFO] Apache Hadoop HDFS ................................. SKIPPED [INFO] Apache Hadoop HttpFS ............................... SKIPPED [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED [INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED [INFO] Apache Hadoop HDFS Project ......................... SKIPPED [INFO] hadoop-yarn ........................................ SKIPPED [INFO] hadoop-yarn-api .................................... SKIPPED [INFO] hadoop-yarn-common ................................. SKIPPED [INFO] hadoop-yarn-server ................................. SKIPPED [INFO] hadoop-yarn-server-common .......................... SKIPPED [INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED [INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED [INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED [INFO] hadoop-yarn-server-tests ........................... SKIPPED [INFO] hadoop-yarn-client ................................. SKIPPED [INFO] hadoop-yarn-applications ........................... SKIPPED [INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED [INFO] hadoop-yarn-site ................................... SKIPPED [INFO] hadoop-yarn-project ................................ SKIPPED [INFO] hadoop-mapreduce-client ............................ SKIPPED [INFO] hadoop-mapreduce-client-core ....................... SKIPPED [INFO] hadoop-mapreduce-client-common ..................... SKIPPED [INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED [INFO] hadoop-mapreduce-client-app ........................ SKIPPED [INFO] hadoop-mapreduce-client-hs ......................... SKIPPED [INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED [INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED [INFO] Apache Hadoop MapReduce Examples ................... SKIPPED [INFO] hadoop-mapreduce ................................... SKIPPED [INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED [INFO] Apache Hadoop Distributed Copy ..................... SKIPPED [INFO] Apache Hadoop Archives ............................. SKIPPED [INFO] Apache Hadoop Rumen ................................ SKIPPED [INFO] Apache Hadoop Gridmix .............................. SKIPPED [INFO] Apache Hadoop Data Join ............................ SKIPPED [INFO] Apache Hadoop Extras ............................... SKIPPED [INFO] Apache Hadoop Pipes ................................ SKIPPED [INFO] Apache Hadoop OpenStack support .................... SKIPPED [INFO] Apache Hadoop Client ............................... SKIPPED [INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED [INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED [INFO] Apache Hadoop Tools Dist ........................... SKIPPED [INFO] Apache Hadoop Tools ................................ SKIPPED [INFO] Apache Hadoop Distribution ......................... SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:51 h [INFO] Finished at: 2016-11-27T06:31:52-08:00 [INFO] Final Memory: 44M/106M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.8.1:jar (module-javadocs) on project hadoop-maven-plugins: MavenReportException: Error while creating archive: [ERROR] Exit code: 1 - /home/grid/release-2.3.0/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java:47: error: unknown tag: String [ERROR] * @param command List<String> containing command and all arguments [ERROR] ^ [ERROR] /home/grid/release-2.3.0/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/Exec.java:48: error: unknown tag: String [ERROR] * @param output List<String> in/out parameter to receive command output [ERROR] ^ [ERROR] /home/grid/release-2.3.0/hadoop-maven-plugins/src/main/java/org/apache/hadoop/maven/plugin/util/FileSetUtils.java:52: error: unknown tag: File [ERROR] * @return List<File> containing every element of the FileSet as a File [ERROR] ^ [ERROR] [ERROR] Command line was: /usr/local/jdk1.8.0_111/jre/../bin/javadoc @options @packages [ERROR] [ERROR] Refer to the generated Javadoc files in '/home/grid/release-2.3.0/hadoop-maven-plugins/target' dir. [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command
官网地址：http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-common/SingleCluster.html
相关文章
- 1. Hadoop1.2.1集群彻底分布式模式环境部署
- 2. hbase彻底分布式集群部署
- 3. 基于CentOS7的Hadoop3.2.0彻底分布式集群部署记录
- 4. Ubuntu14.04安装彻底分布式Hadoop1.2.1
- 5. Hadoop1.X集群彻底分布式模式环境部署
- 6. Hadoop及Zookeeper+HBase彻底分布式集群部署
- 7. Centos7.5搭建Hadoop2.8.5彻底分布式集群部署
- 8. hadoop笔记之部署hadoop集群(彻底分布式)
- 9. Hadoop3.1.0彻底分布式集群部署超详细记录
- 10. Hadoop3.0彻底分布式集群安装部署
- 更多相关文章...
- • ionic 头部与底部 - ionic 教程
- • ionic 头部和底部 - ionic 教程
- • ☆基于Java Instrument的Agent实现
- • 常用的分布式事务解决方案
相关标签/搜索
每日一句
-
每一个你不满意的现在，都有一个你没有努力的曾经。
欢迎关注本站公众号,获取更多信息
相关文章