Hadoop集群搭建1Master、3Slaves

时间 2021-07-13 标签 hadoop集群 hadoop 搭建 jar包运行

master.hadoop 10.9.148.193
slave1.hadoop 10.9.148.190
slave2.hadoop 10.9.148.191
slave3.hadoop 10.9.148.192
root 123456 hadoop 123456
1. 重装系统
Centos6
2. 设置固定IP
IPv4：
手动
地址子网掩码网关
10.9.148.193 255.255.255.0 10.9.148.1
DNS：10.9.148.1
应用
3. 设置sshd启动和开机启动（centos6）
rpm –qa | grep ssh
….
service sshd start
chkconfig sshd on
4. 设置主机名
Ubuntu：10.9.148.193 master.hadoop
[ [email protected] ~]# vi /etc/sysconfig/network
[ [email protected] ~]# vi /etc/hostname
[ [email protected] ~]# vi /etc/hosts
Centos:其它三台
[ [email protected] ~]# vi /etc/hosts
127.0.0.1 localhost

::1 localhost6.localdomain6 localhost6

#hadoop
10.9.148.193 master.hadoop
10.9.148.190 slave1.hadoop
10.9.148.191 slave2.hadoop
10.9.148.192 slave3.hadoop
#hbase Master Slave IP address
10.9.148.193 master
10.9.148.190 slave1
10.9.148.191 slave2
10.9.148.192 slave3
[ [email protected] ~]# vi /etc/sysconfig/network
[ [email protected] ~]# reboot
5. 安装JDK
5.1Centos（为新装的系统，删除自带的java）
[ [email protected] local]# rpm -qa | grep gcj
java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
libgcj-4.4.7-18.el6.x86_64
[ [email protected] local]# yum -y remove java java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
5.2安装
[ [email protected] local]# tar -zxf jdk-7u45-linux-x64.tar.gz
[ [email protected] local]# ls
bin games jdk1.7.0_45 lib libexec share
etc include jdk-7u45-linux-x64.tar.gz lib64 sbin src
[ [email protected] local]# mv jdk1.7.0_45 jdk1.7
[ [email protected] local]# ls
bin games jdk1.7 lib libexec share
etc include jdk-7u45-linux-x64.tar.gz lib64 sbin src
[ [email protected] local]# rm -rf jdk-7u45-linux-x64.tar.gz
[ [email protected] local]# ls
bin etc games include jdk1.7 lib lib64 libexec sbin share src
[ [email protected] local]# vi /etc/profile
#java
export JAVA_HOME=/usr/local/jdk1.7
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$CLASSPATH:
[ [email protected] local]# source /etc/profile
[ [email protected] local]# java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
[ [email protected] local]#
6. SSH无**验证配置（其它的节点同理，只需master和slave之间互相无**登录即可，这里slave和slave之间也配置了，可忽略）
[ [email protected] ~]# ssh-****** -t rsa -P ''
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
43:a0:f3:40:59:27:76:fc:fb:55:c7:d3:8b:66:28:00 [email protected]
The key's randomart image is:
+--[ RSA 2048]----+
| .o=.o |
| ..E =. |
| + . .. ..|
| + o . .=|
| . S .. ..+|
| o.. +.. |
| ..o. |
| . |
| |
+-----------------+
[ [email protected] ~]# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
[ [email protected] ~]# cd .ssh/
[ [email protected] .ssh]# chmod 600 authorized_keys
[ [email protected] .ssh]# vi /etc/ssh/sshd_config
修改内容：
去掉前面的“#”注释
RSAAuthentication yes #启用RSA认证
PubkeyAuthentication yes #启用公钥私钥配对认证方式
[ [email protected] .ssh]# cat ~/.ssh/id_rsa.pub | ssh [email protected] 'cat >> ~/.ssh/authorized_keys'
The authenticity of host 'slave2.hadoop (10.9.148.191)' can't be established.
RSA key fingerprint is 82:e8:77:8c:32:ed:6a:62:6c:c5:55:0d:10:4a:84:30.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave2.hadoop,10.9.148.191' (RSA) to the list of known hosts.
[email protected]'s password:
[ [email protected] .ssh]# cat ~/.ssh/id_rsa.pub | ssh [email protected] 'cat >> ~/.ssh/authorized_keys'
The authenticity of host 'slave1.hadoop (10.9.148.190)' can't be established.
RSA key fingerprint is 96:37:8a:3f:b1:1d:6b:c1:f7:cc:3b:c2:3e:03:09:07.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'slave1.hadoop,10.9.148.190' (RSA) to the list of known hosts.
[email protected]'s password:
[ [email protected] .ssh]# cat ~/.ssh/id_rsa.pub | ssh [email protected] 'cat >> ~/.ssh/authorized_keys'
//追加到主机Master的无**登录配置里
The authenticity of host 'master.hadoop (10.9.148.193)' can't be established.
RSA key fingerprint is fc:eb:53:bd:e9:e2:ec:20:fe:8a:2e:da:f9:f8:e5:b0.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'master.hadoop,10.9.148.193' (RSA) to the list of known hosts.
[email protected]'s password:
[ [email protected] .ssh]# ssh master.hadoop
Last login: Thu Aug 24 13:29:09 2017 from 10.9.148.17
[ [email protected] ~]# ssh slave3.hadoop
Last login: Wed Aug 23 13:35:15 2017 from 10.9.148.17
[ [email protected] ~]#
7. 创建hadoop系统用户并设置密码（Master与Slave相同）（这里直接用root账户安装配置实用）
846 useradd hadoop
847 passwd hadoop
8. 安装hadoop
952 vi slaves
10.9.148.190
10.9.148.191
10.9.148.192
953 vi core-site.xml
<property>
<name>hadoop.tmp.dir</name>
<value>/home/softs/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>

<property>
<name>fs.default.name</name>
<value>hdfs://master.hadoop:9000</value>
</property>
954 vi hdfs-site.xml
<configuration>

<property>
<name>dfs.namenode.secondary.http-address</name>
<value>master.hadoop:50090</value>
</property>

<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/softs/hadoop/tmp/dfs/name</value>
</property>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/softs/hadoop/tmp/dfs/data</value>
</property>

<property>
<name>dfs.replication</name>
<value>3</value>
</property>

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
957 mv mapred-site.xml.template mapred-site.xml
958 cp mapred-site.xml mapred-site.xml.template
959 ls
960 vi mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master.hadoop:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master.hadoop:19888</value>
</property>
</configuration>
961 vi yarn-site.xml
<configuration>


<property>
<name>yarn.resourcemanager.hostname</name>
<value>master.hadoop</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
962 vi masters
master.hadoop
984 tar -zcf hadoop.master.tar.gz ./hadoop
//把master.hadoop配置好的hadoop压缩，分别复制到各slave里
985 scp ./hadoop.master.tar.gz slave1.hadoop:/home/softs/
986 scp ./hadoop.master.tar.gz slave2.hadoop:/home/softs/
987 scp ./hadoop.master.tar.gz slave3.hadoop:/home/softs/

62 tar -zxf hadoop.master.tar.gz
63 chown -R hadoop ./hadoop
<启动时，因为找不到JAVA_HOME，但是明显/etc/profile里面配置了的，所以只好手动添加java的路径到/home/softs/hadoop/etc/hadoop/hadoop-env.sh 里面、>
68 vi /home/softs/hadoop/etc/hadoop/hadoop-env.sh

<注意关闭防火墙>：否则，文件传输过程中会报错
[ [email protected] hadoop]# hadoop fs -copyFromLocal /home/201706/* /I2
17/08/24 15:46:49 INFO hdfs.DataStreamer: Exception in createBlockOutputStream
java.net.NoRouteToHostException: 没有到主机的路由
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:226)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1591)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1547)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:658)
17/08/24 15:46:49 WARN hdfs.DataStreamer: Abandoning BP-748028963-10.9.148.193-1503559871024:blk_1073741833_1009
17/08/24 15:46:49 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[10.9.148.190:50010,DS-e1300bd0-fb4f-488e-9c79-42b47c1ef714,DISK]
17/08/24 15:46:49 INFO hdfs.DataStreamer: Exception in createBlockOutputStream
java.net.NoRouteToHostException: 没有到主机的路由
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:226)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1591)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1547)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:658)
17/08/24 15:46:49 WARN hdfs.DataStreamer: Abandoning BP-748028963-10.9.148.193-1503559871024:blk_1073741834_1010
17/08/24 15:46:49 WARN hdfs.DataStreamer: Excluding datanode DatanodeInfoWithStorage[10.9.148.191:50010,DS-969a3d41-8e2e-475d-943f-f2a02841d174,DISK]
^C17/08/24 15:46:52 INFO fs.FileSystem: Ignoring failure to deleteOnExit for path hdfs://master.hadoop:9000/I2/20170601._COPYING_
copyFromLocal: java.nio.channels.ClosedChannelException
at org.apache.hadoop.hdfs.DataStreamer$LastExceptionInStreamer.throwException4Close(DataStreamer.java:291)
at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:147)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:105)
at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:91)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:63)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:124)
at org.apache.hadoop.fs.shell.CommandWithDestination$TargetFileSystem.writeStreamToFile(CommandWithDestination.java:485)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyStreamToTarget(CommandWithDestination.java:407)
at org.apache.hadoop.fs.shell.CommandWithDestination.copyFileToTarget(CommandWithDestination.java:342)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:277)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPath(CommandWithDestination.java:262)
at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:317)
at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:289)
at org.apache.hadoop.fs.shell.CommandWithDestination.processPathArgument(CommandWithDestination.java:257)
at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
at org.apache.hadoop.fs.shell.CommandWithDestination.processArguments(CommandWithDestination.java:228)
at org.apache.hadoop.fs.shell.CopyCommands$Put.processArguments(CopyCommands.java:286)
at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:119)
at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:315)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:378)

copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed
copyFromLocal: Filesystem closed

9. 格式化namenode
[ [email protected] hadoop]# hadoop namenode –format
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

17/08/24 15:28:29 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: user = root
STARTUP_MSG: host = master.hadoop/10.9.148.193
STARTUP_MSG: args = [–format]
STARTUP_MSG: version = 2.8.1
STARTUP_MSG: classpath = /home/softs/hadoop/etc/hadoop:/home/softs/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/softs/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/softs/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/softs/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/softs/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/softs/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/softs/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/softs/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/softs/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/softs/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/softs/hadoop/share/hadoop/common/lib/hadoop-auth-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jcip-annotations-1.0.jar:/home/softs/hadoop/share/hadoop/common/lib/json-smart-1.1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/softs/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/softs/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/softs/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/softs/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/softs/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jsch-0.1.51.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/home/softs/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/softs/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/softs/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/softs/hadoop/share/hadoop/common/lib/hadoop-annotations-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/softs/hadoop/share/hadoop/common/lib/httpclient-4.5.2.jar:/home/softs/hadoop/share/hadoop/common/lib/httpcore-4.4.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/softs/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/hadoop-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/hadoop-common-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/common/hadoop-nfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/okio-1.4.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/fst-2.24.jar:/home/softs/hadoop/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/home/softs/hadoop/share/hadoop/yarn/lib/objenesis-2.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-math-2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/softs/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1-tests.jar:/home/softs/hadoop/contrib/capacity-scheduler/*.jar:/home/softs/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 20fe5304904fc2f5a18053c389e43cd26f7a70fe; compiled by 'vinodkv' on 2017-06-02T06:14Z
STARTUP_MSG: java = 1.7.0_45
************************************************************/
17/08/24 15:28:29 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/08/24 15:28:29 INFO namenode.NameNode: createNameNode [–format]
Usage: hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReserved<k-v pairs>] ] |
[-upgradeOnly [-clusterid cid] [-renameReserved<k-v pairs>] ] |
[-rollback] |
[-rollingUpgrade <rollback|downgrade|started> ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby [-force] [-nonInteractive] [-skipSharedEditsCheck] ] |
[-recover [ -force] ] |
[-metadataVersion ]

17/08/24 15:28:30 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master.hadoop/10.9.148.193
************************************************************/

========================================================================
[ [email protected] hadoop]# hdfs namenode -format
17/08/24 15:31:09 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: user = root
STARTUP_MSG: host = master.hadoop/10.9.148.193
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.8.1
STARTUP_MSG: classpath = /home/softs/hadoop/etc/hadoop:/home/softs/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-json-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/softs/hadoop/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/softs/hadoop/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/softs/hadoop/share/hadoop/common/lib/activation-1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/home/softs/hadoop/share/hadoop/common/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/common/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/softs/hadoop/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/softs/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/home/softs/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/home/softs/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/home/softs/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/home/softs/hadoop/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/common/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/home/softs/hadoop/share/hadoop/common/lib/hadoop-auth-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/home/softs/hadoop/share/hadoop/common/lib/jcip-annotations-1.0.jar:/home/softs/hadoop/share/hadoop/common/lib/json-smart-1.1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/softs/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/home/softs/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/home/softs/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/home/softs/hadoop/share/hadoop/common/lib/zookeeper-3.4.6.jar:/home/softs/hadoop/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-framework-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-client-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/jsch-0.1.51.jar:/home/softs/hadoop/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/home/softs/hadoop/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/home/softs/hadoop/share/hadoop/common/lib/junit-4.11.jar:/home/softs/hadoop/share/hadoop/common/lib/hamcrest-core-1.3.jar:/home/softs/hadoop/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/softs/hadoop/share/hadoop/common/lib/hadoop-annotations-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/softs/hadoop/share/hadoop/common/lib/xmlenc-0.52.jar:/home/softs/hadoop/share/hadoop/common/lib/httpclient-4.5.2.jar:/home/softs/hadoop/share/hadoop/common/lib/httpcore-4.4.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-net-3.1.jar:/home/softs/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/home/softs/hadoop/share/hadoop/common/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/home/softs/hadoop/share/hadoop/common/hadoop-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/common/hadoop-common-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/common/hadoop-nfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/okio-1.4.0.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/home/softs/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1-tests.jar:/home/softs/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/fst-2.24.jar:/home/softs/hadoop/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/home/softs/hadoop/share/hadoop/yarn/lib/objenesis-2.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jetty-6.1.26.jar:/home/softs/hadoop/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-math-2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/softs/hadoop/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/activation-1.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/guice-3.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/softs/hadoop/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jettison-1.1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/softs/hadoop/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/softs/hadoop/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/home/softs/hadoop/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-api-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-client-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.1.jar:/home/softs/hadoop/share/hadoop/yarn/hadoop-yarn-registry-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/xz-1.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/asm-3.2.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/guice-3.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/javax.inject-1.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/home/softs/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar:/home/softs/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1-tests.jar:/home/softs/hadoop/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 20fe5304904fc2f5a18053c389e43cd26f7a70fe; compiled by 'vinodkv' on 2017-06-02T06:14Z
STARTUP_MSG: java = 1.7.0_45
************************************************************/
17/08/24 15:31:09 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/08/24 15:31:09 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-4b31f9f8-c01d-45e3-b334-91bd86aafefa
17/08/24 15:31:10 INFO namenode.FSEditLog: Edit logging is async:false
17/08/24 15:31:10 INFO namenode.FSNamesystem: KeyProvider: null
17/08/24 15:31:10 INFO namenode.FSNamesystem: fsLock is fair: true
17/08/24 15:31:10 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
17/08/24 15:31:10 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/08/24 15:31:10 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/08/24 15:31:10 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17/08/24 15:31:10 INFO blockmanagement.BlockManager: The block deletion will start around 2017 八月 24 15:31:10
17/08/24 15:31:10 INFO util.GSet: Computing capacity for map BlocksMap
17/08/24 15:31:10 INFO util.GSet: VM type = 64-bit
17/08/24 15:31:10 INFO util.GSet: 2.0% max memory 889 MB = 17.8 MB
17/08/24 15:31:10 INFO util.GSet: capacity = 2^21 = 2097152 entries
17/08/24 15:31:10 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
17/08/24 15:31:10 INFO blockmanagement.BlockManager: defaultReplication = 3
17/08/24 15:31:10 INFO blockmanagement.BlockManager: maxReplication = 512
17/08/24 15:31:10 INFO blockmanagement.BlockManager: minReplication = 1
17/08/24 15:31:10 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
17/08/24 15:31:10 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/08/24 15:31:10 INFO blockmanagement.BlockManager: encryptDataTransfer = false
17/08/24 15:31:10 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
17/08/24 15:31:10 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
17/08/24 15:31:10 INFO namenode.FSNamesystem: supergroup = supergroup
17/08/24 15:31:10 INFO namenode.FSNamesystem: isPermissionEnabled = false
17/08/24 15:31:10 INFO namenode.FSNamesystem: HA Enabled: false
17/08/24 15:31:10 INFO namenode.FSNamesystem: Append Enabled: true
17/08/24 15:31:10 INFO util.GSet: Computing capacity for map INodeMap
17/08/24 15:31:10 INFO util.GSet: VM type = 64-bit
17/08/24 15:31:10 INFO util.GSet: 1.0% max memory 889 MB = 8.9 MB
17/08/24 15:31:10 INFO util.GSet: capacity = 2^20 = 1048576 entries
17/08/24 15:31:10 INFO namenode.FSDirectory: ACLs enabled? false
17/08/24 15:31:10 INFO namenode.FSDirectory: XAttrs enabled? true
17/08/24 15:31:10 INFO namenode.NameNode: Caching file names occurring more than 10 times
17/08/24 15:31:10 INFO util.GSet: Computing capacity for map cachedBlocks
17/08/24 15:31:10 INFO util.GSet: VM type = 64-bit
17/08/24 15:31:10 INFO util.GSet: 0.25% max memory 889 MB = 2.2 MB
17/08/24 15:31:10 INFO util.GSet: capacity = 2^18 = 262144 entries
17/08/24 15:31:10 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/08/24 15:31:10 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/08/24 15:31:10 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000
17/08/24 15:31:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
17/08/24 15:31:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
17/08/24 15:31:10 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17/08/24 15:31:10 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/08/24 15:31:10 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/08/24 15:31:10 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/08/24 15:31:10 INFO util.GSet: VM type = 64-bit
17/08/24 15:31:10 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
17/08/24 15:31:10 INFO util.GSet: capacity = 2^15 = 32768 entries
17/08/24 15:31:11 INFO namenode.FSImage: Allocated new BlockPoolId: BP-748028963-10.9.148.193-1503559871024
17/08/24 15:31:11 INFO common.Storage: Storage directory /home/softs/hadoop/tmp/dfs/name has been successfully formatted.
17/08/24 15:31:11 INFO namenode.FSImageFormatProtobuf: Saving image file /home/softs/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/08/24 15:31:11 INFO namenode.FSImageFormatProtobuf: Image file /home/softs/hadoop/tmp/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 321 bytes saved in 0 seconds.
17/08/24 15:31:11 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/08/24 15:31:11 INFO util.ExitUtil: Exiting with status 0
17/08/24 15:31:11 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master.hadoop/10.9.148.193
************************************************************/
[ [email protected] hadoop]#

10. 启动和验证：
[ [email protected] hadoop]# start-dfs.sh
Starting namenodes on [master.hadoop]
master.hadoop: starting namenode, logging to /home/softs/hadoop/logs/hadoop-root-namenode-master.hadoop.out
10.9.148.191: starting datanode, logging to /home/softs/hadoop/logs/hadoop-root-datanode-slave2.hadoop.out
10.9.148.190: starting datanode, logging to /home/softs/hadoop/logs/hadoop-root-datanode-slave1.hadoop.out
10.9.148.192: starting datanode, logging to /home/softs/hadoop/logs/hadoop-root-datanode-slave3.hadoop.out
Starting secondary namenodes [master.hadoop]
master.hadoop: starting secondarynamenode, logging to /home/softs/hadoop/logs/hadoop-root-secondarynamenode-master.hadoop.out
[ [email protected] hadoop]# start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/softs/hadoop/logs/yarn-root-resourcemanager-master.hadoop.out
10.9.148.192: starting nodemanager, logging to /home/softs/hadoop/logs/yarn-root-nodemanager-slave3.hadoop.out
10.9.148.191: starting nodemanager, logging to /home/softs/hadoop/logs/yarn-root-nodemanager-slave2.hadoop.out

10.9.148.190: starting nodemanager, logging to /home/softs/hadoop/logs/yarn-root-nodemanager-slave1.hadoop.out

在这里，我直接是start-all.sh启动namenode和yarn和dfs的，再加下面的jobhistoryserver

[ [email protected] hadoop]# mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to /home/softs/hadoop/logs/mapred-root-historyserver-master.hadoop.out
[ [email protected] hadoop]# jps
10342 NameNode
10665 SecondaryNameNode
10921 ResourceManager
11259 Jps
11213 JobHistoryServer
[ [email protected] hadoop]# jps
10342 NameNode
10665 SecondaryNameNode
10921 ResourceManager
11213 JobHistoryServer
11279 Jps
[ [email protected] hadoop]# hdfs dfsadmin -report
Configured Capacity: 458424913920 (426.94 GB)
Present Capacity: 428918013952 (399.46 GB)
DFS Remaining: 428917919744 (399.46 GB)
DFS Used: 94208 (92 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0

-------------------------------------------------
Live datanodes (3):

Name: 10.9.148.190:50010 (slave1.hadoop)
Hostname: slave1.hadoop
Decommission Status : Normal
Configured Capacity: 152682475520 (142.20 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 8190533632 (7.63 GB)
DFS Remaining: 142939611136 (133.12 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.62%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Aug 24 15:55:18 CST 2017

Name: 10.9.148.191:50010 (slave2.hadoop)
Hostname: slave2.hadoop
Decommission Status : Normal
Configured Capacity: 152617463808 (142.14 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 8327135232 (7.76 GB)
DFS Remaining: 142738649088 (132.94 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.53%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Aug 24 15:55:18 CST 2017

Name: 10.9.148.192:50010 (slave3.hadoop)
Hostname: slave3.hadoop
Decommission Status : Normal
Configured Capacity: 153124974592 (142.61 GB)
DFS Used: 36864 (36 KB)
Non DFS Used: 8328491008 (7.76 GB)
DFS Remaining: 143239659520 (133.40 GB)
DFS Used%: 0.00%
DFS Remaining%: 93.54%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Thu Aug 24 15:55:18 CST 2017

[ [email protected] hadoop]#

11. 使用
放入HDFS文件
[ [email protected] hadoop]# hadoop fs -copyFromLocal /home/201706/* /I2
查看HDFS放入的文件
[ [email protected] hadoop]# hadoop fs -ls /I2
Found 30 items
-rw-r--r-- 3 root supergroup 275137778 2017-08-24 15:57 /I2/20170601
-rw-r--r-- 3 root supergroup 287210022 2017-08-24 15:57 /I2/20170602
-rw-r--r-- 3 root supergroup 326314540 2017-08-24 15:58 /I2/20170603
-rw-r--r-- 3 root supergroup 327799830 2017-08-24 15:58 /I2/20170604
-rw-r--r-- 3 root supergroup 308526744 2017-08-24 15:59 /I2/20170605
-rw-r--r-- 3 root supergroup 349079308 2017-08-24 15:59 /I2/20170606
-rw-r--r-- 3 root supergroup 417899378 2017-08-24 16:00 /I2/20170607
-rw-r--r-- 3 root supergroup 442854968 2017-08-24 16:01 /I2/20170608
-rw-r--r-- 3 root supergroup 455375694 2017-08-24 16:01 /I2/20170609
-rw-r--r-- 3 root supergroup 486294396 2017-08-24 16:02 /I2/20170610
-rw-r--r-- 3 root supergroup 429505122 2017-08-24 16:03 /I2/20170611
-rw-r--r-- 3 root supergroup 427015202 2017-08-24 16:03 /I2/20170612
-rw-r--r-- 3 root supergroup 469780264 2017-08-24 16:04 /I2/20170613
-rw-r--r-- 3 root supergroup 510235430 2017-08-24 16:05 /I2/20170614
-rw-r--r-- 3 root supergroup 497186788 2017-08-24 16:06 /I2/20170615
-rw-r--r-- 3 root supergroup 558273348 2017-08-24 16:07 /I2/20170616
-rw-r--r-- 3 root supergroup 585348962 2017-08-24 16:07 /I2/20170617
-rw-r--r-- 3 root supergroup 573453072 2017-08-24 16:08 /I2/20170618
-rw-r--r-- 3 root supergroup 571823300 2017-08-24 16:09 /I2/20170619
-rw-r--r-- 3 root supergroup 686884860 2017-08-24 16:10 /I2/20170620
-rw-r--r-- 3 root supergroup 615866930 2017-08-24 16:11 /I2/20170621
-rw-r--r-- 3 root supergroup 738148784 2017-08-24 16:12 /I2/20170622
-rw-r--r-- 3 root supergroup 826302986 2017-08-24 16:14 /I2/20170623
-rw-r--r-- 3 root supergroup 841656010 2017-08-24 16:15 /I2/20170624
-rw-r--r-- 3 root supergroup 849475152 2017-08-24 16:16 /I2/20170625
-rw-r--r-- 3 root supergroup 860776370 2017-08-24 16:17 /I2/20170626
-rw-r--r-- 3 root supergroup 905409004 2017-08-24 16:19 /I2/20170627
-rw-r--r-- 3 root supergroup 978153598 2017-08-24 16:20 /I2/20170628
-rw-r--r-- 3 root supergroup 992471718 2017-08-24 16:22 /I2/20170629
-rw-r--r-- 3 root supergroup 1078413818 2017-08-24 16:23 /I2/20170630
[ [email protected] hadoop]#
运行jar包
[ [email protected] 6mouth]# ls
hdfs-mapreduce-jar-with-dependencies.jar
[ [email protected] 6mouth]# pwd
/home/DATA/6mouth
[ [email protected] 6mouth]# hadoop jar hdfs-mapreduce-jar-with-dependencies.jar /I2/20170601,/I2/20170602,/I2/20170603,/I2/20170604,/I2/20170605,/I2/20170606,/I2/20170607,/I2/20170608,/I2/20170609,/I2/20170610,/I2/20170611,/I2/20170612,/I2/20170613,/I2/20170614,/I2/20170615,/I2/20170616,/I2/20170617,/I2/20170618,/I2/20170619,/I2/20170620,/I2/20170621,/I2/20170622,/I2/20170623,/I2/20170624,/I2/20170625,/I2/20170626,/I2/20170627,/I2/20170628,/I2/20170629,/I2/20170630 /201706/
17/08/24 16:29:33 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.9.148.193:8032
17/08/24 16:29:34 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/08/24 16:29:38 INFO input.FileInputFormat: Total input files to process : 30
17/08/24 16:29:39 INFO mapreduce.JobSubmitter: number of splits:149
17/08/24 16:29:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1503561276940_0001
17/08/24 16:29:39 INFO impl.YarnClientImpl: Submitted application application_1503561276940_0001
17/08/24 16:29:39 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1503561276940_0001/
17/08/24 16:29:39 INFO mapreduce.Job: Running job: job_1503561276940_0001
17/08/24 16:29:53 INFO mapreduce.Job: Job job_1503561276940_0001 running in uber mode : false
17/08/24 16:29:53 INFO mapreduce.Job: map 0% reduce 0%
17/08/24 16:39:45 INFO mapreduce.Job: map 3% reduce 0%
17/08/24 16:39:45 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000004_0, Status : FAILED
Exception from container-launch.
Container id: container_1503561276940_0001_01_000006
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65

17/08/24 16:39:50 INFO mapreduce.Job: map 5% reduce 0%
17/08/24 16:39:50 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000015_0, Status : FAILED
Exception from container-launch.
Container id: container_1503561276940_0001_01_000017
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65

17/08/24 16:46:27 INFO mapreduce.Job: map 7% reduce 0%
17/08/24 16:46:30 INFO mapreduce.Job: map 8% reduce 0%
17/08/24 16:46:34 INFO mapreduce.Job: map 9% reduce 0%
17/08/24 16:46:40 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000008_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000008_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000010
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4146 as user root for container container_1503561276940_0001_01_000010, result=failed

17/08/24 16:46:42 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000012_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000012_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4147 as user root for container container_1503561276940_0001_01_000014, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:42 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000009_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000009_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4141 as user root for container container_1503561276940_0001_01_000011, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:45 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000017_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000017_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000019
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4023 as user root for container container_1503561276940_0001_01_000019, result=failed

17/08/24 16:46:45 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000014_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000014_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4025 as user root for container container_1503561276940_0001_01_000016, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:46 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000007_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000007_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4145 as user root for container container_1503561276940_0001_01_000009, result=failed

17/08/24 16:46:47 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000006_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000006_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000008
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4149 as user root for container container_1503561276940_0001_01_000008, result=failed

17/08/24 16:46:47 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000011_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000011_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4142 as user root for container container_1503561276940_0001_01_000013, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:47 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000018_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000018_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000020
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4022 as user root for container container_1503561276940_0001_01_000020, result=success
Container exited with a non-zero exit code 65

17/08/24 16:46:48 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000020_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000020_0 Timed out after 600 secs
Container killed by the ApplicationMaster.
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4021 as user root for container container_1503561276940_0001_01_000022, result=success
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:48 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000023_0, Status : FAILED
17/08/24 16:46:49 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000016_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000016_0 Timed out after 600 secs
Sent signal OUTPUT_THREAD_DUMP (SIGQUIT) to pid 4024 as user root for container container_1503561276940_0001_01_000018, result=success
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/08/24 16:46:49 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000010_0, Status : FAILED
AttemptID:attempt_1503561276940_0001_m_000010_0 Timed out after 600 secs
Exception from container-launch.
Container id: container_1503561276940_0001_01_000012
Exit code: 65
Stack trace: ExitCodeException exitCode=65:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:972)
at org.apache.hadoop.util.Shell.run(Shell.java:869)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1170)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:236)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:305)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:84)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Container exited with a non-zero exit code 65

17/08/24 16:47:24 INFO mapreduce.Job: map 10% reduce 0%
17/08/24 16:47:38 INFO mapreduce.Job: map 10% reduce 3%
17/08/24 16:47:45 INFO mapreduce.Job: map 11% reduce 3%
17/08/24 16:48:30 INFO mapreduce.Job: map 11% reduce 4%
17/08/24 16:49:57 INFO mapreduce.Job: map 12% reduce 4%
17/08/24 16:51:31 INFO mapreduce.Job: map 13% reduce 4%
17/08/24 16:51:46 INFO mapreduce.Job: map 14% reduce 4%
17/08/24 16:51:49 INFO mapreduce.Job: map 16% reduce 4%
17/08/24 16:51:51 INFO mapreduce.Job: map 18% reduce 4%
17/08/24 16:51:55 INFO mapreduce.Job: map 19% reduce 4%
17/08/24 16:51:56 INFO mapreduce.Job: map 20% reduce 4%
17/08/24 16:52:38 INFO mapreduce.Job: map 21% reduce 4%
17/08/24 16:53:05 INFO mapreduce.Job: map 22% reduce 4%
17/08/24 16:55:22 INFO mapreduce.Job: map 23% reduce 4%
17/08/24 16:56:13 INFO mapreduce.Job: map 26% reduce 7%
17/08/24 16:56:21 INFO mapreduce.Job: map 25% reduce 7%
17/08/24 16:56:23 INFO mapreduce.Job: map 28% reduce 7%
17/08/24 16:56:26 INFO mapreduce.Job: map 31% reduce 7%
17/08/24 16:56:32 INFO mapreduce.Job: map 32% reduce 7%
17/08/24 16:56:39 INFO mapreduce.Job: map 33% reduce 7%
17/08/24 16:56:50 INFO mapreduce.Job: map 33% reduce 11%
17/08/24 17:01:09 INFO mapreduce.Job: map 36% reduce 11%
17/08/24 17:01:20 INFO mapreduce.Job: map 37% reduce 11%
17/08/24 17:01:46 INFO mapreduce.Job: map 38% reduce 11%
17/08/24 17:01:47 INFO mapreduce.Job: map 39% reduce 12%
17/08/24 17:01:55 INFO mapreduce.Job: map 40% reduce 12%
17/08/24 17:02:10 INFO mapreduce.Job: map 40% reduce 13%
17/08/24 17:02:24 INFO mapreduce.Job: map 41% reduce 13%
17/08/24 17:03:06 INFO mapreduce.Job: map 42% reduce 13%
17/08/24 17:05:31 INFO mapreduce.Job: map 44% reduce 13%
17/08/24 17:06:36 INFO mapreduce.Job: map 47% reduce 13%
17/08/24 17:06:52 INFO mapreduce.Job: map 48% reduce 13%
17/08/24 17:06:54 INFO mapreduce.Job: map 49% reduce 14%
17/08/24 17:07:00 INFO mapreduce.Job: map 50% reduce 14%
17/08/24 17:07:02 INFO mapreduce.Job: map 51% reduce 14%
17/08/24 17:07:04 INFO mapreduce.Job: map 52% reduce 14%
17/08/24 17:07:13 INFO mapreduce.Job: map 52% reduce 16%
17/08/24 17:07:19 INFO mapreduce.Job: map 52% reduce 17%
17/08/24 17:08:56 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000092_0, Status : FAILED
17/08/24 17:08:57 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000089_0, Status : FAILED
17/08/24 17:09:00 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000091_0, Status : FAILED
17/08/24 17:09:00 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000088_0, Status : FAILED
17/08/24 17:09:02 INFO mapreduce.Job: map 55% reduce 17%
17/08/24 17:09:47 INFO mapreduce.Job: map 56% reduce 17%
17/08/24 17:09:50 INFO mapreduce.Job: map 56% reduce 18%
17/08/24 17:10:58 INFO mapreduce.Job: map 57% reduce 18%
17/08/24 17:11:16 INFO mapreduce.Job: map 58% reduce 18%
17/08/24 17:12:13 INFO mapreduce.Job: map 59% reduce 18%
17/08/24 17:13:05 INFO mapreduce.Job: map 60% reduce 18%
17/08/24 17:13:12 INFO mapreduce.Job: map 61% reduce 18%
17/08/24 17:13:15 INFO mapreduce.Job: map 62% reduce 18%
17/08/24 17:13:17 INFO mapreduce.Job: map 63% reduce 18%
17/08/24 17:13:25 INFO mapreduce.Job: map 64% reduce 18%
17/08/24 17:13:31 INFO mapreduce.Job: map 64% reduce 20%
17/08/24 17:13:36 INFO mapreduce.Job: map 65% reduce 20%
17/08/24 17:13:45 INFO mapreduce.Job: map 66% reduce 20%
17/08/24 17:13:56 INFO mapreduce.Job: map 67% reduce 20%
17/08/24 17:14:15 INFO mapreduce.Job: map 68% reduce 20%
17/08/24 17:14:23 INFO mapreduce.Job: map 68% reduce 22%
17/08/24 17:16:38 INFO mapreduce.Job: map 69% reduce 22%
17/08/24 17:16:43 INFO mapreduce.Job: map 72% reduce 22%
17/08/24 17:16:48 INFO mapreduce.Job: map 73% reduce 22%
17/08/24 17:16:57 INFO mapreduce.Job: map 73% reduce 23%
17/08/24 17:17:03 INFO mapreduce.Job: map 73% reduce 24%
17/08/24 17:17:07 INFO mapreduce.Job: map 74% reduce 24%
17/08/24 17:17:28 INFO mapreduce.Job: map 75% reduce 24%
17/08/24 17:18:13 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000122_0, Status : FAILED
17/08/24 17:18:13 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000119_0, Status : FAILED
17/08/24 17:18:20 INFO mapreduce.Job: map 76% reduce 24%
17/08/24 17:18:22 INFO mapreduce.Job: map 77% reduce 24%
17/08/24 17:18:24 INFO mapreduce.Job: map 79% reduce 24%
17/08/24 17:18:26 INFO mapreduce.Job: map 79% reduce 25%
17/08/24 17:18:32 INFO mapreduce.Job: map 80% reduce 25%
17/08/24 17:18:42 INFO mapreduce.Job: map 81% reduce 25%
17/08/24 17:18:46 INFO mapreduce.Job: map 81% reduce 27%
17/08/24 17:18:54 INFO mapreduce.Job: map 82% reduce 27%
17/08/24 17:18:55 INFO mapreduce.Job: map 84% reduce 27%
17/08/24 17:19:01 INFO mapreduce.Job: map 85% reduce 27%
17/08/24 17:19:18 INFO mapreduce.Job: map 85% reduce 28%
17/08/24 17:21:21 INFO mapreduce.Job: Task Id : attempt_1503561276940_0001_m_000122_1, Status : FAILED
17/08/24 17:21:33 INFO mapreduce.Job: map 88% reduce 28%
17/08/24 17:21:36 INFO mapreduce.Job: map 89% reduce 28%
17/08/24 17:21:41 INFO mapreduce.Job: map 92% reduce 28%
17/08/24 17:21:51 INFO mapreduce.Job: map 92% reduce 29%
17/08/24 17:21:55 INFO mapreduce.Job: map 93% reduce 29%
17/08/24 17:22:01 INFO mapreduce.Job: map 93% reduce 30%
17/08/24 17:22:04 INFO mapreduce.Job: map 94% reduce 30%
17/08/24 17:22:10 INFO mapreduce.Job: map 96% reduce 30%
17/08/24 17:22:12 INFO mapreduce.Job: map 97% reduce 30%
17/08/24 17:22:15 INFO mapreduce.Job: map 97% reduce 32%
17/08/24 17:22:19 INFO mapreduce.Job: map 98% reduce 32%
17/08/24 17:22:20 INFO mapreduce.Job: map 99% reduce 32%
17/08/24 17:22:21 INFO mapreduce.Job: map 99% reduce 33%
17/08/24 17:22:28 INFO mapreduce.Job: map 100% reduce 33%
17/08/24 17:22:41 INFO mapreduce.Job: map 100% reduce 85%
17/08/24 17:22:44 INFO mapreduce.Job: map 100% reduce 100%
17/08/24 17:22:51 INFO mapreduce.Job: Job job_1503561276940_0001 completed successfully
17/08/24 17:22:53 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
17/08/24 17:22:53 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
17/08/24 17:22:53 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
Exception in thread "main" java.io.IOException: java.io.IOException: Unknown Job job_1503561276940_0001
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:232)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:344)
at org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:381)
at org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:606)
at org.apache.hadoop.mapreduce.Job$8.run(Job.java:812)
at org.apache.hadoop.mapreduce.Job$8.run(Job.java:809)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:809)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1434)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1362)
at com.mapreduce.Driver.runJob(Driver.java:35)
at com.mapreduce.Driver.main(Driver.java:19)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: java.io.IOException: Unknown Job job_1503561276940_0001
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:232)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:110)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.unwrapAndThrowException(MRClientProtocolPBClientImpl.java:291)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getCounters(MRClientProtocolPBClientImpl.java:168)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:325)
... 18 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unknown Job job_1503561276940_0001
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.verifyAndGetJob(HistoryClientService.java:218)
at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:232)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:845)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:788)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2455)

at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1481)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1337)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy21.getCounters(Unknown Source)
at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getCounters(MRClientProtocolPBClientImpl.java:166)
... 23 more
[ [email protected] 6mouth]#
失败太多
运行jar失败原因：
出现以上错误是因为datanode的服务线程连接数都被占用，导致Spark等待超时。可以通过以下方法解决：

根据datanode的资源情况适当增加的服务线程数：在hdfs-default.xml配置文件里面新增或修改dfs.datanode.handler.count，默认是10
增加客户端的超时时间dfs.client.socket-timeout，默认是60000ms，注意，旧版本的Hadoop用的是dfs.socket.timeout，可参考http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

例如以下Java代码设置超时时间为300000ms=300s=5分钟：
运行jar包失败太多解决方法，修改：hdfs-site.xml
该文件与HDFS子项目密切相关，其参数对集群性能调整具有很大影响。
dfs.name.dir 指定name镜像文件存放目录，如不指定则默认为core-site中配置的tmp目录。
dfs.data.dir 真正的datanode数据保存路径，可以写多块硬盘，中间以逗号分隔。
dfs.replication hdfs数据块的复制份数，默认3，理论上份数越多跑数速度越快，但是需要的存储空间也更多。
dfs.permissions 是否需要角色权限验证，上传文件时会用到。一般设置false，设置为true有时候会遇到数据因为权限访问不了。
dfs.block.size 每个文件块的大小，默认是64M，对于大型文件可以设置为128M。
dfs.namenode.handler.count NameNode 节点上面为处理datanode节点来来气的远程调用的服务线程数量。
dfs.datanode.max.xcievers 相当于Linux下的打开文件最大数量，文档中无此参数，当出现DataXceiver报错的时候，需要调大。默认256
dfs.datanode.handler.count datanode节点上为处理datanode节点的远程调用开启的服务线程数量，默认为3。当有很多HDFS客户端时可以设置更大。
<property>
<name>dfs.datanode.handler.count</name>
<value>500</value>
</property>
<property>
<name>dfs.client.socket-timeout</name>
<value>300000</value>
</property>
修改后从新放入数据出现INFO提示：
[ [email protected] hadoop]# hadoop fs -copyFromLocal /home/201706/* /I2
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81911ms (threshold=30000ms); ack: seqno: 1 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 82033ms (threshold=30000ms); ack: seqno: 2 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 82032ms (threshold=30000ms); ack: seqno: 3 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 82033ms (threshold=30000ms); ack: seqno: 4 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81914ms (threshold=30000ms); ack: seqno: 5 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81832ms (threshold=30000ms); ack: seqno: 6 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81832ms (threshold=30000ms); ack: seqno: 7 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81751ms (threshold=30000ms); ack: seqno: 8 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81751ms (threshold=30000ms); ack: seqno: 9 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81741ms (threshold=30000ms); ack: seqno: 10 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81741ms (threshold=30000ms); ack: seqno: 11 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81731ms (threshold=30000ms); ack: seqno: 12 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81730ms (threshold=30000ms); ack: seqno: 13 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81720ms (threshold=30000ms); ack: seqno: 14 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81719ms (threshold=30000ms); ack: seqno: 15 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81710ms (threshold=30000ms); ack: seqno: 16 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81714ms (threshold=30000ms); ack: seqno: 17 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81703ms (threshold=30000ms); ack: seqno: 18 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81702ms (threshold=30000ms); ack: seqno: 19 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81692ms (threshold=30000ms); ack: seqno: 20 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81692ms (threshold=30000ms); ack: seqno: 21 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81681ms (threshold=30000ms); ack: seqno: 22 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81681ms (threshold=30000ms); ack: seqno: 23 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81671ms (threshold=30000ms); ack: seqno: 24 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81671ms (threshold=30000ms); ack: seqno: 25 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81660ms (threshold=30000ms); ack: seqno: 26 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81660ms (threshold=30000ms); ack: seqno: 27 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81649ms (threshold=30000ms); ack: seqno: 28 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81650ms (threshold=30000ms); ack: seqno: 29 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81639ms (threshold=30000ms); ack: seqno: 30 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81639ms (threshold=30000ms); ack: seqno: 31 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81628ms (threshold=30000ms); ack: seqno: 32 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81628ms (threshold=30000ms); ack: seqno: 33 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81615ms (threshold=30000ms); ack: seqno: 34 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81615ms (threshold=30000ms); ack: seqno: 35 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81606ms (threshold=30000ms); ack: seqno: 36 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81607ms (threshold=30000ms); ack: seqno: 37 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81596ms (threshold=30000ms); ack: seqno: 38 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81596ms (threshold=30000ms); ack: seqno: 39 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81583ms (threshold=30000ms); ack: seqno: 40 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81584ms (threshold=30000ms); ack: seqno: 41 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81575ms (threshold=30000ms); ack: seqno: 42 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81575ms (threshold=30000ms); ack: seqno: 43 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81564ms (threshold=30000ms); ack: seqno: 44 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81565ms (threshold=30000ms); ack: seqno: 45 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81553ms (threshold=30000ms); ack: seqno: 46 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81553ms (threshold=30000ms); ack: seqno: 47 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81538ms (threshold=30000ms); ack: seqno: 48 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81539ms (threshold=30000ms); ack: seqno: 49 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81539ms (threshold=30000ms); ack: seqno: 50 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81531ms (threshold=30000ms); ack: seqno: 51 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81531ms (threshold=30000ms); ack: seqno: 52 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81520ms (threshold=30000ms); ack: seqno: 53 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81519ms (threshold=30000ms); ack: seqno: 54 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81510ms (threshold=30000ms); ack: seqno: 55 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81509ms (threshold=30000ms); ack: seqno: 56 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81499ms (threshold=30000ms); ack: seqno: 57 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81499ms (threshold=30000ms); ack: seqno: 58 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81488ms (threshold=30000ms); ack: seqno: 59 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81488ms (threshold=30000ms); ack: seqno: 60 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81477ms (threshold=30000ms); ack: seqno: 61 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81478ms (threshold=30000ms); ack: seqno: 62 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81462ms (threshold=30000ms); ack: seqno: 63 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81462ms (threshold=30000ms); ack: seqno: 64 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81449ms (threshold=30000ms); ack: seqno: 65 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81450ms (threshold=30000ms); ack: seqno: 66 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81439ms (threshold=30000ms); ack: seqno: 67 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81439ms (threshold=30000ms); ack: seqno: 68 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81439ms (threshold=30000ms); ack: seqno: 69 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81422ms (threshold=30000ms); ack: seqno: 70 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81423ms (threshold=30000ms); ack: seqno: 71 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81423ms (threshold=30000ms); ack: seqno: 72 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81411ms (threshold=30000ms); ack: seqno: 73 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81399ms (threshold=30000ms); ack: seqno: 74 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81400ms (threshold=30000ms); ack: seqno: 75 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81400ms (threshold=30000ms); ack: seqno: 76 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81391ms (threshold=30000ms); ack: seqno: 77 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81391ms (threshold=30000ms); ack: seqno: 78 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81380ms (threshold=30000ms); ack: seqno: 79 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 81380ms (threshold=30000ms); ack: seqno: 80 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
17/08/25 09:25:02 INFO hdfs.DataStreamer: Slow ReadProcessor read fields for block BP-748028963-10.9.148.193-1503559871024:blk_1073742043_1219 took 80425ms (threshold=30000ms); ack: seqno: 81 reply: SUCCESS downstreamAckTimeNanos: 0 flag: 0, targets: [DatanodeInfoWithStorage[10.9.148.192:50010,DS-fc570b61-7dae-4958-bdeb-05c0a8534ebd,DISK]]
[ [email protected] hadoop]#
修改后再次运行jar包代码：出错一大堆。解决（有点投机的方法）：重新删掉数据、tmp、log，重新格式化，然后相当于重新启动集群。重新放入数据，运行jar包代码。