CentOS6.4上搭建hadoop-2.4.0集群

时间 2019-12-08

标签 centos6.4 centos 搭建 hadoop 2.4.0 集群栏目 CentOS 繁體版

原文原文链接

公司Commerce Cloud平台上提供申请主机的服务。昨天试了下，申请了3台机器，搭了个hadoop环境。如下是机器的一些配置：html

emi-centos-6.4-x86_64
medium | 6GB 内存| 2 虚拟内核 | 30.0GB 盘java

3个机器的主机和ip规划以下：node

IP地址主机名用途linux

192.168.0.101 hd1     namenode
192.168.0.102 hd2     datanode
192.168.0.103 hd3     datanodeweb

1、系统设置 apache

（全部步骤都须要在全部节点执行）windows

1. 修改主机名及ip地址解析centos

1) 修改主机名oracle

[root@hd1 toughhou]# hostname hd1

[root@hd1 toughhou]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hd1

2) 增长ip和主机映射app

[root@hd1 toughhou]# vi /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.101 hd1
192.168.0.102 hd2
192.168.0.103 hd3

3) 验证是否成功

[toughhou@hd1 ~]$ ping hd2
PING hd2 (192.168.0.102) 56(84) bytes of data.
64 bytes from hd2 (192.168.0.102): icmp_seq=1 ttl=63 time=2.55 ms

[toughhou@hd1 ~]$ ping hd3
PING hd3 (192.168.0.103) 56(84) bytes of data.
64 bytes from hd3 (192.168.0.103): icmp_seq=1 ttl=63 time=2.48 ms

能ping通说明已经OK。

2. 关闭防火墙
[root@hd1 toughhou]# chkconfig iptables off

3. SSH免密码登录

1) 生成密钥与公钥
登录到hd1，把生成的id_rsa.pub（公钥）内容cat到authorized_keys文件中。同时登录到hd2, hd3，生成id_rsa.pub，并把hd2, hd3各自的id_rsa.pub的内容copy到hd1中的authorzied_keys中。最后从hd1中scp到hd2, hd3的.ssh目录中。

[toughhou@hd1 ~]$ ssh-keygen -t rsa
[toughhou@hd1 ~]$ cat id_rsa.pub >> authorized_keys

[toughhou@hd2 ~]$ ssh-keygen -t rsa
[toughhou@hd2 ~]$ cat id_rsa.pub >> authorized_keys

[toughhou@hd3 ~]$ ssh-keygen -t rsa
[toughhou@hd3 ~]$ cat id_rsa.pub >> authorized_keys

2) scp authorized_keys到hd2, hd3

[toughhou@hd1 ~]$ scp authorized_keys 192.168.0.102:/home/toughhou/.ssh/
[toughhou@hd1 ~]$ scp authorized_keys 192.168.0.103:/home/toughhou/.ssh/

3) 验证ssh登录是不是免密码

（第一次须要密码，若配置正确的话以后就不用密码了。）

[toughhou@hd1 ~]$ ssh 192.168.0.102
[toughhou@hd2 ~]$

[toughhou@hd1 ~]$ ssh 192.168.0.103
[toughhou@hd3 ~]$

关于SSH免密码登录，也能够参考文章 “SSH时不需输入密码”，它更具体地说了关于SSH设置。

2、安装jdk、hadoop及设置环境变量

1. 下载jdk、hadoop安装包
download.oracle.com/otn-pub/java/jdk/7u65-b17/jdk-7u65-linux-x64.tar.gz
http://mirrors.cnnic.cn/apache/hadoop/common/hadoop-2.4.0/hadoop-2.4.0.tar.gz

2. 解压

[toughhou@hd1 software]$ tar zxvf jdk-7u65-linux-x64.gz
[toughhou@hd1 software]$ tar zxvf hadoop-2.4.0.tar.gz

[root@hd1 software]# mv hadoop-2.4.0 /opt/hadoop-2.4.0
[root@hd1 software]# mv jdk1.7.0_65    /opt/jdk1.7.0

3. 设置Java环境变量

以root用户登录编辑/etc/profile，加入如下内容：

[root@hd1 software]# vi /etc/profile

#java
export JAVA_HOME=/opt/jdk1.7.0
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=./:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

#hadoop
export HADOOP_HOME=/opt/hadoop-2.4.0
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native

4. 验证环境变量

[toughhou@hd1 hadoop]$ java -version

[toughhou@hd1 hadoop]$ hadoop
Usage: hadoop [--config confdir] COMMAND

3、hadoop集群设置

1. 修改hadoop配置文件

[toughhou@hd1 hadoop]$ cd /opt/hadoop-2.4.0/etc/hadoop

1) hadoop-env.sh、yarn-env.sh 设置JAVA_HOME环境变量

最开始觉得已经在/etc/profile设置了JAVA_HOME，因此在hadoop-env.sh和yarn-env.sh中已经能成功获取到JAVA_HOME，因此就不用再设置了。最终发现这在hadoop-2.4.0中行不通，start-all.sh的时候出错了（hd1: Error: JAVA_HOME is not set and could not be found.）。

找到里面的JAVA_HOME,修改成实际路径

2) slaves
这个文件配置全部datanode节点，以便namenode搜索

[toughhou@hd1 hadoop]$ vi slaves 
hd2
hd3

3) core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hd1:9000</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoop/temp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>hd1</value>
    </property>
    <property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
    </property>
</configuration>

View Code

4) hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>/hadoop/name</value>
    <final>true</final>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>/hadoop/data</value>
    <final>true</final>
    </property>
    <property>
        <name>dfs.replication</name>
    <value>2</value>
    </property>
    <property>
        <name>dfs.permissions</name>
        <value>false</value>
    </property>
</configuration>

View Code

5) mapred-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hd1:9000</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131072</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/hadoop/temp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
        <name>hadoop.proxyuser.root.hosts</name>
        <value>hd1</value>
    </property>
    <property>
        <name>hadoop.proxyuser.root.groups</name>
        <value>*</value>
    </property>
</configuration>

View Code

6) yarn-site.xml

<configuration>
<property>
<name>yarn.resourcemanager.address</name>
<value>hd1:18040</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>hd1:18030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>hd1:18025</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>hd1:18041</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>hd1:8088</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/hadoop/mynode/my</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/hadoop/mynode/logs</value>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>10800</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/logs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>-1</value>
</property>
<property>
<name>yarn.log-aggregation.retain-check-interval-seconds</name>
<value>-1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

View Code

2. 把如下文件复制到其它节点

[root@hd1 toughhou]# scp -R /opt/hadoop-2.4.0/ hd2:/opt/
[root@hd1 toughhou]# scp -R /opt/hadoop-2.4.0/ hd3:/opt/

[root@hd1 toughhou]# scp -R /opt/jdk1.7.0/ hd2:/opt/
[root@hd1 toughhou]# scp -R /opt/jdk1.7.0/ hd3:/opt/

[root@hd1 toughhou]# scp /etc/profile hd2:/etc/profile
[root@hd1 toughhou]# scp /etc/profile hd3:/etc/profile

[root@hd1 toughhou]# scp /etc/hosts hd2:/etc/hosts
[root@hd1 toughhou]# scp /etc/hosts hd3:/etc/hosts

配置完成以后须要重启电脑

3. namenode初始化

只须要第一次的时候初始化，以后就不须要了

[toughhou@hd1 bin]$ hdfs namenode -format

若是“Exiting with status 0”，就说明OK。
14/07/23 03:26:33 INFO util.ExitUtil: Exiting with status 0

4. 启动集群

[toughhou@hd1 sbin]$ cd /opt/hadoop-2.4.0/sbin

[toughhou@hd1 sbin]$ ./start-all.sh 

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [hd1]
hd1: namenode running as process 12580. Stop it first.
hd2: starting datanode, logging to /opt/hadoop-2.4.0/logs/hadoop-toughhou-datanode-hd2.out
hd3: starting datanode, logging to /opt/hadoop-2.4.0/logs/hadoop-toughhou-datanode-hd3.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 12750. Stop it first.
starting yarn daemons
resourcemanager running as process 11900. Stop it first.
hd3: starting nodemanager, logging to /opt/hadoop-2.4.0/logs/yarn-toughhou-nodemanager-hd3.out
hd2: starting nodemanager, logging to /opt/hadoop-2.4.0/logs/yarn-toughhou-nodemanager-hd2.out

5. 查看各节点的状态

[toughhou@hd1 sbin]$ jps
16358 NameNode
16535 SecondaryNameNode
16942 Jps
16683 ResourceManage

[toughhou@hd2 ~]$ jps
2253 NodeManager
2369 Jps
2152 DataNode

[toughhou@hd3 ~]$ jps
2064 NodeManager
2178 Jps
1963 DataNode

以上说明都OK。

6. windows添加快捷访问

为了方便访问，咱们也能够编辑 %systemroot%\system32\drivers\etc\hosts 文件，加入如下的 ip和主机映射

192.168.0.101 hd1
192.168.0.102 hd2
192.168.0.103 hd3

这样，咱们在本身机器上也能够经过 http://hd2:8042/node 方式访问节点，而不必用 http://192.168.0.102:8042/node。

7. wordcount 测试

为了更进一步验证hadoop环境，咱们能够运行hadoop自带的例子。

wordcount是hadoop最经典的mapreduce例子。咱们进入到相应目录运行自带的jar包，来测试hadoop环境是否OK。

具体步骤：

1) hdfs上建立目录

[toughhou@hd1 ~]$ hadoop fs -mkdir /in/wordcount
[toughhou@hd1 ~]$ hadoop fs -mkdir /out/

2) 上传文件到hdfs

[toughhou@hd1 ~]$ cat in1.txt
Hello World , Hello China, Hello Shanghai
I love China
How are you

[toughhou@hd1 ~]$ hadoop fs -put in1.txt /in/wordcount

3) 运行wordcount

[toughhou@hd1 ~]$ cd /opt/hadoop-2.4.0/share/hadoop/mapreduce/

[toughhou@hd2 mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.4.0.jar wordcount /in/wordcount /out/out1

14/07/23 10:42:36 INFO client.RMProxy: Connecting to ResourceManager at hd1/192.168.0.101:18040
14/07/23 10:42:38 INFO input.FileInputFormat: Total input paths to process : 2
14/07/23 10:42:38 INFO mapreduce.JobSubmitter: number of splits:2
14/07/23 10:42:38 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406105556378_0003
14/07/23 10:42:38 INFO impl.YarnClientImpl: Submitted application application_1406105556378_0003
14/07/23 10:42:38 INFO mapreduce.Job: The url to track the job: http://hd1:8088/proxy/application_1406105556378_0003/
14/07/23 10:42:38 INFO mapreduce.Job: Running job: job_1406105556378_0003
14/07/23 10:42:46 INFO mapreduce.Job: Job job_1406105556378_0003 running in uber mode : false
14/07/23 10:42:46 INFO mapreduce.Job: map 0% reduce 0%
14/07/23 10:42:55 INFO mapreduce.Job: map 100% reduce 0%
14/07/23 10:43:01 INFO mapreduce.Job: map 100% reduce 100%

4) 查看运行结果

[toughhou@hd2 mapreduce]$ hadoop fs -cat /out/out4/part-r-00000
, 1
China 1
China, 1
Hello 3
How 1
I 1
Shanghai 1
World 1
are 1
love 1
you 1

到此，所有结束。整个hadoop-2.4.0集群搭建过程所有结束。