[转]hadoop-2.5安装与配置

安装以前准备4台机器：bluejoe0,bluejoe4,bluejoe5,bluejoe9

bluejoe0做为master，bluejoe4,5,9做为slave

bluejoe0做为namenode

bluejoe9为secondary namenode

bluejoe4,5,9做为datanode

安装hadoop

首先在bluejoe0机器上下载Hadoop：

wget http://mirrors.cnnic.cn/apache/hadoop/common/stable2/hadoop-2.5.2.tar.gz

保存至/usr/local/，tar之；

ln之，/usr/local/hadoop；

配置hdfs

配置core-site.xml：

 

[html] view plain copy

 print?

1. <configuration>  
2.         <property>     
3.                 <name>fs.default.name</name>     
4.                 <value>hdfs://bluejoe0:9000</value>     
5.         </property>  
6.         <property>  
7.                 <name>io.file.buffer.size</name>  
8.                 <value>4096</value>  
9.         </property>  
10.   
11.         <property>  
12.                  <name>hadoop.tmp.dir</name>  
13.                  <value>/data/hdfs/tmp</value>  
14.         </property>  
15. </configuration>  

配置hdfs-site.xml：

 

 

[html] view plain copy

 print?

1. <?xml version="1.0" encoding="UTF-8"?>  
2. <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>  
3.   
4. <configuration>  
5.         <property>    
6.                 <name>dfs.name.dir</name>    
7.                 <value>file:/data/hdfs/name</value>    
8.         </property>    
9.         <property>    
10.                 <name>dfs.data.dir</name>     
11.                 <value>file:/data/hdfs/data</value>    
12.         </property>    
13.         <property>    
14.                 <name>dfs.permissions</name>    
15.                 <value>false</value>    
16.        </property>  
17.         <property>  
18.                 <name>dfs.replication</name>  
19.                 <value>2</value>  
20.         </property>  
21.         <property>  
22.                 <name>dfs.webhdfs.enabled</name>  
23.                 <value>true</value>  
24.         </property>  
25.         <property>  
26.                 <name>dfs.namenode.rpc-address</name>  
27.                 <value>bluejoe0:9000</value>  
28.         </property>  
29.         <property>  
30.                 <name>dfs.namenode.secondary.http-address</name>  
31.                 <value>bluejoe9:50090</value>  
32.         </property>  
33. </configuration>  

 

注意，dfs.namenode.rpc-address要和fs.default.name一致。

 

设置/usr/local/hadoop/etc/hadoop/hadoop-env.sh：

export JAVA_HOME=/usr/lib/jvm/Java-1.6.0-openjdk-1.6.0.0.x86_64

 

scp，将hadoop目录复制到其它机器；

 

配置slaves：

 

[html] view plain copy

 print?

1. bluejoe4  
2. bluejoe5  
3. bluejoe9  

 

namenode格式化：

hdfs namenode -format

启动hdfs：

./sbin/start-dfs.sh

能够看到输出信息：

 

[html] view plain copy

 print?

1. Starting namenodes on [bluejoe0]  
2. bluejoe0: starting namenode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-namenode-bluejoe0.out  
3. bluejoe9: starting datanode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-datanode-bluejoe9.out  
4. bluejoe4: starting datanode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-datanode-bluejoe4.out  
5. bluejoe5: starting datanode, logging to /usr/local/hadoop-2.5.2/logs/hadoop-root-datanode-bluejoe5.out  
6. Starting secondary namenodes [bluejoe9]  

 

 

接下来，能够查看Web界面(http://bluejoe0:50070/)，其datanodes截图以下：

目前为止，hdfs安装完毕！

配置mapreduce

修改 yarn-site.xml：

 

[html] view plain copy

 print?

1. <?xml version="1.0"?>  
2. <configuration>  
3.         <property>  
4.                 <name>yarn.nodemanager.aux-services</name>  
5.                 <value>mapreduce_shuffle</value>  
6.         </property>  
7.   
8.         <property>  
9.                 <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>  
10.                 <value>org.apache.hadoop.mapred.ShuffleHandler</value>  
11.         </property>  
12.   
13.         <property>  
14.                 <name>yarn.resourcemanager.address</name>  
15.                 <value>bluejoe0:8032</value>  
16.         </property>  
17.   
18.         <property>  
19.                 <name>yarn.resourcemanager.scheduler.address</name>  
20.                 <value>bluejoe0:8030</value>  
21.         </property>  
22.   
23.         <property>  
24.                 <name>yarn.resourcemanager.resource-tracker.address</name>  
25.                 <value>bluejoe0:8031</value>  
26.          </property>  
27.   
28.          <property>  
29.                 <name>yarn.resourcemanager.admin.address</name>  
30.                 <value>bluejoe0:8033</value>  
31.         </property>  
32.   
33.         <property>  
34.                 <name>yarn.resourcemanager.webapp.address</name>  
35.                 <value>bluejoe0:8088</value>  
36.         </property>  
37. </configuration>  

scp如上配置文件至其它节点；

 

启动mapreduce框架：

/usr/local/hadoop-2.5.2/sbin/start-yarn.sh

启动浏览器，访问http://bluejoe0:8088：

执行测试程序：

hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar pi 100 1000

Job Finished in 12.885 seconds
Estimated value of Pi is 3.14120000000000000000

设置/usr/local/hadoop/etc/hadoop/hadoop-env.sh：

export JAVA_HOME=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64