最近本身写了一个Hadoop自动化部署脚本，包括Hadoop集群自动化部署脚本和Hadoop增长单节点自动化部署脚本。须要快速部署Hadoop集群的童鞋能够使用该脚本。这些脚本我在用5台虚拟机进行了测试，若是在使用中还有bug，欢迎指出。本文主要介绍Hadoop集群自动化部署脚本，安装的Hadoop版本为2.6.0。html
安装Hadoop2.6.0集群须要依赖JDK和Zookeeper。本文安装的JDK版本为jdk-7u60-linux-x64，Zookeeper版本为zookeeper-3.4.6。java
该部署脚本由两部分构成：root用户下执行的脚本和Hadoop启动用户下执行的脚本。这些脚本都只须要在一台服务器上执行便可，执行脚本的服务器做为Hadoop的Master服务器。下面分别进行说明。node
root脚本的目录结构以下：linux
该目录下的init.conf文件为root执行脚本使用的配置文件，在执行脚本以前须要对该配置文件进行修改。文件内容以下：shell
#jdk file and version JDK_FILE_TAR=jdk-7u60-linux-x64.tar.gz #jdk unpack name JDK_FILE=jdk1.7.0_60 #java home JAVAHOME=/usr/java #Whether install the package for dependence,0 means no,1 means yes IF_INSTALL_PACKAGE=1 #host conf ALLHOST="hadoop1master hadoop1masterha hadoop1slave1 hadoop1slave2 hadoop1slave3" ALLIP="192.168.0.180 192.168.0.184 192.168.0.181 192.168.0.182 192.168.0.183" #zookeeper conf ZOOKEEPER_TAR=zookeeper-3.4.6.tar.gz ZOOKEEPERHOME=/usr/local/zookeeper-3.4.6 SLAVELIST="hadoop1slave1 hadoop1slave2 hadoop1slave3" #hadoop conf HADOOP_TAR=hadoop-2.6.0.tar.gz HADOOPHOME=/usr/local/hadoop-2.6.0 HADOOP_USER=hadoop2 HADOOP_PASSWORD=hadoop2 #root conf: $MASTER_HA $SLAVE1 $SLAVE2 $SLAVE3 ROOT_PASSWORD="hadoop hadoop hadoop hadoop"
下面是个别参数的解释及注意事项：bootstrap
该目录下包含password.expect、scp.expect、otherInstall.expect三个文件。password.expect用来设置hadoop启动用户的密码；scp.expect用来远程传输文件；otherInstall.expect用来远程执行其余服务器上的installRoot.sh。这三个文件都在installRoot.sh中被调用。
password.expect文件内容以下：bash
#!/usr/bin/expect -f set user [lindex $argv 0] set password [lindex $argv 1] spawn passwd $user expect "New password:" send "$password\r" expect "Retype new password:" send "$password\r" expect eof
其中argv 0和argv 1都是在installRoot.sh脚本中进行传值的。其余两个文件argv *也是这样传值的。
scp.expect文件内容以下：服务器
#!/usr/bin/expect -f # set dir, host, user, password set dir [lindex $argv 0] set host [lindex $argv 1] set user [lindex $argv 2] set password [lindex $argv 3] set timeout -1 spawn scp -r $dir $user@$host:/root/ expect { "(yes/no)?" { send "yes\n" expect "*assword:" { send "$password\n"} } "*assword:" { send "$password\n" } } expect eof
otherInstall.expect文件内容以下：ssh
#!/usr/bin/expect -f # set dir, host, user, password set dir [lindex $argv 0] set name [lindex $argv 1] set host [lindex $argv 2] set user [lindex $argv 3] set password [lindex $argv 4] set timeout -1 spawn ssh -q $user@$host "$dir/$name" expect { "(yes/no)?" { send "yes\n" expect "*assword:" { send "$password\n"} } "*assword:" { send "$password\n" } } expect eof
这里就是安装Hadoop集群及其依赖所需的安装包。oop
该脚本是在root用户下须要执行的脚本，文件内容以下：
#!/bin/bash if [ $USER != "root" ]; then echo "[ERROR]:Must run as root"; exit 1 fi # Get absolute path and name of this shell readonly PROGDIR=$(readlink -m $(dirname $0)) readonly PROGNAME=$(basename $0) hostname=`hostname` source /etc/profile # import init.conf source $PROGDIR/conf/init.conf echo "install start..." # install package for dependence if [ $IF_INSTALL_PACKAGE -eq 1 ]; then yum -y install expect >/dev/null 2>&1 echo "expect install successful." # yum install openssh-clients #scp fi #stop iptables or open ports, now stop iptables service iptables stop chkconfig iptables off FF_INFO=`service iptables status` if [ -n "`echo $FF_INFO | grep "Firewall is not running"`" ]; then echo "Firewall is already stop." else echo "[ERROR]:Failed to shut down the firewall.Exit shell." exit 1 fi #stop selinux setenforce 0 SL_INFO=`getenforce` if [ $SL_INFO == "Permissive" -o $SL_INFO == "disabled" ]; then echo "selinux is already stop." else