Cloudera Manager 大数据集群搭建离线安装详细过程

时间  2019-11-21
标签 cloudera manager 数据 集群 搭建 离线 安装 详细 过程 栏目 云服务 繁體版
原文   原文链接

软件环境

本教程搭建5个CDH节点，分别为html

cdh1 192.168.10.101
cdh2 192.168.10.102
cdh3 192.168.10.103
cdh4 192.168.10.104
cdh5 192.168.10.105
复制代码

步骤

安装虚拟机

用vsphere client新建cdh1一台虚拟机。 注意：虚拟机安装选择上海时区，并修改时间。 修改配置：操做系统ip、防火墙、seliux、hosts、内核参数。java

IP配置

BOOTPROTO=static
IPADDR=192.168.10.101
NETMASK=255.255.255.0
GATEWAY=192.168.10.1
DNS1=8.8.8.8
DNS2=114.114.114.114
ONBOOT=yes
复制代码

关闭防火墙和禁止开机自启

systemctl stop firewalld & systemctl disable firewalld
复制代码

关闭防火墙和禁止开机自启

seliux配置(重启生效)

getenforce Enforcing
Enforcing
复制代码

修改SELINUX为permissive或disabled。此配置重启才生效。mysql

vi /etc/selinux/config
SELINUX=disabled
复制代码

修改本地域名

若是采用局域网内域名服务器，只须要在第一步将DNS服务器配置为内网域名服务器便可。此步骤能够忽略。linux

vi /etc/hosts
192.168.10.101 cdh1
192.168.10.102 cdh2
192.168.10.103 cdh3
192.168.10.104 cdh4
192.168.10.105 cdh5
复制代码

文件打开数(重启生效)

vi /etc/security/limits.confsql

* soft nofile 65536 
* hard nofile 65536 
* soft nproc 131072 
* hard nproc 131072
复制代码

修改系统整体文件打开数shell

echo  6553560 > /proc/sys/fs/file-max
复制代码

修改swappiness (重启生效)

echo 'vm.swappiness=10'>> /etc/sysctl.conf数据库

关闭大叶内存(重启生效)

cat << EOF >> /etc/rc.d/rc.local 
echo never > /sys/kernel/mm/transparent_hugepage/defrag 
echo never > /sys/kernel/mm/transparent_hugepage/enabled 
EOF
复制代码

重启

以上全部配置完毕，关机并验证以上配置设置无误。json

复制虚拟机

因为以上配置对全部虚拟机都适用，用vsphere client 复制4个虚拟机。 处处OVF文件前关机vim

导出 OVF文件

导出 OVF文件

再部署OVF文件，依次修改各虚拟机IP。centos

chd1-5

修改主机名(每一个节点)

vi /etc/sysconfig/network
cdhx
复制代码

免密登陆

（全部节点执行）

ssh-keygen   #（全部节点执行）
ssh-copy-id -i ~/.ssh/id_rsa.pub root@cdh1 #（其余非cdh1节点执行）
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys #（cdh1执行)
scp ~/.ssh/authorized_keys root@cdhx:/root/.ssh/ #(cdh1执行)
复制代码

集群脚本(cdh1执行)

集群分发命令

yum install -y rsync #(全部节点执行)
复制代码
vi /usr/local/bin/xsync
#!/bin/bash
# $#：表示传递给脚本或函数的参数个数。
#1 获取输入参数个数，若是没有参数，直接退出
pcount=$#
if((pcount==0)); then
echo no args;
exit;
fi
 
#2 获取文件名称
p1=$1
fname=`basename $p1`
echo fname=$fname
 
#3 获取上级目录到绝对路径
pdir=`cd -P $(dirname $p1); pwd`
echo pdir=$pdir
 
#4 获取当前用户名称
user=`whoami`
 

#5 循环
for host in {cdh2,cdh3,cdh4,cdh5}; do
        #echo $pdir/$fname $user@$host:$pdir
        echo --------------- $host ----------------
        rsync -rvl $pdir/$fname $user@$host:$pdir
done
复制代码
集群调用命令

vi /usr/local/bin/xcall

##!/bin/sh
pcount=$#
if((pcount==0));then
        echo no args...;
        exit;
fi
echo ==================cdh1==================
$@
for host in {cdh2,cdh3,cdh4,cdh5}; do
        echo ==================$host==================
        ssh $host $@
done
复制代码

赋权

chmod 777 xsync.sh
chmod 777 xcall.sh
复制代码

时钟同步

查看时区

[root@cdh1 ~]# xcall 'timedatectl '
==================cdh1==================
      Local time: Fri 2019-10-11 17:24:57 CST
  Universal time: Fri 2019-10-11 09:24:57 UTC
        RTC time: Fri 2019-10-11 09:24:57
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: no
NTP synchronized: no
 RTC in local TZ: no
      DST active: n/a
复制代码

若是时区不对，先调整时区

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
复制代码

安装ntp (全部节点)

xcall 'yum install -y ntp'
复制代码

修改ntp配置文件(cdh1)

vim /etc/ntp.conf
# 容许内网其余机器同步时间
restrict 192.168.10.0 mask 255.255.255.0 nomodify notrap
server 0.cn.pool.ntp.org iburst
server 1.cn.pool.ntp.org iburst
server 2.cn.pool.ntp.org iburst
server 3.cn.pool.ntp.org iburst

复制代码

修改ntp配置文件(cdh2-5)

vim /etc/ntp.conf
# 容许内网其余机器同步时间
server cdh1
复制代码

重启ntpd

xcall systemctl start ntpd
xcall systemctl enable ntpd
复制代码

本地yum库

安装httpd,并启动

yum install -y httpd
systemctl start httpd
systemctl enable httpd
复制代码

建立/var/www/html/pub/cm/5.12.1/RPMS/x86_64/目录，将如下文件上传

mkdir -p /var/www/html/pub/cm/5.12.1/RPMS/x86_64/
复制代码
cloudera-manager-agent-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
cloudera-manager-daemons-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
cloudera-manager-server-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
cloudera-manager-server-db-2-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
enterprise-debuginfo-5.12.1-1.cm5121.p0.6.el7.x86_64.rpm
复制代码

建立/var/www/html/pub/parcels/5.12.1/目录，将如下文件上传

mkdir -p /var/www/html/pub/parcels/5.12.1/
复制代码
CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel
CDH-5.12.1-1.cdh5.12.1.p0.3-el7.parcel.sha1
manifest.json
复制代码

建立/var/www/html/pub/kudu/1.4.0目录，将如下文件上传

mkdir -p /var/www/html/pub/kudu/1.4.0
复制代码
KUDU-1.4.0-1.cdh5.12.1.p0.10-el7.parcel  
KUDU-1.4.0-1.cdh5.12.1.p0.10-el7.parcel.sha1  
manifest.json
复制代码

制做元数据

yum -y install createrepo
createrepo  /var/www/html/pub/
复制代码

会生成/var/www/html/pub/repodata/repomd.xml文件

建立/etc/yum.repos.d/cm.repo

[cm] 
name=cm5.12 
baseurl=http://cdh1/pub/
enabled=1 
gpgcheck=0 
priority=1 
复制代码

yum 更新(全部节点)

xcall yum clean all
xcall yum makecache
复制代码

搜索 cloudera 肯定本地源配置无误

yum search cloudera
复制代码

分发cm.repo文件

xsync /etc/yum.repos.d/cm.repo
复制代码

jdk安装

xsync /opt/jdk-8u131-linux-x64.tar.gz
xcall 'tar zxf /opt/jdk-8u131-linux-x64.tar.gz -C /opt/'
xcall  'mv /opt/jdk1.8.0_131 /opt/jdk'
复制代码

编辑环境变量并分发

vim /etc/profile
export JAVA_HOME=/opt/jdk
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$PATH:$JAVA_HOME/bin

xsync /etc/profile
source /etc/profile #(各节点执行)
xcall java -version #(各节点执行)

#连接jdk 该CDH版本即便系统配置了JAVA_HOME也找不到
xcall 'mkdir /usr/lib/jvm/'
xcall 'ln -s /opt/jdk/ /usr/lib/jvm/java-8-openjdk'
复制代码

mariadb 安装

yum install -y mariadb-server
#安装完成MariaDB首先启动MariaDB
systemctl start mariadb
#设置开机启动
systemctl enable mariadb

update user set host='%' where user='root';
update user set password=password('123456') where user='root'; 
flush privileges;
复制代码

建立数据库

create database  cmf default character set utf8;
grant all privileges on cmf.* to 'cmf'@'%' identified by 'Cmf_123!';

create database  metastore default character set latin1;
grant all privileges on metastore.* to 'hive'@'%' identified by 'Hive_123!';

create database  amon default character set utf8;
grant all privileges on amon.* to 'amon'@'%' identified by 'Amon_123!';


create database  rman default character set utf8;
grant all privileges on rman.* to 'rman'@'%' identified by 'Rman_123!';

create database  nas default character set utf8;
grant all privileges on nas.* to 'nas'@'%' identified by 'Nas_123!';

create database  nms default character set utf8;
grant all privileges on nms.* to 'nms'@'%' identified by 'Nms_123!';

create database  oozie default character set utf8;
grant all privileges on oozie.* to 'oozie'@'%' identified by 'Oozie_123!';

create database  hue default character set utf8;
grant all privileges on hue.* to 'hue'@'%' identified by 'Hue_123!';

flush privileges;
复制代码

mysql 驱动安装

xcall 'mkdir -p /usr/share/java'
cd /usr/share/java
cp /opt/mysql-connector-java-5.1.48-bin.jar /usr/share/java
mv /usr/share/java/mysql-connector-java-5.1.48-bin.jar /usr/share/java/mysql-connector-java.jar 
xsync  /usr/share/java/mysql-connector-java.jar
xcall 'chmod a+x /usr/share/java/mysql-connector-java.jar'
复制代码

CDM部署

安装

yum install -y cloudera-manager-server 

复制代码

修改db.properties

vim /etc/cloudera-scm-server/db.properties
# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
com.cloudera.cmf.db.host=localhost

# The database name
com.cloudera.cmf.db.name=cmf

# The database user
com.cloudera.cmf.db.user=cmf

# The database user's password
com.cloudera.cmf.db.password=Cmf_123!

# The db setup type
# By default, it is set to INIT
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL
com.cloudera.cmf.db.setupType=EXTERNAL
复制代码

启动

/etc/init.d/cloudera-scm-server restart 
/etc/init.d/cloudera-scm-server status 
复制代码

打开网址 http://cdh1:7180 帐户名密码都是admin

部署cdm-agent

一、agent安装 (时间长，能够不用xcall)

xcall 'yum -y install cloudera-manager-daemons cloudera-manager-agent' 
#二、修改客户端配置文件config.ini 
xcall 'cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.inibak' 
sed -i '/server_host/s/localhost/cdh1/' /etc/cloudera-scm-agent/config.ini
#三、启动客户端 
xcall '/etc/init.d/cloudera-scm-agent restart'
xcall '/etc/init.d/cloudera-scm-agent status'
复制代码

填写parcel的地址

下载 分配

安装服务

依次安装服务

问题

若是有目录权限问题按照如下命令修改

xcall chown  -R hdfs:hdfs /var/lib/hadoop-hdfs
xcall chown  -R httpfs.httpfs /var/lib/hadoop-httpfs
xcall chown  -R kms.kms /var/lib/hadoop-kms
xcall chown  -R mapred:mapred /var/lib/hadoop-mapreduce
xcall chown  -R yarn:yarn /var/lib/hadoop-yarn
xcall chmod -R 755 /var/lib/hadoop-*
复制代码
相关文章
相关标签/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公众号
   欢迎关注本站公众号,获取更多信息
联系我们 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程