大数据平台CDH搭建

时间 2019-12-01

标签数据平台 cdh 搭建繁體版

原文原文链接

1. CDH简介

来自官网简介: html

CDH is Cloudera’s 100% open source platform distribution, including Apache Hadoop and built specifically to meet enterprise demands. CDH delivers everything you need for enterprise use right out of the box. By integrating Hadoop with more than a dozen other critical open source projects, Cloudera has created a functionally advanced system that helps you perform end-to-end Big Data workflows.前端

www.cloudera.com/java

2. 环境准备

2.1 机器准备

三台虚拟机便可，博主这里使用的一台xenserver虚拟服务器分出来的三台虚拟机。node

| IP地址       |   主机名           | 系统版本   |  软件                |
| 192.168.1.80 | devcdh1.juejin.org| centos7.6 |  ClouderaManager Mysql  
| 192.168.1.81 | devcdh1.juejin.org| centos7.6 |  hadoopNameNode  HBaseMaster NodeManagre
| 192.168.1.82 | devcdh1.juejin.org| centos7.6 |  hadoopNode HRegionServer sparkSlave复制代码

2.2 软件准备(离线安装)

因公网安装速度很慢，而常常会下载断线，最好的方式是离线安装，事前将rpm包下载到服务器上.mysql

2.2.1 下图全部rpm软件下载到本地git

archive.cloudera.com/cm6/6.1.1/r…
sql

2.2.2 下图中红框内rpm软件下载到本地数据库

archive.cloudera.com/cdh6/6.1.1/…
vim

2.3 系统初始化(三台机器都执行)

1. 安装经常使用软件
yum install wget iptables-services telnet net-tools git curl unzip sysstat lsof ntpdate lrzsz vim  -y
2. 安装ntp时间服务器并启动
yum install ntp -y
systemctl start ntpd
systemctl enable ntpd
3. 为了后续方便操做，先关闭防火墙并禁止开机启动(iptables为上图安装,firewalld为centos7默认防火墙)
systemctl stop iptables
systemctl disable iptables
systemctl stop firewalld
systemctl disable firewalld
复制代码

2.4 安装JDK(三台机器都执行)

将下载好的oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm 上传到服务器centos

执行命令安装jdk:

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm复制代码

写入系统全局环境文件，并声明:

echo "export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera"  >> /etc/profile
echo "export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib"  >> /etc/profileecho "export PATH=$PATH:$JAVA_HOME/bin"  >> /etc/profile

source /etc/profile   #声明
java -version         #查看版本复制代码

2.5 设置hosts(三台机器都执行)

服务器主机名看上去和二级域名同样。只是没有解析，这是CDH官方文档要求:

docs.cloudera.com/documentati…

注* 主机名称尽可能使用自已公司域名

echo "192.168.1.80 devcdh1.juejin.org" >> /etc/hosts
echo "192.168.1.81 devcdh2.juejin.org" >> /etc/hostsecho "192.168.1.82 devcdh3.juejin.org" >> /etc/hosts复制代码

2.6 安装mysql(官方支持多种数据库，博主mysql用得比较多，这里选择mysql)

支持的数据库:

docs.cloudera.com/documentati…

执行下面命令安装mysql:

wget http://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm  #下载官方rpm包
yum localinstall mysql57-community-release-el7-11.noarch.rpm -y            #安装rpm包
yum repolist enabled | grep "mysql.*-community.*"  -y                      
yum install mysql-community-server -y                                      #安装mysql-server
systemctl start mysqld                                                     #启动mysql
grep 'temporary password' /var/log/mysqld.log                              #查看mysql 初始密码仓库复制代码

建立cdh集群所须要的库和帐密(帐密请自行设定)

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'XXXXXXX#####';

CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'XXXXXXX#####';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'XXXXXXX#####';复制代码

2.7 安装cloudera-manager

将下载好的 cloudera-manager 软件上传到服务器上

dev-cdh1.juejin.org  安装  ->   cloudera-manager-daemons cloudera-manager-agent cloudera-manager-serverdev-cdh2.juejin.org  安装  ->   cloudera-manager-daemons cloudera-manager-agent dev-cdh3.juejin.org  安装  ->    cloudera-manager-daemons cloudera-manager-agent 复制代码

登录到dev-cdh1.juejin.org 执行命令:

yum localinstall cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server -y复制代码

登录到dev-cdh2.juejin.org 执行命令:

yum localinstall cloudera-manager-daemons cloudera-manager-agent -y复制代码

登录到dev-cdh3.juejin.org 执行命令:

yum localinstall cloudera-manager-daemons cloudera-manager-agent -y复制代码

注* yum localinstall 为安装本地rpm包文件 yum install 是下载yum仓库软件再安装

2.8 启动manager

在dev-cdh1.juejin.org 上执行启动命令:

service cloudera-scm-server restart复制代码

本地绑一下hosts, 博主这里使用的SwitchHosts:

192.168.1.80 devcdh1.juejin.org
192.168.1.81 devcdh2.juejin.org
192.168.1.82 devcdh3.juejin.org复制代码

访问:

devchd1.juejin.org:7180 打开管理页面，CDH默认帐密为: admin admin

3. 集群安装

3.1 将主机组成集群，安装cdh agent

3.1.1 打开管理后台，登录后并接受协议点击继续

3.1.2 选择Cloudera Express版本点击继续

3.1.3 Specify Hosts 将三台主机名填入，搜索后选择所有点击继续

3.1.4 选择存储库默认既可点击继续

3.1.5 JDK 安装选项因咱们已经安装jdk 这里不用勾选点击继续

3.1.6 提供 SSH 登陆凭据。将服务器密码填入既可点击继续

3.1.7 Install Agents 点击继续

3.1.8 Install Parcels 点击继续

3.1.8 检查主机正确性若有报错根据提示修正后再从新检查既可点击继续

3.2 配置CDH服务部署

3.2.1 选择安装的服务自定义安装, 选择hadoop hbase spark hive hue zookeepr 既可，其余或后续添加

3.2.2 自定义角色分配，选择安装在那个节点上须要根据服务器配置来选择，这里由于cdh安装在devcdh1上。因此这里将hadoopNamenode hbaseMaster sparkMaster等所有安装在devcdh2上

3.2.3 数据设置。将前端设置好的帐密和主机地址填入，点击测试便可

3.2.4 后续一路默认，直到安装完成

参考:

官方安装文档:

docs.cloudera.com/documentati…

官方对服务器要求文档:

docs.cloudera.com/documentati…