大数据平台 CDH5.12.2 安装部署

---

• 一：初始化环境部署配置
• 二： CDH 5.12.2 部署处理

---

一： 初始化环境部署处理

1.1 环境准备：

Centos7.3x64 全部主机名配置 配置好 全部机器的无密钥登录

---

1.2 主机名配置

172.17.100.131  node01.yangyang.com node01
172.17.100.132  node02.yangyang.com node02
172.17.100.133  node03.yangyang.com node03
172.17.100.134  node04.yangyang.com node04
172.17.100.135  node05.yangyang.com node05
172.17.100.136  node06.yangyang.com node06

---

1.3 无密钥登陆配置

作root用户无密钥认证

 ssh-keygen ---一直敲回车到最后

 cat id_rsa.pub >> authorized_keys 

 将全部的公钥导入authorized_keys  分发到 全部的 机器的.ssh/ 下面

 而后测试

---

1.4 关闭 firewalld，iptables 与 selinux

yum install -y firewall* iptable* 

  (1) 关闭firewalld

 systemctl stop firewalld.service
 systemctl disable firewalld.service
 firewall-cmd --state

 (2) 关闭iptables 
 iptables -F
 systemctl stop iptables.service 
 service iptables save
 systemctl disable iptables.service 

 (3) 禁用 selinux 

  sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config  
  setenforce 0
  getenforce 0
  sestatus

  重启机器

1.4 全部机器更改centos7.3x64的文件句柄数

vim /etc/security/limits.conf

---

*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited

1.5 全部机器 配置好jdk环境 同步全部机器

(1) 卸载原有jdk
rpm -e java-1.7.0-openjdk-1.7.0.141-2.6.10.5.el7.x86_64 java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 java-1.8.0-openjdk-headless-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-headless-1.7.0.141-2.6.10.5.el7.x86_64 copy-jdk-configs-2.2-3.el7.noarch java-1.8.0-openjdk-1.8.0.131-11.b12.el7.x86_64 java-1.7.0-openjdk-devel-1.7.0.141-2.6.10.5.el7.x86_64 java-1.8.0-openjdk-devel-1.8.0.131-11.b12.el7.x86_64 java-1.6.0-openjdk-1.6.0.41-1.13.13.1.el7_3.x86_64 tzdata-java-2017b-1.el7.noarch python-javapackages-3.4.1-11.el7.noarch javapackages-tools-3.4.1-11.el7.noarch --nodeps

(2) 安装jdk环境
rpm -ivh jdk-8u151-linux-x64.rpm

---

1.6 生产环境配置chrony 时间同步服务器

全部机器 yum install chrony*
1. node01:

vim /etc/chrony.conf 增长
----
注释掉相关默认的server
server ntp1.aliyun.com iburst

allow 172.17.100.0.0/16

service chronyd start
chkconfig chronyd on 

2. node02---node05 同步node01

vim /etc/chrony.conf
----
注释掉默认的server
server 172.17.100.131 iburst
----

service chronyd start 
chkconfig chronyd on

---

1.7 安装mysql5.7数据库主节点上面安装

卸载：mariadb-libs 的包：
----
 ＃rpm -e mariadb-libs mariadb-devel --nodeps
---
安装mysql5.7 所须要依赖包

 # yum install -y perl-JSON libaio 

解压 mysql 5.7.21 集成安装包

tar -xvf mysql-5.7.21-1.el7.x86_64.rpm-bundle.tar

删掉一个包mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm 这个minimal包否则装不了mysql5.7 

# rm -rf mysql-community-server-minimal-5.7.21-1.el7.x86_64.rpm
# rpm -ivh mysql-community-* 
# mysqld --initialize-insecure --user=mysql

注：上面的安装步骤是用忽略安装mysql包安装顺序，系统将自动识别安装顺序

启动mysql5.7 

#service mysqld start
# chkconfig mysqld on

注：初次使用mysql的时候,须要更改mysql 的密码，不然查询不了mysql数据库当中的数据。 
#mysql -uroot -p
mysql > set password = password('Bigdata_1234');
mysql> flush privileges;

1.8 修改系统参数

vim /etc/sysctl.conf

##内核默认参数 
kernel.sysrq = 0 
kernel.core_uses_pid = 1 
kernel.msgmnb = 65536 
kernel.msgmax = 65536 
kernel.shmmax = 68719476736 
kernel.shmall = 4294967296 
##打开文件数参数(20*1024*1024) 
fs.file-max= 20971520 
##WEB Server参数 
net.ipv4.tcp_tw_reuse=1 
net.ipv4.tcp_tw_recycle=1 
net.ipv4.tcp_fin_timeout=30 
net.ipv4.tcp_keepalive_time=1200 
net.ipv4.ip_local_port_range = 1024 65535 
net.ipv4.tcp_rmem=4096 87380 8388608 
net.ipv4.tcp_wmem=4096 87380 8388608 
net.ipv4.tcp_max_syn_backlog=8192 
net.ipv4.tcp_max_tw_buckets = 5000 

##TCP补充参数 
net.ipv4.ip_forward = 0 
net.ipv4.conf.default.rp_filter = 1 
net.ipv4.conf.default.accept_source_route = 0 
net.ipv4.tcp_syncookies = 1 
net.ipv4.tcp_sack = 1 
net.ipv4.tcp_window_scaling = 1 
net.core.wmem_default = 8388608 
net.core.rmem_default = 8388608 
net.core.rmem_max = 16777216 
net.core.wmem_max = 16777216 
net.core.netdev_max_backlog = 262144 
net.ipv4.tcp_max_orphans = 3276800 
net.ipv4.tcp_timestamps = 0 
net.ipv4.tcp_synack_retries = 1 
net.ipv4.tcp_syn_retries = 1 
net.ipv4.tcp_mem = 94500000 915000000 927000000 
##禁用ipv6 
net.ipv6.conf.all.disable_ipv6 =1 
net.ipv6.conf.default.disable_ipv6 =1 
##swap使用率优化 
vm.swappiness=0 
---
执行如下命令使参数生效: 
# sysctl -p 
同步到全部机器

---

二： CDH5.12.2 安装与配置

2.1 安装CM 所依赖的包：(所有节点都安装)

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb postgresql* portmap mod_ssl openssl-devel python-psycopg2 MySQL-python

2.2 安装cloudera-server 与更改配置 （节点所有安装）

rpm -ivh cloudera-manager-* 
rpm -ivh enterprise-debuginfo-5.12.0-1.cm5120.p0.120.el7.x86_64.rpm

在mysql 当中建立CM 的库

# mysql -uroot -pBigdata_1234
mysql> create database cmf character set utf8;
mysql> grant all privileges on cmf.* to 'cmf'@'%' identified by 'cmf_1234' with grant option;
mysql> grant all privileges on cmf.* to 'cmf'@'node01.yangyang.com' identified by 'cmf_1234' with grant option;

修改cm数据库
#cp /etc/cloudera-scm-server/db.properties /etc/cloudera-scm-server/db.propertiesbak 
# vim /etc/cloudera-scm-server/db.properties

2.3 安装 离线cdh parcel-repo

#cd /root/software
#cp -ap CDH-5.12.2-1.cdh5.12.2.p0.4-el7.parcel* /opt/cloudera/parcel-repo/
#cp -p manifest.json /opt/cloudera/parcel-repo/
#cd /opt/cloudera/parcel-repo
mv CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha1  (去掉 这个 1) CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha   
chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/ （全部主机都要执行这条命令）

---

2.4 jdbc 配置

拷贝jdbc 到 java的目录
  tar -zxvf mysql-connector-java-5.1.44.tar.gz
  cd mysql-connector-java-5.1.44
  cp mysql-connector-java-5.1.44-bin.jar  /usr/share/java/
  cd /usr/share/java/
  mv mysql-connector-java-5.1.44-bin.jar mysql-connector-java.jar

2.5 修改config.ini 文件

#cp /etc/cloudera-scm-agent/config.ini /etc/cloudera-scm-agent/config.inibak 
#vim /etc/cloudera-scm-agent/config.ini
修改server_host 为主节点的IP 地址
---

server_host=172.17.100.131

---
同步到全部主机节点 
scp /etc/cloudera-scm-agent/config.ini node02:/etc/cloudera-scm-agent/
scp /etc/cloudera-scm-agent/config.ini node03:/etc/cloudera-scm-agent/

2.6 启动CM-server与agent-server

node01.yangyang.com 主机

cd /etc/init.d
./cloudera-scm-server start

---

查看启动日志

查看启动结果：
        须要经过查看server的启动日志文件
        /var/log/cloudera-scm-server
    tail -f cloudera-scm-server.log

    看到jetty 表示启动成功 会有 7180 端口启动：

2.7打开web浏览器：

http://172.17.100.131:7180

用户名： admin
密码： admin

启动 全部的agent 节点
cd /etc/init.d/
./cloudera-scm-agent start

一直到分发结束 -- 而后继续

全部节点执行：
#echo never > /sys/kernel/mm/transparent_hugepage/defrag
#echo never > /sys/kernel/mm/transparent_hugepage/enabled

vim /etc/rc.local 

在最下面增长：

---
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
---
同步全部机器

从新运行界面：

配置角色：

数据库受权：
mysql -uroot -pBigdata_1234
---
建立hive 所须要的库并受权：

mysql >create database cdh_hive default character set latin1; 
字符必须为拉丁(latin1),若是uft8可能会报错Specified key was too long; max key length is 767 bytes 
mysql >grant all privileges on cdh_hive.* to 'hive'@'%' identified by 'Hive_123' with grant option;
mysql >grant all privileges on cdh_hive.* to 'hive'@'node01.yangyang.com' identified by 'Hive_123' with grant option;
mysql >flush privileges;

建立oozie 所须要的库
mysql >create database cdh_oozie default character set utf8; 
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'%' identified by 'Oozie_123' with grant option;
mysql >grant all privileges on cdh_oozie.* to 'oozie'@'node01.yangyang.com' identified by 'Oozie_123' with grant option;
mysql >flush privileges;

建立hue所须要的库：
mysql >create database cdh_hue default character set utf8; 
mysql >grant all privileges on cdh_hue.* to ‘hue’@'%' identified by 'Hue_123' with grant option;
mysql >grant all privileges on cdh_hue.* to 'hue'@'node01.yangyang.com' identified by 'Hue_123' with grant option;
mysql >flush privileges;

建立CM 的 监控端Activity Monitor 的库
Activity Monitor:
mysql >create database cdh_am default character set utf8; 
mysql >grant all privileges on cdh_am.* to ‘am’@'%' identified by 'Am_123' with grant option;
mysql >grant all privileges on cdh_am.* to 'am'@'node01.yangyang.com' identified by 'Am_123' with grant option;
mysql >flush privileges;