Ambari2.7.4+HDP3.1.4安装 Centos7离线安装

时间  2020-12-10
标签 html java node mysql linux web sql 数据库 json vim 栏目 Tomcat 繁體版
原文   原文链接

Ambari2.7.4+HDP3.1.4安装 Centos7离线安装

标签（空格分隔）： 大数据平台构建html

  • 一： 关于HDP的介绍
  • 二： 系统环境介绍
  • 三： 系统环境初始化
  • 四： 安装HDP3.1.4平台

一： 关于HDP的介绍

1.1 ambari的介绍

Ambari是一种基于Web的工具，支持Apache Hadoop集群的建立 、管理和监控。java

Ambari已支持大多数Hadoop组件，包括HDFS、MapReduce、Hive、Pig、 Hbase、Zookeeper、Sqoop和Hcatalog等。Apache Ambari 支持HDFS、MapReduce、Hive、Pig、Hbase、Zookeepr、Sqoop和Hcatalog等的集中管理。也是5个顶级hadoop管理工具之一。node

Ambari 自身也是一个分布式架构的软件，主要由两部分组成：Ambari Server 和 Ambari Agent。简单来讲，用户经过 Ambari Server 通知 Ambari Agent 安装对应的软件；Agent 会定时地发送各个机器每一个软件模块的状态给 Ambari Server，最终这些状态信息会呈如今 Ambari 的 GUI，方便用户了解到集群的各类状态，并进行相应的维护。mysql

1.2 hdp的介绍

HDP是hortonworks的软件栈，里面包含了hadoop生态系统的全部软件项目，好比HBase,Zookeeper,Hive,Pig等等。linux

1.3 HDP-UTILS

HDP-UTILS是工具类库。web

二：系统环境安装介绍

2.1 操做系统与环境

2.1.1：系统主机名
系统： Centos7.8 X64
---
192.168.100.11  node01.flyfish
192.168.100.12  node02.flyfish
192.168.100.13  node03.flyfish
192.168.100.14  node04.flyfish
192.168.100.15  node05.flyfish
192.168.100.16  node06.flyfish
192.168.100.17  node07.flyfish

本次安装采用前4台虚拟主机

2.2 系统包下载地址

Ambari在线安装特别慢，因此使用离线安装，建议使用迅雷下载sql

参照文档：
https://docs.cloudera.com/HDPDocuments/Ambari-2.7.4.0/bk_ambari-installation/content/hdp_314_repositories.html
ambari:
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.4.0/ambari-2.7.4.0-centos7.tar.gz
HDP:
http://public-repo-1.hortonworks.com/HDP/centos7/3.x/updates/3.1.4.0/HDP-3.1.4.0-centos7-rpm.tar.gz
HDP-UTILS:
http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz
HDP-GPL：
http://public-repo-1.hortonworks.com/HDP-GPL/centos7/3.x/updates/3.1.4.0/HDP-GPL-3.1.4.0-centos7-gpl.tar.gz数据库

三：系统环境初始化

3.1：无密钥登陆配置

作root用户无密钥认证json

ssh-keygen ---一直敲回车到最后vim

cat id_rsa.pub >> authorized_keys

将全部的公钥导入authorized_keys 分发到 全部的 机器的.ssh/ 下面

而后测试

image_1ec7t75c11kn4n3jpbf2913gmp.png-80.6kB

3.2: 关闭 firewalld，iptables 与 selinux

yum install -y firewall iptable

(1) 关闭firewalld

systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state

(2) 关闭iptables
iptables -F
systemctl stop iptables.service
service iptables save
systemctl disable iptables.service

(3) 禁用 selinux

sed -i '/SELINUX/s/enforcing/disabled/' /etc/selinux/config
setenforce 0
getenforce 0
sestatus

重启机器

reboot

image_1ec7t8565u2i16lh1vp71fdt1gem16.png-103.4kB

3.3 全部机器更改centos7.5x64的文件句柄数

vim /etc/security/limits.conf
---
*               soft    nofile          65535
*               hard    nofile          1029345
*               soft    nproc           unlimited
*               hard    nproc           unlimited
*               soft    memlock         unlimited
*               hard    memlock         unlimited
---

image_1ec7t8uhg105r131h1hjnfh01rm71j.png-59kB

3.4 生产环境配置chrony 时间同步服务器

全部机器 yum install chrony*
1. node01.flyfish:

vim /etc/chrony.conf 增长
----
注释掉相关默认的server:
全部节点均同步阿里云时间服务器
server ntp1.aliyun.com iburst

service chronyd start
chkconfig chronyd on

image_1ec7tc1201dlj1s7bftbnu1gn02d.png-106.2kB

image_1ec7tbhi41t61oi08sdf1cpj20.png-113.3kB

3.6 安装jdk1.8.

rpm -ivh jdk-8u162-linux-x64.rpm
---
export JAVA_HOME=/usr/java/jdk1.8.0_162
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin

---
source /etc/profile 

java -version

image_1ec7th44r1mug1clm1ild15bvt629.png-37.8kB

image_1ec7thugo1bu8r5612gu1am7dvep.png-59.1kB

3.7 设置swap

sh ssh_do_all.sh node.list "echo vm.swappiness = 10 >> /etc/sysctl.conf"
sh ssh_do_all.sh node.list "sysctl -p |grep vm.swappiness"

image_1efqmiptsrh61l5n13ne1vpm1mnj9.png-82.8kB

3.8 设置透明大页面

sh ssh_do_all.sh node.list "echo never > /sys/kernel/mm/transparent_hugepage/defrag"

sh ssh_do_all.sh node.list  "echo never > /sys/kernel/mm/transparent_hugepage/enabled"
设置开机自关闭

将以下脚本添加到/etc/rc.local文件中

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi

四： 安装 hdp3.1.4

4.1 : 配置ambari所需安装源

安装httpd 

yum install -y httpd

service httpd start

chkconfig httpd on

cd /var/www/html/

1. ambari 的 yum源

cd /root/software
tar -zxvf ambari-2.7.4.0-centos7.tar
mv ambari /var/www/html/

image_1efqn24gg1jjpe591hrchvu16g1m.png-271.1kB

image_1efqn2htd3lv1ahh1kc91t0tkk913.png-30.2kB

cd /var/www/html/ambari/centos7/2.7.4.0-118
cp -p ambari.repo /etc/yum.repos.d/

cd /etc/yum.repos.d

vim ambari.repo

----
#VERSION_NUMBER=2.7.4.0-118
[ambari-2.7.4.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.4.0
baseurl=http://192.168.100.11/ambari/centos7/2.7.4.0-118
gpgcheck=0
enabled=1
priority=1
----

image_1ec7ubh9t1d181mvn11gv17i31fguc.png-175.1kB

4.2 配置hdp和hdp-utils 的源

cd /root/software

tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz

mv HDP /var/www/html/

cd /var/www/html/HDP/centos7/3.1.4.0-315

cp -p hdp.repo /etc/yum.repos.d/

cd /root/software

tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz

mv HDP-UTILS /var/www/html/

cd /etc/yum.repos.d/

vim hdp.repo

---
#VERSION_NUMBER=3.1.4.0-315
[HDP-3.1.4.0]
name=HDP Version - HDP-3.1.4.0
baseurl=http://192.168.100.11/HDP/centos7/3.1.4.0-315
gpgcheck=0
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://192.168.100.11/HDP-UTILS/centos7/1.1.0.22
gpgcheck=0
enabled=1
priority=1
---

----

tar -zxvf HDP-GPL-3.1.4.0-centos7-gpl.tar.gz

mv HDP-GPL /var/www/html/

cd /var/www/html/HDP-GPL/centos7/3.1.4.0-315

cp -p hdp.gpl.repo /etc/yum.repos.d/
cd /etc/yum.repos.d/

vim hdp.gpl.repo
----
#VERSION_NUMBER=3.1.4.0-315
[HDP-GPL-3.1.4.0]
name=HDP-GPL Version - HDP-GPL-3.1.4.0
baseurl=http://192.168.100.11/HDP-GPL/centos7/3.1.4.0-315
gpgcheck=0
enabled=1
priority=1
---

cd /etc/yum.repos.d/
scp ambari.repo hdp.repo hdp.gpl.repo node02.flyfish:/etc/yum.repos.d/
scp ambari.repo hdp.repo hdp.gpl.repo node03.flyfish:/etc/yum.repos.d/
scp ambari.repo hdp.repo hdp.gpl.repo node04.flyfish:/etc/yum.repos.d/

yum clean all

yum makecache fast

image_1ec7vo68h1os1g8515k84saillp.png-177.5kB

image_1ec7voiooohnofb1djciu61pa316.png-174.6kB

image_1ec7vp1q1192p919ol1287fr71m.png-166.2kB

image_1efqoh16g178d5751gsqt2p12ge1g.png-214.5kB

4.3 安装mariadb-server 数据库

yum install -y mariadb-server 

systemctl start mariadb

systemctl enable mariadb

设置密码为：flyfish225.com

image_1efqoka15sds18acv301ncv77n2a.png-273.9kB

image_1efqojp26m37otkrgo1udtofd1t.png-126.5kB

image_1efqolsuupe6p29ruc1r9uiej2n.png-200kB

image_1efqomco9ff613m919uj1tfh21734.png-131.4kB

mysql -uroot -pflyfish225.com

ambari 数据库：

create database ambari default character set utf8; 
grant all privileges on ambari.* to 'ambari'@'%' identified by 'Ambari123' with grant option;
grant all privileges on ambari.* to 'ambari'@'node01.flyfish' identified by 'Ambari123' with grant option;
flush privileges; 

use mysql;

UPDATE user SET PASSWORD=PASSWORD('Ambari123') where USER='ambari';

flush privileges;

hive 数据库：

create database hive character set utf8 ;  
CREATE USER 'hive'@'%'IDENTIFIED BY 'Hive-123';
GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
FLUSH PRIVILEGES;

oozie 数据库：

create database oozie character set utf8 ;  
CREATE USER 'oozie'@'%'IDENTIFIED BY 'Oozie-123';
GRANT ALL PRIVILEGES ON *.* TO 'oozie'@'%';
FLUSH PRIVILEGES;

image_1ec800j5h4u0vuc4j4i3e1716.png-150.1kB

image_1ec8010ah1jpt1jf515emp86v1r1j.png-123.5kB

4.3 安装jdbc

拷贝jdbc 到 java的目录
  mkdir /usr/share/java
  mv mysql-connector-java-5.1.49-bin.jar /usr/share/java/
  cd /usr/share/java/
  mv mysql-connector-java-5.1.49-bin.jar mysql-connector-java.jar
  chmod 777 mysql-connector-java.jar

image_1efqor75n1srg1brjnmh50j1r1v3h.png-206.6kB

4.4 安装ambari

yum install -y ambari-server

image_1ec80lj431rn31sha077on1c743n.png-258.9kB

vim /etc/ambari-server/conf/ambari.properties

----
增长：
  server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

---

ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar

image_1ec80o23d1o0onec16dt1lo419g24h.png-101.6kB

ambari-server setup

image_1ec8135c5b321oqdb11fgf6cf4u.png-242.5kB

image_1ec8144k0gn5p2714n91sdjeu25b.png-240.7kB

image_1ec815j1i12br5bpteh1itlr4s65.png-114.7kB

mysql -uroot -pflyfish225.com
use ambari
source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql

image_1ec814t0k167c12o612l57cj3af5o.png-146.7kB

4.5 启动ambari

ambari-server start

image_1ec818bven3bgigtuc16ijq8d6i.png-139.2kB

image_1ec819aigamqinj1chk1nnlglq6v.png-43kB

4.6 打开ambari的web

http://192.168.100.11:8080

用户名：admin
密码： admin

image_1ec81apfr13br1chr95kbj1ve77c.png-119.8kB

image_1ec81bji81kpng3cs4310jg4c27p.png-243.7kB

image_1ec81d0a08csls2boq1dtchqb8i.png-209.3kB

image_1ec81hslg1ljr4fnv9318j818cb8v.png-212.4kB

image_1efqrd3mauqh14551ba0ntp1pp89.png-293.6kB

image_1efqrfk0517ts5el1irngsg1pi8m.png-442.5kB

image_1efqrg73d1qc81n4d1ma21qbm5n713.png-311kB

image_1efqri75cq321g8c17ici1i1n881g.png-339.3kB

选择了hdfs、yarn、zookeeper三个服务，仅供参考：

image_1efr1ff4l22a1njb75fqj31k3d5v.png-538.4kB

image_1efr1esgl1ujbklr1spu1lek7035i.png-342.2kB

image_1efr1gvis1s7914s4rdrrbqao6c.png-355.2kB

image_1efr1ho2p8vb1m4iiobv2f64n6p.png-245.1kB

image_1efr1ihai1j9ntc718jmubm1puj76.png-287.9kB

image_1efr1j59510r61dh64n81gedma97j.png-316.6kB

image_1efr1jkqb19ei1p1516fh145v1n6p80.png-400.6kB

这一步分发要等很长的一段时间

image_1efr1keai11s0joa1lsi1q771fnu8d.png-325.4kB

image_1efr35m76vb41b9j1hmmqg71th38q.png-328.3kB

image_1efr36565pdr1m3amvmpusfg97.png-350.8kB

image_1efrpj80d1huke6r11ok14hronh9.png-571.3kB

相关文章
相关标签/搜索
每日一句
    每一个你不满意的现在，都有一个你没有努力的曾经。
本站公众号
   欢迎关注本站公众号,获取更多信息
联系我们 最近搜索 最新文章 沪ICP备13005482号-10 MyBatis教程 SQL 教程 MySQL教程 Java 教程 Thymeleaf 教程 Hibernate教程 Spring教程 Redis教程