最少三台Centos7系统虚拟机用于部署Ceph集群。硬件配置:2C4G,另外每台机器最少挂载三块硬盘(每块盘5G).如下为本次环境规划:node
HOST | IP | Service | Mark |
---|---|---|---|
cephmanager01 | public network:192.168.10.51--ens224 <br>cluster network:172.168.10.51--ens192 | mon, mgr,mds | 8c8g<br>系统盘:200G |
cephmanager02 | public network:192.168.10.72--ens224 <br/>cluster network:172.168.10.72--ens192 | mon, mgr,mds | 8c8g<br/>系统盘:200G |
cephmanager03 | public network:192.168.10.75--ens224 <br/>cluster network:172.168.10.75--ens192 | mon, mgr,mds | 8c8g<br/>系统盘:200G |
cephnode001 | public network:192.168.10.76--ens224 <br/>cluster network:172.168.10.76--ens192 | osd | 4c8g<br>系统盘:200G<br>数据盘:50G/块x4 |
cephnode002 | public network:192.168.10.82--ens224 <br/>cluster network:172.168.10.82--ens192 | osd | 4c8g<br/>系统盘:200G<br/>数据盘:50G/块x4 |
cephnode003 | public network:192.168.10.101--ens224 <br/>cluster network:172.168.10.101--ens192 | osd | 4c8g<br/>系统盘:200G<br/>数据盘:50G/块x4 |
组件 | 版本 | 建议配置 | 备注 |
CentOS | CentOS Linux release 7.4.1708 | 管理节点CPU要好 | |
Ceph | 13.2.10 mimic | 数据节点磁盘空间大 |
[在manager节点和node节点执行]python
关闭防火墙并设置开机关闭linux
# systemctl stop firewalld && systemctl disable firewalld && systemctl status firewalld
或者配置防火墙端口(根据状况添加对应端口)shell
# firewall-cmd --zone=public --add-port=6789/tcp --permanent # firewall-cmd --zone=public --add-port=6800-7100/tcp --permanent # firewall-cmd --reload # firewall-cmd --zone=public --list-all
永久关闭(须要重启)bootstrap
# sed -i 's/enforcing/disabled/' /etc/selinux/config
临时关闭vim
# setenforce 0
确认关闭后端
# getenforce
# systemctl disable NetworkManager && systemctl stop NetworkManager && systemctl status NetworkManager
设置主机名[在对应节点上执行]服务器
# hostnamectl --static set-hostname cephmanager01 # hostnamectl --static set-hostname cephmanager02 # hostnamectl --static set-hostname cephmanager03 # hostnamectl --static set-hostname cephnode001 # hostnamectl --static set-hostname cephnode002 # hostnamectl --static set-hostname cephnode003
修改hosts网络
# cat >> /etc/hosts << EOF 192.168.10.51 cephmanager01 192.168.10.72 cephmanager02 192.168.10.75 cephmanager03 192.168.10.76 cephnode001 192.168.10.82 cephnode002 192.168.10.101 cephnode003 EOF
1)设置时区并同步互联网时间
# yum install -y ntpdate chrony # timedatectl set-timezone Asia/Shanghai # ntpdate time1.aliyun.com
2)配置全部服务器的时间和时间服务器同步或者同某一台服务器同步[本例同步cephmanager01节点时间]
编辑 /etc/chrony.conf 文件,添加(互联网上的时间服务器能够删除):
server 192.168.10.51 iburst
重启服务并配置开机启动
# systemctl restart chronyd.service && systemctl enable chronyd.service && systemctl status chronyd.service
验证:是否与cephmanager01同步
# chronyc sources
# echo "ulimit -SHn 102400" >> /etc/rc.local # cat >> /etc/security/limits.conf << EOF * soft nofile 65535 * hard nofile 65535 EOF
# cat >> /etc/sysctl.conf << EOF kernel.pid_max = 4194303 vm.swappiness = 0 EOF # sysctl -p
经过数据预读而且记载到随机访问内存方式提升磁盘读操做
# echo "8192" > /sys/block/sda/queue/read_ahead_kb
SSD要用noop,SATA/SAS使用deadline
# echo "deadline" >/sys/block/sd[x]/queue/scheduler # echo "noop" >/sys/block/sd[x]/queue/scheduler
# yum install wget vim lrzsz -y
# useradd -d /home/cephuser -m cephuser # passwd cephuser Changing password for user cephuser. New password: ceph_pass Retype new password: ceph_pass
修改visudo文件,使cephuser用户在sudo列表中;
# 在92行” root ALL=(ALL) ALL”下新增一行:” cephuser ALL=(ALL) ALL” # visudo cephuser ALL=(ALL) ALL
设置cephuser用户具有无密码sudo(root)权限;切换到cephuser用户下操做
# su - cephuser [cephuser@cephmanager01 ~]$ echo "cephuser ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephuser [sudo] password for cephuser: ceph_pass cephuser ALL = (root) NOPASSWD:AL [cephuser@cephmanager01 ~]$ sudo chmod 0440 /etc/sudoers.d/cephuser
1)生成秘钥
ceph-deploy不支持密码输入,须要在管理控制节点生成ssh秘钥,并将公钥分发到各ceph节点;
在用户cephuser下生成秘钥,不能使用sudo或root用户;默认在用户目录下生成~/.ssh目录,含生成的秘钥对;“Enter passphrase”时,回车,口令为空;
备注:3个manager节点均设置为ceph管理控制节点,应该使3个manager节点均能ssh免密登录到其余全部manager节点与node节点
# 分别在cephmanager01,cephmanager02,cephmanager03执行 [root@cephmanager01 ~]# su - cephuser Last login: Sun Jun 28 14:49:09 CST 2020 on pts/4 [cephuser@cephmanager01 ~]$ ssh-keygen -t rsa
2)分发密钥
确认各控制manger与node节点已生成相关用户cephuser;分发成功后,在~/.ssh/下生成known_hosts文件,记录相关登录信息;
# 须要分别在cephmanager0一、cephmanager0二、cephmanager03上分发到全部的manager和node节点 [cephuser@cephmanager01 ~]$ ssh-copy-id cephuser@cephmanager02
配置内网yum源服务器,或者配置对应的国内源,这里配置ceph源和epel源
# cat > /etc/yum.repos.d/ceph.repo << EOF [ceph] name=ceph baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/x86_64/ gpgcheck=0 priority=1 [ceph-noarch] name=cephnoarch baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch/ gpgcheck=0 priority=1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-mimic/el7/SRPMS gpgcheck=0 priority=1 EOF # wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo # yum clean all # yum makecache # 查看yum源 # yum repolist
[在规划的所有manager节点安装ceph-deploy工具,本例以cephmanager01为例]
推配置文件、变动维护,添加硬盘等操做都须要用到ceph-deploy,并确认ceph-deploy版本是否为2.0.1
[root@cephmanager01 ~]# yum install ceph-deploy python-setuptools -y # 确认版本 [root@cephmanager01 ~]# ceph-deploy --version
[在规划的所有manager节点执行,特殊说明处例外,本例以cephmanager01为例]
在cephuser帐户下操做,切忌使用sudo操做;在manager节点上建立一个目录用于存放集群相关配置文件;
[root@cephmanager01 ~]# su - cephuser [cephuser@cephmanager01 ~]$ mkdir cephcluster # 后续ceph-deploy相关操做所有在所建立的目录执行 [cephuser@cephmanager01 ~]$ cd cephcluster/ # 将规划中的MON(monitor)节点归入集群,即建立集群, # [该操做任意一个cephmanager执行,以cephmanage01为例] [cephuser@cephmanager01 cephcluster]$ ceph-deploy new cephmanager01 cephmanager02 cephmanager03 [cephuser@cephmanager01 cephcluster]$ ls ceph.conf ceph-deploy-ceph.log ceph.mon.keyrin
生成集群后在集群目录下生成3个文件,其中ceph.conf便是配置文件;
默承认不修改,为使服务按规划启动,可作适当修改;
如下红色字体部分是在默认生成的conf文件上新增的配置
[该操做任意一个cephmanager执行,以cephmanage01为例]
[cephde@controller01 cephcluster]$ cat ceph.conf [global] fsid = 6a71324c-6949-42bc-92ca-325c9d45ae06 mon_initial_members = cephmanager01, cephmanager02, cephmanager03 mon_host = 192.168.10.51,192.168.10.72,192.168.10.75 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx # public network:前端mon网络,client访问网络;确保public network与mon_host在相同网段,不然初始化时可能会有错误; # cluster network:后端osd心跳,数据/流复制恢复等网络 public network = 192.168.10.0/24 cluster network = 172.168.10.0/24 # 默认的副本数为3,根据实际状况变动 osd pool default size = 3 # 默认保护机制不容许删除pool,根据状况设置 mon_allow_pool_delete = true
[在所有manager和node节点执行]
# 建议使用该命令安装在每一个节点安装 [root@cephmanager01 ~]# yum -y install epel-release [root@cephmanager01 ~]# yum -y install ceph # 不建议使用,该命令会在对应节点安装,可是源会自动更换为国外官方源,网速会很慢 [root@cephmanager01 ~]# ceph-deploy install --release mimic cephnode01 cephnode02 cephnode03 # 查看版本 [root@controller01 ~]# ceph -v
[该操做在任意一个cephmanager执行,以cephmanage01为例]
[root@cephmanager01 ~]# su - cephuser Last login: Mon Jun 29 02:17:20 CST 2020 on pts/0 [cephuser@cephmanager01 ~]$ cd cephcluster/ [cephuser@cephmanager01 cephcluster]$ ceph-deploy mon create-initial # 初始化完成后,在集群目录下新增多个秘钥文件 [cephuser@cephmanager01 cephcluster]$ ls -l
# 查看状态 [cephuser@cephmanager01 cephcluster]$ sudo systemctl status ceph-mon@cephmanager01
# 分发ceph配置文件与秘钥到其余控制manager节点与node节点; # 注意分发节点自己也须要包含在内,默认没有秘钥文件,须要分发; # 若是被分发节点已经配置文件(统一变动配置文件场景),可使用以下命令:ceph-deploy --overwrite-conf admin xxx # 分发的配置文件与秘钥到各节点/etc/ceph/目录 [cephuser@cephmanager01 cephcluster]$ ceph-deploy admin cephmanager01 cephmanager02 cephmanager03 cephnode001 cephnode002 cephnode003
完后可使用ceph -s命令,tail -f ceph-deploy-ceph.log查看安装日志
将cephcluster目录下面的文件拷贝到其他的manager节点:
[cephuser@cephmanager01 cephcluster]$ pwd /home/cephuser/cephcluster [cephuser@cephmanager01 cephcluster]$ scp ceph.conf ceph-deploy-ceph.log ceph.mon.keyring cephuser@cephmanager03:~/cephcluster/ [cephuser@cephmanager01 cephcluster]$ scp ceph.bootstrap-* cephuser@cephmanager02:~/cephcluster/
注意:L版以后才须要部署
[在任意manager节点执行,以cephmanager01为例]
[cephuser@cephmanager01 cephcluster]$ ceph-deploy mgr create cephmanager01:cephmanager01_mgr cephmanager02:cephmanager02_mgr cephmanager03:cephmanager03_mgr # 查看状态 [cephuser@cephmanager01 cephcluster]$ systemctl status ceph-mgr@cephmanager01_mgr [cephuser@cephmanager01 cephcluster]$ sudo ss -tunlp |grep mgr
[manager和node节点都可执行]
# 查看monitor状态 [cephuser@cephmanager01 ~]$ sudo ceph mon stat # 查看ceph状态 [cephuser@cephmanager01 ~]$ sudo ceph -s
osd位于node节点,可查看node节点磁盘情况,以cephnode001节点为例;
或在manager节点采用命令:ceph-deploy disk list cephnode001 cephnode001 … cephnode00N;
[root@cephnode001 ~]# lsblk
实际建立osd时,可经过manager节点使用ceph-deploy建立[任意一manager节点执行,以cephmanager01为例];
# 参数"--data"指定数据盘,参数"--journal"指定journal日志盘,日志盘能够是logical volume(vg/lv)或GPT patition,是option操做; # 另有命令(L版本):ceph-deploy osd prepare {NODE}:{DISK}[:{/PATH/TO/JOURNAL}]; ceph-deploy osd activate {NODE}:{DISK}[:{/PATH/TO/JOURNAL}],其中[:{/PATH/TO/JOURNAL}]是option操做,可建立具备journal日志的osd守护进程(节点) [cephuser@cephmanager01 cephcluster]$ ceph-deploy osd create cephnode001 --data /dev/sdb [cephuser@cephmanager01 cephcluster]$ ceph-deploy osd create cephnode001 --data /dev/sdc [cephuser@cephmanager01 cephcluster]$ ceph-deploy osd create cephnode001 --data /dev/sdd [cephuser@cephmanager01 cephcluster]$ ceph-deploy osd create cephnode001 --data /dev/sde
本例中有3个node节点(osd),每一个node节点运行4个osd进程(在6800~7300端口范围内,每进程监听1个本地端口)[以cephnode001为例];
[root@cephnode001 ~]# lsblk [root@cephnode001 ~]# ps -aux |grep osd # osd进程端口号; [root@cephnode001 ~]# netstat -tunlp | grep osd # ceph-osd进程,根据启动顺序,每一个osd进程有特定的序号 [root@cephnode001 ~]# systemctl status ceph-osd@0
若是节点磁盘有数据,可经过命令初始化(谨慎):ceph-deploy disk zap {NODE} {DISK},这里{DISK}能够是物理盘符,也但是分区
在manager节点查看,以cephmanager01为例
[cephuser@cephmanager01 cephcluster]$ ceph-deploy osd list cephnode001 [cephuser@cephmanager01 cephcluster]$ sudo ceph osd stat [cephuser@cephmanager01 cephcluster]$ sudo ceph osd tree [cephuser@cephmanager01 cephcluster]$ sudo ceph df
在node节点查看,以cephnode001为例
[root@cephnode001 ~]# lsblk [root@cephnode001 ~]# ps -aux |grep osd [root@cephnode001 ~]# netstat -tunlp | grep osd [root@cephnode001 ~]# systemctl status ceph-osd@0