heartbeat+drbd实现磁盘共享自动挂载切换

时间 2019-12-04 标签 heartbeat+drbd heartbeat drbd 实现磁盘共享自动切换

heartbeat 实现 DRBD 主从自动切换

介绍 :heartbeat 和 drbd

若是主服务器宕机,形成的损失是不可估量的。要保证主服务器不间断服务,就须要对服务器实现冗余。在众多的实现服务器冗余的解决方案中,heartbeat 为咱们提供了廉价的、可伸缩的高可用集群方案。咱们经过 heartbeat+drbd 在 Linux下建立一个高可用(HA)的集群服务器。
DRBD 是一种块设备,能够被用于高可用(HA)之中。它相似于一个网络 RAID-1 功能。当你将数据写入本地文件系统时,数据还将会被发送到网络中另外一台主机上。以相同的形式记录在一个文件系统中。本地(主节点)与远程主机(备节点)的数据能够保证明时同步。当本地系统出现故障时,远程主机上还会保留有一份相同的数据,能够继续使用。在高可用(HA)中使用 DRBD 功能,能够代替使用一个共享盘阵。由于数据同时存在于本地主机和远程主机上。切换时,远程主机只要使用它上面的那份备份数据,就能够继续进行服务了。

操做环境 redhat6.5 ，iptables and selinux disabled

服务机：server1 172.25.32.1

server2 172.25.32.2

drdb安装

serever2：

[root@server2 ~]# ls
anaconda-ks.cfg  drbd-8.4.3.tar.gz  nginx  server2
[root@server2 ~]# tar zxf drbd-8.4.3.tar.gz 
[root@server2 ~]# ls
anaconda-ks.cfg  drbd-8.4.3  drbd-8.4.3.tar.gz  nginx  server2
[root@server2 ~]# cd drbd-8.4.3
[root@server2 drbd-8.4.3]# ls
autogen.sh    documentation        filelist-redhat  preamble-sles11
benchmark     drbd                 filelist-suse    README
ChangeLog     drbd_config.h        Makefile.in      rpm-macro-fixes
configure     drbd-kernel.spec.in  preamble         scripts
configure.ac  drbd-km.spec.in      preamble-rhel5   user
COPYING       drbd.spec.in         preamble-sles10
[root@server2 drbd-8.4.3]# ./configure --help |grep enable #过滤编译须要的参数
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-spec           Rather than creating Makefiles, create an RPM spec
                          architecture (makes sense only with --enable-spec,
[root@server2 drbd-8.4.3]# alias grep='grep --color' 
[root@server2 drbd-8.4.3]# ./configure --help |grep km #过滤编译须要的参数
  --with-km               Enable kernel module
[root@server2 drbd-8.4.3]# ./configure --enable-spec --with-km^C
[root@server2 drbd-8.4.3]# yum install rpm-build -y #安转rpm-buid，能够经过生成rpm包的方式安装drbd
[root@server2 drbd-8.4.3]# ./configure --enable-spec --with-km
...
configure: error: no acceptable C compiler found in $PATH  #提示须要gcc编译器
[root@server2 drbd-8.4.3]# yum install gcc -y #安装gcc
[root@server2 drbd-8.4.3]# ./configure --enable-spec --with-km #编译
...
config.status: creating drbd.spec
config.status: creating drbd-km.spec
config.status: creating drbd-kernel.spec
[root@server2 drbd-8.4.3]# rpmbuild --help |grep bb #过滤编译须要的参数
  -bb                           build binary package only from <specfile>
[root@server2 drbd-8.4.3]# rpmbuild -bb drbd.spec
[root@server2 drbd-8.4.3]# cd ..
[root@server2 ~]# ls
anaconda-ks.cfg  drbd-8.4.3  drbd-8.4.3.tar.gz  nginx  rpmbuild  server2
[root@server2 ~]# cp drbd-8.4.3.tar.gz /root/rpmbuild/SOURCES/
[root@server2 ~]# cd -
/root/drbd-8.4.3
[root@server2 drbd-8.4.3]# rpmbuild -bb drbd.spec #生成rpm包
...
Wrote: /root/rpmbuild/RPMS/x86_64/drbd-8.4.3-2.el6.x86_64.rpm   #生成的rpm包存放位置
Wrote: /root/rpmbuild/RPMS/x86_64/drbd-utils-8.4.3-2.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/drbd-xen-8.4.3-2.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/drbd-udev-8.4.3-2.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/drbd-pacemaker-8.4.3-2.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/drbd-heartbeat-8.4.3-2.el6.x86_64.rpm
Wrote: /root/rpmbuild/RPMS/x86_64/drbd-bash-completion-8.4.3-2.el6.x86_64.rpm
...
[root@server2 drbd-8.4.3]# rpmbuild -bb drbd-km.spec
error: Failed build dependencies:
    kernel-devel is needed by drbd-km-8.4.3-2.el6.x86_64    #缺乏kernel-devel依赖包
[root@server2 drbd-8.4.3]# yum install kernel-devel -y #安装依赖包
[root@server2 drbd-8.4.3]# rpmbuild -bb drbd-km.spec
...
Wrote: /root/rpmbuild/RPMS/x86_64/drbd-km-2.6.32_431.el6.x86_64-8.4.3-2.el6.x86_64.rpm  #生成的和内核相关的rpm包
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.SW52f3
...
[root@server2 drbd-8.4.3]# cd /root/rpmbuild/RPMS/x86_64/
[root@server2 x86_64]# ls
drbd-8.4.3-2.el6.x86_64.rpm                  drbd-km-2.6.32_431.el6.x86_64-8.4.3-2.el6.x86_64.rpm  drbd-utils-8.4.3-2.el6.x86_64.rpm
drbd-bash-completion-8.4.3-2.el6.x86_64.rpm  drbd-pacemaker-8.4.3-2.el6.x86_64.rpm                 drbd-xen-8.4.3-2.el6.x86_64.rpm
drbd-heartbeat-8.4.3-2.el6.x86_64.rpm        drbd-udev-8.4.3-2.el6.x86_64.rpm
[root@server2 x86_64]# rpm -ivh drbd-* #安装生成的rpm包 
Preparing...                ########################################### [100%]
   1:drbd-utils             ########################################### [ 13%]
   2:drbd-bash-completion   ########################################### [ 25%]
   3:drbd-heartbeat         ########################################### [ 38%]
   4:drbd-pacemaker         ########################################### [ 50%]
   5:drbd-udev              ########################################### [ 63%]
   6:drbd-xen               ########################################### [ 75%]
   7:drbd                   ########################################### [ 88%]
   8:drbd-km-2.6.32_431.el6.########################################### [100%]
[root@server2 x86_64]# scp .* root@server1:/mnt/ #将生成的rpm包发给server1
并在server1上安装rpm包

drbd的配置

在server1和server2各添加一块磁盘

[root@server2 x86_64]# fdisk -l
Disk /dev/vdc: 5368 MB, 5368709120 bytes   #server2添加的5g的磁盘
16 heads, 63 sectors/track, 10402 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
[root@server1 drbd-8.4.3]# fdisk -l
Disk /dev/vda: 5368 MB, 5368709120 bytes  #server1添加的5g的磁盘
16 heads, 63 sectors/track, 10402 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

server2

[root@server2 x86_64]# cat /etc/drbd.d/drdb.res #建立drbd的资源文件
resource example {
meta-disk internal;
device /dev/drbd1;
syncer {
verify-alg sha1;
}
net {
allow-two-primaries;
}
on server1 {
disk /dev/vda;
address 172.25.32.1:7789;
}
on server2 {
disk /dev/vdc;
address 172.25.32.2:7789;
}
}
[root@server2 x86_64]# scp /etc/drbd.d/drdb.res root@server1:/etc/drbd.d/ #将配置文件发给server1一份

在两台主机上分别执行如下命令

[root@server2 x86_64]# drbdadm create-md example
[root@server2 x86_64]# /etc/init.d/drbd start
Starting DRBD resources: [
     create res: example
   prepare disk: example
    adjust disk: example
     adjust net: example
]
..........    #此时若是server2上没有没有启动drbd，会等待server启动drbd并给与提示
[root@server1 x86_64]# drbdadm create-md example
Writing meta data...
initializing activity log
NOT initializing bitmap
New drbd meta data block successfully created.
[root@server1 drbd-8.4.3]# /etc/init.d/drbd start
Starting DRBD resources: [
     create res: example
   prepare disk: example
    adjust disk: example
     adjust net: example
]
.
[root@server1 drbd-8.4.3]# drbdsetup /dev/drbd1 primary --force #强制将server1设为primary

server1设定为primary时，server自动论为sencondary，而且primary将数据同步给secondary

[root@server2 x86_64]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@server2, 2017-09-21 14:42:19

 1: cs:SyncTarget ro:Secondary/Primary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:4903936 dw:4903936 dr:0 al:0 bm:299 lo:1 pe:4 ua:0 ap:0 ep:1 wo:f oos:338748
    [=================>..] sync'ed: 93.6% (328/5116)M    #同步中.....
    finish: 0:00:04 speed: 76,584 (76,624) want: 57,840 K/sec
[root@server1 drbd-8.4.3]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@server1, 2017-09-21 14:08:20

 1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- #同步完成
    ns:5242684 nr:0 dw:0 dr:5243348 al:0 bm:320 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

同步完成了，就能够正常的使用drbd下的磁盘了,格式化和挂在以及写入数据，格式化，挂载，写入，只能在一边进行，不支持两边同时挂载写入。

[root@server1 drbd-8.4.3]# mkfs.ext4 /dev/drbd1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
327680 inodes, 1310671 blocks
65533 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1342177280
40 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 32 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@server1 drbd-8.4.3]# mount /dev/drbd1 /var/lib/mysql/
[root@server1 drbd-8.4.3]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root   18G  1.9G   15G  12% /
tmpfs                         499M   23M  477M   5% /dev/shm
/dev/sda1                     485M   33M  427M   8% /boot
/dev/drbd1                    5.0G  159M  4.6G   4% /var/lib/mysql
[root@server1 x86_64]# mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)  #数据库服务没有开启
[root@server1 x86_64]# /etc/init.d/mysqld start
Starting mysqld:                                           [  OK  ]
[root@server1 drbd-8.4.3]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3
Server version: 5.1.71 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+---------------------+
| Database            |
+---------------------+
| information_schema  |
| #mysql50#lost+found |
| mysql               |
| test                |
+---------------------+
4 rows in set (0.00 sec)
mysql> create database hh;
Query OK, 1 row affected (0.00 sec)

mysql> show databases;
+---------------------+
| Database            |
+---------------------+
| information_schema  |
| hh                  |
| #mysql50#lost+found |
| mysql               |
| test                |
+---------------------+
5 rows in set (0.00 sec)

mysql> Ctrl-C -- exit!
Aborted
[root@server1 drbd-8.4.3]# df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup-lv_root   18G  1.9G   15G  12% /
tmpfs                         499M   23M  477M   5% /dev/shm
/dev/sda1                     485M   33M  427M   8% /boot
/dev/drbd1                    5.0G  159M  4.6G   4% /var/lib/mysql
[root@server1 drbd-8.4.3]# /etc/init.d/mysqld stop
Stopping mysqld:                                           [  OK  ]
[root@server1 drbd-8.4.3]# umount /var/lib/mysql
[root@server1 drbd-8.4.3]# drbdadm secondary example 
#将server1变为secondary，server2变为primary，server2上才能挂载
[root@server1 drbd-8.4.3]# cat /proc/drbd #查看是否更新过，以及是主仍是辅
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@server1, 2017-09-21 14:08:20

 1: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
    ns:5483972 nr:0 dw:241288 dr:5248344 al:51 bm:320 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

[root@server2 x86_64]# drbdadm primary example --force
[root@server2 x86_64]# cat /proc/drbd 
version: 8.4.3 (api:1/proto:86-101)
GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by root@server2, 2017-09-21 14:42:19

 1: cs:Connected ro:Primary/Primary ds:UpToDate/UpToDate C r-----
    ns:0 nr:5483972 dw:5483972 dr:664 al:0 bm:320 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0  
[root@server2 x86_64]# mount /dev/drbd1 /var/lib/mysql/
[root@server2 x86_64]# /etc/init.d/mysqld start
Starting mysqld:                                           [  OK  ]
[root@server2 x86_64]# mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
[root@server2 x86_64]# ls /var/lib/mysql/
hh  ibdata1  ib_logfile0  ib_logfile1  lost+found  mysql  passwd  test
[root@server2 x86_64]# /etc/init.d/mysqld restart
Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]
[root@server2 x86_64]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.71 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;
+---------------------+
| Database            |
+---------------------+
| information_schema  |
| hh                  | #能够看到server1上新建的数据库hh
| #mysql50#lost+found |
| mysql               |
| test                |
+---------------------+
5 rows in set (0.01 sec)

mysql> Ctrl-C -- exit!
Aborted

注意:两台主机上的/dev/drbd1 不能同时挂载,只有状态为 primary 时,才能被挂载使
用,而此时另外一方的状态为 secondary。node