MariaDB Galera Cluster集群搭建

时间 2021-01-09

标签 html node python mysql linux c++ sql 数据库 bootstrap ubuntu 栏目负载均衡繁體版

原文原文链接

MariaDB Galera Cluster是什么？ html

Galera Cluster是由第三方公司Codership所研发的一套免费开源的集群高可用方案，实现了数据零丢失，官网地址为http://galeracluster.com/node

其在MySQLInnoDB存储引擎基础上打了wrep(虚拟全同步复制)，Percona/MariaDB已捆绑在各自的发行版本中python

MariaDB Galera Cluster是MariaDB同步多主机集群。它仅支持XtraDB/InnoDB存储引擎（虽然有对MyISAM实验支持，具体看wsrep_replicate_myisam系统变量）mysql

MariaDB Galera Cluster主要功能linux

1.同步复制c++

2.真正的multi-master，即全部节点能够同时读写数据库sql

3. 自动的节点成员控制，失效节点自动被清除数据库

4. 新节点加入数据自动复制bootstrap

5.真正的并行复制，行级ubuntu

6.用户能够直接链接集群，使用感觉上与MySQL彻底一致

优点

由于是多主，因此不存在Slavelag(延迟)
不存在丢失事务的状况
同时具备读和写的扩展能力
更小的客户端延迟
节点间数据是同步的，而Master/Slave模式是异步的，不一样slave上的binlog多是不一样的

缺点

加入新节点时开销大，须要复制完整的数据
不能有效地解决写扩展的问题，全部的写操做都发生在全部的节点
有多少个节点，就有多少份重复的数据
因为事务提交须要跨节点通讯，即涉及分布式事务操做，所以写入会比主从复制慢不少，节点越多，写入越慢，死锁和回滚也会更加频繁
对网络要求比较高，若是网络出现波动不稳定，则可能会形成两个节点失联，Galera Cluster集群会发生脑裂，服务将不可用

存在局限

仅支持InnoDB/XtraDB存储引擎，任何写入其余引擎的表，包括mysql.*表都不会被复制。可是DDL语句能够复制，可是insert into mysql.user(MyISAM存储引擎)之类的插入数据不会被复制
Delete操做不支持没有主键的表，由于没有主键的表在不一样的节点上的顺序不一样，若是执行select … limit …将出现不一样的结果集
LOCK/UNLOCK TABLES/FLUSH TABLES WITH READ LOCKS不支持单表所锁，以及锁函数GET_LOCK()、RELEASE_LOCK()，但FLUSH TABLES WITH READ LOCK支持全局表锁
General Query Log日志不能保存在表中，若是开始查询日志，则只能保存到文件中
不能有大事务写入，不能操做wsrep_max_ws_rows=131072(行)，且写入集不能超过wsrep_max_ws_size=1073741824(1GB)，不然客户端直接报错
因为集群是乐观锁并发控制，所以，在commit阶段会有事务冲突发生。若是两个事务在集群中的不一样节点上对同一行写入并提交，则失败的节点将回滚，客户端返回死锁报错
XA分布式事务不支持Codership Galera Cluster，在提交时可能会回滚
整个集群的写入吞吐量取决于最弱的节点限制，集群要使用同一的配置

服务器环境说明

系统版本	MariaDB版本	主机IP	节点名称
CentOS 7.9	mariadb-10.3.27	192.168.40.50	MariaDB-Node1
CentOS 7.9	mariadb-10.3.27	192.168.40.60	MariaDB-Node2
CentOS 7.9	mariadb-10.3.27	192.168.40.70	MariaDB-Node3

环境准备

[root@MariaDB-Node1 ~]# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
[root@MariaDB-Node1 ~]# setenforce 0
 setenforce: SELinux is disabled
[root@MariaDB-Node1 ~]# systemctl stop firewalld.service
[root@MariaDB-Node1 ~]# systemctl disable firewalld.service
[root@MariaDB-Node1 ~]# vi /etc/security/limits.conf

* soft nofile 65536  
* hard nofile 65536  

[root@MariaDB-Node1 ~]#sysctl -p

安装MariaDB

参考我以前的博客安装，采用二进制安装

https://www.cnblogs.com/mike666/p/13666469.html

三台数据库主机同时安装 MariaDB 10.3.27数据库

备注：从MariaDB 10.1.20 版本开始，Galera Cluster就已经包含在MariaDB包里面了，不须要单独部署MariaDB-Galera-server 和galera 包

安装 Galera 依赖包

安装依赖包
[root@MariaDB-Node1 ~]# yum -y install  lsof cmake  gcc gcc-c++  autoconf automake zlib*  libxml* \
ncurses ncurses-devel libtool  libtool-ltdl-devel* make  bison bison-devel  \
openssl-devel libevent-devel \
libaio libaio-devel pam-devel boost boost-devel valgrind-devel \
libnl-devel popt-devel popt-static bzr nmap-* check-devel 


安装Scons
[root@MariaDB-Node1 ~]# wget http://prdownloads.sourceforge.net/scons/scons-2.5.0.tar.gz
[root@MariaDB-Node1 ~]# tar zxvf scons-2.5.0.tar.gz -C /usr/local/ && rm -rf scons-2.5.0.tar.gz
[root@MariaDB-Node1 ~]# cd /usr/local/scons-2.5.0/
[root@MariaDB-Node1 /usr/local/scons-2.5.0]# python setup.py install
[root@MariaDB-Node1 /usr/local/scons-2.5.0]# cd 
[root@MariaDB-Node1 ~]# scons -v
SCons by Steven Knight et al.:
    script: v2.5.0.rel_2.5.0:3544:95d356f188a3[MODIFIED], 2016/04/09 14:38:50, by bdbaddog on ubuntu1404-32bit
    engine: v2.5.0.rel_2.5.0:3544:95d356f188a3[MODIFIED], 2016/04/09 14:38:50, by bdbaddog on ubuntu1404-32bit
    engine path: ['/usr/lib/scons-2.5.0/SCons']
Copyright (c) 2001 - 2016 The SCons Foundation
[root@MariaDB-Node1 ~]#

三台都要安装这个依赖包，否则启动集群可能会报错

修改配置文件my.cnf

下面咱们开始配置MariaDB Galera Cluster，分别修改MariaDB Galera集群的每一个节点上的/etc/my.cnf文件，具体每一个节点的内容以下：

192.168.40.50 配置文件修改：

[root@MariaDB-Node1 ~]# vim /etc/my.cnf


[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
user = mysql
datadir = /data/mysql/data
pid-file = /data/mysql/mysql.pid
server-id = 50
relay_log =/data/logs/mysql/mysql_relay.log
bind-address = 0.0.0.0

tmpdir=/data/tmp

init_connect ='SET NAMES utf8'
character-set-server = utf8
skip-name-resolve
back_log = 300


max_connections = 8019
max_connect_errors = 1024000
open_files_limit = 65535
table_open_cache = 2048
max_allowed_packet = 50M
max_heap_table_size = 512M
tmp_table_size = 256M


read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
key_buffer_size = 512M
thread_cache_size = 128


query_cache_type = 2
query_cache_size = 128M
query_cache_limit = 5M
thread_stack = 192k
ft_min_word_len = 4


log_bin = /data/mysql/mysql_bin.log
binlog_format = ROW
expire_logs_days = 7

log_error = /data/logs/mysql/mysql_error.log
slow_query_log = 1
long_query_time = 1
log_slow_verbosity=query_plan
slow_query_log_file = /data/logs/mysql/mysql_slow.log
performance_schema = 0


skip-external-locking #跳过外部锁定，避免external locking

bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1


default_storage_engine = InnoDB
innodb_file_per_table = 1
innodb_open_files = 500
innodb_buffer_pool_size = 2048M
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_purge_threads = 1
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 10
innodb_autoinc_lock_mode = 2
innodb_doublewrite = 1
###New ADD
innodb_rollback_on_timeout = 1
innodb_force_recovery=0
interactive_timeout = 28800
wait_timeout = 120


[galera]
wsrep_provider=/usr/local/mysql/lib/galera/libgalera_smm.so                        # Galera library 安装包自带有
wsrep_cluster_address="gcomm://192.168.40.50,192.168.40.60,192.168.40.70"          # Galera cluster 集群地址
wsrep_cluster_name='Galera_Cluster'                                                # Galera集群名称
wsrep_node_address='192.168.40.50'                                                 # 该节点的地址
wsrep_node_name='MariaDB-Node1'                                                    # 该节点的主机名
wsrep_slave_threads = 8                                   #开启并行复制线程,根据CPU核数设置
wsrep_log_conflicts=1
wsrep_on=ON                                               #开启全同步复制模式
wsrep_provider_options="gcache.size=2G"                   #同步复制缓冲池
wsrep_forced_binlog_format=ROW
wsrep_drupal_282555_workaround=1
wsrep_max_ws_size=2147483647
wsrep_sst_method=rsync                                    #拷贝模式
wsrep_sst_auth=galera:galera                              #galera集群认证用户:密码
query_cache_size=0                                        #关闭查询缓存
default_storage_engine=InnoDB                             #Mariadb存储引擎
#wsrep_dirty_reads=0    #当值是1的时候这个节点是只读节点
#wsrep_sst_method=[mysqldump|xtrabackup|xtrabackup-v2|rsync|rsync_wan]
#wsrep_sst_method=xtrabackup-v2



[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 8M
read_buffer = 4M
write_buffer = 4M


[root@MariaDB-Node1 ~]#

192.168.40.60配置文件修改：

[root@MariaDB-Node2 ~]# vim /etc/my.cnf

[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
user = mysql
datadir = /data/mysql/data
pid-file = /data/mysql/mysql.pid
server-id = 60
relay_log =/data/logs/mysql/mysql_relay.log
bind-address = 0.0.0.0

tmpdir=/data/tmp



init_connect ='SET NAMES utf8'
character-set-server = utf8
skip-name-resolve
back_log = 300


max_connections = 8019
max_connect_errors = 1024000
open_files_limit = 65535
table_open_cache = 2048
max_allowed_packet = 50M
max_heap_table_size = 512M
tmp_table_size = 256M


read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
key_buffer_size = 512M
thread_cache_size = 128


query_cache_type = 2
query_cache_size = 128M
query_cache_limit = 5M
thread_stack = 192k
ft_min_word_len = 4


log_bin = /data/mysql/mysql_bin.log
binlog_format = ROW
expire_logs_days = 7

log_error = /data/logs/mysql/mysql_error.log
slow_query_log = 1
long_query_time = 1
log_slow_verbosity=query_plan
slow_query_log_file = /data/logs/mysql/mysql_slow.log
performance_schema = 0


skip-external-locking #跳过外部锁定，避免external locking

bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1


default_storage_engine = InnoDB
innodb_file_per_table = 1
innodb_open_files = 500
innodb_buffer_pool_size = 2048M
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_purge_threads = 1
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 10
innodb_autoinc_lock_mode = 2
innodb_doublewrite = 1

innodb_rollback_on_timeout = 1
innodb_force_recovery=0
interactive_timeout = 28800
wait_timeout = 120



[galera]
wsrep_provider=/usr/local/mysql/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://192.168.40.50,192.168.40.60,192.168.40.70"
wsrep_cluster_name='Galera_Cluster'
wsrep_node_address='192.168.40.60'
wsrep_node_name='MariaDB-Node2'
wsrep_slave_threads = 8                                   #开启并行复制线程,根据CPU核数设置
wsrep_log_conflicts=1
wsrep_on=ON                                               #开启全同步复制模式
wsrep_provider_options="gcache.size=2G"                   #同步复制缓冲池
wsrep_forced_binlog_format=ROW
wsrep_drupal_282555_workaround=1
wsrep_max_ws_size=2147483647
wsrep_sst_method=rsync                                    #拷贝模式
wsrep_sst_auth=galera:galera                              #galera集群认证用户:密码
query_cache_size=0                                        #关闭查询缓存
default_storage_engine=InnoDB                             #Mariadb存储引擎
#wsrep_dirty_reads=0    #当值是1的时候这个节点是只读节点
#wsrep_sst_method=[mysqldump|xtrabackup|xtrabackup-v2|rsync|rsync_wan]
#wsrep_sst_method=xtrabackup-v2

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 8M
read_buffer = 4M
write_buffer = 4M



[root@MariaDB-Node2 ~]#

192.168.40.70配置文件修改：

[root@MariaDB-Node3 ~]# vim /etc/my.cnf

[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
user = mysql
datadir = /data/mysql/data
pid-file = /data/mysql/mysql.pid
server-id = 70
relay_log =/data/logs/mysql/mysql_relay.log
bind-address = 0.0.0.0

tmpdir=/data/tmp



init_connect ='SET NAMES utf8'
character-set-server = utf8
skip-name-resolve
back_log = 300


max_connections = 8019
max_connect_errors = 1024000
open_files_limit = 65535
table_open_cache = 2048
max_allowed_packet = 50M
max_heap_table_size = 512M
tmp_table_size = 256M


read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
key_buffer_size = 512M
thread_cache_size = 128


query_cache_type = 2
query_cache_size = 128M
query_cache_limit = 5M
thread_stack = 192k
ft_min_word_len = 4


log_bin = /data/mysql/mysql_bin.log
binlog_format = ROW
expire_logs_days = 7

log_error = /data/logs/mysql/mysql_error.log
slow_query_log = 1
long_query_time = 1
log_slow_verbosity=query_plan
slow_query_log_file = /data/logs/mysql/mysql_slow.log
performance_schema = 0


skip-external-locking #跳过外部锁定，避免external locking

bulk_insert_buffer_size = 64M
myisam_sort_buffer_size = 128M
myisam_max_sort_file_size = 10G
myisam_repair_threads = 1


default_storage_engine = InnoDB
innodb_file_per_table = 1
innodb_open_files = 500
innodb_buffer_pool_size = 2048M
innodb_write_io_threads = 8
innodb_read_io_threads = 8
innodb_thread_concurrency = 16
innodb_purge_threads = 1
innodb_flush_log_at_trx_commit = 1
innodb_log_buffer_size = 8M
innodb_log_file_size = 256M
innodb_log_files_in_group = 3
innodb_max_dirty_pages_pct = 90
innodb_lock_wait_timeout = 10
innodb_autoinc_lock_mode = 2
innodb_doublewrite = 1

innodb_rollback_on_timeout = 1
innodb_force_recovery=0
interactive_timeout = 28800
wait_timeout = 120




[galera]
wsrep_provider=/usr/local/mysql/lib/galera/libgalera_smm.so
wsrep_cluster_address="gcomm://192.168.40.50,192.168.40.60,192.168.40.70"
wsrep_cluster_name='Galera_Cluster'
wsrep_node_address='192.168.40.70'
wsrep_node_name='MariaDB-Node3'
wsrep_slave_threads = 8                                   #开启并行复制线程,根据CPU核数设置
wsrep_log_conflicts=1
wsrep_on=ON                                               #开启全同步复制模式
wsrep_provider_options="gcache.size=2G"                   #同步复制缓冲池
wsrep_forced_binlog_format=ROW
wsrep_drupal_282555_workaround=1
wsrep_max_ws_size=2147483647
wsrep_sst_method=rsync                                    #拷贝模式
wsrep_sst_auth=galera:galera                              #galera集群认证用户:密码
query_cache_size=0                                        #关闭查询缓存
default_storage_engine=InnoDB                             #Mariadb存储引擎
#wsrep_dirty_reads=0    #当值是1的时候这个节点是只读节点
#wsrep_sst_method=[mysqldump|xtrabackup|xtrabackup-v2|rsync|rsync_wan]
#wsrep_sst_method=xtrabackup-v2

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer_size = 512M
sort_buffer_size = 8M
read_buffer = 4M
write_buffer = 4M


[root@MariaDB-Node3 ~]#

有几个参数须要修改的：

server-id = 50
wsrep_node_address='192.168.40.50'
wsrep_node_name='MariaDB-Node1'

修改对应的 server ID 、IP地址和主机名，其余的复制第一份到别的机器修改就能够

初始化集群

在MariaDB一个节点初始化安装 192.168.40.50 （其余节点不用操做）

[root@MariaDB-Node1 ~]# /usr/local/mysql/scripts/mysql_install_db --defaults-file=/etc/my.cnf --user=mysql

在 192.168.40.50 节点上经过bootstrap启动（第一次启动必定要使用 --wsrep-new-cluster，再次启动就不须要）

[root@MariaDB-Node1 ~]#  mysqld_safe --defaults-file=/etc/my.cnf --user=mysql  --wsrep-new-cluster &

其余节点操做

初始化节点（192.168.40.50）操做启动以后，别的主机只要正常启动就能够了

192.168.40.60操做：

[root@MariaDB-Node2 ~]# /etc/init.d/mysql start

192.168.40.70操做：

[root@MariaDB-Node3 ~]# /etc/init.d/mysql start

192.168.40.50操做：

初始化节点等待其余节点正常启动以后，再关闭带参数启动数据库，再已正常方式启动

[root@MariaDB-Node1 ~]# ps -ef|grep mysql
[root@MariaDB-Node1 ~]# kill 8316
[root@MariaDB-Node1 ~]# ps -ef|grep mysql
[root@MariaDB-Node1 ~]# /etc/init.d/mysql start

查看集群状态

[root@MariaDB-Node1 ~]# mysql -uroot -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 17
Server version: 10.3.27-MariaDB-log MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> 
MariaDB [(none)]> 
MariaDB [(none)]> SHOW STATUS LIKE 'wsrep_cluster_size';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 3     |
+--------------------+-------+
1 row in set (0.001 sec)

MariaDB [(none)]> show global status like 'ws%';
+-------------------------------+----------------------------------------------------------+
| Variable_name                 | Value                                                    |
+-------------------------------+----------------------------------------------------------+
| wsrep_applier_thread_count    | 8                                                        |
| wsrep_apply_oooe              | 0.000000                                                 |
| wsrep_apply_oool              | 0.000000                                                 |
| wsrep_apply_window            | 0.000000                                                 |
| wsrep_causal_reads            | 0                                                        |
| wsrep_cert_deps_distance      | 0.000000                                                 |
| wsrep_cert_index_size         | 0                                                        |
| wsrep_cert_interval           | 0.000000                                                 |
| wsrep_cluster_conf_id         | 5                                                        |
| wsrep_cluster_size            | 3                                                        |
| wsrep_cluster_state_uuid      | a4d25ee0-515c-11eb-b451-66b7a1b050e2                     |
| wsrep_cluster_status          | Primary                                                  |
| wsrep_cluster_weight          | 3                                                        |
| wsrep_commit_oooe             | 0.000000                                                 |
| wsrep_commit_oool             | 0.000000                                                 |
| wsrep_commit_window           | 0.000000                                                 |
| wsrep_connected               | ON                                                       |
| wsrep_desync_count            | 0                                                        |
| wsrep_evs_delayed             |                                                          |
| wsrep_evs_evict_list          |                                                          |
| wsrep_evs_repl_latency        | 0.000308077/0.00394484/0.0110307/0.00501106/3            |
| wsrep_evs_state               | OPERATIONAL                                              |
| wsrep_flow_control_active     | false                                                    |
| wsrep_flow_control_paused     | 0.000000                                                 |
| wsrep_flow_control_paused_ns  | 0                                                        |
| wsrep_flow_control_recv       | 0                                                        |
| wsrep_flow_control_requested  | false                                                    |
| wsrep_flow_control_sent       | 0                                                        |
| wsrep_gcomm_uuid              | 4fd60274-523f-11eb-b7dd-476f9c9f0398                     |
| wsrep_gmcast_segment          | 0                                                        |
| wsrep_incoming_addresses      | 192.168.40.60:3306,192.168.40.70:3306,192.168.40.50:3306 |
| wsrep_last_committed          | 0                                                        |
| wsrep_local_bf_aborts         | 0                                                        |
| wsrep_local_cached_downto     | 18446744073709551615                                     |
| wsrep_local_cert_failures     | 0                                                        |
| wsrep_local_commits           | 0                                                        |
| wsrep_local_index             | 2                                                        |
| wsrep_local_recv_queue        | 0                                                        |
| wsrep_local_recv_queue_avg    | 0.000000                                                 |
| wsrep_local_recv_queue_max    | 1                                                        |
| wsrep_local_recv_queue_min    | 0                                                        |
| wsrep_local_replays           | 0                                                        |
| wsrep_local_send_queue        | 0                                                        |
| wsrep_local_send_queue_avg    | 0.000000                                                 |
| wsrep_local_send_queue_max    | 1                                                        |
| wsrep_local_send_queue_min    | 0                                                        |
| wsrep_local_state             | 4                                                        |
| wsrep_local_state_comment     | Synced                                                   |
| wsrep_local_state_uuid        | a4d25ee0-515c-11eb-b451-66b7a1b050e2                     |
| wsrep_open_connections        | 0                                                        |
| wsrep_open_transactions       | 0                                                        |
| wsrep_protocol_version        | 9                                                        |
| wsrep_provider_name           | Galera                                                   |
| wsrep_provider_vendor         | Codership Oy <info@codership.com>                        |
| wsrep_provider_version        | 25.3.31(r0ede97d)                                        |
| wsrep_ready                   | ON                                                       |
| wsrep_received                | 2                                                        |
| wsrep_received_bytes          | 306                                                      |
| wsrep_repl_data_bytes         | 0                                                        |
| wsrep_repl_keys               | 0                                                        |
| wsrep_repl_keys_bytes         | 0                                                        |
| wsrep_repl_other_bytes        | 0                                                        |
| wsrep_replicated              | 0                                                        |
| wsrep_replicated_bytes        | 0                                                        |
| wsrep_rollbacker_thread_count | 1                                                        |
| wsrep_thread_count            | 9                                                        |
+-------------------------------+----------------------------------------------------------+
66 rows in set (0.001 sec)

MariaDB [(none)]>

能够看到集群正常使用有三个节点

注释：

wsrep_cluster_status为Primary，表示节点为主节点，正常读写

wsrep_ready为ON，表示集群正常运行

wsrep_cluster_size为 3，表示集群有三个节点

遇到问题

若是数据库 所有关机 了，再次启动能够会遇到以下报错

[root@MariaDB-Node1 ~]# cd /data/logs/mysql/
[root@MariaDB-Node1 /data/logs/mysql]# tail -100f mysql_error.log

2021-01-09 13:49:33 0 [Note] WSREP: Read nil XID from storage engines, skipping position init
2021-01-09 13:49:33 0 [Note] WSREP: wsrep_load(): loading provider library '/usr/local/mysql/lib/galera/libgalera_smm.so'
2021-01-09 13:49:33 0 [Note] WSREP: wsrep_load(): Galera 25.3.31(r0ede97d) by Codership Oy <info@codership.com> loaded successfully.
2021-01-09 13:49:33 0 [Note] WSREP: CRC-32C: using 64-bit x86 acceleration.
2021-01-09 13:49:33 0 [Note] WSREP: Found saved state: a4d25ee0-515c-11eb-b451-66b7a1b050e2:0, safe_to_bootstrap: 0
2021-01-09 13:49:33 0 [Note] WSREP: Passing config to GCS: base_dir = /data/mysql/data/; base_host = 192.168.40.50; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = yes; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/data/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /data/mysql/data//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = 
2021-01-09 13:49:33 0 [Note] WSREP: Assign initial position for certification: 0, protocol version: -1
2021-01-09 13:49:33 0 [Note] WSREP: wsrep_sst_grab()
2021-01-09 13:49:33 0 [Note] WSREP: Start replication
2021-01-09 13:49:33 0 [Note] WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster
2021-01-09 13:49:33 0 [Note] WSREP: Setting initial position to a4d25ee0-515c-11eb-b451-66b7a1b050e2:0
2021-01-09 13:49:33 0 [ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1 .
2021-01-09 13:49:33 0 [ERROR] WSREP: wsrep::connect(gcomm://192.168.40.50,192.168.40.60,192.168.40.70) failed: 7
2021-01-09 13:49:33 0 [ERROR] Aborting

解决办法

[root@MariaDB-Node1 /data/logs/mysql]# cd /data/mysql/data/
[root@MariaDB-Node1 /data/mysql/data]# cat grastate.dat 
# GALERA saved state
version: 2.1
uuid:    a4d25ee0-515c-11eb-b451-66b7a1b050e2
seqno:   -1
safe_to_bootstrap: 0
[root@MariaDB-Node1 /data/mysql/data]# 

把 safe_to_bootstrap: 0 修改成 1

[root@MariaDB-Node1 /data/mysql/data]# vim grastate.dat 
# GALERA saved state
version: 2.1
uuid:    a4d25ee0-515c-11eb-b451-66b7a1b050e2
seqno:   -1
safe_to_bootstrap: 1
[root@MariaDB-Node1 /data/mysql/data]# mysqld_safe --defaults-file=/etc/my.cnf --user=mysql  --wsrep-new-cluster &

把 safe_to_bootstrap: 0 修改成 1，而后再用带参数启动就能够，其余节点使用正常方法启动

mysql galera 集群常见问题处理

1、mysql HA集群在断网太久或者全部节点都down了以后的恢复有如下的方法：
解决方案1：
一、等三台机器恢复网络通信后，由于此时的mysql已经异常没法加入集群，所以须要先保证全部的mysql都是down的，再上台执行 /usr/local/mysql/bin/mysqld --wsrep-new-cluster --wsrep-cluster-address='gcomm://' & 这条命令，并进入mysql（只有一台机器可以成功执行，其余机器执行了过几秒钟都会异常退出这个进程，咱们这里把可以成功执行的机器称为master）
二、此时三台只有一台可以成功进入mysql（即执行mysql这条命令），在非master上的两台上一台一台的执行/etc/init.d/mysql start，必须等一台成功了，另外一台才能执行

三、在mysql中执行show status like "wsrep%";

咱们须要保证的第一项为synced，以及第二项必须为三个mysql的ip

四、保证3的结果是想要的说明集群已经恢复了，此时须要将master机器上面的 /usr/local/mysql/bin/mysqld --wsrep-new-cluster --wsrep-cluster-address='gcomm://'这个进程kill掉，而后再执行 /etc/init.d/mysql start 便可

2、mysql HA集群某个节点无端down了而且有一段时间处于down的状况经过如下方式恢复：

一、若日志里面出现如下日志

[Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (eb9f50c6-bc95-11e5-a735-9f48e437dc03): 1 (Operation not permitted)

解决方法：删除/var/lib/mysql/grastate.dat 文件（若还存在没法同步的状况则删除galera.cache文件）

二、若那个down了的节点出现如下日志

（异常状况集群挂了）[ERROR] Found 1 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions

解决方法：

一、/usr/local/mysql/bin/mysqld start --innodb_force_recovery=6

(SRV_FORCE_IGNORE_CORRUPT):忽略检查到的corrupt页
(SRV_FORCE_NO_BACKGROUND):阻止主线程的运行，如主线程须要执行full purge操做，会致使crash
(SRV_FORCE_NO_TRX_UNDO):不执行事务回滚操做
(SRV_FORCE_NO_IBUF_MERGE):不执行插入缓冲的合并操做
(SRV_FORCE_NO_UNDO_LOG_SCAN):不查看重作日志，InnoDB存储引擎会将未提交的事务视为已提交
(SRV_FORCE_NO_LOG_REDO):不执行前滚的操做

若是配置后出现如下状况：
130507 14:14:01 InnoDB: Waiting for the background threads to start
130507 14:14:02 InnoDB: Waiting for the background threads to start
130507 14:14:03 InnoDB: Waiting for the background threads to start
130507 14:14:04 InnoDB: Waiting for the background threads to start
130507 14:14:05 InnoDB: Waiting for the background threads to start
130507 14:14:06 InnoDB: Waiting for the background threads to start
130507 14:14:07 InnoDB: Waiting for the background threads to start
130507 14:14:08 InnoDB: Waiting for the background threads to start
130507 14:14:09 InnoDB: Waiting for the background threads to start

须要在galera.cfg中添加这一下：
若是在设置 innodb_force_recovery >2 的同时innodb_purge_thread = 0
二、mysqld --tc-heuristic-recover=ROLLBACK
三、删除/var/lib/mysql/ib_logfile*
四、当某个mysql节点挂了，而且存在三个mysql所在host有不一样的网段，当mysql想从新加入须要一个sst的过程，sst时会须要知道集群中某个节点的ip所以须要制定参数--wsrep-sst-receive-address不然可能出现同步的ip不在三台机器所共有的网段

3、一个mysql节点若down了一段时间。从新启动的时候须要一些时间去同步数据，服务的启动超时时间不够，致使服务没法启动，解决方法以下：
The correct way to adjust systemd settings so they don't get overwritten is to create a directory and file as such:
/etc/systemd/system/mariadb.service.d/timeout.conf
[Service]

TimeoutStartSec=12min

或者直接修改/usr/lib/systemd/system/mariadb.service
[Service]

TimeoutStartSec=12min
这里的时间最少要大于90s，默认是90s以后执行 systemctl daemon-reload再重启服务便可

4、日志中出现相似以下错误：
160428 13:54:49 [ERROR] Slave SQL: Error 'Table 'manage_operations' already exists' on query. Default database: 'horizon'. Query: 'CREATE TABLE `manage_operations` (
`id` integer AUTO_INCREMENT NOT NULL PRIMARY KEY,
`name` varchar(50) NOT NULL,
`type` varchar(20) NOT NULL,
`operation` varchar(20) NOT NULL,
`status` varchar(20) NOT NULL,
`time` date NOT NULL,
`operator` varchar(50) NOT NULL
) default charset=utf8', Error_code: 1050
160428 13:54:49 [Warning] WSREP: RBR event 1 Query apply warning: 1, 28585
160428 13:54:49 [Warning] WSREP: Ignoring error for TO isolated action: source: 752eecd1-0ce0-11e6-83fc-3e0502d0bdd2 version: 3 local: 0 state: APPLYING flags: 65 conn_id: 24053 trx_id: -1 seqnos (l: 28668, g: 28585, s: 28584, d: 28584, ts: 80224119986850)
致使进程异常关闭，
此时能够经过执行mysqladmin flush-tables来刷新表项，这个问题的缘由是三个节点之间的表同步存在问题，刷新一下表便可

5、日志出现如下错误：

160820 3:13:41 [ERROR] Error in accept: Too many open files
160820 3:19:42 [ERROR] Error in accept: Too many open files
160827 3:16:24 [ERROR] Error in accept: Too many open files
160831 17:20:52 [ERROR] Error in accept: Too many open files
160831 19:54:29 [ERROR] Error in accept: Too many open files
160831 20:21:53 [ERROR] Error in accept: Too many open files
160901 11:25:57 [ERROR] Error in accept: Too many open files

解决方法

vim /usr/lib/systemd/system/mariadb.service

[Service]
LimitNOFILE=10000

默认的mysql的open_file_limits是1024将该项增大，而且修改vim /etc/my.cnf.d/server.cnf该文件的open_files_limit值

systemctl daemon-reload

systemctl restart mysqld

查看mysql的open_file_limits值是否调整成功

cat /proc/$pid/limit

其中$pid为mysql进程的pid看看值是否调整成功，并看看日志是否还会出现上述错误

总结一下：

当全部节点都宕机的状况下，再次启动节点，Mariadb-Galera集群会出现没法启动的现象
Mariadb-Galera集群启动是有顺序的，遵循一个原则：最后宕机的最早启动，由于集群认为这个节点的数据是最新的
在生产环境下应该避免使用大事务，不建议在高并发写入场景下使用Galera Cluster架构，会致使集群限流，从而引发整个集群hang住，出现生产故障。针对这种状况能够考虑主从，实现读写分离等手段
对数据一致性要求较高，而且数据写入不频繁，数据库容量也不大（50GB左右），网络情况良好的状况下，能够考虑使用Galera方案

本文分享完毕，感谢支持点赞~~