Zabbix DB历史表由Innodb转换成Tokudb引擎总结

时间 2020-08-05

标签 zabbix 历史 innodb 转换 tokudb 引擎总结栏目 Zabbix 繁體版

原文原文链接

线上Zabbix监控服务器DB采用的是MySQL一主一从。监控历史数据公司要求保留2年，历史数据相关的表作了表分区，过时数据自动删除。随着线上Zabbix监控设备数的不断增长，服务器磁盘空间增加的很厉害，3.7T的磁盘空间目前已经用了3.3T。尝试用pt-online工具压缩数据表，因为历史数据没有主键不能压缩。mysql

最后考虑把Zabbix历史相关的数据表由Innodb引擎转换成TokuDB引擎从而来减小磁盘空间。因为采用TokuDB压缩历史数据会损耗必定的CPU资源，申请一台HP G10服务器，内存64G 8块600G硬盘作Raid5专门作Zabbix DB服务器。sql

1.Percona数据库安装并开启Tokudb存储引擎。shell

1）安装percona数据库，采用官方最新的RPM进行安装。数据库

[root@ZBXTKDB01 software]# wget https://www.percona.com/downloads/Percona-Server-5.6/Percona-Server-5.6.42-84.2/binary/redhat/6/x86_64/Percona-Server-5.6.42-84.2-r6b2b987-el6-x86_64-bundle.tar
[root@ZBXTKDB01 software]# tar -xvf Percona-Server-5.6.42-84.2-r6b2b987-el6-x86_64-bundle.tar
[root@ZBXTKDB01 software]# rpm -ivh Percona-Server-server-56-5.6.42-rel84.2.el6.x86_64.rpm\ Percona-Server-client-56-5.6.42-rel84.2.el6.x86_64.rpm \Percona-Server-shared-56-5.6.42-rel84.2.el6.x86_64.rpm
[root@ZBXTKDB01 software]# service mysql start

2）安装LIBJEMALLOC库。TokuDB存储引擎须要libjemalloc library 3.3.0或者更新，RPM安装库文件默认在/usr/lib64/libjemalloc.so.1。服务器

[root@ZBXTKDB01 software]# rpm -ivh jemalloc-3.6.0-1.el6.art.x86_64.rpmapp

3）安装Tokudb插件，RPM 安装的软件包Tokudb插件默认在 /usr/lib64/mysql/plugin/ha_tokudb.so。socket

[root@ZBXTKDB01 software]# rpm -ivh Percona-Server-tokudb-56-5.6.42-rel84.2.el6.x86_64.rpmide

4）加载Tokudb引擎须要修改下面的内核参数并写到/etc/rc.local里面。工具

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

5）安装Tokudb引擎。测试

[root@ZBXTKDB01 ~]# ps_tokudb_admin --enable -uroot
Checking if Percona Server is running with jemalloc enabled...
INFO: Percona Server is running with jemalloc enabled.

Checking transparent huge pages status on the system...
INFO: Transparent huge pages are currently disabled on the system.

Checking if thp-setting=never option is already set in config file...
INFO: Option thp-setting=never is set in the config file.

Checking TokuDB engine plugin status...
INFO: TokuDB engine plugin is not installed.

Installing TokuDB engine...
INFO: Successfully installed TokuDB engine plugin.

6）能够经过手工的方式加载Tokudb存储引擎。

INSTALL PLUGIN tokudb SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_file_map SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_fractal_tree_info SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_fractal_tree_block_map SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_trx SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_locks SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_lock_waits SONAME 'ha_tokudb.so';
INSTALL PLUGIN tokudb_background_job_status SONAME 'ha_tokudb.so';

7）验证MySQL数据库的Tokudb引擎是否安装成功。经过show engines或者show plugins命令均可以查看。

mysql> show engines;
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
| Engine             | Support | Comment                                                                    | Transactions | XA   | Savepoints |
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                                         | NO           | NO   | NO         |
| MRG_MYISAM         | YES     | Collection of identical MyISAM tables                                      | NO           | NO   | NO         |
| CSV                | YES     | CSV storage engine                                                         | NO           | NO   | NO         |
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears)             | NO           | NO   | NO         |
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables                  | NO           | NO   | NO         |
| TokuDB             | YES     | Percona TokuDB Storage Engine with Fractal Tree(tm) Technology             | YES          | YES  | YES        |
| MyISAM             | YES     | MyISAM storage engine                                                      | NO           | NO   | NO         |
| ARCHIVE            | YES     | Archive storage engine                                                     | NO           | NO   | NO         |
| FEDERATED          | NO      | Federated MySQL storage engine                                             | NULL         | NULL | NULL       |
| InnoDB             | DEFAULT | Percona-XtraDB, Supports transactions, row-level locking, and foreign keys | YES          | YES  | YES        |
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+

8）因为原来Zabbix数据库版本5.5.20的安装的，能够经过下面的命令update数据库。

[root@ZBXTKDB01 ~]# mysql_upgrade -uroot -p
Enter password:
Looking for 'mysql' as: mysql
Looking for 'mysqlcheck' as: mysqlcheck
Running 'mysqlcheck' with connection arguments: '--socket=/mysql/3307/mysql.sock' '--port=3307'

9）MySQL my.conf配置文件的参数优化。主要优化了Innodb和Tokubdb参数。若是是源码安装的数据库，Tokudb扩展及LIBJEMALLOC库须要在my.conf配置文件里面指定Tokudb插件和LIBJEMALLOC库的路径地址。

[root@ZBXTKDB01 ~]# cat /etc/my.cnf
[client]
socket                 = /mysql/3307/mysql.sock
port                   = 3307

[mysql]
no-auto-rehash

[mysqld]
server-id              = 6243
port                   = 3307
user                   = mysql
basedir                = /usr
datadir                = /mysql/3307/data
socket                 = /mysql/3307/mysql.sock
skip-name-resolve
default_storage_engine = innodb 

binlog-do-db           = zabbix   
log-bin                = /mysql/3307/binlog/mysql-bin
binlog_format          = ROW
binlog_cache_size      = 20M
expire_logs_days       = 10

#slow_query_log         = 1
slow_query_log_file    = /mysql/3307/data/mysql-slow.log
long_query_time        = 20
log_error              = /mysql/3307/mysql_error.log
pid-file               = /mysql/3307/mysql.pid

max_connections     = 3000
back_log            = 100
max_connect_errors  = 1000000
max_allowed_packet  = 64M
max_heap_table_size = 300M
sort_buffer_size    = 2M
join_buffer_size    = 2M
thread_cache_size   = 400
query_cache_type    = 1
query_cache_size    = 128M
query_cache_limit   = 2M
tmp_table_size      = 1024M
#table_cache         = 1000

key_buffer_size           = 128M
read_buffer_size          = 8M
read_rnd_buffer_size      = 8M
bulk_insert_buffer_size   = 10M
myisam_sort_buffer_size   = 16M
myisam_max_sort_file_size = 4G
myisam_repair_threads     = 1

innodb_file_per_table           = 1
innodb_buffer_pool_size         = 10G
innodb_use_sys_malloc           = 1 
innodb_data_file_path           = ibdata01:400M:autoextend
innodb_log_buffer_size          = 10M
innodb_log_file_size            = 400M
innodb_log_files_in_group       = 2
innodb_open_files               = 2048
innodb_flush_log_at_trx_commit  = 2
innodb_lock_wait_timeout        = 60

tokudb_directio = 1
tokudb_row_format = tokudb_lzma
tokudb_cache_size=30G
optimizer_switch=index_condition_pushdown=off
tokudb_read_buf_size = 4K
tokudb_commit_sync = 0
tokudb_fsync_log_period = 60000
tokudb_fs_reserve_percent = 2
tokudb-data-dir = /mysql/3307/tokudb
tokudb-log-dir=

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer_size  = 512M
sort_buffer_size = 512M
read_buffer      = 8M
write_buffer     = 8M

[mysqlhotcopy]
interactive-timeout

[mysqld_safe]
thp-setting = never
open-files-limit = 8192

2.在安装Tokudb存储引擎的服务器上面开启Zabbix数据库新的从库。主从同步和MySQL二进制文件有关，和存储引擎无关，即Innodb存储引擎的MySQL表的数据能够同步到Tokudb存储引擎的数据表里面。

1）中止该服务器和原从库MySQL服务，将data目录拷贝到该服务器上面。

2）根据原来从库服务器数据库data目录里面/data/mysql/3307/data/master.info记录主从复制的信息，开启新服务器的主从复制。

#主库上面添加主从复制的帐号
grant replication slave on *.* to 'xxx'@'192.168.6.243'identified by 'xxxxxx'
#新服务器上面开启主从同步
change master to
master_host='92.168.6.244',
master_port=3307,
master_user='xxx',
master_password='xxxxxx’,
master_log_file='msql-bin.xxxx’,  
master_log_pos=xxxx;

3）验证新从库的数据和主库的数据是否正常同步，若是出现数据不一致能够经过下面的命令解决。

stop slave;
set global sql_slave_skip_counter=1;
start slave ;

set global slave_exec_mode=idempotent;
set global slave_exec_mode=STRICT;

4）Tokudb存储引擎的转换，将Zabbix数据库历史数据相关的表(history、history_uint、trends、tredns_uint 、history_str、history_log、history_text)所有转换成Tokudb存储引擎。

mysql> alter table history_uint engine=TokuDB;

由Innodb存储引擎转换成Tokudb存储引擎应该注意如下事项：

1）在执行alter table history_uint engine=TokuDB 命令以后，首先数据库库会执行MDL（Meta Data lock元数据锁），把Innodb表里面每条数据拿出来放存放到tokudb-data-dir目录里面的临时文件里。等全部的数据取出来存储到临时文件以后，在合并临时文件里面的数据到对应的Tokudb数据表里面。而后删除Innodb数据文件并释放磁盘空间。

mysql> show processlist;
+----+-------------+-----------+--------+---------+-------+-----------------------------------------------------------+----------------------------------------+-----------+---------------+
| Id | User        | Host      | db     | Command | Time  | State                                                     | Info                                   | Rows_sent | Rows_examined |
+----+-------------+-----------+--------+---------+-------+-----------------------------------------------------------+----------------------------------------+-----------+---------------+
|  1 | system user |           | NULL   | Connect | 13776 | Waiting for table metadata lock                           | NULL                                   |         0 |             0 |
|  2 | system user |           | NULL   | Connect | 13936 | Waiting for master to send event                          | NULL                                   |         0 |             0 |
|  7 | root        | localhost | zabbix | Query   | 13767 | Fetched about 8502619000 rows, loading data still remains | alter table history_uint engine=TokuDB |         0 |             0 |
| 37 | root        | localhost | NULL   | Query   |     0 | init                                                      | show processlist                       |         0 |             0 |
+----+-------------+-----------+--------+---------+-------+-----------------------------------------------------------+----------------------------------------+-----------+---------------+

[root@ZBXTKDB01 ~]# ls -l /mysql/3307/tokudb/ |more
total 214125632
-rw------- 1 mysql mysql 0 Dec 20 17:35 __tokudb_lock_dont_delete_me_data
-rw------- 1 mysql mysql 0 Dec 20 17:35 __tokudb_lock_dont_delete_me_temp
-rw------- 1 mysql mysql 1921350 Dec 24 15:07 tokuld102q5N
-rw------- 1 mysql mysql 1948116 Dec 24 13:16 tokuld106Z4m
-rw------- 1 mysql mysql 1936454 Dec 24 14:10 tokuld10AgeQ
-rw------- 1 mysql mysql 1936518 Dec 24 14:42 tokuld10DHEW
-rw------- 1 mysql mysql 1938374 Dec 24 14:42 tokuld10DJqt
-rw------- 1 mysql mysql 1943288 Dec 24 14:51 tokuld10FlFb
-rw------- 1 mysql mysql 1943969 Dec 24 12:42 tokuld10jIQP

mysql> show processlist;
+-----+-------------+-----------+--------+---------+-------+---------------------------------------------------------------------+----------------------------------------+-----------+---------------+
| Id  | User        | Host      | db     | Command | Time  | State                                                               | Info                                   | Rows_sent | Rows_examined |
+-----+-------------+-----------+--------+---------+-------+---------------------------------------------------------------------+----------------------------------------+-----------+---------------+
|   1 | system user |           | NULL   | Connect | 97410 | Waiting for table metadata lock                                     | NULL                                   |         0 |             0 |
|   2 | system user |           | NULL   | Connect | 97570 | Waiting for master to send event                                    | NULL                                   |         0 |             0 |
|   7 | root        | localhost | zabbix | Query   | 97401 | Loading of data t ./zabbix/#sql-2822_7#P#p20180419 about 15.1% done | alter table history_uint engine=TokuDB |         0 |             0 |
| 827 | root        | localhost | NULL   | Query   |     0 | init                                                                | show processlist                       |         0 |             0 |
+-----+-------------+-----------+--------+---------+-------+---------------------------------------------------------------------+----------------------------------------+-----------+---------------+

2）因为在转换成TokuDB存储引擎的时候，会生成成千上万的临时文件，确保数据库分区空间可用性。建议1000G的Innodb转换成Tokudb的时候至少预留300G磁盘空间。

3）Innodb大表在转换成TokuDB存储引擎的时候，特别耗时。经测试history_uint表总共有230亿左右的数据，Fetched到临时表总共花费了10小时30分钟，Loading合成临时文件也须要26个小时左右。

建议把命令放在后台以脚本的方式去执行，若是出错还能够查看具体出错的日志。

[root@ZBXTKDB01 shell]# nohup ChangeInnodbToTokudb.sh &;

4）在后台执行alter table history_uint engine=TokuDB的时候，程序忽然中断报下面的错误。须要调大innodb_buffer_pool_size 的数值。

ERROR 1206 (HY000) at line 1: The total number of locks exceeds the lock table size

1. zabbix 数据库更换 TokuDB 引擎
2. 【总结】InnoDB存储引擎
3. 【MySQL】InnoDB引擎TokuDB引擎对比测试
4. TokuDB存储引擎
5. TokuDB引擎笔记
6. TokuDB · 引擎特性 · HybridDB for MySQL高压缩引擎TokuDB 揭秘
7. 转换表的引擎
8. MySql 转换表的引擎
9. [转]TOKUDB® VS. INNODB FLASH MEMORY
10. 《MySQL + Innodb存储引擎》三刷总结
更多相关文章...
• RSS 历史 - RSS 教程
• XSLT - 转换 - XSLT 教程
• 算法总结-双指针
• 算法总结-回溯法