专职DBA-MySQL日志管理1 MySQL日志种类: 错误日志error log : 当数据库启动、运行、中止时产生该日志。 普通查询日志general query log : 客户端链接数据库后执行语句时产生该日志。 二进制日志binary log : 当数据库内容发生改变时产生该日志。 中继日志relay log : 从库上收到主库的数据更新时产生该日志。 慢查询日志slow query log : SQL语句在数据库查询超出指定时间时产生该日志。 DDL日志metadata log : 执行DDL语句操做元数据时产生该日志。 对日志进行轮训切割常见的命令: mysqladmin flush-logs mysqldump -F/--master-data [root@db01 ~]# mysqld --defaults-file=/data/mysql/3306/my.cnf & [1] 6758 [root@db01 ~]# ps -ef | grep mysql mysql 6758 6711 0 Jul20 pts/0 00:00:01 mysqld --defaults-file=/data/mysql/3306/my.cnf root 6816 6711 0 00:02 pts/0 00:00:00 grep --color=auto mysql [root@db01 ~]# mysql -S /data/mysql/3306/mysql.sock -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 2 Server version: 5.7.26-log MySQL Community Server (GPL) Copyright (c) 2000, 2019, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> 1.错误日志error log MySQL错误日志用于记录MySQL服务进程mysqld在启动关闭运行过程当中遇到的错误信息。 [root@db01 ~]# grep "log_error" /data/mysql/3306/my.cnf log_error = /data/mysql/3306/error.log [root@db01 ~]# ls -l /data/mysql/3306/error.log -rw-r----- 1 mysql mysql 26338 Jul 21 00:00 /data/mysql/3306/error.log mysql> show global variables like "log_error"; +---------------+----------------------------+ | Variable_name | Value | +---------------+----------------------------+ | log_error | /data/mysql/3306/error.log | +---------------+----------------------------+ 1 row in set (0.00 sec) 默认状况下日志的文件名部分和系统主机名是一致的,扩展名为:.err [root@db01 ~]# hostname db01 ==========================》db01.err 错误日志轮询(按天轮询) [root@db01 ~]# cd /data/mysql/3306/ [root@db01 /data/mysql/3306]# mv error.log error_$(date +%F).log [root@db01 /data/mysql/3306]# mysqladmin -S /data/mysql/3306/mysql.sock -p flush-logs Enter password: [root@db01 /data/mysql/3306]# ls -l total 44 -rw-r----- 1 mysql mysql 5 Jul 21 00:00 3306.pid drwxr-xr-x 5 mysql mysql 161 Jul 21 00:00 data -rw-r----- 1 mysql mysql 26338 Jul 21 00:00 error_2019-07-21.log -rw-r----- 1 mysql mysql 0 Jul 21 00:10 error.log drwxr-xr-x 2 mysql mysql 173 Jul 21 00:10 logs -rw-r--r-- 1 mysql mysql 1147 Jul 15 05:42 my.cnf srwxrwxrwx 1 mysql mysql 0 Jul 21 00:00 mysql.sock -rw------- 1 mysql mysql 5 Jul 21 00:00 mysql.sock.lock -rw-r----- 1 mysql mysql 1050 Jul 21 00:10 slow.log drwxr-xr-x 2 mysql mysql 6 Jul 21 00:00 tmp 2.普通查询日志general query log 普通查询日志记录客户端链接信息,以及执行的SQL语句信息。 [root@db01 ~]# grep "general" /data/mysql/3306/my.cnf # general log general_log = off general_log_file = /data/mysql/3306/general.log mysql> show global variables like "%general%"; +------------------+------------------------------+ | Variable_name | Value | +------------------+------------------------------+ | general_log | OFF | | general_log_file | /data/mysql/3306/general.log | +------------------+------------------------------+ 2 rows in set (0.00 sec) mysql> set global general_log = on; Query OK, 0 rows affected (0.00 sec) mysql> show global variables like "%general%"; +------------------+------------------------------+ | Variable_name | Value | +------------------+------------------------------+ | general_log | ON | | general_log_file | /data/mysql/3306/general.log | +------------------+------------------------------+ 2 rows in set (0.00 sec) [root@db01 ~]# ls -l /data/mysql/3306/ total 48 -rw-r----- 1 mysql mysql 5 Jul 21 00:00 3306.pid drwxr-xr-x 5 mysql mysql 161 Jul 21 00:00 data -rw-r----- 1 mysql mysql 26338 Jul 21 00:00 error_2019-07-21.log -rw-r----- 1 mysql mysql 0 Jul 21 00:10 error.log -rw-r----- 1 mysql mysql 254 Jul 21 00:15 general.log drwxr-xr-x 2 mysql mysql 173 Jul 21 00:10 logs -rw-r--r-- 1 mysql mysql 1147 Jul 15 05:42 my.cnf srwxrwxrwx 1 mysql mysql 0 Jul 21 00:00 mysql.sock -rw------- 1 mysql mysql 5 Jul 21 00:00 mysql.sock.lock -rw-r----- 1 mysql mysql 1050 Jul 21 00:10 slow.log drwxr-xr-x 2 mysql mysql 6 Jul 21 00:00 tmp [root@db01 ~]# cat /data/mysql/3306/general.log mysqld, Version: 5.7.26-log (MySQL Community Server (GPL)). started with: Tcp port: 3306 Unix socket: /data/mysql/3306/mysql.sock Time Id Command Argument 2019-07-20T16:15:45.036658Z 2 Query show global variables like "%general%" 对于general.log在高并发数据库场景下,应该是关闭的,由于查询日志信息量很大,容易致使磁盘I/O性能问题。 当访问量不是很大,并且企业又有审计执行的SQL语句时,能够开启general.log 3.二进制日志binary log 二进制日志记录数据库里数据被修改的SQL语句。 通常为DDL和DML语句:insert/delete/update/create/drop/alter等关键字的语句。 做用:(1).记录MySQL数据的增量数据,用来作增量数据库恢复。(2).实现主从复制。 [root@db01 ~]# grep "bin" /data/mysql/3306/my.cnf # for binlog binlog_format = row log_bin = /data/mysql/3306/logs/mysql-bin 默认状况下binlog日志前缀是"主机名-bin" ======》db01-bin mysql> show global variables like "%log_bin%"; +---------------------------------+---------------------------------------+ | Variable_name | Value | +---------------------------------+---------------------------------------+ | log_bin | ON | | log_bin_basename | /data/mysql/3306/logs/mysql-bin | | log_bin_index | /data/mysql/3306/logs/mysql-bin.index | | log_bin_trust_function_creators | OFF | | log_bin_use_v1_row_events | OFF | +---------------------------------+---------------------------------------+ 5 rows in set (0.00 sec) MySQL5.6临时不记录binlog参数是: set session sql_log_bin = off; #临时中止记录binlog,session级别不影响其余会话。 之后你用mysql恢复数据时不但愿恢复的数据SQL记录到binlog里,就能够关闭这个参数。 binlog文件刷新为新文件的条件: (1).数据库重启会自动刷新binlog为新文件。 (2).执行mysqladmin -F (3).执行mysqladmin flush-log (4).binlog文件达到1GB左右时。 (5).人为配置切割及调整。 mysql> show global variables like "%binlog_size%"; +-----------------+------------+ | Variable_name | Value | +-----------------+------------+ | max_binlog_size | 1073741824 | +-----------------+------------+ 1 row in set (0.01 sec) binlog的索引文件 [root@db01 ~]# ls -l /data/mysql/3306/logs/ total 28 -rw-r----- 1 mysql mysql 177 Jul 15 05:43 mysql-bin.000001 -rw-r----- 1 mysql mysql 461 Jul 15 05:53 mysql-bin.000002 -rw-r----- 1 mysql mysql 241 Jul 15 05:53 mysql-bin.000003 -rw-r----- 1 mysql mysql 217 Jul 15 05:53 mysql-bin.000004 -rw-r----- 1 mysql mysql 241 Jul 21 00:10 mysql-bin.000005 -rw-r----- 1 mysql mysql 194 Jul 21 00:10 mysql-bin.000006 -rw-r----- 1 mysql mysql 234 Jul 21 00:10 mysql-bin.index [root@db01 ~]# cat /data/mysql/3306/logs/mysql-bin.index /data/mysql/3306/logs/mysql-bin.000001 /data/mysql/3306/logs/mysql-bin.000002 /data/mysql/3306/logs/mysql-bin.000003 /data/mysql/3306/logs/mysql-bin.000004 /data/mysql/3306/logs/mysql-bin.000005 /data/mysql/3306/logs/mysql-bin.000006 mysql> show global variables like "%log_bin_index%"; +---------------+---------------------------------------+ | Variable_name | Value | +---------------+---------------------------------------+ | log_bin_index | /data/mysql/3306/logs/mysql-bin.index | +---------------+---------------------------------------+ 1 row in set (0.01 sec) binlog日志删除方法: 理论上来说天天数据库全备时刻之前的binlog都是无用的,可是工做中会根据需求保留3到7天的本地binlog文件。 (1). expire_logs_days #删除7天前的binlog日志 能够写入到my.cnf配置文件里 expire_logs_days = 7 mysql> show global variables like "%logs_days%"; +------------------+-------+ | Variable_name | Value | +------------------+-------+ | expire_logs_days | 0 | +------------------+-------+ 1 row in set (0.00 sec) mysql> set global expire_logs_days = 7; Query OK, 0 rows affected (0.00 sec) mysql> show global variables like "%logs_days%"; +------------------+-------+ | Variable_name | Value | +------------------+-------+ | expire_logs_days | 7 | +------------------+-------+ 1 row in set (0.01 sec) (2).从最开始一直删除到指定的文件位置(不包含指定文件) mysql> show binary logs; +------------------+-----------+ | Log_name | File_size | +------------------+-----------+ | mysql-bin.000001 | 177 | | mysql-bin.000002 | 461 | | mysql-bin.000003 | 241 | | mysql-bin.000004 | 217 | | mysql-bin.000005 | 241 | | mysql-bin.000006 | 194 | +------------------+-----------+ 6 rows in set (0.00 sec) mysql> purge binary logs to 'mysql-bin.000002'; Query OK, 0 rows affected (0.01 sec) mysql> show binary logs; +------------------+-----------+ | Log_name | File_size | +------------------+-----------+ | mysql-bin.000002 | 461 | #序列000002以前的binlog就没了 | mysql-bin.000003 | 241 | | mysql-bin.000004 | 217 | | mysql-bin.000005 | 241 | | mysql-bin.000006 | 194 | +------------------+-----------+ 5 rows in set (0.00 sec) (3).按照时间删除binlog日志 [root@db01 ~]# ls -l --time-style=long-iso /data/mysql/3306/logs/mysql-bin.* -rw-r----- 1 mysql mysql 461 2019-07-15 05:53 /data/mysql/3306/logs/mysql-bin.000002 -rw-r----- 1 mysql mysql 241 2019-07-15 05:53 /data/mysql/3306/logs/mysql-bin.000003 -rw-r----- 1 mysql mysql 217 2019-07-15 05:53 /data/mysql/3306/logs/mysql-bin.000004 -rw-r----- 1 mysql mysql 241 2019-07-21 00:10 /data/mysql/3306/logs/mysql-bin.000005 -rw-r----- 1 mysql mysql 194 2019-07-21 00:10 /data/mysql/3306/logs/mysql-bin.000006 -rw-r----- 1 mysql mysql 195 2019-07-21 00:51 /data/mysql/3306/logs/mysql-bin.index 删除2019-07-21 00:10之前的binlog文件 mysql> purge master logs before "2019-07-21 00:10"; Query OK, 0 rows affected (0.00 sec) mysql> system ls -l --time-style=long-iso /data/mysql/3306/logs/mysql-bin.*; -rw-r----- 1 mysql mysql 241 2019-07-21 00:10 /data/mysql/3306/logs/mysql-bin.000005 -rw-r----- 1 mysql mysql 194 2019-07-21 00:10 /data/mysql/3306/logs/mysql-bin.000006 -rw-r----- 1 mysql mysql 78 2019-07-21 01:04 /data/mysql/3306/logs/mysql-bin.index (4).清除全部的binlog,并从000001开始记录。 mysql> reset master; Query OK, 0 rows affected (0.02 sec) mysql> system ls -l --time-style=long-iso /data/mysql/3306/logs/mysql-bin.* -rw-r----- 1 mysql mysql 154 2019-07-21 01:06 /data/mysql/3306/logs/mysql-bin.000001 -rw-r----- 1 mysql mysql 39 2019-07-21 01:06 /data/mysql/3306/logs/mysql-bin.index 下面是binlog相关参数的设置和优化思路 mysql> show global variables like "%binlog%"; +--------------------------------------------+----------------------+ | Variable_name | Value | +--------------------------------------------+----------------------+ | binlog_cache_size | 32768 | | binlog_checksum | CRC32 | | binlog_direct_non_transactional_updates | OFF | | binlog_error_action | ABORT_SERVER | | binlog_format | ROW | | binlog_group_commit_sync_delay | 0 | | binlog_group_commit_sync_no_delay_count | 0 | | binlog_gtid_simple_recovery | ON | | binlog_max_flush_queue_time | 0 | | binlog_order_commits | ON | | binlog_row_image | FULL | | binlog_rows_query_log_events | OFF | | binlog_stmt_cache_size | 32768 | | binlog_transaction_dependency_history_size | 25000 | | binlog_transaction_dependency_tracking | COMMIT_ORDER | | innodb_api_enable_binlog | OFF | | innodb_locks_unsafe_for_binlog | OFF | | log_statements_unsafe_for_binlog | ON | | max_binlog_cache_size | 18446744073709547520 | | max_binlog_size | 1073741824 | | max_binlog_stmt_cache_size | 18446744073709547520 | | sync_binlog | 1 | +--------------------------------------------+----------------------+ 22 rows in set (0.00 sec) binlog_cache_size #binlog日志缓存是数据库为每个客户端链接分配的内存空间。 max_binlog_size #默认是1GB,可是并不能严格控制binlog日志大小,若binlog大小已经接近1GB,而此时又在执行一个较大的事务时,为了保证事务的完整性,数据库不会作日志刷新动做,而是直到该事务的日志所有记录进入当前binlog日志后才会进行刷新。 sync_binlog #控制binlog何时同步到磁盘。 sync_binlog = 0 表示正在事务提交以后,数据库不会将binlog_cache中的数据刷新到磁盘,而是让文件系统自行决定何时来作刷新或者在缓存满了以后才刷新到磁盘。 sync_binlog = n 表示每进行n次事务提交以后,数据库都会进行一次将缓存数据强制刷新到磁盘。 mysql> show global variables like "%log_bin%"; +---------------------------------+---------------------------------------+ | Variable_name | Value | +---------------------------------+---------------------------------------+ | log_bin | ON | | log_bin_basename | /data/mysql/3306/logs/mysql-bin | | log_bin_index | /data/mysql/3306/logs/mysql-bin.index | | log_bin_trust_function_creators | OFF | | log_bin_use_v1_row_events | OFF | +---------------------------------+---------------------------------------+ 5 rows in set (0.00 sec) 记录binlog日志的三种模式: (1).语句模式statement(MySQL5.6默认的模式) 就是每一条被修改数据的SQL语句都会记录到binlog中。 (2).行级模式row 将数据被修改的每一行的状况记录为一条语句。 (3).混合模式mixed 默认采用statement模式记录日志,在一些特殊状况下会转换为row模式。 线网环境建议使用row模式。 mysql> show global variables like "%binlog_format%"; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | binlog_format | ROW | +---------------+-------+ 1 row in set (0.01 sec) [root@db01 ~]# grep "binlog" /data/mysql/3306/my.cnf # for binlog binlog_format = row --base64-output=decode-rows -v #以row模式解析binlog日志 [root@db01 ~]# mysqlbinlog --base64-output=decode-rows -v /data/mysql/3306/logs/mysql-bin.000001 /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; # at 4 #190721 1:06:07 server id 113306 end_log_pos 123 CRC32 0x391e6633 Start: binlog v 4, server v 5.7.26-log created 190721 1:06:07 at startup # Warning: this binlog is either in use or was not closed properly. ROLLBACK/*!*/; # at 123 #190721 1:06:07 server id 113306 end_log_pos 154 CRC32 0x5a6e058c Previous-GTIDs # [empty] SET @@SESSION.GTID_NEXT= 'AUTOMATIC' /* added by mysqlbinlog */ /*!*/; DELIMITER ; # End of log file /*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/; 4.慢查询日志slow.log 慢查询日志就是记录执行时间超出指定值 或 其余指定条件的SQL语句。 [root@db01 ~]# grep "query" /data/mysql/3306/my.cnf # for slow query log slow_query_log = on slow_query_log_file = /data/mysql/3306/slow.log long_query_time = 1.000000 mysql> show global variables like "%slow%"; +---------------------------+---------------------------+ | Variable_name | Value | +---------------------------+---------------------------+ | log_slow_admin_statements | OFF | | log_slow_slave_statements | OFF | | slow_launch_time | 2 | | slow_query_log | ON | | slow_query_log_file | /data/mysql/3306/slow.log | +---------------------------+---------------------------+ 5 rows in set (0.00 sec) mysql> show global variables like "%long_query_time%"; +-----------------+----------+ | Variable_name | Value | +-----------------+----------+ | long_query_time | 1.000000 | +-----------------+----------+ 1 row in set (0.01 sec) mysql> show global variables like "%log_qu%"; +-------------------------------+-------+ | Variable_name | Value | +-------------------------------+-------+ | log_queries_not_using_indexes | OFF | +-------------------------------+-------+ 1 row in set (0.00 sec) mysql> show global variables like "%row_limit%"; +------------------------+-------+ | Variable_name | Value | +------------------------+-------+ | min_examined_row_limit | 0 | +------------------------+-------+ 1 row in set (0.00 sec) mysql> show global variables like "%log_th%"; +----------------------------------------+-------+ | Variable_name | Value | +----------------------------------------+-------+ | log_throttle_queries_not_using_indexes | 0 | +----------------------------------------+-------+ 1 row in set (0.01 sec) 对于天天所产生的大量慢查询日志,如何进行刷新切割呢? 利用定时任务按天对慢查询日志进行切割。 [root@db01 ~]# mkdir -p /disk/sh [root@db01 ~]# cd /disk/sh/ [root@db01 /disk/sh]# vim cut_slow_log.sh #!/bin/bash cd /data/mysql/3306/ && \ mv slow.log slow_$(date +%F).log && \ mysqladmin flush-log -S /data/mysql/3306/mysql.sock -p123 >/dev/null 2>&1 [root@db01 /disk/sh]# sh cut_slow_log.sh [root@db01 /disk/sh]# ls -l /data/mysql/3306/ total 56 -rw-r----- 1 mysql mysql 5 Jul 21 00:00 3306.pid drwxr-xr-x 5 mysql mysql 161 Jul 21 00:00 data -rw-r----- 1 mysql mysql 26338 Jul 21 00:00 error_2019-07-21.log -rw-r----- 1 mysql mysql 0 Jul 21 00:10 error.log -rw-r----- 1 mysql mysql 4571 Jul 21 02:05 general.log drwxr-xr-x 2 mysql mysql 125 Jul 21 02:05 logs -rw-r--r-- 1 mysql mysql 1147 Jul 15 05:42 my.cnf srwxrwxrwx 1 mysql mysql 0 Jul 21 00:00 mysql.sock -rw------- 1 mysql mysql 5 Jul 21 00:00 mysql.sock.lock -rw-r----- 1 mysql mysql 175 Jul 21 02:04 slow_2019-07-21.log -rw-r----- 1 mysql mysql 175 Jul 21 02:05 slow.log drwxr-xr-x 2 mysql mysql 6 Jul 21 00:00 tmp 天天0点执行切割任务 [root@db01 ~]# crontab -e # cut mysql slow log by zhouwanchun at 2019-07-20 00 00 * * * /bin/sh /disk/sh/cut_slow_log.sh >/dev/null 2>&1 [root@db01 ~]# cat /var/spool/cron/root # cut mysql slow log by zhouwanchun at 2019-07-20 00 00 * * * /bin/sh /disk/sh/cut_slow_log.sh >/dev/null 2>&1 使用工具分析慢查询日志 分析慢查询日志的工具备: mysqldumpslow mysqlsla pt-query-diges myprofi mysql-explain-slow-log mysqllogfilter ELK(可视化展现) https://yq.aliyun.com/articles/59260
[root@db01 ~]# mysqldumpslow --help 推荐使用mysqlsla工具 [root@db01 ~]# wget http://hackmysql.com/scripts/mysqlsla-2.03.tar.gz [root@db01 ~]# yum install perl-devel perl-DBI perl-DBD-MySQL -y [root@db01 ~]# rpm -qa perl-devel perl-DBI perl-DBD-MySQL perl-DBI-1.627-4.el7.x86_64 perl-devel-5.16.3-294.el7_6.x86_64 perl-DBD-MySQL-4.023-6.el7.x86_64 [root@db01 /disk]# ls -l total 631080 -rw-r--r-- 1 root root 644869837 Jul 15 05:30 mysql-5.7.26-linux-glibc2.12-x86_64.tar.gz -rw-r--r-- 1 root root 1352558 Jul 21 02:51 mysqlsla.tar.gz drwxr-xr-x 2 root root 29 Jul 21 02:05 sh drwxr-xr-x. 2 root root 6 Jul 12 03:35 soft drwxr-xr-x. 2 root root 6 Jul 12 03:35 tools tar -xf mysqlsla-2.03.tar.gz cd mysqlsla-2.03 perl Makefile.PL make make install [root@db01 /disk]# tar -xf mysqlsla.tar.gz [root@db01 /disk]# cd mysqlsla/ [root@db01 /disk/mysqlsla]# perl Makefile.PL Checking if your kit is complete... Looks good Writing Makefile for mysqlsla [root@db01 /disk/mysqlsla]# make cp lib/mysqlsla.pm blib/lib/mysqlsla.pm cp bin/mysqlsla blib/script/mysqlsla /usr/bin/perl -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/mysqlsla Manifying blib/man3/mysqlsla.3pm [root@db01 /disk/mysqlsla]# make install Installing /usr/local/share/perl5/mysqlsla.pm Installing /usr/local/share/man/man3/mysqlsla.3pm Installing /usr/local/bin/mysqlsla Appending installation info to /usr/lib64/perl5/perllocal.pod 写切割日志的脚本 [root@db01 ~]# cd /disk/sh/ [root@db01 /disk/sh]# vim slow_log_analyze.sh #!/bin/bash Date=`date +%F -d -1day` # cut slow log cd /data/mysql/3306/ && \ mv slow.log slow_$Date.log && \ mysqladmin flush-log -S /data/mysql/3306/mysql.sock -p123 >/dev/null 2>&1 # analyze slow log Time=`date +%F` Path=/usr/local/bin/ cd /data/mysql/3306/ && \ $Path/mysqlsla -lt show slow_$Date.log >/tmp/analyze_slow_$Date.log 2>&1 # rsync analyzed_slow to backup server. # send analyzed slow log to admin on backup server by mail. 写定时任务 [root@db01 ~]# cat /var/spool/cron/root # analyzed mysql slow log by zhouwanchun at 2019-07-20 00 00 * * * /bin/sh /disk/sh/slow_log_analyze.sh >/dev/null 2>&1