PostgreSQL的pg_xlog文件数计算和在线清理

postgresql的pg_xlog是记录数据库事务信息用的，相似oracle的redo信息，也叫wal日志(write ahead log)，就是在写数据到磁盘里成为固定数据以前，先写入到日志里，而后必定条件下触发调用fsync()将此数据刷到磁盘。好处很少说，下面主要看看postgresql一般会保留多少个pg_xlog文件，以及如何在线清理。

1、环境及参数

环境:
postgresql 9.2.4(单台，未作流复制)
vmware 6.0
centos 6.2

参数:
postgres=# show checkpoint_segments ;   --设置相对大一点,默认过小，防止频繁checkpoint
checkpoint_segments
---------------------
30
(1 row)

postgres=# show checkpoint_timeout ;
checkpoint_timeout
--------------------
5min
(1 row)

postgres=# show checkpoint_completion_target ;
checkpoint_completion_target
------------------------------
0.5
(1 row)

postgres=# show wal_keep_segments ;     --未开启流复制预留xlog参数
 wal_keep_segments 
-------------------
 0
(1 row)

postgres=# show archive_mode ;             --关闭归档
archive_mode 
--------------
off
(1 row)

理论上合理的pg_xlog通常在(2+checkpoint_completion_target)*checkpoint_segment+1左右浮动，可是高并发环境下若是checkpoint_segment设置比较大，且checkpoint较少被触发的时候，pg_xlog下会存有不少的事务日志，严重会耗掉磁盘存储，因此设置checkpoint_segment的大小要视磁盘容量和每一个pg_xlog文件的大小来看，有些时候须要手工调整参数来收缩。

2、测试过程:

1.模拟pg_xlog文件增加
模拟大数据量载入,开启多个窗口插入数据,每一个窗口插入1000W数据,能够看到pg_xlog增加不少，原来只有8个文件

postgres=# create table tbl_kenyon(id int,cname varchar(50),remark text);
CREATE TABLE

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

此时检查pg_xlog下面的文件数量

[postgres@localhost pg_xlog]$ ll
total 4063240
-rw-------  1 postgres postgres 67108864 Jul 14 23:05 00000001000000020000002F
-rw-------  1 postgres postgres 67108864 Jul 14 23:13 000000010000000200000030
-rw-------  1 postgres postgres 67108864 Jul 14 23:18 000000010000000200000031
-rw-------  1 postgres postgres 67108864 Jul 14 23:25 000000010000000200000032
-rw-------  1 postgres postgres 67108864 Jul 14 22:42 000000010000000200000033
-rw-------  1 postgres postgres 67108864 Jul 14 22:42 000000010000000200000034
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000200000035
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000200000036
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000200000037
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000200000038
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000200000039
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 00000001000000020000003A
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000020000003B
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 00000001000000020000003C
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000020000003D
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 00000001000000020000003E
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000000
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000300000001
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000002
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000300000003
-rw-------. 1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000004
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000005
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000006
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000300000007
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000008
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000009
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000000A
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 00000001000000030000000B
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000030000000C
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000000D
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000030000000E
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 00000001000000030000000F
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000010
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000011
-rw-------  1 postgres postgres 67108864 Jul 14 22:44 000000010000000300000012
-rw-------  1 postgres postgres 67108864 Jul 14 22:43 000000010000000300000013
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000014
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000015
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000016
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000017
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000018
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000019
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000001A
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000001B
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000001C
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000001D
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000001E
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 00000001000000030000001F
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 000000010000000300000020
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000021
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000022
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000023
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000024
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000025
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000026
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 000000010000000300000027
-rw-------  1 postgres postgres 67108864 Jul 14 22:45 000000010000000300000028
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 000000010000000300000029
-rw-------. 1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000002A
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000002B
-rw-------  1 postgres postgres 67108864 Jul 14 22:46 00000001000000030000002C
-rw-------  1 postgres postgres 67108864 Jul 14 22:56 00000001000000030000002D
drwx------. 2 postgres postgres     4096 Jun 12 23:19 archive_status
[postgres@localhost pg_xlog]$ ll|wc -l
64

2.模拟pg_xlog文件清理
这类文件对DB是很重要的，不能手工直接清理。每一个文件64MB，是源码编译时--with-wal-segsize时指定的，或者登录查看

postgres=# show wal_segment_size ;  
wal_segment_size 
------------------
 64MB
(1 row)

要收缩这个事务日志空间，减小事务日志数，能够调整参数 checkpoint_segments，本次调为3

postgres=# show checkpoint_segments ;
 checkpoint_segments 
---------------------
 3
(1 row)


postgres=# checkpoint;
CHECKPOINT

 xlog文件有可能不是当即减小的，若是想模拟当即减小，能够模拟数据库频繁checkpoint，如从新作上述的大数据库加载或者更新操做，此时去看pg_xlog的文件数，能看到在减小了，开启两个窗口操做:

postgres=# insert into tbl_kenyon select generate_series(1,10000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 10000000

postgres=# insert into tbl_kenyon select generate_series(1,2000000),'kenyon good boy',md5('kenyon good boy');
INSERT 0 2000000

[postgres@localhost pg_xlog]$ ll|wc -l
62
[postgres@localhost pg_xlog]$ ll|wc -l
49
[postgres@localhost pg_xlog]$ ll|wc -l
46
[postgres@localhost pg_xlog]$ ll|wc -l
46
[postgres@localhost pg_xlog]$ ll|wc -l
43
[postgres@localhost pg_xlog]$ ll|wc -l
43
[postgres@localhost pg_xlog]$ ll|wc -l
43
[postgres@localhost pg_xlog]$ ll|wc -l
40
[postgres@localhost pg_xlog]$ ll|wc -l
40
[postgres@localhost pg_xlog]$ ll|wc -l
31
[postgres@localhost pg_xlog]$ ll|wc -l
25

后续发生的checkpoint，将会逐步收缩pg_xlog的文件数量，直到知足条件为止。以前写过另一种较为激进的清理pg_xlog的方式，须要停机清理，参考 http://my.oschina.net/Kenyon/blog/101432

3.其余
若是是流复制，还有wal_keep_segments这个参数也要考虑一下。后续。

4.说明

checkpoint_segments (integer)
Maximum number of log file segments between automatic WAL checkpoints (each segment
is normally 16 megabytes). The default is three segments. Increasing this parameter can increase
the amount of time needed for crash recovery. This parameter can only be set in the
postgresql.conf file or on the server command line.

5.参考:
http://my.oschina.net/Kenyon/blog/101432
http://francs3.blog.163.com/blog/static/4057672720112182158386/