是一款开源的 多功能 快速的全量和增量备份软件算法
a 等价于本地复制命令(cp)vim
b 等价于远程复制命令(scp)服务器
c 等价于数据删除命令(rm)网络
d 等价于数据查看命令(ls)ssh
1. 能够实现排除同步数据socket
2. 能够实现加密传输数据(借助ssh协议隧道)tcp
3. 能够实现数据传输认证机制ide
4. 能够实现socket守护进程方式,传输数据(重点)测试
5. 能够保持数据传输属性信息不变(-p)网站
6. 能够传输普通文件,也能够传输特殊文件(连接文件 或者 设备文件)
7. 能够数据增量传输
基于一种特殊的算法“quick check”,实现增量复制传输数据
1. 先实现两台主机之间网络互通
2. 对要传输的数据信息,进行属性信息比对(好比 文件大小 时间信息 属主 属组 权限属性等)
3. 只传输变化的数据信息
1. 实现本地数据传输备份
2. 实现远程数据传输备份
3. 实现守护进程方式数据传输备份(重点)
第一步:检查软件是否安装
第二步:编写rsync软件配置文件
第三步:建立备份目录的管理用户
第四步:建立认证时所须要密码文件(对密码文件权限进行修改成600权限)
第五步:建立数据备份目录,修改备份目录属主和属组权限为管理用户(rsync)
第六步:启动rsync服务(rsync --daemon)
进行数据传输测试
说明:若是想实现免交互方式传输数据,客户端要完成如下操做步骤
第一步:建立密码文件,并进行修改权限为600
echo "wuhuang123" >>/etc/rsync.password
chmod 600 /etc/rsync.password
[root@nfs01 ~]# echo "wuhuang123" >>/etc/rsync.password [root@nfs01 ~]# chmod 600 /etc/rsync.password [root@nfs01 ~]# ll /etc/rsync.password -rw------- 1 root root 10 Jan 23 14:45 /etc/rsync.password |
第二步:进行免交互传输数据测试
rsync -avz /etc/hosts rsync_backup@backup::backup --password-file=/etc/rsync.password
[root@nfs01 ~]# rsync -avz /etc/hosts rsync_backup@backup::backup --password-file=/etc/rsync.password sending incremental file list hosts
sent 201 bytes received 27 bytes 456.00 bytes/sec total size is 372 speedup is 1.63 [root@backup ~]# ll /backup/ total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts |
1.7 rsync守护进程方式传输数据原理过程
在备份服务器上部署rsync软件服务,采用推的方式(push:即从本地机器拷贝文件到远程rsync服务器中)进行数据备份传输。
采用定时备份:备份的数据信息大部分是内部人员产生的数据
采用实时备份:备份的数据信息大部分是外部人员产生的数据
命令参数 |
参数 |
-v, --verbose |
详细模式输出,传输时的信息。 |
-z,--compress |
传输时进行压缩提供传输效率,--compress-level=NUM 可按级别压缩,局域网能够不用压缩 |
-a, --archive 重要 |
归档模式,表示以递归方式传输文件,并保持全部文件属性,等于-rlptgoD。 |
-r, --recursive 归类于-a参数 |
对子目录以递归模式处理,即目录下的全部目录都一样传输,注意是小写 |
-t, --times 归类于-a参数 |
保持文件时间信息。 |
-o, --owner 归类于-a参数 |
保持文件属主信息。 |
-p, --perms 归类于-a参数 |
保持文件权限。 |
-g, --group 归类于-a参数 |
保持文件属组信息。 |
-P --progress |
等同于 --partial,显示备份过程。 |
-D, --devices 归类于-a参数 |
保持设备文件信息。 |
-l, --links 归类于-a参数 |
保留软链结。(小写字母) |
-e, --rsh=command |
指定使用rsh、ssh方式进行数据同步 |
--exclude=PATTETN |
用来指定排除那些不但愿传输的文件(和tar参数同样) |
--exclude-from=FILE |
文件名所在的目录文件,便可以实现排除多个文件(和tar参数同样) |
--bwlimit=RATE |
limit I/O bandwidth: KBytes per second limit socket I/O bandwidth限速功能 案例:某DBA作数据同步,带宽占满,致使用户没法访问网站 |
--delete |
让目标目录SRC和源目录数据DST一致,即无差别同步数据 |
保持同步目录及文件属性: 这里的-avzP 至关于 -vzrtopgDIP(还多了DI功能),生产环境经常使用的参数选项为 -avzP 或 -vzrtopgP 若是是放入脚本中,也能够把 -v 和 -P去掉。这里的 --progress 能够用 -P代替。 |
例:-e, --rsh=command 指定使用rsh、ssh方式进行数据同步
[root@nfs01 ~]# rsync -avz -e "ssh -p22" /wuhuang/a 172.16.1.41:/backup The authenticity of host '172.16.1.41 (172.16.1.41)' can't be established. RSA key fingerprint is 57:3f:64:68:95:4d:99:54:01:33:ab:47:a0:72:da:bf. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '172.16.1.41' (RSA) to the list of known hosts. root@172.16.1.41's password: sending incremental file list a/ a/1 a/2 a/3
sent 171 bytes received 73 bytes 28.71 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# ll /backup/ total 8 drwxr-xr-x 2 root root 4096 Jan 23 17:54 a |
第4章 rsync服务常见错误
【客户端的错误】 No route to host 【错误演示过程】 [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup rsync: failed to connect to 172.16.1.41: No route to host (113) rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6] 【异常问题解决】 关闭rsync服务端的防火墙服务(iptables) 也可作防火墙设置,此处很少讲 [root@backup mnt]# /etc/init.d/iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ] [root@backup mnt]# /etc/init.d/iptables status iptables: Firewall is not running. |
客户端的错误现象: [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::/backup ERROR: The remote path must start with a module name not a / rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: rsync命令语法理解错误,::/backup是错误的语法,应该为::backup(rsync模块) |
客户端的错误现象: [root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: auth failed on module backup rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: 1. 密码真的输入错误,用户名真的错误 2. secrets file = /etc/rsync.password指定的密码文件和实际密码文件名称不一致 3. /etc/rsync.password文件权限不是600 4. rsync_backup:123456密码配置文件后面注意不要有空格 5. rsync客户端密码文件中只输入密码信息便可,不要输入虚拟认证用户名称 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup @ERROR: Unknown module 'backup' rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: 1. /etc/rsyncd.conf配置文件模块名称书写错误 2. 模块对应的目录建立有误 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: sending incremental file list hosts rsync: mkstemp ".hosts.5z3AOA" (in backup) failed: Permission denied (13)
sent 196 bytes received 27 bytes 63.71 bytes/sec total size is 349 speedup is 1.57 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6] 异常问题解决: 1. 共享目录的属主和属组不正确,不是rsync 2. 共享目录的权限不正确,不是755 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: chdir failed rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: 1. 备份存储目录没有创建 2. 创建的备份存储目录和配置文件定义不一致 说明:若是没有备份存储目录 |
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: @ERROR: invalid uid rsync rsync error: error starting client-server protocol (code 5) at main.c(1503) [sender=3.0.6] 异常问题解决: rsync服务对应rsync虚拟用户不存在了 |
password file must not be other-accessible
[root@nfs01 tmp]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password password file must not be other-accessible continuing without password file Password: sending incremental file list
sent 26 bytes received 8 bytes 5.23 bytes/sec total size is 349 speedup is 10.26 异常问题解决: rsync客户端的秘钥文件也必须是600权限 |
4.9 rsync客户端链接慢问题
错误日志输出 2017/03/08 20:14:43 [3422] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors 2017/03/08 20:14:43 [3422] name lookup failed for 172.16.1.31: Name or service not known 2017/03/08 20:14:43 [3422] connect from UNKNOWN (172.16.1.31) 2017/03/08 20:14:43 [3422] rsync to backup/ from rsync_backup@unknown (172.16.1.31) 2017/03/08 20:14:43 [3422] receiving file list 2017/03/08 20:14:43 [3422] sent 76 bytes received 83 bytes total size 349 正确日志输出 2017/03/08 20:16:45 [3443] params.c:Parameter() - Ignoring badly formed line in configuration file: ignore errors 2017/03/08 20:16:45 [3443] connect from nfs02 (172.16.1.31) 2017/03/08 20:16:45 [3443] rsync to backup/ from rsync_backup@nfs02 (172.16.1.31) 2017/03/08 20:16:45 [3443] receiving file list 2017/03/08 20:16:45 [3443] sent 76 bytes received 83 bytes total size 349 客户端链接慢缘由:主机名未被解析 异常问题解决: 查看日志进行分析,配置/etc/hosts文件,写入对应主机名与IP的映射关系 |
4.10 rsync服务没有正确启动
Connection refused (111)
[root@wuhuang-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup rsync: failed to connect to 172.16.1.41: Connection refused (111) rsync error: error in socket IO (code 10) at clientserver.c(124) [sender=3.0.6] 解决 rsync服务没开启 [root@wuhuang-muban ~]# rsync --daemon [root@wuhuang-muban ~]# ss -lntup |grep rsync tcp LISTEN 0 5 :::873 :::* users:(("rsync",1434,5)) tcp LISTEN 0 5 *:873 *:* users:(("rsync",1434,4)) [root@wuhuang-muban ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup Password: sending incremental file list hosts
sent 196 bytes received 27 bytes 49.56 bytes/sec total size is 349 speedup is 1.57 |
第5章 rsync扩展应用说明
1. 让rsync守护进程开机自启动
echo "rsync --daemon" >>/etc/rc.local
利用xinetd服务启动rsync守护进程服务流程
rpm -qa|grep xinetd
yum install -y xinetd
[root@backup ~]# rpm -qa|grep xinetd [root@backup ~]# yum install xinetd -y |
5.1.2 第二个里程碑:配置xinetd文件,让rsync服务容许被超级守护进程管理
vim /etc/xinetd.d/rsync
disable = yes ===> disable = no
[root@backup ~]# vim /etc/xinetd.d/rsync # default: off # description: The rsync server is a good addition to an ftp server, as it \ # allows crc checksumming etc. service rsync { disable =no flags = IPv6 socket_type = stream wait = no user = root server = /usr/bin/rsync server_args = --daemon log_on_failure += USERID } |
5.1.3 第三个里程碑:确保rsync守护进程服务关闭了
[root@backup ~]# netstat -lntup|grep rsync tcp 0 0 0.0.0.0:873 0.0.0.0:* LISTEN 1294/rsync tcp 0 0 :::873 :::* LISTEN 1294/rsync [root@backup ~]# killall rsync [root@backup ~]# killall rsync rsync: no process killed |
5.1.4 第四个里程碑:启动xinetd服务
[root@backup ~]# /etc/init.d/xinetd start Starting xinetd: [ OK ] [root@backup ~]# netstat -lntup|grep 873 tcp 0 0 :::873 :::* LISTEN 1353/xinetd |
5.2 守护进程多模块功能配置
[root@backup ~]# cat /etc/rsyncd.conf #rsync_config #created by HQ at 2017 ##rsyncd.conf start##
uid = rsync gid = rsync use chroot = no max connections = 200 timeout = 300 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore errors read only = false list = false hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32 auth users = rsync_backup secrets file = /etc/rsync.password [backup] comment = "backup dir by wuhuang" path = /backup [devbackup] comment = "backup dir by wuhuang" path = /devbackup [sabackup] comment = "backup dir by wuhuang" path = /sabackup |
[root@backup ~]# mkdir /{backup,devbackup,sabackup}/ -p [root@backup ~]# chown -R rsync.rsync /{backup,devbackup,sabackup}/ [root@backup ~]# ll /backup/ -d drwxr-xr-x 3 rsync rsync 4096 Jan 23 21:08 /backup/ [root@backup ~]# ll /devbackup/ -d drwxr-xr-x 2 rsync rsync 4096 Jan 23 23:17 /devbackup/ [root@backup ~]# ll /sabackup/ -d drwxr-xr-x 2 rsync rsync 4096 Jan 23 23:17 /sabackup/ |
[root@backup ~]# /etc/init.d/xinetd restart Stopping xinetd: [ OK ] Starting xinetd: [ OK ] |
1. 若是是由rsync --daemon启动的,重启时使用 killall rsync && rsync --daemon
2. 若是是由 /etc/init.d/xinetd start启动的,重启时使用 /etc/init.d/xinetd restart
[root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::devbackup --password-file=/etc/rsync.passwordsending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::sabackup --password-file=/etc/rsync.password sending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list hosts
sent 205 bytes received 27 bytes 464.00 bytes/sec total size is 372 speedup is 1.60 [root@backup ~]# ll /backup/ /devbackup/ /sabackup/ /backup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts /devbackup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts /sabackup/: total 4 -rw-r--r-- 1 rsync rsync 372 Jan 19 11:55 hosts |
建立一个测试环境:nfs服务器(客户端)上操做
[root@nfs01 ~]# tree /wuhuang/ /wuhuang/ ├── a │ ├── 1 │ ├── 2 │ └── 3 ├── b │ ├── 1 │ ├── 2 │ └── 3 ├── c │ ├── 1 │ ├── 2 │ └── 3 └── d ├── 1 ├── 2 └── 3 4 directories, 12 files
|
实现数据同步传输排除方法
1. --exclude 参数指定排除的文件或目录信息
2. --exclude-from 参数指定排除的多个数据信息文件
--exclude实现排除需求:不要a和b,只要c目录全部数据,d目录中只要1和3文件
[root@nfs01 ~]# rsync -avz /wuhuang/ --exclude=a --exclude=b --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ c/ c/1 c/2 c/3 d/ d/1 d/3
sent 292 bytes received 114 bytes 270.67 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# ll -R /backup/ /backup/: total 8 drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 c drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 d
/backup/c: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 2 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3
/backup/d: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 |
以上命令精简化:
rsync -avz /wuhuang/ --exclude={a..b} --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password 或 rsync -avz /wuhuang/ --exclude={a,b} --exclude=d/2 rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password 5.3.2 --exclude-from=FILE |
--exclude-from实现排除需求:不要a和b 只要c目录全部数据 d目录中只要1和3文件
[root@nfs01 wuhuang]# pwd /wuhuang [root@nfs01 wuhuang]# vim exclude.txt [root@nfs01 wuhuang]# cat exclude.txt a b d/2 [root@nfs01 wuhuang]# rsync -avz /wuhuang/ --exclude-from=/wuhuang/exclude.txt rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ exclude.txt c/ c/1 c/2 c/3 d/ d/1 d/3
sent 368 bytes received 133 bytes 1002.00 bytes/sec total size is 8 speedup is 0.02 [root@backup ~]# ll -R /backup/ /backup/: total 12 drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 c drwxr-xr-x 2 rsync rsync 4096 Jan 23 17:54 d -rw-r--r-- 1 rsync rsync 8 Jan 23 23:54 exclude.txt
/backup/c: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 2 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 /backup/d: total 0 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 1 -rw-r--r-- 1 rsync rsync 0 Jan 23 17:54 3 |
说明:若是在排除同步过程当中,出现了指定的exclude.txt文件,两种方法解决:
01. 将exclude.txt文件从同步数据目录中移出
02. 在exclude.txt文件中写入排除自身文件信息
[root@backup ~]# ll /backup/ total 0 [root@nfs01 ~]# rsync -av /wuhuang/a rsync_backup@172.16.1.41::backup/01/ --password-file=/etc/rsync.password sending incremental file list created directory 01 a/ a/1 a/2 a/3
sent 176 bytes received 69 bytes 163.33 bytes/sec total size is 0 speedup is 0.00 [root@nfs01 ~]# rsync -avz /wuhuang/b rsync_backup@172.16.1.41::backup/02/ --password-file=/etc/rsync.password sending incremental file list created directory 02 b/ b/1 b/2 b/3
sent 167 bytes received 69 bytes 472.00 bytes/sec total size is 0 speedup is 0.00 [root@backup ~]# tree /backup/ /backup/ ├── 01 │ └── a │ ├── 1 │ ├── 2 │ └── 3 └── 02 └── b ├── 1 ├── 2 └── 3
4 directories, 6 files
|
[root@nfs01 ~]# rsync -avz /wuhuang/c rsync_backup@172.16.1.41::backup/03/w/h --password-file=/etc/rsync.password sending incremental file list rsync: mkdir "03/w/h" (in backup) failed: No such file or directory (2) rsync error: error in file IO (code 11) at main.c(576) [receiver=3.0.6] rsync: connection unexpectedly closed (5 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6] |
[root@backup ~]# cat /etc/rsyncd.conf #rsync_config #created by HQ at 2017 ##rsyncd.conf start##
uid = rsync gid = rsync use chroot = no max connections = 200 timeout = 300 pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsyncd.log ignore errors read only = false list = false hosts allow = 172.16.1.0/24 hosts deny = 0.0.0.0/32 auth users = rsync_backup secrets file = /etc/rsync.password [backup] comment = "backup dir by wuhuang" path = /backup [devbackup] comment = "backup dir by wuhuang" path = /devbackup [sabackup] comment = "backup dir by wuhuang" path = /sabackup |
第三种状况测试
[root@nfs01 ~]# rsync --delete -avz /etc/hosts rsync_backup@10.0.0.41::backup --password-file=/etc/rsync.password sending incremental file list hosts
sent 205 bytes received 27 bytes 154.67 bytes/sec total size is 372 speedup is 1.60 10.0.0.41网段能够实现传输 |
5.4 守护进程无差别同步配置
无差别同步通俗的讲:
1. 我有的数据,你也有;我没有的数据,你也不能有(删除)
2. 存储与备份服务数据彻底一致(如出一辙)
我有的数据,你也有:实践过程
rsync客户端操做命令: [root@nfs01 ~]# rsync -avz --delete /wuhuang/ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ exclude.txt a/ a/1 a/2 a/3 b/ b/1 b/2 b/3 c/ c/1 c/2 c/3 d/ d/1 d/2 d/3
sent 705 bytes received 274 bytes 1958.00 bytes/sec total size is 8 speedup is 0.01 rsync服务端检查: [root@backup ~]# tree /backup/ /backup/ ├── a │ ├── 1 │ ├── 2 │ └── 3 ├── b │ ├── 1 │ ├── 2 │ └── 3 ├── c │ ├── 1 │ ├── 2 │ └── 3 ├── d │ ├── 1 │ ├── 2 │ └── 3 └── exclude.txt
4 directories, 13 files |
我没有的数据,你也不能有(删除):实践过程
[root@nfs01 ~]# cd /wuhuang/ [root@nfs01 wuhuang]# rm -rf a [root@nfs01 wuhuang]# ll total 16 drwxr-xr-x 2 root root 4096 Jan 23 17:54 b drwxr-xr-x 2 root root 4096 Jan 23 17:54 c drwxr-xr-x 2 root root 4096 Jan 23 17:54 d -rw-r--r-- 1 root root 8 Jan 23 23:54 exclude.txt [root@nfs01 wuhuang]# rsync -avz --delete /wuhuang/ rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password sending incremental file list ./ deleting a/3 deleting a/2 deleting a/1 deleting a/
sent 177 bytes received 14 bytes 382.00 bytes/sec total size is 8 speedup is 0.04 rsync服务端检查: [root@backup ~]# tree /backup/ /backup/ ├── b │ ├── 1 │ ├── 2 │ └── 3 ├── c │ ├── 1 │ ├── 2 │ └── 3 ├── d │ ├── 1 │ ├── 2 │ └── 3 └── exclude.txt
3 directories, 10 files |