redis-cluster

时间 2019-11-17

标签 redis cluster 栏目 Redis 繁體版

原文原文链接

环境：
   centos 7.3
   ip:10.36.8.106(这里是单台测试，因此没有用bind，直接是127.0.0.1，若是是实际环境得加bind绑定本身的地址)
安装：
   yum -y install redis redis-tribnode

配置：
   mkdir -p /opt/redis-cluster/700{0,1,2,3,4,5}
   cd /opt/redis-cluster/7000
   vim redis.conf
       port 7000
       daemonize yes
       cluster-enabled yes
       cluster-config-file nodes.conf
       cluster-node-timeout 5000
       appendonly yesredis

#配置复制及修改全部节点
for i in `seq 1 5`;do cp redis.conf ../700$i/redis.conf;sed -i "s/7000/700$i/" ../700$i/redis.conf;done算法

测试：
   cd /opt/redis-cluster
   #由于配置文件没有加后台机器，这样批量启动会卡住
   for dir in `ls -r`;do cd /opt/redis-cluster/$dir;redis-server redis.conf;done
   #加入后台以守护进程方式在批量启动
   for dir in `ls -r`;do sed -i "/port/a\daemonize yes" $dir/redis.conf;donevim

建立集群：
   [root@host-192-168-1-100 7000]# redis-trib create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
   >>> Creating cluster
   >>> Performing hash slots allocation on 6 nodes...
   Using 3 masters:
   127.0.0.1:7000
   127.0.0.1:7001
   127.0.0.1:7002
   Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
   Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
   Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
   M: bc37b785100a2fe0b4575c977cb587908b15f2d6 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   M: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   S: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
   replicates bc37b785100a2fe0b4575c977cb587908b15f2d6
   S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   S: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
   replicates 575f24b2fe784e2b6d18591e90ff7e760dc272df
   Can I set the above configuration? (type 'yes' to accept): y
   这里必须输入yes，不然会失败
   [root@host-192-168-1-100 7000]# redis-trib create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
   >>> Creating cluster
   >>> Performing hash slots allocation on 6 nodes...
   Using 3 masters:
   127.0.0.1:7000
   127.0.0.1:7001
   127.0.0.1:7002
   Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
   Adding replica 127.0.0.1:7004 to 127.0.0.1:7001
   Adding replica 127.0.0.1:7005 to 127.0.0.1:7002
   M: bc37b785100a2fe0b4575c977cb587908b15f2d6 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   M: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   S: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
   replicates bc37b785100a2fe0b4575c977cb587908b15f2d6
   S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   S: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
   replicates 575f24b2fe784e2b6d18591e90ff7e760dc272df
   Can I set the above configuration? (type 'yes' to accept): yes
   >>> Nodes configuration updated
   >>> Assign a different config epoch to each node
   >>> Sending CLUSTER MEET messages to join the cluster
   Waiting for the cluster to join...
   >>> Performing Cluster Check (using node 127.0.0.1:7000)
   M: bc37b785100a2fe0b4575c977cb587908b15f2d6 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
   M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
   M: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
   S: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
   slots: (0 slots) slave
   replicates bc37b785100a2fe0b4575c977cb587908b15f2d6
   S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   S: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
   slots: (0 slots) slave
   replicates 575f24b2fe784e2b6d18591e90ff7e760dc272df
   [OK] All nodes agree about slots configuration.
   >>> Check for open slots...
   >>> Check slots coverage...
   [OK] All 16384 slots covered.centos

   查看集群状态：(也能够 redis-trib check 127.0.0.1:7000)
   [root@host-192-168-1-100 7000]# redis-cli -p 7000 cluster nodes
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524020822039 2 connected 5461-10922
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 master - 0 1524020823042 3 connected 10923-16383
   bc37b785100a2fe0b4575c977cb587908b15f2d6 127.0.0.1:7000 myself,master - 0 0 1 connected 0-5460
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 slave bc37b785100a2fe0b4575c977cb587908b15f2d6 0 1524020822540 4 connected
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524020823543 5 connected
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 slave 575f24b2fe784e2b6d18591e90ff7e760dc272df 0 1524020821537 6 connected服务器

验证添加数据：
   必须加-c 以集群模式进入，不然添加key会报错(从7000进入，根据算法直接保存在了7002)
   [root@host-192-168-1-100 ~]# redis-cli -p 7000
   127.0.0.1:7000>
   127.0.0.1:7000>
   127.0.0.1:7000> set foo bar
   (error) MOVED 12182 127.0.0.1:7002app

   [root@host-192-168-1-100 ~]# redis-cli -c -p 7000
   127.0.0.1:7000> set foo bar
   -> Redirected to slot [12182] located at 127.0.0.1:7002
   OK
   127.0.0.1:7002> 异步

Redis 集群没有使用一致性hash, 而是引入了哈希槽的概念.
Redis 集群有16384个哈希槽,每一个key经过CRC16校验后对16384取模来决定放置哪一个槽.集群的每一个节点负责一部分hash槽,socket

主动主从切换：(进入须要切换的从)
   redis-cli -p 7005
   cluster failover
   主动完成主从切换，不会丢失数据测试

添加新节点：
新启动的服务加入集群一个主节点的从节点(把maser-id及后面的node-id去掉随机分配一个主，若是--slave也去掉就是新添加一个主节点)
redis-trib add-node --slave --master-id 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7006 127.0.0.1:7000

从更换主节点：
   已经是从节点能够主动改变本身归属的主节点
   进入从节点 (网上说主节点必须空，特地测试发现不用为空也是能够切换的）
   redis-cli -c -p 7006
   cluster replicate 00a1138c9e4358f5bbd734c6f194bed04a37e999

插入数据：
for i in `seq 1000`;do redis-cli -c -p 7001 set foot$i $i ;done

   再次切换
   [root@host-192-168-1-100 7007]# redis-cli -c -p 7007
   127.0.0.1:7007>
   127.0.0.1:7007> cluster replicate 0b1580a5305041bde69108e438653e1bfccefed0
   OK
   能够切换成功，对应的主id已经变换为切换后的主
   [root@host-192-168-1-100 7007]#
   [root@host-192-168-1-100 7007]# redis-cli -c -p 7000 cluster nodes
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524111358911 8 connected
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524111357909 2 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524111359413 8 connected
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 master - 0 1524111359413 8 connected 10923-16383
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 myself,slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 0 0 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524111357909 7 connected 0-5460
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524111358409 8 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524111359413 2 connected 5461-10922

   而且切换到新的主下后，与新主的数据是一致的
   redis-cli -c -p 7005 keys "*" |awk '{print $0}' |sort -n |uniq -c
   *** 经过这种链接方式去get指定值，会关联全部master作查询，任何一个节点存在都会返回，set 设置特定值也是根据CRC16校验后对16384取模来决定放置哪一个槽

移除集群从新加入：
   移出集群会直接把服务干掉(主若是非空不行，必须得先reshard，从能够干掉)
   redis-trib del-node 127.0.0.1:7000 "455a070d53f205d89de8c252ff61997b04852976"
   从新启动加入以前的集群时，因为以前产生了集群的惟一id节点信息，因此得干掉node.conf文件在重启

   未删掉node.conf时直接加入报错
   redis-trib add-node --slave 127.0.0.1:7007 127.0.0.1:7000
   >>> Check slots coverage...
   [OK] All 16384 slots covered.
   Automatically selected master 127.0.0.1:7005
   [ERR] Node 127.0.0.1:7007 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

   中止删除node.conf后启动在加入成功
   netstat -tnlp |grep 7007 |awk '{print $NF}'|cut -d"/" -f1|sort -n |uniq
   kill -INT 30523
   rm -f nodes.conf
   redis-server redis.conf
   redis-trib add-node --slave 127.0.0.1:7007 127.0.0.1:7000
   [OK] All nodes agree about slots configuration.
   >>> Check for open slots...
   >>> Check slots coverage...
   [OK] All 16384 slots covered.
   Automatically selected master 127.0.0.1:7005
   >>> Send CLUSTER MEET to node 127.0.0.1:7007 to make it join the cluster.
   Waiting for the cluster to join.
   >>> Configure node as replica of 127.0.0.1:7005.
   [OK] New node added correctly.

   [root@host-192-168-1-100 7007]# redis-cli -c -p 7000 cluster nodes
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524110722826 8 connected
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524110724330 2 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524110723829 8 connected
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 master - 0 1524110722826 8 connected 10923-16383
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 myself,slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 0 0 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524110723328 7 connected 0-5460
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524110722325 8 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524110723829 2 connected 5461-10922

故障转移或者升级能够先把从切换为主在对应的从上cluster failover
扩容-添加节点：(启动新节点7008,添加add-node不在说明，这里以master身份加入)
   [root@host-192-168-1-100 7008]# redis-server redis.conf
   [root@host-192-168-1-100 7008]# redis-cli -p 7008 cluster nodes
   29b6f39557e3c4da1505545de0bd8bf0abb6a5bb :7008 myself,master - 0 0 0 connected
   [root@host-192-168-1-100 7008]#
   [root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118789706 5 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524118788703 2 connected 5461-10922
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10923-16383
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118788201 8 connected
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524118788201 7 connected
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118789205 8 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524118790206 8 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524118789205 7 connected 0-5460
   [root@host-192-168-1-100 7008]#
   [root@host-192-168-1-100 7008]# redis-trib add-node 127.0.0.1:7008 127.0.0.1:7005
   >>> Adding node 127.0.0.1:7008 to cluster 127.0.0.1:7005
   >>> Performing Cluster Check (using node 127.0.0.1:7005)
   M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
   S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   3 additional replica(s)
   S: 22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007
   slots: (0 slots) slave
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   S: 790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000
   slots: (0 slots) slave
   replicates 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
   S: 018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006
   slots: (0 slots) slave
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   S: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
   slots: (0 slots) slave
   replicates 00a1138c9e4358f5bbd734c6f194bed04a37e999
   M: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
   [OK] All nodes agree about slots configuration.
   >>> Check for open slots...
   >>> Check slots coverage...
   [OK] All 16384 slots covered.
   >>> Send CLUSTER MEET to node 127.0.0.1:7008 to make it join the cluster.
   [OK] New node added correctly.
   [root@host-192-168-1-100 7008]#
   [root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
   29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524118842819 0 connected
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118842319 5 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524118843820 2 connected 5461-10922
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10923-16383
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118843320 8 connected
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524118842318 7 connected
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524118843320 8 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524118842319 8 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524118844322 7 connected 0-5460

   上述只是添加到集群成功，可是尚未分配槽不能正常工做
   [root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
   29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524119218246 9 connected 0-999
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524119219249 5 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524119219249 2 connected 5461-10922
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10923-16383
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524119220249 8 connected
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524119219749 7 connected
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 0 1524119218247 9 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524119218747 8 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524119219749 7 connected 1000-5460
   [root@host-192-168-1-100 7008]# redis-trib reshard 127.0.0.1:7005 #分配slot
   >>> Performing Cluster Check (using node 127.0.0.1:7005)
   M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
   M: 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008
   slots:0-999 (1000 slots) master
   1 additional replica(s)
   S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   2 additional replica(s)
   S: 22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007
   slots: (0 slots) slave
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   S: 790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000
   slots: (0 slots) slave
   replicates 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
   S: 018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006
   slots: (0 slots) slave
   replicates 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
   S: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
   slots: (0 slots) slave
   replicates 00a1138c9e4358f5bbd734c6f194bed04a37e999
   M: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
   slots:1000-5460 (4461 slots) master
   1 additional replica(s)
   [OK] All nodes agree about slots configuration.
   >>> Check for open slots...
   >>> Check slots coverage...
   [OK] All 16384 slots covered.
   How many slots do you want to move (from 1 to 16384)? 20 #选择分配多少数量的slot
   What is the receiving node ID? 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb #上述数量的slot分配给谁
   Please enter all the source node IDs.
   Type 'all' to use all the nodes as source nodes for the hash slots.
   Type 'done' once you entered all the source nodes IDs.
   Source node #1:all                                           #all表明从全部的master里面根据算法选择上述数量slot槽给receiving node ID
   Ready to move 20 slots.                                       #done表明指定从某个master选择上述slot给receiving node ID
   Source nodes:
       M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
       M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   2 additional replica(s)
       M: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
   slots:1000-5460 (4461 slots) master
   1 additional replica(s)
   Destination node:
       M: 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008
   slots:0-999 (1000 slots) master
   1 additional replica(s)
   Resharding plan:
       Moving slot 5461 from 0b1580a5305041bde69108e438653e1bfccefed0
       Moving slot 5462 from 0b1580a5305041bde69108e438653e1bfccefed0
       Moving slot 5463 from 0b1580a5305041bde69108e438653e1bfccefed0
       Moving slot 5464 from 0b1580a5305041bde69108e438653e1bfccefed0
       Moving slot 5465 from 0b1580a5305041bde69108e438653e1bfccefed0
       Moving slot 5466 from 0b1580a5305041bde69108e438653e1bfccefed0
       Moving slot 5467 from 0b1580a5305041bde69108e438653e1bfccefed0
       Moving slot 5468 from 0b1580a5305041bde69108e438653e1bfccefed0
       Moving slot 10923 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
       Moving slot 10924 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
       Moving slot 10925 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
       Moving slot 10926 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
       Moving slot 10927 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
       Moving slot 10928 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
       Moving slot 10929 from 00a1138c9e4358f5bbd734c6f194bed04a37e999
       Moving slot 1000 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
       Moving slot 1001 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
       Moving slot 1002 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
       Moving slot 1003 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
       Moving slot 1004 from 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
   Do you want to proceed with the proposed reshard plan (yes/no)?
   Moving slot 5461 from 127.0.0.1:7001 to 127.0.0.1:7008:
   Moving slot 5462 from 127.0.0.1:7001 to 127.0.0.1:7008:
   Moving slot 5463 from 127.0.0.1:7001 to 127.0.0.1:7008:
   Moving slot 5464 from 127.0.0.1:7001 to 127.0.0.1:7008:
   Moving slot 5465 from 127.0.0.1:7001 to 127.0.0.1:7008:
   Moving slot 5466 from 127.0.0.1:7001 to 127.0.0.1:7008:
   Moving slot 5467 from 127.0.0.1:7001 to 127.0.0.1:7008:
   Moving slot 5468 from 127.0.0.1:7001 to 127.0.0.1:7008:
   Moving slot 10923 from 127.0.0.1:7005 to 127.0.0.1:7008:
   Moving slot 10924 from 127.0.0.1:7005 to 127.0.0.1:7008:
   Moving slot 10925 from 127.0.0.1:7005 to 127.0.0.1:7008:
   Moving slot 10926 from 127.0.0.1:7005 to 127.0.0.1:7008:
   Moving slot 10927 from 127.0.0.1:7005 to 127.0.0.1:7008:
   Moving slot 10928 from 127.0.0.1:7005 to 127.0.0.1:7008:
   Moving slot 10929 from 127.0.0.1:7005 to 127.0.0.1:7008:
   Moving slot 1000 from 127.0.0.1:7003 to 127.0.0.1:7008:
   Moving slot 1001 from 127.0.0.1:7003 to 127.0.0.1:7008:
   Moving slot 1002 from 127.0.0.1:7003 to 127.0.0.1:7008:
   Moving slot 1003 from 127.0.0.1:7003 to 127.0.0.1:7008:
   Moving slot 1004 from 127.0.0.1:7003 to 127.0.0.1:7008:

   那若是你先选择从某一个指定节点分配到另一个节点的话
   Please enter all the source node IDs.
   Type 'all' to use all the nodes as source nodes for the hash slots.
   Type 'done' once you entered all the source nodes IDs.
   Source node #1:all
   在这里先填入指定节点回车在输入done回车便可

   第一次转移1000选择的是done，从7003转移;后续转移20,从7003转移5个，7001转移8个：5461-5468，7005转移7个：10923-10929
   [root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
   29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524119466369 9 connected 0-1004 5461-5468 10923-10929
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524119465867 5 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524119466871 2 connected 5469-10922
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10930-16383
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524119466870 8 connected
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524119467371 7 connected
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 0 1524119467873 9 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524119467873 8 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524119466369 7 connected 1005-5460

缩容-缩减节点：
前提：必须得先转移master节点上所分配的全部槽以及转移全部的从节点(若是从节点也干掉的话就先reshard master,在干掉这个这个master和slave)

   [root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
   29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524120043364 9 connected 5461-5468 10923-10929
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120044363 5 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524120044363 2 connected 5469-10922
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 8 connected 10930-16383
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120045367 8 connected
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524120043864 10 connected
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 0 1524120044865 9 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120044865 8 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524120043364 10 connected 0-5460
   [root@host-192-168-1-100 7008]#
   上面采起reshard+done 指定转移的方式把1005 转移给了7003，测试删除一下发现报错，非空的，必须得想Ianreshard data
   [root@host-192-168-1-100 7008]# redis-trib del-node 127.0.0.1:7000 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
   >>> Removing node 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb from cluster 127.0.0.1:7000
   [ERR] Node 127.0.0.1:7008 is not empty! Reshard data away and try again.

   [root@host-192-168-1-100 7008]# redis-trib reshard 127.0.0.1:7005
   >>> Performing Cluster Check (using node 127.0.0.1:7005)
   M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
   slots:10930-16383 (5454 slots) master
   1 additional replica(s)
   M: 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008
   slots:5461-5468,10923-10929 (15 slots) master
   1 additional replica(s)
   S: 02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   M: 0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001
   slots:5469-10922 (5454 slots) master
   2 additional replica(s)
   S: 22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007
   slots: (0 slots) slave
   replicates 0b1580a5305041bde69108e438653e1bfccefed0
   S: 790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000
   slots: (0 slots) slave
   replicates 3df3a3bfd7489854d3dcbd2549e17639b6aa049c
   S: 018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006
   slots: (0 slots) slave
   replicates 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
   S: 575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002
   slots: (0 slots) slave
   replicates 00a1138c9e4358f5bbd734c6f194bed04a37e999
   M: 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
   [OK] All nodes agree about slots configuration.
   >>> Check for open slots...
   >>> Check slots coverage...
   [OK] All 16384 slots covered.
   How many slots do you want to move (from 1 to 16384)? 15
   What is the receiving node ID? 00a1138c9e4358f5bbd734c6f194bed04a37e999
   Please enter all the source node IDs.
   Type 'all' to use all the nodes as source nodes for the hash slots.
   Type 'done' once you entered all the source nodes IDs.
   Source node #1:29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
   Source node #2:done

   Ready to move 15 slots.
   Source nodes:
       M: 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008
   slots:5461-5468,10923-10929 (15 slots) master
   1 additional replica(s)
   Destination node:
       M: 00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005
   slots:10930-16383 (5454 slots) master
   1 additional replica(s)
   Resharding plan:
       Moving slot 5461 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 5462 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 5463 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 5464 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 5465 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 5466 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 5467 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 5468 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 10923 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 10924 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 10925 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 10926 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 10927 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 10928 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
       Moving slot 10929 from 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
   Do you want to proceed with the proposed reshard plan (yes/no)? yes
   Moving slot 5461 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 5462 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 5463 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 5464 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 5465 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 5466 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 5467 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 5468 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 10923 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 10924 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 10925 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 10926 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 10927 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 10928 from 127.0.0.1:7008 to 127.0.0.1:7005:
   Moving slot 10929 from 127.0.0.1:7008 to 127.0.0.1:7005:

   再次reshard以后，查看集群状况7008已是空的，测试删除成功(上述缩减reshard的时候注意只能分配给指定的）
   [root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
   29b6f39557e3c4da1505545de0bd8bf0abb6a5bb 127.0.0.1:7008 master - 0 1524120244373 9 connected
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120246378 5 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524120245376 2 connected 5469-10922
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 11 connected 5461-5468 10923-16383
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120245877 8 connected
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524120246378 10 connected
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120244875 11 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120246378 11 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524120244875 10 connected 0-5460
   [root@host-192-168-1-100 7008]#
   [root@host-192-168-1-100 7008]# redis-trib del-node 127.0.0.1:7000 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb
   >>> Removing node 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb from cluster 127.0.0.1:7000
   >>> Sending CLUSTER FORGET messages to the cluster...
   >>> SHUTDOWN the node.
   [root@host-192-168-1-100 7008]# redis-cli -p 7005 cluster nodes
   bc37b785100a2fe0b4575c977cb587908b15f2d6 :0 master,fail,noaddr - 1524025875825 1524025874721 1 disconnected
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120264923 5 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524120265424 2 connected 5469-10922
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 myself,master - 0 0 11 connected 5461-5468 10923-16383
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524120266929 8 connected
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 slave 3df3a3bfd7489854d3dcbd2549e17639b6aa049c 0 1524120265927 10 connected
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120264923 11 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 slave 00a1138c9e4358f5bbd734c6f194bed04a37e999 0 1524120266427 11 connected
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 master - 0 1524120266929 10 connected 0-5460

前面说了主从主动切换cluster failover，如今是集群自动熔断：
   为了方便拍错，记录日志
   cd /opt/redis-cluster
   for i in `ls -r`;do echo "logfile \"/var/log/redis/$i.log\"" >> /opt/redis-cluster/$i/redis.conf;done
   重启服务
   pkill redis-server
   for i in `ls -r`;do cd /opt/redis-cluster/$i;redis-server redis.conf;done
   如今down掉主：
       当都是1主1从时，从直接使用failover选举，并宣告本身是主
       当down的是1主1从，其余是1主多从时，选举这个从为主以后，其它多从的会选择一个历来成为这个新主的附属
   [root@host-192-168-1-100 ~]# redis-cli -p 7000 cluster nodes
   3df3a3bfd7489854d3dcbd2549e17639b6aa049c 127.0.0.1:7003 slave 790b56a8774e24455f6da822aae1c06c8898d45b 0 1524125881979 13 connected
   575f24b2fe784e2b6d18591e90ff7e760dc272df 127.0.0.1:7002 master - 0 1524125880975 14 connected 5461-5468 10923-16383
   02f0216b114632f4d25dcc742693c814e1bdfdd4 127.0.0.1:7004 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524125880975 2 connected
   22628a3a484345de1ae0323b26c679eef5688cf8 127.0.0.1:7007 slave 0b1580a5305041bde69108e438653e1bfccefed0 0 1524125881979 8 connected
   0b1580a5305041bde69108e438653e1bfccefed0 127.0.0.1:7001 master - 0 1524125880473 2 connected 5469-10922
   018f59aedc474b8589674ff9454269dbfdefaa22 127.0.0.1:7006 slave 575f24b2fe784e2b6d18591e90ff7e760dc272df 0 1524125880473 14 connected
   00a1138c9e4358f5bbd734c6f194bed04a37e999 127.0.0.1:7005 master,fail - 1524125827918 1524125826314 11 disconnected
   790b56a8774e24455f6da822aae1c06c8898d45b 127.0.0.1:7000 myself,master - 0 0 13 connected 0-5460

   31105:S 19 Apr 16:17:07.887 # Connection with master lost.
   31105:S 19 Apr 16:17:07.888 * Caching the disconnected master state.
   31105:S 19 Apr 16:17:08.021 * Connecting to MASTER 127.0.0.1:7005
   31105:S 19 Apr 16:17:08.021 * MASTER <-> SLAVE sync started
   31105:S 19 Apr 16:17:08.021 # Error condition on socket for SYNC: Connection refused
   31105:S 19 Apr 16:17:09.022 * Connecting to MASTER 127.0.0.1:7005
   31105:S 19 Apr 16:17:09.022 * MASTER <-> SLAVE sync started
   31105:S 19 Apr 16:17:09.022 # Error condition on socket for SYNC: Connection refused
   31105:S 19 Apr 16:17:10.025 * Connecting to MASTER 127.0.0.1:7005
   31105:S 19 Apr 16:17:10.026 * MASTER <-> SLAVE sync started
   31105:S 19 Apr 16:17:10.026 # Error condition on socket for SYNC: Connection refused
   31105:S 19 Apr 16:17:11.028 * Connecting to MASTER 127.0.0.1:7005
   31105:S 19 Apr 16:17:11.028 * MASTER <-> SLAVE sync started
   31105:S 19 Apr 16:17:11.028 # Error condition on socket for SYNC: Connection refused
   31105:S 19 Apr 16:17:12.031 * Connecting to MASTER 127.0.0.1:7005
   31105:S 19 Apr 16:17:12.031 * MASTER <-> SLAVE sync started
   31105:S 19 Apr 16:17:12.031 # Error condition on socket for SYNC: Connection refused
   31105:S 19 Apr 16:17:13.033 * Connecting to MASTER 127.0.0.1:7005
   31105:S 19 Apr 16:17:13.033 * MASTER <-> SLAVE sync started
   31105:S 19 Apr 16:17:13.033 # Error condition on socket for SYNC: Connection refused
   31105:S 19 Apr 16:17:13.337 * FAIL message received from 790b56a8774e24455f6da822aae1c06c8898d45b about 00a1138c9e4358f5bbd734c6f194bed04a37e999
   31105:S 19 Apr 16:17:13.337 # Cluster state changed: fail
   31105:S 19 Apr 16:17:13.434 # Start of election delayed for 827 milliseconds (rank #0, offset 3389).
   31105:S 19 Apr 16:17:14.035 * Connecting to MASTER 127.0.0.1:7005
   31105:S 19 Apr 16:17:14.035 * MASTER <-> SLAVE sync started
   31105:S 19 Apr 16:17:14.035 # Error condition on socket for SYNC: Connection refused
   31105:S 19 Apr 16:17:14.209 * Ignoring FAIL message from unknown node 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb about 00a1138c9e4358f5bbd734c6f194bed04a37e999
   31105:S 19 Apr 16:17:14.335 # Starting a failover election for epoch 14.
   31105:S 19 Apr 16:17:14.366 # Failover election won: I'm the new master.
   31105:S 19 Apr 16:17:14.366 # configEpoch set to 14 after successful failover
   31105:M 19 Apr 16:17:14.366 * Discarding previously cached master state.
   31105:M 19 Apr 16:17:14.367 # Cluster state changed: ok
   31105:M 19 Apr 16:17:19.946 * Slave 127.0.0.1:7006 asks for synchronization
   31105:M 19 Apr 16:17:19.946 * Full resync requested by slave 127.0.0.1:7006 #这个是从其余一主多从的有一个从新请求作为这个新主的从
   31105:M 19 Apr 16:17:19.946 * Starting BGSAVE for SYNC with target: disk
   31105:M 19 Apr 16:17:19.949 * Background saving started by pid 31269
   31269:C 19 Apr 16:17:19.965 * DB saved on disk
   31269:C 19 Apr 16:17:19.966 * RDB: 2 MB of memory used by copy-on-write
   31105:M 19 Apr 16:17:20.063 * Background saving terminated with success
   31105:M 19 Apr 16:17:20.063 * Synchronization with slave 127.0.0.1:7006 succeeded

   31113:M 19 Apr 16:17:13.335 * Marking node 00a1138c9e4358f5bbd734c6f194bed04a37e999 as failing (quorum reached).
   31113:M 19 Apr 16:17:13.335 # Cluster state changed: fail
   31113:M 19 Apr 16:17:14.208 * Ignoring FAIL message from unknown node 29b6f39557e3c4da1505545de0bd8bf0abb6a5bb about 00a1138c9e4358f5bbd734c6f194bed04a37e999
   31113:M 19 Apr 16:17:14.366 # Failover auth granted to 575f24b2fe784e2b6d18591e90ff7e760dc272df for epoch 14
   31113:M 19 Apr 16:17:14.406 # Cluster state changed: ok
   31113:M 19 Apr 16:17:19.443 # Connection with slave 127.0.0.1:7006 lost.

   31324:M 19 Apr 16:31:00.714 * Clear FAIL state for node 0b1580a5305041bde69108e438653e1bfccefed0: master without slots is reachable again.
   31324:M 19 Apr 16:31:01.670 * Slave 127.0.0.1:7001 asks for synchronization
   31324:M 19 Apr 16:31:01.670 * Full resync requested by slave 127.0.0.1:7001
   31324:M 19 Apr 16:31:01.670 * Starting BGSAVE for SYNC with target: disk
   31324:M 19 Apr 16:31:01.672 * Background saving started by pid 31375
   31375:C 19 Apr 16:31:01.690 * DB saved on disk
   31375:C 19 Apr 16:31:01.691 * RDB: 2 MB of memory used by copy-on-write
   31324:M 19 Apr 16:31:01.715 * Background saving terminated with success
   31324:M 19 Apr 16:31:01.715 * Synchronization with slave 127.0.0.1:7001 succeeded

Redis 集群的一致性保证(Redis Cluster consistency guarantees)
Redis 集群不保证强一致性。实践中，这意味着在特定的条件下，Redis 集群可能会丢掉一些被系统收到的写入请求命令。
Redis 集群为何会丢失写请求的第一个缘由，是由于采用了异步复制。这意味着在写期间下面的事情发生了：

 你的客户端向主服务器 B 写入。
 主服务器 B 回复 OK 给你的客户端。
 主服务器 B 传播写入操做到其从服务器 B1，B2 和 B3。

第二种就是用CLUSTER FAILOVER 命令这里须要登陆从而后输入命令就能把对应的主换成从了，这种方法不会丢数据 redis-cluster选举: (1)领着选举过程是集群中全部master参与,若是半数以上master节点与master节点通讯超过(cluster-node-timeout),认为当前master节点挂掉. (2):何时整个集群不可用(cluster_state:fail),当集群不可用时,全部对集群的操做作都不可用，收到((error) CLUSTERDOWN The cluster is down)错误 a:若是集群任意master挂掉,且当前master没有slave.集群进入fail状态,也能够理解成进群的slot映射[0-16383]不完成时进入fail状态. b:若是进群超过半数以上master挂掉，不管是否有slave集群进入fail状态.