Redis cluster集群：原理及搭建

1.为何使用redis？

redis是一种典型的no-sql 即非关系数据库 像python的字典同样 存储key-value键值对 工做在memory中

因此很适合用来充当整个互联网架构中各级之间的cache 好比lvs的4层转发层 nginx的7层代理层

尤为是lnmp架构应用层如php-fpm或者是Tomcat到mysql之间 作一个cache 以减轻db的压力

由于有至关一部分的数据 只是简单的key-value对应关系，并且在实际的业务中经常在短期内迅速变更 

若是用关系数据库mysql之类存储 会大大增长对db的访问 致使db的负担很重 由于全部的require中的大部分最后都要汇聚到db

因此若是想要业务稳定 那么解决db的压力 就是关键 因此如今大部分的解决方案就是在db层之上的各级使用多级的no-sql 

像memcache redis 等 来为db提供缓冲  

2.为何使用redis-cluster？

为了在大流量访问下提供稳定的业务，集群化是存储的必然形态

将来的发展趋势确定是云计算和大数据的紧密结合 只有分布式架构能知足要求

若是没有集群化 何来的分布式？

3.顺带一提总结一波今天的redis原理之数据持久化

虽然redis这种no-sql通常都是做为cache来服务  可是若是彻底没有数据可持久化的方法 那么显得有些单薄

就像memcache 因为这种no-sql是工做在memory的 那么因为memory的实体是ram

因此若是重启或者宕机 memory中的数据就全没了 数据的一致性的不到保障 

可是 redis不一样 redis有相对的数据持久化的方案 由两种方式构成 aof & rdb

aof就像关系数据库中的binlog同样  把每一次写操做以追加的形式记录在其中以文件的形式刷到磁盘里 

而且可使用不一样的fsync策略     无fsync,每秒fsync,每次写的时候fsync.

使用默认的每秒fsync策略,Redis的性能依然很好(fsync是由后台线程进行处理的,主线程会尽力处理客户端请求)

一旦出现故障，最多丢失1秒的数据.

可是缺点也随之而来 那就是aof文件的大小会随着时间线性增加 一段时间以后 就会变得很大 

若是要在一端以AOF的形式来恢复数据 那么因为AOF文件的巨大致积 可能会让进程如同假死同样 十分的慢

rdb则是一种快照机制 

redis工做在内存中 rdb就是每隔一段时间 对内存中的数据作一次快照  保存在rdb文件中 

并且redis的主从同步能够实现异步 也是因为rdb的机制 他在作快照时会fork出一个子进程 由子进程来作快照

父进程彻底处理请求 绝不影响 很适合数据的备份 

可是问题是 若是数据量很大的话 rdb它要保存一个完整的数据集 是一个大的工做 若是时间间隔设置的过短 

那么严重影响redis的性能 可是按照常规设置的话 如5分钟一次 那么若是宕机或者重启 就会基于上次作rdb的时间

从而丢失分钟级的数据

point：在redis4.0的新特性中 采用了aof-rdb的混合方案来保障数据的持久性 可是官方的说法是还不成熟

是一个长期的工做  因此有待观察吧 

4.redis集群实现方案： 

关于redis的集群化方案 目前有三种 

（1）Twitter开发的twemproxy

（2）豌豆荚开发的codis

（3）redis官方的redis-cluster

简介：twemproxy架构简单 就是用proxy对后端redis server进行代理 可是因为代理层的消耗性能很低 并且一般涉及多个key的操做都是不支持的 并且自己不支持动态扩容和透明的数据迁移 并且也失去维护 Twitter内部已经不使用了

          redis-cluster是三个里性能最强大的 由于他使用去中心化的思想 使用hash slot方式 将16348个hash slot 覆盖到全部节点上 对于存储的每一个key值 使用CRC16(KEY)&16348=slot 获得他对应的hash slot 并在访问key时就去找他的hash slot在哪个节点上 而后由当前访问节点从实际被分配了这个hash slot的节点去取数据 节点之间使用轻量协议通讯 减小带宽占用 性能很高 自动实现负载均衡与高可用 自动实现failover  而且支持动态扩展 官方已经玩到能够1000个节点 实现的复杂度低 总之我的比较喜欢这个架构 由于他的去中心化思想免去了proxy的消耗 是全新的思路

           可是它也有一些不足 例如官方没有提供图形化管理工具 运维体验差 全手工数据迁移 而且本身对本身自己的redis命令支持也不彻底等 可是这些问题 我以为不能掩盖他关键的新思想所带来的的优点 随着官方的推动 这些问题应该都能在必定时间内获得解决 那么这时候去中心化思想带来的高性能就会表现出他巨大的优点 

          codis使用的也是proxy思路 可是作的比较好 是这两种之间的一个中间级 并且支持redis命令是最多的 有图形化GUI管理和监控工具 运维友好 这个过段时间会详细另外写出来原理 工做机制和搭建实现

 

 

Redis集群

基本介绍

Redis 集群是一个能够在多个 Redis 节点之间进行数据共享的设施installation。

Redis 集群不支持那些须要同时处理多个键的 Redis 命令， 由于执行这些命令须要在多个 Redis 节点之间移动数据， 而且在高负载的状况下， 这些命令将下降Redis集群的性能， 并致使不可预测的行为。

Redis 集群经过分区partition来提供必定程度的可用性availability： 即便集群中有一部分节点失效或者没法进行通信， 集群也能够继续处理命令请求。

Redis集群提供了如下两个好处：

• 将数据自动切分split到多个节点的能力。
• 当集群中的一部分节点失效或者没法进行通信时， 仍然能够继续处理命令请求的能力。

集群原理

redis-cluster架构图

1. 全部的redis节点彼此互联(PING-PONG机制),内部使用二进制协议优化传输速度和带宽。

2. 节点的fail是经过集群中超过半数的节点检测失效时才生效。

3. 客户端与redis节点直连,不须要中间proxy层.客户端不须要链接集群全部节点,链接集群中任何一个可用节点便可。

4. redis-cluster把全部的物理节点映射到[0-16383]slot上,cluster 负责维护node<->slot<->value

Redis集群中内置了 16384 个哈希槽，当须要在 Redis 集群中放置一个 key-value 时，redis 先对key 使用 crc16 算法算出一个结果，而后把结果对 16384 求余数，这样每一个 key 都会对应一个编号在 0-16383 之间的哈希槽，redis 会根据节点数量大体均等的将哈希槽映射到不一样的节点

redis-cluster投票:容错

1. 投票过程是集群中全部master参与,若是半数以上master节点与master节点通讯超时(cluster-node-timeout),认为当前master节点挂掉.

2. 何时整个集群不可用(cluster_state:fail)?

   • 若是集群任意master挂掉,且当前master没有slave.集群进入fail状态,也能够理解成集群的slot映射[0-16383]不完整时进入fail状态. 
     
     • redis-3.0.0.rc1加入cluster-require-full-coverage参数,默认关闭,打开集群兼容部分失败.
   • 若是集群超过半数以上master挂掉，不管是否有slave，集群进入fail状态.

Redis集群搭建

Redis单机版的安装见博客《redis入门——安装篇》，安装好以后，将redis复制成6份，注意要将.rdb和.aof后缀的文件删除，若是有的话。

Ruby环境

使用

1.  
   yum -y install ruby
2.  
   yum -y install rubygems

安装ruby环境。 
网上下载redis-3.0.0.gem，执行gem install redis-3.0.0.gem安装。

redis配置文件修改

如今已经准备好了，6份干净的redis，以下所示

1.  
   [root @localhost redis-cluster]# pwd
2.  
   /usr/ local/redis/redis-cluster
3.  
   [root @localhost redis-cluster]# ll
4.  
   total 72
5.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:17 redis1
6.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis2
7.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis3
8.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis4
9.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis5
10.  
    drwxr-xr- x 2 root root 4096 Nov 2 00:25 redis6
11.  
    -rwxr-xr- x 1 root root 48141 Nov 2 00:16 redis-trib.rb
12.  
    [root @localhost redis-cluster]#

PS：注意，这里已经将redis源文件src目录下的redis-trib.rb文件拷贝过来了。 
redis-trib.rb这个文件是redis集群的管理文件，ruby脚本。

将六个节点的redis.conf配置文件按照以下进行修改

1.  
   ################################ GENERAL #####################################
2.  
    
3.  
   # By default Redis does not run as a daemon. Use 'yes' if you need it.
4.  
   # Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
5.  
   daemonize yes
6.  
    
7.  
   # Accept connections on the specified port, default is 6379.
8.  
   # If port 0 is specified Redis will not listen on a TCP socket.
9.  
   port *
10.  
     
11.  
    ################################ REDIS CLUSTER ###############################
12.  
    #
13.  
    # ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
14.  
    # WARNING EXPERIMENTAL: Redis Cluster is considered to be stable code, however
15.  
    # in order to mark it as "mature" we need to wait for a non trivial percentage
16.  
    # of users to deploy it in production.
17.  
    # ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
18.  
    #
19.  
    # Normal Redis instances can't be part of a Redis Cluster; only nodes that are
20.  
    # started as cluster nodes can. In order to start a Redis instance as a
21.  
    # cluster node enable the cluster support uncommenting the following:
22.  
    #
23.  
    cluster-enabled yes

PS：端口号若是是同一台主机的话，必须不一样。不一样主机能够相同。

PS：我这里是使用一台主机，因此我将六个节点的端口号修改成7001-7006

编写集群启动脚本和中止脚本

启动脚本start-all.sh

1.  
   cd redis1
2.  
   ./redis-server redis.conf
3.  
   cd ..
4.  
   cd redis2
5.  
   ./redis-server redis.conf
6.  
   cd ..
7.  
   cd redis3
8.  
   ./redis-server redis.conf
9.  
   cd ..
10.  
    cd redis4
11.  
    ./redis-server redis.conf
12.  
    cd ..
13.  
    cd redis5
14.  
    ./redis-server redis.conf
15.  
    cd ..
16.  
    cd redis6
17.  
    ./redis-server redis.conf
18.  
    cd ..

中止脚本stop-all.sh

1.  
   ./redis1/redis-cli -p 7001 shutdown
2.  
   ./redis1/redis-cli -p 7002 shutdown
3.  
   ./redis1/redis-cli -p 7003 shutdown
4.  
   ./redis1/redis-cli -p 7004 shutdown
5.  
   ./redis1/redis-cli -p 7005 shutdown
6.  
   ./redis1/redis-cli -p 7006 shutdown

PS：两个脚本都放在以下所属目录

1.  
   [root @localhost redis-cluster]# pwd
2.  
   /usr/ local/redis/redis-cluster
3.  
   [root @localhost redis-cluster]# ll
4.  
   total 80
5.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:52 redis1
6.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:51 redis2
7.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:53 redis3
8.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:53 redis4
9.  
   drwxr-xr- x 2 root root 4096 Nov 2 00:53 redis5
10.  
    drwxr-xr- x 2 root root 4096 Nov 2 00:53 redis6
11.  
    -rwxr-xr- x 1 root root 48141 Nov 2 00:16 redis-trib.rb
12.  
    -rw-r--r-- 1 root root 252 Nov 2 00:55 start-all.sh
13.  
    -rw-r--r-- 1 root root 216 Nov 2 00:57 stop-all.sh
14.  
    [root @localhost redis-cluster]#

修改权限

[root@localhost redis-cluster]# chmod -u+x start-all.sh stop-all.sh 

启动节点

1.  
   [root @localhost redis-cluster]# ./start-all.sh
2.  
   [root @localhost redis-cluster]# ps aux | grep redis
3.  
   root 2924 0.8 0.1 33932 2048 ? Ssl Nov01 3:53 ./redis-server *:6379 [cluster]
4.  
   root 11924 0.0 0.1 33936 1948 ? Ssl 01:01 0:00 ./redis-server *:7001 [cluster]
5.  
   root 11928 0.0 0.1 33936 1952 ? Ssl 01:01 0:00 ./redis-server *:7002 [cluster]
6.  
   root 11932 0.0 0.1 33936 1948 ? Ssl 01:01 0:00 ./redis-server *:7003 [cluster]
7.  
   root 11936 0.0 0.1 33936 1952 ? Ssl 01:01 0:00 ./redis-server *:7004 [cluster]
8.  
   root 11940 0.0 0.1 33936 1952 ? Ssl 01:01 0:00 ./redis-server *:7005 [cluster]
9.  
   root 11944 0.0 0.1 33936 1948 ? Ssl 01:01 0:00 ./redis-server *:7006 [cluster]
10.  
    root 11948 0.0 0.0 4360 748 pts/2 S+ 01:01 0:00 grep redis
11.  
    [root @localhost redis-cluster]#

执行建立集群命令

1.  
   [root @localhost redis-cluster]# pwd
2.  
   /usr/ local/redis/redis-cluster
3.  
   [root @localhost redis-cluster]# ./redis-trib.rb create --replicas 1 192.168.37.131:7001 192.168.37.131:7002 192.168.37.131:7003 192.168.37.131:7004 192.168.37.131:7005 192.168.37.131:7006

成功启动信息

1.  
   >>> Creating cluster
2.  
   Connecting to node 192.168.37.131:7001: OK
3.  
   Connecting to node 192.168.37.131:7002: OK
4.  
   Connecting to node 192.168.37.131:7003: OK
5.  
   Connecting to node 192.168.37.131:7004: OK
6.  
   Connecting to node 192.168.37.131:7005: OK
7.  
   Connecting to node 192.168.37.131:7006: OK
8.  
   >>> Performing hash slots allocation on 6 nodes...
9.  
   Using 3 masters:
10.  
    192.168.37.131:7001
11.  
    192.168.37.131:7002
12.  
    192.168.37.131:7003
13.  
    Adding replica 192.168.37.131:7004 to 192.168.37.131:7001
14.  
    Adding replica 192.168.37.131:7005 to 192.168.37.131:7002
15.  
    Adding replica 192.168.37.131:7006 to 192.168.37.131:7003
16.  
    M: 8b153503b52f83634e04b0077f32ef629ad91ee6 192.168.37.131:7001
17.  
    slots: 0-5460 (5461 slots) master
18.  
    M: f89799066dd8ecaaa1430559be4ce9c8c87055d8 192.168.37.131:7002
19.  
    slots: 5461-10922 (5462 slots) master
20.  
    M: 53d698ad56b09f89cfef34850213e2d0a44154dd 192.168.37.131:7003
21.  
    slots: 10923-16383 (5461 slots) master
22.  
    S: e73204399d08c14def1f71d0c5377cbc757dc4b8 192.168.37.131:7004
23.  
    replicates 8b153503b52f83634e04b0077f32ef629ad91ee6
24.  
    S: 1d5dcc8d1ccb6bce55efc3e3aadc690dc77808d8 192.168.37.131:7005
25.  
    replicates f89799066dd8ecaaa1430559be4ce9c8c87055d8
26.  
    S: e9458233cb85bd897ff694003e6d8a834eba2b44 192.168.37.131:7006
27.  
    replicates 53d698ad56b09f89cfef34850213e2d0a44154dd
28.  
    Can I set the above configuration? (type 'yes' to accept): y
29.  
    *** Aborting...
30.  
    [root@localhost redis-cluster] # ./redis-trib.rb create --replicas 1 192.168.37.131:7001 192.168.37.131:7002 192.168.37.131:7003 192.168.37.131:7004 192.168.37.131:7005 192.168.37.131:7006
31.  
    >>> Creating cluster
32.  
    Connecting to node 192.168.37.131:7001: OK
33.  
    Connecting to node 192.168.37.131:7002: OK
34.  
    Connecting to node 192.168.37.131:7003: OK
35.  
    Connecting to node 192.168.37.131:7004: OK
36.  
    Connecting to node 192.168.37.131:7005: OK
37.  
    Connecting to node 192.168.37.131:7006: OK
38.  
    >>> Performing hash slots allocation on 6 nodes...
39.  
    Using 3 masters:
40.  
    192.168.37.131:7001
41.  
    192.168.37.131:7002
42.  
    192.168.37.131:7003
43.  
    Adding replica 192.168.37.131:7004 to 192.168.37.131:7001
44.  
    Adding replica 192.168.37.131:7005 to 192.168.37.131:7002
45.  
    Adding replica 192.168.37.131:7006 to 192.168.37.131:7003
46.  
    M: 8b153503b52f83634e04b0077f32ef629ad91ee6 192.168.37.131:7001
47.  
    slots: 0-5460 (5461 slots) master
48.  
    M: f89799066dd8ecaaa1430559be4ce9c8c87055d8 192.168.37.131:7002
49.  
    slots: 5461-10922 (5462 slots) master
50.  
    M: 53d698ad56b09f89cfef34850213e2d0a44154dd 192.168.37.131:7003
51.  
    slots: 10923-16383 (5461 slots) master
52.  
    S: e73204399d08c14def1f71d0c5377cbc757dc4b8 192.168.37.131:7004
53.  
    replicates 8b153503b52f83634e04b0077f32ef629ad91ee6
54.  
    S: 1d5dcc8d1ccb6bce55efc3e3aadc690dc77808d8 192.168.37.131:7005
55.  
    replicates f89799066dd8ecaaa1430559be4ce9c8c87055d8
56.  
    S: e9458233cb85bd897ff694003e6d8a834eba2b44 192.168.37.131:7006
57.  
    replicates 53d698ad56b09f89cfef34850213e2d0a44154dd
58.  
    Can I set the above configuration? (type 'yes' to accept): yes
59.  
    >>> Nodes configuration updated
60.  
    >>> Assign a different config epoch to each node
61.  
    >>> Sending CLUSTER MEET messages to join the cluster
62.  
    Waiting for the cluster to join.....
63.  
    >>> Performing Cluster Check ( using node 192.168.37.131:7001)
64.  
    M: 8b153503b52f83634e04b0077f32ef629ad91ee6 192.168.37.131:7001
65.  
    slots: 0-5460 (5461 slots) master
66.  
    M: f89799066dd8ecaaa1430559be4ce9c8c87055d8 192.168.37.131:7002
67.  
    slots: 5461-10922 (5462 slots) master
68.  
    M: 53d698ad56b09f89cfef34850213e2d0a44154dd 192.168.37.131:7003
69.  
    slots: 10923-16383 (5461 slots) master
70.  
    M: e73204399d08c14def1f71d0c5377cbc757dc4b8 192.168.37.131:7004
71.  
    slots: ( 0 slots) master
72.  
    replicates 8b153503b52f83634e04b0077f32ef629ad91ee6
73.  
    M: 1d5dcc8d1ccb6bce55efc3e3aadc690dc77808d8 192.168.37.131:7005
74.  
    slots: ( 0 slots) master
75.  
    replicates f89799066dd8ecaaa1430559be4ce9c8c87055d8
76.  
    M: e9458233cb85bd897ff694003e6d8a834eba2b44 192.168.37.131:7006
77.  
    slots: ( 0 slots) master
78.  
    replicates 53d698ad56b09f89cfef34850213e2d0a44154dd
79.  
    [OK] All nodes agree about slots configuration.
80.  
    >>> Check for open slots...
81.  
    >>> Check slots coverage...
82.  
    [OK] All 16384 slots covered.
83.  
    [root@localhost redis-cluster] #

异常

1.  
   >>> Creating cluster
2.  
   Connecting to node 192.168.37.131:7001: OK
3.  
   [ERR] Node 192.168.37.131:7001 is not empty. Either the node already knows other nodes (check with CLUSTER NODES) or contains some key in database 0.

解决方法是删除生成的配置文件nodes.conf，若是不行则说明如今建立的结点包括了旧集群的结点信息，须要删除redis的持久化文件后再重启redis，好比：appendonly.aof、dump.rdb

首先，使用stop-all.sh中止服务

1.  
   [root @localhost redis-cluster]# ./stop-all.sh
2.  
   [root @localhost redis-cluster]# ps aux | grep redis
3.  
   root 2924 0.8 0.1 33932 2048 ? Ssl Nov01 3:54 ./redis-server *:6379 [cluster]
4.  
   root 11969 0.0 0.0 4360 744 pts/2 S+ 01:10 0:00 grep redis
5.  
   [root @localhost redis-cluster]#

而后每一个节点中的appendonly.aof、dump.rdb 、nodes.conf。

1.  
   [root @localhost redis-cluster]# rm -f redis*/dump.rdb
2.  
   [root @localhost redis-cluster]# rm -f redis*/appendonly.aof
3.  
   [root @localhost redis-cluster]# rm -f redis*/nodes.conf

而后使用脚本start-all.sh启动，再启动集群管理服务。

1.  
   [root @localhost redis-cluster]# ./start-all.sh
2.  
   [root @localhost redis-cluster]# ps aux | grep redis
3.  
   root 2924 0.8 0.1 33932 2048 ? Ssl Nov01 3:54 ./redis-server *:6379 [cluster]
4.  
   root 11980 0.0 0.1 33936 1952 ? Ssl 01:12 0:00 ./redis-server *:7001 [cluster]
5.  
   root 11982 0.0 0.1 33936 1952 ? Ssl 01:12 0:00 ./redis-server *:7002 [cluster]
6.  
   root 11984 0.0 0.1 33936 1952 ? Ssl 01:12 0:00 ./redis-server *:7003 [cluster]
7.  
   root 11986 0.0 0.1 33936 1952 ? Ssl 01:12 0:00 ./redis-server *:7004 [cluster]
8.  
   root 11988 0.0 0.1 33936 1948 ? Ssl 01:12 0:00 ./redis-server *:7005 [cluster]
9.  
   root 11990 0.0 0.1 33936 1948 ? Ssl 01:12 0:00 ./redis-server *:7006 [cluster]
10.  
    root 12004 0.0 0.0 4360 748 pts/2 S+ 01:12 0:00 grep redis
11.  
    [root @localhost redis-cluster]# ./redis-trib.rb create --replicas 1 192.168.37.131:7001 192.168.37.131:7002 192.168.37.131:7003 192.168.37.131:7004 192.168.37.131:7005 192.168.37.131:7006
12.  
    [root @localhost redis-cluster]#

Redis集群节点的操做

查询集群信息

集群建立成功登录任意redis结点查询集群中的节点状况。

客户端以集群方式登录：

1.  
   [root @localhost redis-cluster]# ./redis1/redis-cli -c -h 192.168.37.131 -p 7001
2.  
   192.168.37.131:7001>

说明： 
./redis1/redis-cli -c -h 192.168.37.131 -p 7001 ，其中-c表示以集群方式链接redis，-h指定ip地址，-p指定端口号

cluster nodes 查询集群结点信息

1.  
   192 .168.37.131:7001> cluster nodes
2.  
   8 e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003 master - 0 1478085160899 3 connected 10923-16383
3.  
   80 de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004 master - 0 1478085156858 8 connected 0-5460
4.  
   652 caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005 slave 3adba62fdcc331ce231ca580cd2c8701e047bc6d 0 1478085158876 9 connected
5.  
   1 cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001 myself,slave 80de7003738f74134a3403fc939fed253b7774f2 0 0 1 connected
6.  
   3 adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002 master - 0 1478085159889 9 connected 5461-10922
7.  
   cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006 slave 8e1186475f87c928e8a146d3804d9a2697246ad0 0 1478085157867 6 connected
8.  
   192 .168.37.131:7001>

cluster info 查询集群状态信息

1.  
   192 .168.37.131:7001> cluster info
2.  
   cluster_state:ok
3.  
   cluster_slots_assigned:16384
4.  
   cluster_slots_ok:16384
5.  
   cluster_slots_pfail:0
6.  
   cluster_slots_fail:0
7.  
   cluster_known_nodes:6
8.  
   cluster_size:3
9.  
   cluster_current_epoch:9
10.  
    cluster_my_epoch:8
11.  
    cluster_stats_messages_sent:12727
12.  
    cluster_stats_messages_received:10820
13.  
    192 .168.37.131:7001>

添加主节点

集群建立成功后能够向集群中添加节点，下面是添加一个master主节点。

首先，准备一个干净的redis节点。按上面集群版修改redis配置文件。开启该redis节点。

查看redis进程

1.  
   [root @localhost redis-cluster]# !ps
2.  
   ps aux | grep redis
3.  
   root 2924 0.6 0.1 33932 1800 ? Ssl Nov01 4:08 ./redis-server *:6379 [cluster]
4.  
   root 11980 0.0 0.2 33936 2216 ? Ssl 01:12 0:09 ./redis-server *:7001 [cluster]
5.  
   root 11982 0.0 0.2 33936 2244 ? Ssl 01:12 0:09 ./redis-server *:7002 [cluster]
6.  
   root 11984 0.0 0.2 33936 2220 ? Ssl 01:12 0:09 ./redis-server *:7003 [cluster]
7.  
   root 11986 0.0 0.2 33936 2216 ? Ssl 01:12 0:09 ./redis-server *:7004 [cluster]
8.  
   root 11988 0.0 0.2 33936 2228 ? Ssl 01:12 0:09 ./redis-server *:7005 [cluster]
9.  
   root 11990 0.0 0.2 33936 2212 ? Ssl 01:12 0:09 ./redis-server *:7006 [cluster]
10.  
    root 13913 0.0 0.1 33936 1952 ? Ssl 04:21 0:00 ./redis7/redis-server *:7007 [cluster]
11.  
    root 13917 0.0 0.0 4360 728 pts/3 S+ 04:21 0:00 grep redis
12.  
    [root @localhost redis-cluster]#

执行下边命令：

1.  
   [root@localhost redis-cluster] # ./redis-trib.rb add-node 192.168.37.131:7007 192.168.37.131:7001
2.  
   >>> Adding node 192.168.37.131:7007 to cluster 192.168.37.131:7001
3.  
   Connecting to node 192.168.37.131:7001: OK
4.  
   Connecting to node 192.168.37.131:7003: OK
5.  
   Connecting to node 192.168.37.131:7004: OK
6.  
   Connecting to node 192.168.37.131:7005: OK
7.  
   Connecting to node 192.168.37.131:7002: OK
8.  
   Connecting to node 192.168.37.131:7006: OK
9.  
   >>> Performing Cluster Check ( using node 192.168.37.131:7001)
10.  
    S: 1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001
11.  
    slots: ( 0 slots) slave
12.  
    replicates 80de7003738f74134a3403fc939fed253b7774f2
13.  
    M: 8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003
14.  
    slots: 10923-16383 (5461 slots) master
15.  
    1 additional replica(s)
16.  
    M: 80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004
17.  
    slots: 0-5460 (5461 slots) master
18.  
    1 additional replica(s)
19.  
    S: 652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005
20.  
    slots: ( 0 slots) slave
21.  
    replicates 3adba62fdcc331ce231ca580cd2c8701e047bc6d
22.  
    M: 3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002
23.  
    slots: 5461-10922 (5462 slots) master
24.  
    1 additional replica(s)
25.  
    S: cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006
26.  
    slots: ( 0 slots) slave
27.  
    replicates 8e1186475f87c928e8a146d3804d9a2697246ad0
28.  
    [OK] All nodes agree about slots configuration.
29.  
    >>> Check for open slots...
30.  
    >>> Check slots coverage...
31.  
    [OK] All 16384 slots covered.
32.  
    Connecting to node 192.168.37.131:7007: OK
33.  
    >>> Send CLUSTER MEET to node 192.168.37.131:7007 to make it join the cluster.
34.  
    [OK] New node added correctly.
35.  
    [root@localhost redis-cluster] #

查看集群结点发现7007已添加到集群中：

1.  
   [root@localhost redis-cluster] # ./redis1/redis-cli -c -h 192.168.37.131 -p 7001
2.  
   192.168.37.131:7001> cluster nodes
3.  
   8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003 master - 0 1478085870038 3 connected 10923-16383
4.  
   80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004 master - 0 1478085868020 8 connected 0-5460
5.  
   652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005 slave 3adba62fdcc331ce231ca580cd2c8701e047bc6d 0 1478085874075 9 connected
6.  
   1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001 myself,slave 80de7003738f74134a3403fc939fed253b7774f2 0 0 1 connected
7.  
   3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002 master - 0 1478085873064 9 connected 5461-10922
8.  
   cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006 slave 8e1186475f87c928e8a146d3804d9a2697246ad0 0 1478085875086 6 connected
9.  
   5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007 master - 0 1478085872056 0 connected
10.  
    192.168.37.131:7001>

hash槽从新分配

添加完主节点须要对主节点进行hash槽分配这样该主节才能够存储数据。 
redis集群有16384个槽，集群中的每一个master结点分配一些槽，经过查看集群结点能够看到槽占用状况。

给刚添加的7007结点分配槽：

第一步：链接上集群 
./redis-trib.rb reshard 192.168.37.131:7001（链接集群中任意一个可用结点都行）

1.  
   [root@localhost redis-cluster] # ./redis-trib.rb reshard 192.168.37.131:7001
2.  
   Connecting to node 192.168.37.131:7001: OK
3.  
   Connecting to node 192.168.37.131:7003: OK
4.  
   Connecting to node 192.168.37.131:7004: OK
5.  
   Connecting to node 192.168.37.131:7005: OK
6.  
   Connecting to node 192.168.37.131:7002: OK
7.  
   Connecting to node 192.168.37.131:7006: OK
8.  
   Connecting to node 192.168.37.131:7007: OK
9.  
   >>> Performing Cluster Check ( using node 192.168.37.131:7001)
10.  
    S: 1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001
11.  
    slots: ( 0 slots) slave
12.  
    replicates 80de7003738f74134a3403fc939fed253b7774f2
13.  
    M: 8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003
14.  
    slots: 10923-16383 (5461 slots) master
15.  
    1 additional replica(s)
16.  
    M: 80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004
17.  
    slots: 0-5460 (5461 slots) master
18.  
    1 additional replica(s)
19.  
    S: 652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005
20.  
    slots: ( 0 slots) slave
21.  
    replicates 3adba62fdcc331ce231ca580cd2c8701e047bc6d
22.  
    M: 3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002
23.  
    slots: 5461-10922 (5462 slots) master
24.  
    1 additional replica(s)
25.  
    S: cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006
26.  
    slots: ( 0 slots) slave
27.  
    replicates 8e1186475f87c928e8a146d3804d9a2697246ad0
28.  
    M: 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007
29.  
    slots: ( 0 slots) master
30.  
    0 additional replica(s)
31.  
    [OK] All nodes agree about slots configuration.
32.  
    >>> Check for open slots...
33.  
    >>> Check slots coverage...
34.  
    [OK] All 16384 slots covered.
35.  
    How many slots do you want to move (from 1 to 16384)?

第二步：输入要分配的槽数量

1.  
   How many slots do you want to move (from 1 to 16384)? 1000
2.  
   What is the receiving node ID?

输入1000表示要分配1000个槽

第三步：输入接收槽的结点id

这里准备给7007分配槽，经过cluster nodes查看7007结点id为5d6c61ecff23bff3b0fb01a86c66d882f2d402a0

输入：5d6c61ecff23bff3b0fb01a86c66d882f2d402a0

1.  
   What is the receiving node ID? 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0
2.  
   Please enter all the source node IDs.
3.  
   Type 'all' to use all the nodes as source nodes for the hash slots.
4.  
   Type 'done' once you entered all the source nodes IDs.
5.  
   Source node #1:

第四步：输入源结点id 
输入源结点id，槽将从源结点中拿，分配后的槽在源结点中就不存在了。 
输入all表示从全部源结点中获取槽。 
输入done取消分配。

这里输入all

1.  
   Source node #1:all
2.  
    
3.  
   省略了不少
4.  
    
5.  
   Moving slot 11253 from 8e1186475f87c928e8a146d3804d9a2697246ad0
6.  
   Moving slot 11254 from 8e1186475f87c928e8a146d3804d9a2697246ad0
7.  
   Moving slot 11255 from 8e1186475f87c928e8a146d3804d9a2697246ad0
8.  
   Do you want to proceed with the proposed reshard plan (yes/no)?

第五步：输入yes开始移动槽到目标结点id

1.  
   省略了不少
2.  
    
3.  
   Moving slot 11253 from 192.168.37.131:7003 to 192.168.37.131:7007:
4.  
   Moving slot 11254 from 192.168.37.131:7003 to 192.168.37.131:7007:
5.  
   Moving slot 11255 from 192.168.37.131:7003 to 192.168.37.131:7007:
6.  
   [root@localhost redis-cluster]#

第六步：查看结点信息

1.  
   [root@localhost redis-cluster] # ./redis1/redis-cli -c -h 192.168.37.131 -p 7001
2.  
   192.168.37.131:7001> cluster nodes
3.  
   8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003 master - 0 1478086754466 3 connected 11256-16383
4.  
   80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004 master - 0 1478086758509 8 connected 333-5460
5.  
   652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005 slave 3adba62fdcc331ce231ca580cd2c8701e047bc6d 0 1478086756490 9 connected
6.  
   1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001 myself,slave 80de7003738f74134a3403fc939fed253b7774f2 0 0 1 connected
7.  
   3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002 master - 0 1478086757500 9 connected 5795-10922
8.  
   cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006 slave 8e1186475f87c928e8a146d3804d9a2697246ad0 0 1478086755477 6 connected
9.  
   5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007 master - 0 1478086759518 10 connected 0-332 5461-5794 10923-11255
10.  
    192.168.37.131:7001>

查看结点信息，能够发现7007结点已经从三个主节点中获取了slot。

添加从节点

集群建立成功后能够向集群中添加节点，下面是添加一个slave从节点。 
添加7008从结点，将7008做为7007的从结点。

首先，准备一个干净的redis节点。按上面集群版修改redis配置文件。开启该redis节点。

查看redis进程

1.  
   [root @localhost redis-cluster]# ps aux | grep redis
2.  
   root 2924 0.5 0.1 33932 1800 ? Ssl Nov01 4:12 ./redis-server *:6379 [cluster]
3.  
   root 11980 0.0 0.2 33936 2308 ? Ssl 01:12 0:13 ./redis-server *:7001 [cluster]
4.  
   root 11982 0.0 0.2 33936 2288 ? Ssl 01:12 0:13 ./redis-server *:7002 [cluster]
5.  
   root 11984 0.0 0.2 33936 2236 ? Ssl 01:12 0:13 ./redis-server *:7003 [cluster]
6.  
   root 11986 0.0 0.2 33936 2288 ? Ssl 01:12 0:13 ./redis-server *:7004 [cluster]
7.  
   root 11988 0.0 0.2 33936 2248 ? Ssl 01:12 0:13 ./redis-server *:7005 [cluster]
8.  
   root 11990 0.0 0.2 33936 2244 ? Ssl 01:12 0:13 ./redis-server *:7006 [cluster]
9.  
   root 13913 0.1 0.2 33936 2092 ? Ssl 04:21 0:04 ./redis7/redis-server *:7007 [cluster]
10.  
    root 14000 0.1 0.1 33936 1948 ? Ssl 05:24 0:00 ./redis-server *:7008 [cluster]
11.  
    root 14006 0.0 0.0 4360 732 pts/3 S+ 05:24 0:00 grep redis
12.  
    [root @localhost redis-cluster]#

命令格式为：

./redis-trib.rb add-node --slave --master-id 主节点id 添加节点的ip和端口 集群中已存在节点ip和端口

执行以下命令：

1.  
   [root@localhost redis-cluster] # ./redis-trib.rb add-node --slave --master-id 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7008 192.168.37.131:7001
2.  
   >>> Adding node 192.168.37.131:7008 to cluster 192.168.37.131:7001
3.  
   Connecting to node 192.168.37.131:7001: OK
4.  
   Connecting to node 192.168.37.131:7003: OK
5.  
   Connecting to node 192.168.37.131:7004: OK
6.  
   Connecting to node 192.168.37.131:7005: OK
7.  
   Connecting to node 192.168.37.131:7002: OK
8.  
   Connecting to node 192.168.37.131:7006: OK
9.  
   Connecting to node 192.168.37.131:7007: OK
10.  
    >>> Performing Cluster Check ( using node 192.168.37.131:7001)
11.  
    S: 1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001
12.  
    slots: ( 0 slots) slave
13.  
    replicates 80de7003738f74134a3403fc939fed253b7774f2
14.  
    M: 8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003
15.  
    slots: 11256-16383 (5128 slots) master
16.  
    1 additional replica(s)
17.  
    M: 80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004
18.  
    slots: 333-5460 (5128 slots) master
19.  
    1 additional replica(s)
20.  
    S: 652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005
21.  
    slots: ( 0 slots) slave
22.  
    replicates 3adba62fdcc331ce231ca580cd2c8701e047bc6d
23.  
    M: 3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002
24.  
    slots: 5795-10922 (5128 slots) master
25.  
    1 additional replica(s)
26.  
    S: cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006
27.  
    slots: ( 0 slots) slave
28.  
    replicates 8e1186475f87c928e8a146d3804d9a2697246ad0
29.  
    M: 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007
30.  
    slots: 0-332,5461-5794,10923-11255 (1000 slots) master
31.  
    0 additional replica(s)
32.  
    [OK] All nodes agree about slots configuration.
33.  
    >>> Check for open slots...
34.  
    >>> Check slots coverage...
35.  
    [OK] All 16384 slots covered.
36.  
    Connecting to node 192.168.37.131:7008: OK
37.  
    >>> Send CLUSTER MEET to node 192.168.37.131:7008 to make it join the cluster.
38.  
    Waiting for the cluster to join.
39.  
    >>> Configure node as replica of 192.168.37.131:7007.
40.  
    [OK] New node added correctly.
41.  
    [root@localhost redis-cluster] #

5d6c61ecff23bff3b0fb01a86c66d882f2d402a0是7007结点的id，能够经过cluster nodes查看。

查看集群中的结点，刚添加的7008为7007的从节点：

1.  
   [root@localhost redis-cluster] # ./redis1/redis-cli -p 7001 -c 127.0.0.1:7001> cluster nodes
2.  
   8e1186475f87c928e8a146d3804d9a2697246ad0 192.168.37.131:7003 master - 0 1478089964769 3 connected 11256-16383
3.  
   80de7003738f74134a3403fc939fed253b7774f2 192.168.37.131:7004 master - 0 1478089966584 8 connected 333-5460
4.  
   652caf5daf7971135679951324eba7b50e99251a 192.168.37.131:7005 slave 3adba62fdcc331ce231ca580cd2c8701e047bc6d 0 1478089963748 9 connected
5.  
   1cd6482fd7038d78ad556b52b0cb9e2590ad5598 192.168.37.131:7001 myself,slave 80de7003738f74134a3403fc939fed253b7774f2 0 0 1 connected
6.  
   3adba62fdcc331ce231ca580cd2c8701e047bc6d 192.168.37.131:7002 master - 0 1478089966787 9 connected 5795-10922
7.  
   cf23ca6d78cba3d3924065e1f7a394f6c51d4b28 192.168.37.131:7006 slave 8e1186475f87c928e8a146d3804d9a2697246ad0 0 1478089962729 6 connected
8.  
   5c97e8eab019c40ea3df4925c7400fe7df1846bb 192.168.37.131:7008 slave 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 0 1478089961713 10 connected
9.  
   5d6c61ecff23bff3b0fb01a86c66d882f2d402a0 192.168.37.131:7007 master - 0 1478089965777 10 connected 0-332 5461-5794 10923-11255
10.  
    127.0.0.1:7001>

删除结点

./redis-trib.rb del-node 192.168.37.131:7007 5d6c61ecff23bff3b0fb01a86c66d882f2d402a0

删除已经占有hash槽的结点会失败，报错以下：

[ERR] Node 192.168.37.131:7007 is not empty! Reshard data away and try again.

须要将该结点占用的hash槽分配出去，请参考《hash槽从新分配》这段内容。

客户端对Redis集群的使用方法

使用redis命令行客户端链接

1.  
   [root@localhost redis-cluster] # ./redis1/redis-cli -p 7001 -c
2.  
   127.0.0.1:7001> get a
3.  
   -> Redirected to slot [15495] located at 192.168.37.131:7003
4.  
   ( nil)
5.  
   192.168.37.131:7003>

必定要加-c参数,节点之间就能够互相跳转

使用jedis链接

1.  
   package com.pc.jedis.test;
2.  
    
3.  
   import java.util.HashSet;
4.  
   import java.util.Set;
5.  
    
6.  
   import redis.clients.jedis.HostAndPort;
7.  
   import redis.clients.jedis.JedisCluster;
8.  
    
9.  
   /**
10.  
    * Jedis集群测试
11.  
    *
12.  
    * @author Switch
13.  
    * @data 2017年2月11日
14.  
    * @version V1.0
15.  
    */
16.  
    public class JedisClusterTest {
17.  
    public static void main(String[] args) {
18.  
    // 建立并填充节点信息
19.  
    Set<HostAndPort> nodes = new HashSet<>();
20.  
    nodes.add( new HostAndPort("192.168.37.131", 7001));
21.  
    nodes.add( new HostAndPort("192.168.37.131", 7002));
22.  
    nodes.add( new HostAndPort("192.168.37.131", 7003));
23.  
    nodes.add( new HostAndPort("192.168.37.131", 7004));
24.  
    nodes.add( new HostAndPort("192.168.37.131", 7005));
25.  
    nodes.add( new HostAndPort("192.168.37.131", 7006))
26. // 建立JedisCluster对象
27. JedisCluster jedisCluster = new JedisCluster(nodes);
28.  
     
29.  
    // 使用jedisCluster操做redis
30. String key = "jedisCluster";
31. String setResult = jedisCluster.set(key, "hello redis!");
32. System.out.println(setResult);
33.  
34. String getResult = jedisCluster.get(key);
35.  
    System.out.println(getResult);
36.  
37. // 关闭jedisCluster（程序执行完后才能关闭，内部封装了链接池）
38.  
    jedisCluster.close();
39.  
    }
40.  
    }

——参考《Redis官方文档》