redis cluster初探之部署

1、简介

在3.0版本以前，redis经过哨兵实现主从的高可用，在3.0版本以后，redis官方推出了高可用的redis集群解决方案。

重点知识：数据分区
分布式数据库是将数据根据分区规则划分到多个节点上，每一个节点负责存储一部分数据；经常使用的分区规则有哈希分区与顺序分区两种，两种方式各有优点，redis集群使用的是哈希分区的虚拟槽分区方式
redis集群采用的虚拟槽分区，全部的键经过计算（slot=CRC16（key）&16383）映射到0～16383整数槽位内，每个节点负责一部分的槽位。

2、redis安装

前提：节点间公钥推送 互免密

2.一、配置ansible hosts

ansible所在节点：192.168.143.130

vim /etc/ansible/hosts 
[redis-cluster]
192.168.143.131
192.168.143.132

2.二、在ansible所在节点安装ruby环境

yum install rubygem-psych ruby-irb ruby-devel rubygem-rdoc rubygem-io-console rubygems rubygem-abrt rebygem-json ruby-libs rubygem-bigdecimal -y

2.三、在ansible所在节点下载redis

Redis：4.0.14 https://redis.io/download
Redis-gem：3.3.5 https://rubygems.org/gems/redis/versions/

2.四、在ansible所在节点安装redis-gem（redis的ruby客户端）

[root@node1 local]# gem install redis
Successfully installed redis-3.3.5
Parsing documentation for redis-3.3.5
Installing ri documentation for redis-3.3.5
1 gem installed

2.五、在ansible所在节点安装redis节点（集群模式）

解压

cd /usr/local/
tar -xzvf redis-4.0.14.tar.gz -C .

编译安装

cd redis-4.0.14/
make && make install

准备工做

mkdir redis-cluster/
cp redis-4.0.14/src/redis-server redis-cluster/
cp redis-4.0.14/src/redis-cli redis-cluster/

准备公共配置文件

vim redis-cluster/redis_comm.conf 
#集群相关的配置
cluster-enabled yes
cluster-node-timeout 15000
cluster-require-full-coverage no

在全部节点建立redis集群工做目录

ansible redis-cluster -m file -a 'path=/home/redis/redis_cluster/redis_node state=directory'

将redis-server和redis-cli放在集群各节点的/usr/bin目录下

ansible redis-cluster -m copy -a 'src=/usr/local/redis-cluster/redis-server dest=/usr/bin/redis-server mode=755'
ansible redis-cluster -m copy -a 'src=/usr/local/redis-cluster/redis-cli dest=/usr/bin/redis-cli mode=755'

将公共配置文件放在集群各节点的/home/redis/redis_cluster/redis_node下

ansible redis-cluster -m copy -a 'src=/usr/local/redis-cluster/redis_comm.conf dest=/home/redis/redis_cluster/redis_node/redis_comm.conf mode=644'`

建立redis节点脚本（redis_node_build.sh ）

#!/bin/bash
#
#Redis can open up to 16
redis_node=16
redis_workdir="/home/redis/redis_cluster/redis_node"
start_port=9001
redis_conf="redis.conf"
#Initialize the redis process
rm -rf ${redis_workdir}/90*
ps -ef|grep redis-server|grep -v grep
if [ $? -eq 0 ];then
    killall redis-server
    echo "The node has been reset successfully"
else
    echo "Node initialization succeeded"
Fi
#Modify kernel parameters: maximum listening queue length per port
conn_num=`sysctl -a|grep net.core.somaxconn|awk -F "=" '{print $2}'`
if [ $conn_num -ne 16384 ];then
    sed -i 's/net.core.somaxconn//g' /etc/sysctl.conf
    echo "net.core.somaxconn=16384" >> /etc/sysctl.conf
    sysctl -p
    sysctl -w net.core.somaxconn=16384
else
    echo "net.core.somaxconn = ${conn_num}"
fi
#Modify the maximum number of file descriptors that a process can open
echo 'ulimit -n 16384' >> /etc/profile
source /etc/profile
#Create a redis node
for ((i=0;i<$redis_node;i++));
do
    next_port=$(expr $start_port + $i)
    #Create an node directory
    mkdir -p $redis_workdir/$next_port
    cd $redis_workdir/$next_port
    #Prepare each node configuration file
    echo "include $redis_workdir/redis_comm.conf" >> $redis_conf
    echo "port ${next_port}" >> $redis_conf
    echo "pidfile ${redis_workdir}/${next_port}/redis.pid" >> $redis_conf
    echo "dir ${redis_workdir}/${next_port}" >> $redis_conf
    echo "logfile ${redis_workdir}/${next_port}/redis.log" >> $redis_conf
    #Run the redis node
    /usr/bin/redis-server redis.conf
done

将建立redis节点的脚本放在集群各节点的/home/redis/redis_cluster/redis_node下

ansible redis-cluster -m copy -a 'src=/usr/local/redis-cluster/redis_node_build.sh dest=/home/redis/redis_cluster/redis_node/redis_node_build.sh  mode=755'

在各节点执行建立redis节点脚本

ansible redis-cluster -m shell -a 'cd /home/redis/redis_cluster/redis_node/;./redis_node_build.sh &> /dev/null'

查看进程

3、redis-cluster安装

3.一、准备

cp /usr/local/redis-4.0.14/src/redis-trib.rb /usr/local/redis-cluster/

3.二、生成集群ip列表（全主集群）

cd /usr/local/redis-cluster/
vim redis_cluster_build.py
#!/usr/bin/python
#
ret = "/usr/local/redis-cluster/redis-trib.rb create --replicas 0 "
ip=open('./iplist-redis')
for i in ip:
    for j in range(16):
        ret += i.replace('\n','')+":"+str(9001+j)+" "
print ret

3.三、建立集群

/usr/local/redis-cluster/redis-trib.rb create --replicas 0 192.168.143.131:9001 192.168.143.131:9002 192.168.143.131:9003 192.168.143.131:9004 192.168.143.131:9005 192.168.143.131:9006 192.168.143.131:9007 192.168.143.131:9008 192.168.143.131:9009 192.168.143.131:9010 192.168.143.131:9011 192.168.143.131:9012 192.168.143.131:9013 192.168.143.131:9014 192.168.143.131:9015 192.168.143.131:9016 192.168.143.132:9001 192.168.143.132:9002 192.168.143.132:9003 192.168.143.132:9004 192.168.143.132:9005 192.168.143.132:9006 192.168.143.132:9007 192.168.143.132:9008 192.168.143.132:9009 192.168.143.132:9010 192.168.143.132:9011 192.168.143.132:9012 192.168.143.132:9013 192.168.143.132:9014 192.168.143.132:9015 192.168.143.132:9016

Ps：create参数是redis-trib.rb命令建立集群的参数 --replicas 1参数意思是主从各占节点总数的一半，节点列表的第一个节点是master，第二个节点是第一个节点的slave，依次日后；--replicas 0参数为建立全master集群，意思是节点列表的全部节点都是master

3.四、查看集群状态

[root@node2 ~]# redis-cli -h 192.168.143.131 -p 9001
192.168.143.131:9001> cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:32
cluster_size:32
cluster_current_epoch:32
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1198
cluster_stats_messages_pong_sent:1174
cluster_stats_messages_sent:2372
cluster_stats_messages_ping_received:1143
cluster_stats_messages_pong_received:1198
cluster_stats_messages_meet_received:31
cluster_stats_messages_received:2372

参考文章：
Redis-cluster docs： https://redis.io/topics/cluster-tutorial
replicas参数： https://redis.io/commands/cluster-replicas
Redis-cluster command： https://redis.io/commands#cluster
Redis-cluster-create scripts： https://github.com/antirez/redis/blob/unstable/utils/create-cluster/create-cluster