centos7中keepalived+nginx作双机热备和反向代理

时间 2019-11-17

标签 centos7 centos keepalived+nginx keepalived nginx 反向代理栏目 Tomcat 繁體版

原文原文链接

我虚拟了三台centos7，要将one（192.168.56.31）、two（192.168.56.32）配置成nginx+keepalived双机热备（思路是当一台服务器的nginx挂掉时，能够自动用另外一台的nginx,就是当主服务器nginx的端口不通时，自动关闭本身的keepalived服务，这样虚拟ip就会指到备用ip上），three（192.168.56.33）作tomcat服务器。实验环境下先把防火墙和selinux关闭：setenforce 0 && systemctl stop firewalld
首先进行nginx的安装，我用的是最简单的yum安装方式，自动的yum源中没有nginx，咱们须要安装一下yum源而后再来安装nginx：yum -y install epel-release && yum -y install nginx。而后再来安装一下keepalived：yum -y install keepalived。安装完成后可使用keepalived --help 来进行keepalived的命令帮助。二者的配置文件都在etc目录下。再将jdk和tomcat上传到three中并安装，我这设置了三个tomcat实例，端口分别为：8180、8280、8380。
而后先来配置nginx，配置文件在/etc/nginx/中：html

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;                          #运行用户，避免权限问题，就用root吧
worker_processes auto;               #工做进程数，表示自动选择，cpu多的话，能够手动设置通常为cpu的倍数。
error_log /var/log/nginx/error.log;  #错误日志路径
pid /run/nginx.pid;                  #PID文件路径

# Load dynamic modules. See /usr/share/nginx/README.dynamic.  
include /usr/share/nginx/modules/*.conf;   #包含这个文件的信息

events {       
    worker_connections 1024;         #工做链接数，指一个进程能够产生多少个链接，用 ulimit -n 来查看，可是这是理论上，不必设置这么大
}

http {     
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '   #定义访问日志的显示格式
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;                                #访问日志的路径

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;                                 #将指定的配置文件引入到这个文件中来
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {                                          #服务器设置，能够设置多个服务器
        listen       80 default_server;               #设置监听的端口
        listen       [::]:80 default_server;            
        server_name  _;                               #服务器名称
        root         /usr/share/nginx/html;           #服务器默认网站的目录

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {                                  #设置请求转向
        proxy_pass http://cs;
        proxy_set_header HOST $host;                   #不添加下面三行时，有次转发后点击链接前面域名会变为cs
        proxy_set_header X-Real-IP $remote_addr; 
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 
        
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }
upstream cs {                      #这是一个反向链接池,指向了三个tomcat实例
 server 192.168.56.33:8180;
 server 192.168.56.33:8280;
 server 192.168.56.33:8380;
}
# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers HIGH:!aNULL:!MD5;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

配置完成后把，nginx和三个tomcat服务起来后，就能成功访问了，接下来咱们再来配置一下keepalived，配置文件keeplived.conf在安装好后的/etc/keeplived中，下面是配置文件的解释：node

! Configuration File for keepalived

global_defs {                                                       #全局定义块
   notification_email {                                             #指定keepalived在发生切换时须要发送email到的对象，一行一个。
     acassen@firewall.loc
     failover@firewall.loc
     sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc            #发件人是谁
   smtp_server 192.168.200.1                                        #指定smtp服务器地址                              
   smtp_connect_timeout 30                                          #指定smtp链接超时时间
   router_id LVS_DEVEL                                              #Lvs负载均衡器标识（lvs_id）。在一个网络内，它应该是惟一的。
   vrrp_skip_check_adv_addr
   vrrp_strict
   vrrp_garp_interval 0
   vrrp_gna_interval 0
}

vrrp_instance VI_1 {                                               #虚拟vrrp设置模块                                        
    state MASTER                                                   #指定那个为master，那个为backup，若是设置了nopreempt这个值不起做用，主备靠priority决定。
    interface eth0                                                 #设置实例绑定的网卡
    virtual_router_id 51                                           #虚拟路由的id
    priority 100                                                   #优先级，高优先级竞选为master
    advert_int 1                                                   #检查间隔，默认1秒
    authentication {                                               #认证设置
        auth_type PASS                                             #认证方式
        auth_pass 1111                                             #认证密码
    }
    virtual_ipaddress {                                            #虚拟ip地址设置
        192.168.200.16
        192.168.200.17
        192.168.200.18
    }
}

#虚拟服务器virtual_server定义块 ，该部分是用来管理LVS的，是实现keepalive和LVS相结合的模块。ipvsadm命令能够实现的管理在这里均可以经过参数配置实现。
virtual_server 192.168.200.100 443 {                             #虚拟IP地址，要和vrrp_instance模块中的virtual_ipaddress地址一致
    delay_loop 6                                                 #健康检查时间间隔          
    lb_algo rr                                                   #lvs调度算法有rr|wrr|lc|wlc|lblc|sh|dh 
    lb_kind NAT                                                  #负载均衡转发规则NAT|DR|RUN                                                       
    persistence_timeout 50                                       #会话保持时间
    protocol TCP                                                 #使用的协议

    real_server 192.168.201.100 443 {                            #真实ip地址
        weight 1                                                 #默认为1,0为失效
        SSL_GET {
            url {
              path /
              digest ff20ad2481f97b1754ef3e12ecd3a9cc
            }
            url {
              path /mrtg/
              digest 9b3a0c85a887a256d6939da88aabd8cd
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

virtual_server 10.10.10.2 1358 {
    delay_loop 6
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP

    sorry_server 192.168.200.200 1358

    real_server 192.168.200.2 1358 {
        weight 1
        HTTP_GET {
            url {
              path /testurl/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            url {
              path /testurl2/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            url {
              path /testurl3/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 192.168.200.3 1358 {
        weight 1
        HTTP_GET {
            url {
              path /testurl/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334c
            }
            url {
              path /testurl2/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334c
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

virtual_server 10.10.10.3 1358 {
    delay_loop 3
    lb_algo rr
    lb_kind NAT
    persistence_timeout 50
    protocol TCP

    real_server 192.168.200.4 1358 {
        weight 1
        HTTP_GET {
            url {
              path /testurl/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            url {
              path /testurl2/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            url {
              path /testurl3/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }

    real_server 192.168.200.5 1358 {
        weight 1
        HTTP_GET {
            url {
              path /testurl/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            url {
              path /testurl2/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            url {
              path /testurl3/test.jsp
              digest 640205b7b0fc66c1ea91c463fac6334d
            }
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        }
    }
}

配置文件有点多，咱们用不到的功能能够不往上配，这样能使性能更好！个人脚本以下：linux

global_defs {
   notification_email {
     652179279@qq.com
   }
   notification_email_from
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id one
}
vrrp_script chk_nginx {                  #由于要检测nginx服务状态，因此建立一个检查脚本
    script "/usr/local/check_ng.sh"
    interval 3
}
vrrp_instance VI_1 {
    state MASTER
    interface enp0s8
    virtual_router_id 60
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }

    virtual_ipaddress {
        192.168.56.100
    }
    track_script {
        chk_nginx
    }
}

接下来就是去上面配置的路径建立脚本了vi /usr/local/check_ng.sh ：nginx

#!/bin/bash
d=`date --date today +%Y%m%d_%H:%M:%S`
n=`ps -C nginx --no-heading|wc -l`
if [ $n -eq "0" ]; then
        systemctl start nginx
        n2=`ps -C nginx --no-heading|wc -l`
        if [ $n2 -eq "0"  ]; then
                echo "$d nginx down,keepalived will stop" >> /var/log/check_ng.log
                systemctl stop keepalived
        fi
fi

而后增长执行权限。再配置备用机：算法

global_defs {
   notification_email {
     652179279@qq.com
   }
   notification_email_from
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id two
}
vrrp_script chk_nginx {
    script "/usr/local/sbin/check_ng.sh"
    interval 3
}
vrrp_instance VI_1 {
    state BACKUP
    interface enp0s8
    virtual_router_id 60
    priority 90
    advert_int 1
    authentication {                                               
        auth_type PASS              
        auth_pass 1111                                             
    }
    
    virtual_ipaddress {
        192.168.56.100
    }
    track_script {
        chk_nginx
    }
}

配置检测脚本和配置主机同样！生产环境下须要开放112端口，是keepalived的！centos