用swoole实现nginx日志解析

时间 2019-12-01

原文原文链接

1.原技术路线解析

在nging配置中将日志信息交给syslog处理，rsyslog配置中将数据传递给了514端口解析，而后将解析好的数据传入elasticsearch中。php

nginx配置html

server {
        listen 80; 
        listen [::]:80;
        server_name test.86dev.wrddns.com;
        
        # 如下两行将日志写入syslog
        access_log syslog:server=unix:/dev/log,facility=local5,tag=web_1,severity=info main;
        error_log syslog:server=unix:/dev/log,facility=local5,tag=web_1,severity=error warn;
        
        # ....其余配置
    }

/etc/rsyslog.confnginx

# 配置文件，存放解析规则xxx.conf和ruleBase文件xx.rb
$IncludeConfig /etc/rsyslog.d/*.conf

# 配置将日志放到哪一个端口解析
local5.* @10.3.19.86:514

在实际应用过程当中有一些问题，不能和php上面的一些配置进行配合记录，解析规则很差配置，有些内容解析很差，因此探索使用新的技术路线。laravel

2.新技术路线

尝试使用新技术路线，经过swoole起一个服务，而后监听9502端口，rsyslog将日志推向该端口，对日志进行解析后推入elasticsearch，此时能够获取到php端的一些配置。如下是大致思路git

2.1 选择swoole的server类型。

这里再也不赘述swoole的安装，首要考虑的问题是原推向514的协议类型。github

先查看端口的协议类型web

# root @ WENGINE in ~ [9:48:51] 
$ netstat -antup | grep 514 
udp        0      0 0.0.0.0:514                 0.0.0.0:*                               23560/rsyslogd      
udp        0      0 :::514                      :::*                                    23560/rsyslogd

能够看到是udp协议，因此选用swoole的 upd服务正则表达式

结合laravel的commands来编写服务端数据库

<?php

namespace App\Console\Swoole;

use Illuminate\Console\Command;
use swoole_websocket_server;
use swoole_server;
use swoole_process;
use swoole_sock_udp;
use UAParser\Parser;
use GeoIp2\Database\Reader;
use Wrd\Framework\Models\SysConfig;
use Elasticsearch\ClientBuilder;

class SwooleServer extends Command
{

    protected $signature = 'swoole-server start
                            {cmd=start : can use start}
                            {--daemon : set to run in daemonize mode}
                            ';

    protected $description = 'swoole server control';

    public $access_buffer = [];


    public function __construct()
    {
        parent::__construct();
    }

    public function handle()
    {
        $command = $this->argument('cmd');
        $option = $this->option('daemon');
        switch ($command) {
            case 'start':
                $this->initWs($option);
                break;
            default:
                $this->info('请按照下面格式输入命令：php artisan swoole-server {start}');
                break;
        }

    }
        
    public function initWs($daemonize = false) 
    {
        if ($daemonize) {
            $this->info('Starting Websocket server in daemon mode...');
        } else {
            $this->info('Starting Websocket server in interactive mode...');
        }

        $server = new swoole_server('0.0.0.0', 9502, SWOOLE_PROCESS, SWOOLE_SOCK_UDP);
        $server->set([
            'daemonize' => $daemonize,
            'log_file' => '/var/www/html/storage/logs/websocket.log',
            'worker_num' => 1,
            'task_worker_num' => 1,
        ]);

        $server->on('Packet', function($serv, $data, $clientInfo) 
        {
            $serv->task($data);
        });
        
        $server->on('Task', function ($serv, $task_id, $from_id, $data) {

            //经过正则表达式提取出须要的信息，不一样的日志格式须要不一样的正则，这里只写一种状况
            $rule = '/\<\d*\>.*\d{2}\:\d{2}\:\d{2}\s[^\s]*\s[^\s]*\s(\w*\_\d*)\:\s\[Customize-format\]/';

            preg_match($rule, $data, $matches);

            if (empty($matches)) {
                $this->writeLog($data); //记录下没法解析的日志，更正正则
                return false;
            }
            
            $vhost = $matches[1];
            $ip = $matches[2];
            //...更多参数
            
            $ua = $matches[12];
            //解析UA，这里使用的解析库https://github.com/ua-parser/uap-php
            $parser = Parser::create();
            $parser_ua = $parser->parse($ua);
            $browser = $parser_ua->ua->family;
            $os = $parser_ua->os->family;
            $device = $parser_ua->device->family; 

            //解析IP，这里使用的解析库https://github.com/maxmind/GeoIP2-php
            $reader = new Reader(public_path().'/geoip2/GeoLite2-City.mmdb');

            try{
                $record = $reader->city($ip);
                $country = $record->country->isoCode;
                $continent = $record->continent->names['zh-CN'];
                $subdivisions = $record->mostSpecificSubdivision->names['zh-CN'];
                $city = $record->city->names['zh-CN'];
                $geoip = array(
                    'location' => array($record->location->longitude, $record->location->latitude)
                );

            } catch (\Exception $e) {
               //若是ip没有被收录（项目有不少内网ip），则拿数据库中的提早配置项，进行解析
            }
            
            $res = array(
                'vhost' => $vhost,
                'ip' => $ip,
                 // ...其它项
                'token' => $token,
                'browser' => $browser,
                'os' => $os,
                'device' => $device,
                'continent' => $continent,
                'country' => $country,
                'subdivisions' => $subdivisions,
                'city' => $city,
                'geoip' => $geoip,
            );

            $this->access_buffer[] = $res;

            //每隔一段时间，写入到elasticsearch
            if (count($this->access_buffer) > 0 && time() - strtotime($this->access_buffer[0]['@timestamp']) > 10) {
                $insert_data = $this->access_buffer;
                $this->access_buffer = [];

                $this->insertElasticsearch('access', $insert_data);
            }

            //return 数据 给 Finish
            return "Task {$task_id}'s result";
        });
        
        $server->on('Finish', function ($serv,$task_id, $data) {
            echo "Task {$task_id} finish\n";
        });

        $server->start();
                
    }
    
    public function insertElasticsearch($type='access', $data){
        foreach($data as $item){
            $params['body'][] = [
                'index' => [
                    '_index' => $type.'-'.date('Y.m.d', time()),
                    '_type' => 'events',
                ]
            ];
            $params['body'][] = $item;
        }

        extract(\Config::get('app.elastic', [
            'host' => '127.0.0.1',
            'port' => '9200'
        ]));

        //往elasticsearch写数据，这里使用的库https://github.com/elastic/elasticsearch-php
        $helper = ClientBuilder::create()
            ->setHosts([$host.":".$port])
            ->build();

        if (!empty($params['body'])) {
            $response = $helper->bulk($params);
            //var_dump($response);
        }

    }
    
    public function writeLog($info){
        $alert_message = array(
            'error' => '此条信息未能命中日志格式，未写入elasticsearch',
            'info' => $info
        );
        \Log::alert($alert_message);
    }

2.2 经过更改nginx的main日志格式简化正则表达式

nginx的配置中的apache

log_format main [$proxy_add_x_forwarded_for]-[$remote_user]-[$time_local]-[$request]-[$status]-[$bytes_sent]-[$http_host]-[$http_referer]-[$http_user_agent]-[$cookie_wengine_ticket]-[archer-main];

加了特殊符号，而且最后给了一个标识,这样能提升命中准确度

2.3 更改rsyslog的端口

local5.* @10.3.19.86:9502

2.4 其它一些问题

能够经过nc来检测swoole的udp服务是否通

yum install -y nc

# root @ WENGINE in ~ [10:17:07] C:130
$ nc -u 127.0.0.1 9502
ceshi

能够写supervisor的脚原本使swoole服务器一直启动

[program:swooleserver]
directory = /var/www/html
command=php artisan swoole-server
user=apache
autostart=true
startsecs=2
autorestart=true
redirect_stderr=true
stopsignal=INT
stderr_logfile_maxbytes=1MB
stderr_logfile_backups=10
stderr_capture_maxbytes=1MB
stderr_events_enabled=false