elasticsearch搭建并经过go-mysql-elasticsearch同步db数据达到搜索引擎的目的

logstash-input-jdbc/elasticsearch-jdbc缺点：删除记录没有办法同步，只能两边执行本身的删除命令，版本16年后未更新。
go-mysql-elasticsearch缺点：国内的不是很稳定
elasticsearch-river-MYSQL 版本太旧，12年以后未更新。

es安装：

下载

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.6.zip

解压

unzip elasticsearch-5.6.6.zip

移动到/usr/local目录

mv elasticsearch-5.6.6 /usr/local/elsearch

按需求修改config/elasticsearch.yml文件

vim /usr/local/elsearch/config/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /usr/local/elsearch/data
#
# Path to log files:
#
path.logs: /usr/local/elsearch/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 172.21.0.15
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
http.cors.enabled: true
http.cors.allow-origin: "*"

安装中文分词设置

cd /usr/local/elsearch/
./bin/elasticsearch-plugin install https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v5.6.6/elasticsearch-analysis-ik-5.6.6.zip

测试

curl -X PUT 'localhost:9200/accounts' -d '
{
  "mappings": {
    "person": {
      "properties": {
        "user": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_max_word"
        },
        "title": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_max_word"
        },
        "desc": {
          "type": "text",
          "analyzer": "ik_max_word",
          "search_analyzer": "ik_max_word"
        }
      }
    }
  }
}'

参考连接 http://www.ruanyifeng.com/blog/2017/08/elasticsearch.html

es-head安装

下载

wget https://github.com/mobz/elasticsearch-head/archive/master.zip

解压

unzip master.zip

放到/usr/local/下

mv elasticsearch-head-master /usr/local/elsearch-head

安装node
于head插件本质上仍是一个nodejs的工程，所以须要安装node，使用npm来安装依赖的包

yum -y install gcc make gcc-c++ openssl-devel wget
wget http://nodejs.org/dist/v0.10.26/node-v0.10.26.tar.gz
tar -zvxf node-v0.10.26.tar.gz
cd node-v0.10.26
make && make install
验证是否安装配置成功：
node -v

安装grunt
grunt是一个很方便的构建工具，能够进行打包压缩、测试、执行等等的工做，5.6里的head插件就是经过grunt启动的。所以须要安装一下grunt：

cd /usr/local/elsearch-head
npm install -g grunt-cli  //执行后会生成node_modules文件夹
npm install

修改vim Gruntfile.js文件：增长hostname属性，设置为*

connect: {
    server: {
        options: {
            port: 9100,
            hostname: '*',
            base: '.',
            keepalive: true
        }
    }
}

修改 vim _site/app.js 文件：修改head的链接地址:

this.base_uri = this.config.base_uri || this.prefs.get(“app-base_uri”) || “http://192.168.0.153:9200“;

启动

cd elsearch-head
grunt server &  #后台启动

安装go-mysql-elasticsearch进行同步

yum install go
go get github.com/tools/godep
go get github.com/siddontang/go-mysql-elasticsearch
cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch 
make

修改配置文件

vim /root/go/src/github.com/siddontang/go-mysql-elasticsearch/etc/river.toml

# MySQL address, user and password
# user must have replication privilege in MySQL.
my_addr = "172.21.0.15:3308"#mysql地址
my_user = "root"#用户名
my_pass = "123456"#密码
# Elasticsearch address
es_addr = "172.21.0.15:9200"#es安装地址
# Path to store data, like master.info, and dump MySQL data
data_dir = "./var"
# Inner Http status address
stat_addr = "172.21.0.15:12800"
# pseudo server id like a slave
server_id = 1
# mysql or mariadb
flavor = "mysql"#mysql数据库
# mysqldump execution path
# if not set or empty, ignore mysqldump.
mysqldump = "mysqldump"
# MySQL data source
[[source]]
schema = "test"#数据库名称
# Only below tables will be synced into Elasticsearch.
# "test_river_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023
# I don't think it is necessary to sync all tables in a database.
tables = ["travelrecord10","travelrecord4","travelrecord5","travelrecord1"]#数据表名称，能够多个
# Below is for special rule mapping
#[[rule]]
#schema = "test"
#table = "test_river"
#index = "river"
#type = "river"
# only MySQL field in filter will be synced
#filter=["title","tags"]
# title is MySQL test_river field name, es_title is the customized name in Elasticsearch
#[rule.field]
# This will map column title to elastic search my_title
#title="es_title"
# This will map column tags to elastic search my_tags and use array type
#tags="my_tags,list"
# This will map column keywords to elastic search keywords and use array type
#keywords=",list"
# wildcard table rule, the wildcard table must be in source tables
[[rule]]
schema = "test"#数据库名称
table = "travelrecord10"#数据表名称
index = ""#生成es数据索引名称，对应schema
type = ""#生成es数据类型，对应table

#[[rule]]
#schema = "test"#数据库名称
#table = "travelrecord8"#数据表名称
#index = ""#生成es数据索引名称，对应schema
#type = ""#生成es数据类型，对应table
# title is MySQL test_river field name, es_title is the customized name in Elasticsearch
[[rule.fields]]
mysql = "title"
elastic = "es_title"

运行

cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch 
./bin/go-mysql-elasticsearch -config=./etc/river.toml  >  /home/heron/wwwlogs/go-mysql.log 2>&1 &

测试：

db增长数据，查看页面是否显示。




curl -XPOST -u elastic '10.168.2.10:9200/_xpack/security/user/demo' -d '{   "password" : "123456",   "full_name" : " elastic",   "email" : “elastic@163.com",  "roles" : [ "clicks_admin" ] }'

不稳定现况，会本身挂掉，暂时没找到解决办法，只能挂掉当即重启