ElasticSearch ik,elasticsearch-jdbc 使用和 yii2 实例

时间 2019-11-07

标签 elasticsearch jdbc 使用 yii2 yii 实例栏目日志分析繁體版

原文原文链接

1.ES安装+ik插件

mac安装和使用:php

a.因为ES为java开发因此首先安装下载jdk,因为最新版本对jdk要求>1.8.0,因此能够经过brew 升级jdkhtml

brew cask install caskroom/versions/java8

b.brew 安装ESjava

$ brew update
$ brew search elasticsearch
elasticsearch            elasticsearch@2.4        elasticsearch@5.6        
$ brew install elasticsearch@2.4  #版本不须要安装过高,否则有的插件用不了
....
$ brew info elasticsearch@2.4 #安装成功后信息
elasticsearch@2.4: stable 2.4.6 [keg-only]
Distributed search & analytics engine
https://www.elastic.co/products/elasticsearch
/usr/local/Cellar/elasticsearch@2.4/2.4.6 (78 files, 38.5MB)
  Built from source on 2017-12-15 at 11:49:26
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/elasticsearch@2.4.rb
==> Requirements
Required: java >= 1.7 ✔
==> Caveats
Data:    /usr/local/var/elasticsearch/elasticsearch_lph/
Logs:    /usr/local/var/log/elasticsearch/elasticsearch_lph.log
Plugins: /usr/local/opt/elasticsearch@2.4/libexec/plugins/
Config:  /usr/local/etc/elasticsearch/
plugin script: /usr/local/opt/elasticsearch@2.4/libexec/bin/plugin

This formula is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.

If you need to have this software first in your PATH run:
  echo 'export PATH="/usr/local/opt/elasticsearch@2.4/bin:$PATH"' >> ~/.bash_profile


To have launchd start elasticsearch@2.4 now and restart at login:
  brew services start elasticsearch@2.4
Or, if you don't want/need a background service you can just run:
  elasticsearch

$ elasticsearch #启动elasticsearch

启动 elasticsearch 后,在浏览器地址:127.0.0.1:9200,浏览器会显示一段json数据node

这里出现一个错误: 再次用brew 启动 elasticsearch时候mysql

 $ brew services start elasticsearch@2.4 
 或者
 $ elasticsearch
 #报错
 ...
 failed to obtain node locks, tried [[/usr/local/var/lib/elasticsearch/elasticsearch_lph]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
 ...

127.0.0.1:9200不能进行访问,这是由于已经有个节点在线程中跑了,能够查看线程linux

$ ps aux | grep "elastics"
lph              86699   0.0  0.0  2442020    816 s000  S+    2:02下午   0:00.00 grep elastics
lph              86662   0.0 11.0  5079720 920492   ??  S     2:01下午   0:19.68 /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home....
$ kill -9 86662 #杀死jdk跑的elasticsearch 线程

以后 elasticsearch就能正常启动了git

 $ brew services start elasticsearch@2.4

修改elasticsearch配置github

在安装完成后 brew info elasticsearch,显示了对于配置文件目录位置 /usr/local/etc/elasticsearch 打开elasticsearch.ymlweb

$ vim /usr/local/etc/elasticsearch/elasticsearch.yml
...
...
cluster.name : test  #集群名称
...
node.name: master-1   #当前节点
...
network.host: 127.0.0.1

保存修改后,启动elasticsearchsql

$ brew services start elasticsearch@2.4

外网访问elasticsearch配置

1.elasticsearch.yml 中network.host: 127.0.0.1 改成 network.host: 0.0.0.0

2.开启unbuntu 防火墙对9200的容许

$ sudo ufw status #查看防火墙状态
22                         ALLOW       Anywhere
80                         ALLOW       Anywhere

$ sudo ufw allow from youallowip to any port 9200 #youallowip为你容许操做的ip,若是没有ip操做限制直接写成:sudo ufw allow 9200

3.用云平台的朋友须要考虑云端是否设置了9200端口是否被容许访问

安装中文分词插件 elasticsearch-analysis-ik

Versions

IK version	ES version
master	6.x -> master
6.0.0	6.0.0
5.6.4	5.6.4
5.5.3	5.5.3
5.4.3	5.4.3
5.3.3	5.3.3
5.2.2	5.2.2
5.1.2	5.1.2
1.10.6	2.4.6
1.9.5	2.3.5
1.8.1	2.2.1
1.7.0	2.1.1
1.5.0	2.0.0
1.2.6	1.0.0
1.2.5	0.90.x
1.1.3	0.20.x
1.0.0	0.16.2 -> 0.19.0

本例子安装ES 版本为2.4.6,因此对应下载1.10.6插件, 点击releases 查看v1.10.6版本复制下载地址

$ cd /usr/local/opt/elasticsearch@2.4/libexec/plugins/ #切换到elasticsearch插件位置
$ cd mkdir ik #建立ik目录存放插件
$ cd ik
$ wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v1.10.6/elasticsearch-analysis-ik-1.10.6.zip #下载安装包
$ unzip elasticsearch-analysis-ik-1.10.6.zip #解压安装包
$ rm elasticsearch-analysis-ik-1.10.6.zip #删除压缩包
$ brew services restart elasticsearch@2.4  #重启elasticsearch@2.4

重启以后查看log 表示插件加载成功 plugins [analysis-ik]

[2017-12-15 13:28:50,371][INFO ][plugins                  ] [master1] modules [reindex, lang-expression, lang-groovy], plugins [analysis-ik], sites []

对比 ik 和自带对比测试

$ curl -XPOST "http://127.0.0.1:9200/_analyze?analyzer=ik&pretty" -d "你好世界"
{
  "tokens" : [ {
    "token" : "你好",
    "start_offset" : 0,
    "end_offset" : 2,
    "type" : "CN_WORD",
    "position" : 0
  }, {
    "token" : "世界",
    "start_offset" : 2,
    "end_offset" : 4,
    "type" : "CN_WORD",
    "position" : 1
  } ]
}

$ curl -XPOST "http://127.0.0.1:9200/_analyze?analyzer=standard&pretty" -d "你好世界"
{
  "tokens" : [ {
    "token" : "你",
    "start_offset" : 0,
    "end_offset" : 1,
    "type" : "<IDEOGRAPHIC>",
    "position" : 0
  }, {
    "token" : "好",
    "start_offset" : 1,
    "end_offset" : 2,
    "type" : "<IDEOGRAPHIC>",
    "position" : 1
  }, {
    "token" : "世",
    "start_offset" : 2,
    "end_offset" : 3,
    "type" : "<IDEOGRAPHIC>",
    "position" : 2
  }, {
    "token" : "界",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "<IDEOGRAPHIC>",
    "position" : 3
  } ]
}

ubuntu安装:

首先安装jdk

sudo add-apt-repository ppa:webupd8team/java
##报错:
The program 'add-apt-repository' is currently not installed. You can install it by typing:
apt-get install software-properties-common
须要先执行:apt-get install software-properties-common 再执行sudo add-apt-repository ppa:webupd8team/java


sudo apt-get update
sudo apt-get install oracle-java8-installer #都选yes
sudo apt-get install oracle-java8-set-default   #都选yes
java -version
whereis java
which java （java執行路徑）
echo $JAVA_HOME

导入 Elasticsearch 公共 GPG 密钥到 apt：

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

建立 Elasticsearch 资源列表：

$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list

安装Elasticsearch

$ sudo apt-get update
$ sudo apt-get -y install elasticsearch
$ service elasticsearch start  #启动elasticsearch

运行如下命令将在系统启动时启动 Elasticsearch：

$ sudo update-rc.d elasticsearch defaults 95 10

查看版本

root@iZwz92p0vf7mrdbn3qwl40Z:~# curl -XGET localhost:9200
{
  "name" : "masterr-1",
  "cluster_name" : "deraceur",
  "cluster_uuid" : "VV0D289YTbamI939oKzkOA",
  "version" : {
    "number" : "2.4.6",#为es版本号 ik 对应版本1.10.6
    "build_hash" : "5376dca9f70f3abef96a77f4bb22720ace8240fd",
    "build_timestamp" : "2017-07-18T12:17:44Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.4"
  },
  "tagline" : "You Know, for Search"
}

安装maven (ik须要打包须要先按照maven)

1 下载安装

$ wget http://mirror.bit.edu.cn/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz 
$ tar -zxvf apache-maven-3.3.9-bin.tar.gz 
$ sudo mv apache-maven-3.3.9 /usr/local/

2 为maven配置环境变量

将以下内容添加到/etc/profile和/root/.bashrc中

export M2_HOME=/usr/local/apache-maven-3.3.9 
export PATH=$PATH:$M2_HOME/bin

执行以下指令使其上述配置生效或者重启系统：

source /etc/profile

检查安装是否生效

$ mvn -v 
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00) 
Maven home: /usr/local/apache-maven-3.3.9 
Java version: 1.8.0_65, vendor: Oracle Corporation 
Java home: /usr/local/jdk/jdk1.8.0_65/jre 
Default locale: en_US, platform encoding: UTF-8 
OS name: “linux”, version: “3.19.0-25-generic”, arch: “amd64”, family: “unix”

安装ik插件

git clone https://github.com/medcl/elasticsearch-analysis-ik.git
cd elasticsearch-analysis-ik/
git checkout -b v1.10.6
/usr/local/apache-maven-3.3.9/bin/mvn package  #mvn执行
mkdir /usr/share/elasticsearch/plugins/ik #建立ik
cp  target/releases/elasticsearch-analysis-ik-1.10.6.zip  /usr/share/elasticsearch/plugins/ik
cd  /usr/share/elasticsearch/plugins/ik
unzip elasticsearch-analysis-ik-1.10.6.zip

$ service elasticsearch restart  #从新启动elasticsearch

修改elasticsearch配置

vim /etc/elasticsearch/elasticsearch.yml

# 添加如下内容
index.analysis.analyzer.default.type: ik

测试

root@dsfsd:~# curl -XGET 'http://127.0.0.1:9200/_analyze?pretty&analyzer=ik_max_word' -d '电动车不贴反光膜是违法行为'
{
  "tokens" : [ {
    "token" : "电动车",
    "start_offset" : 0,
    "end_offset" : 3,
    "type" : "CN_WORD",
    "position" : 0
  }, {
    "token" : "电动",
    "start_offset" : 0,
    "end_offset" : 2,
    "type" : "CN_WORD",
    "position" : 1
  }, {
    "token" : "车",
    "start_offset" : 2,
    "end_offset" : 3,
    "type" : "CN_CHAR",
    "position" : 2
  }, {
    "token" : "不",
    "start_offset" : 3,
    "end_offset" : 4,
    "type" : "CN_CHAR",
    "position" : 3
  }, {
    "token" : "贴反",
    "start_offset" : 4,
    "end_offset" : 6,
    "type" : "CN_WORD",
    "position" : 4
  }, {
    "token" : "贴",
    "start_offset" : 4,
    "end_offset" : 5,
    "type" : "CN_WORD",
    "position" : 5
  }, {
    "token" : "反光膜",
    "start_offset" : 5,
    "end_offset" : 8,
    "type" : "CN_WORD",
    "position" : 6
  }, {
    "token" : "反光",
    "start_offset" : 5,
    "end_offset" : 7,
    "type" : "CN_WORD",
    "position" : 7
  }, {
    "token" : "膜",
    "start_offset" : 7,
    "end_offset" : 8,
    "type" : "CN_WORD",
    "position" : 8
  }, {
    "token" : "违法行为",
    "start_offset" : 9,
    "end_offset" : 13,
    "type" : "CN_WORD",
    "position" : 9
  }, {
    "token" : "违法",
    "start_offset" : 9,
    "end_offset" : 11,
    "type" : "CN_WORD",
    "position" : 10
  }, {
    "token" : "违",
    "start_offset" : 9,
    "end_offset" : 10,
    "type" : "CN_WORD",
    "position" : 11
  }, {
    "token" : "法",
    "start_offset" : 10,
    "end_offset" : 11,
    "type" : "CN_CHAR",
    "position" : 12
  }, {
    "token" : "行为",
    "start_offset" : 11,
    "end_offset" : 13,
    "type" : "CN_WORD",
    "position" : 13
  } ]
}

4.经过elasticsearch-jdbc操做数据(不推荐使用,这边我只是记录下)

使用elasticsearch-jdbc将现有MySQL数据批量导入至ElasticSearch(ubuntu环境运行)

下载并生成 mysql-import-product.sh

# wget http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/2.3.4.0/elasticsearch-jdbc-2.3.4.0-dist.zip  #若是下载速度很慢 能够尝试迅雷下载
# unzip elasticsearch-jdbc-2.3.4.0-dist.zip #解压
# cd elasticsearch-jdbc-2.3.4.0-dist
# cd bin
# cp mysql-blog.sh mysql-import-product.sh
# vim mysql-import-product.sh

修改 mysql-import-product.sh

#!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
bin=${DIR}/../bin
lib=${DIR}/../lib

echo '
{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://localhost:3306/test", #目标数据库为test
        "schedule" : "0 0-59 0-23 ? * *", #定时任务
        "user" : "root", #数据库user
        "password" : "12345678",
        "sql" : "select productid,title from product" #导入数据语句
        "index" : "test", 
        "type" : "products",
       
        "elasticsearch" : {
             "cluster" : "test",#为 elasticsearch config 中的 cluster.name
             "host" : "localhost",
             "port" : 9300 
        }   
    }
}
' | java \
    -cp "${lib}/*" \
    -Dlog4j.configurationFile=${bin}/log4j2.xml \
    org.xbib.tools.Runner \
    org.xbib.tools.JDBCImporter

保存后执行该脚本

# cd ../..
# bin/mysql-import-product.sh
bin/mysql-import-product.sh: 3: bin/mysql-import-product.sh: Bad substitution
bin/mysql-import-product.sh: 4: bin/mysql-import-product.sh: num: not found

#报错 为路劲错误,google不少说是jdk path问题,这边没有时间研究,直接修改路劲
# pwd  #显示当前路劲
/root/elasticsearch-jdbc-2.3.4.0
# vim bin/mysql-import-product.sh  #修改以下
...
 bin=/root/elasticsearch-jdbc-2.3.4.0/bin
lib=/root/elasticsearch-jdbc-2.3.4.0/lib
....
# 退出保存
# bin/mysql-import-product.sh #继续报错 google后 为jdk版本>1.8要求,本机为1.7
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/xbib/tools/Runner : Unsupported major.minor version 52.0
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:803)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:442)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:64)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:354)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:312)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
# .... #papapa~ 升级jdk 升级方式请参考 http://blog.csdn.net/cuiaamay/article/details/51822308
# chmod 777 bin/mysql-import-product.sh #添加执行权限
# bin/mysql-import-product.sh 
# curl -XGET "http://127.0.0.1:9200/test/_search?pretty"
 #验证导入时候成功
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 378,
  .......
  #有数据表示成功

指令记录:

$  curl -XPOST "http://127.0.0.1:9200/test/_search?pretty"   #获取全部
$  curl -XPOST "http://127.0.0.1:9200/test/_search?pretty" -d”@search.json”   #经过search.json检索数据
$  curl -XDELETE "http://127.0.0.1:9200/test"    #删除数据

查看更多 es指令操做

elasticsearch-jdbc自动导入增量数据

# cp mysql-import-product.sh mysql-delta-import-product.sh
# vim mysql-delta-import-product.sh

#!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
bin=${DIR}/../bin
lib=${DIR}/../lib

echo '
{
    "type" : "jdbc",
    "jdbc" : {
        "url" : "jdbc:mysql://localhost:3306/test", #目标数据库为test
        "schedule" : "0 0-59 0-23 ? * *", #定时任务
        "user" : "root", #数据库user
        "password" : "12345678",
        "sql" :[{
                   "statement": "select productid,title from product.updated_at > unix_timestamp(?) group by bshop_product_lang.productid",#当updated_at时间大于当前时间 执行数据导入
                    "parameter" : ["$metrics.lastexecutionstart"] #当前时间戳
        }],
        "index" : "test", 
        "type" : "products",
       
        "elasticsearch" : {
             "cluster" : "test",#为 elasticsearch config 中的 cluster.name
             "host" : "localhost",
             "port" : 9300 
        }   
    }
}
' | java \
    -cp "${lib}/*" \
    -Dlog4j.configurationFile=${bin}/log4j2.xml \
    org.xbib.tools.Runner \
    org.xbib.tools.JDBCImporter

其中updated_at为product更新或者建立时候的记录时间戳.

添加mysql-delta-import-product.sh 脚本权限和写进开机执行脚本

# chmod -X  mysql-delta-import-product.sh
# vim /etc/rc.local
....
sh /youpath/bin/mysql-delta-import-product.sh #ES数据增量 youpath为文件绝对位置
exit 0
....

这样开机将自动执行该脚本

5.经过 yii2-elasticsearch操做数据(推荐方法)

打开 yii2 根目录下 composer.json

...
   "require": {
        ....
        "yiisoft/yii2-elasticsearch": "^2.0"
    },
...

保存后

$ composer update

完成后在yii2 配置文件中写入

'components' => [
        'elasticsearch' => [
            'class' => 'yii\elasticsearch\Connection',
            'nodes' => [
                ['http_address' => '127.0.0.1:9200'],
                // configure more hosts if you have a cluster
            ],
        ],

建立common\models\ESProduct类继承 yii\elasticsearch\ActiveRecord

<?php

namespace common\models;

use yii\elasticsearch\ActiveRecord;
/**
 * ProductSearch represents the model behind the search form about `common\models\Product`.
 */
class ESProductSearch extends ActiveRecord
{

 /**
     * @inheritdoc
     */

    //public  $index = "bshop";
    // public $type = "products";

    # 属性
    public function attributes()
    {
        $mapConfig = self::mapConfig();
        return array_keys($mapConfig['properties']);
    }

    public function rules()
    {
        return [
            [["productid","title"],'required']
        ];
    }

    # mapping配置
  
    public static function mapConfig(){
        return [
            'properties' => [
                'productid'  => ['type' => 'double'],
                
                'title' => ['type' => 'string','analyzer'=>'ik', 'searchAnalyzer' => 'ik']
            ]
        ];
    }



    public static function mapping()
    {
        return [
            static::type() => self::mapConfig(),
        ];
    }



    //索引名称
    public static function index()
    {
        return "test";
    }
    //索引类型
    public static function type()
    {
        return "products";
    }

    /**
     * Set (update) mappings for this model
     */
    public static function updateMapping()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->setMapping(static::index(), static::type(), static::mapping());
    }


    /**
     * Create this model's index
     */
    public static function createIndex()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->createIndex(static::index(), [
            'settings' => [ 'index' => ['refresh_interval' => '1s'] ],
            'mappings' => static::mapping(),
            //'warmers' => [ /* ... */ ],
            //'aliases' => [ /* ... */ ],
            //'creation_date' => '...'
        ]);
    }


    /**
     * Delete this model's index
     */
    public static function deleteIndex()
    {
        $db = static::getDb();
        $command = $db->createCommand();
        $command->deleteIndex(static::index(), static::type());
    }

    public static function updateRecord($productid, $columns){
        try {
            $record = self::get($productid);
            foreach($columns as $key => $value){
                $record->$key = $value;
            }
            return $record->update();
        }catch (\Exception $e){
            return false;
        }
    }

    //删除记录
    public static function deleteRecord($productid){
        try {
            $record = self::get($productid);
            $record->delete();
            return true;
        }catch (\Exception $e){
            return false;
        }
    }

    //保存数据(添加 || 更新)
    public function saveRecord($data){

        //加载数据
        if ($this->load($data) && $this->validate()){
            $isExist = false;
            try {
                $record = self::get($this->productid);
                //不存在 为更新
                if (!$record){
                    $record = new self();
                    $record->setPrimaryKey($this->productid);
                }else{
                    $isExist = true;
                }
            }catch (\Exception $e){
                $record = new self();
                $record->setPrimaryKey($this->productid);
            }
            //遍历数据到对象
            foreach ($data['ESProductSearch'] as $key => $value) {
                $record->$key = $value;
            }

            try {

                if (!$isExist){
                    //插入
                    $result = $record->insert();
                }else{
                    //更新
                    $result = $record->update();
                }

            }catch (\Exception $e){
                return false;

            }
            return true;
        }
    }



   


}

建立测试TestControlle.php

<?php
/**
 * Created by PhpStorm.
 * User: yidashi
 * Date: 2017/4/14
 * Time: 下午11:33
 */

namespace api\modules\v1\models;
use common\models\ESProductSearch;
class TestController extends Controller{

    //查询
    public function actionTestES(){
        //es 测试
        //搜索词
        $keyword = "测试";
        $res = ESProductSearch::find()->query([
            "multi_match" => [
                "query" => $keyword,
                "fields" => ["title"] //检索属性
            ],
        ])->all();
        foreach ($res as $k){
            var_dump($k->productid);//输出id
        }
        exit;
    }

    //建立 或更新
    public function actionCreateES(){
        $ESArray = [
          'ESProductSearch' => [
                'productid' => '1',
                'title'     => 'hello world'
           ]
        ];

        $eSProductSearch = new ESProductSearch();
         if(!$eSProductSearch -> saveRecord($ESArray)){
                   throw new \Exception(Common::getLayoutMessageHtml("操做失败"));
        }        
       
    }
    
    //删除
    public function actionDeteleES($id){
        
        $eSProductSearch = new ESProductSearch();
         if(!$eSProductSearch->deleteByProduct($id)){
                   throw new \Exception(Common::getLayoutMessageHtml("删除失败"));
        }        
       
    }

}

ElasticSearch ik,elasticsearch-jdbc 使用 和 yii2 实例

1.ES安装+ik插件

Versions

检查安装是否生效

ElasticSearch ik,elasticsearch-jdbc 使用和 yii2 实例