Spring boot整合ElasticSearch案例分享

时间 2019-11-08

标签 spring boot 整合 elasticsearch 案例分享栏目 Spring 繁體版

原文原文链接

The best elasticsearch highlevel java rest api-----bboss java

Spring boot整合ElasticSearch HighLevel Rest Client案例分享，本文涉及内容mysql

集成bboss es starter
单es集群整合
多es集群整合

本文内容适合于:git

spring boot 1.x,2.x
elasticsearch 1.x,2.x,5.x,6.x,+

1.集成bboss es starter

在spring boot项目中导入bboss es starter github

maven工程spring

<dependency>
            <groupId>com.bbossgroups.plugins</groupId>
            <artifactId>bboss-elasticsearch-spring-boot-starter</artifactId>
            <version>5.6.2</version>
        </dependency>

gradle工程sql

compile "com.bbossgroups.plugins:bboss-elasticsearch-spring-boot-starter:5.6.2"

2.建立spring boot启动类

新建Application类：shell

package org.bboss.elasticsearchtest.springboot;


import org.frameworkset.elasticsearch.ElasticSearchHelper;
import org.frameworkset.elasticsearch.client.ClientInterface;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.context.annotation.Bean;

/**
 *  @author yinbp [122054810@qq.com]
 *  
 */

@SpringBootApplication

public class Application {

    private Logger logger = LoggerFactory.getLogger(Application.class);

   
    public static void main(String[] args) {

        SpringApplication.run(Application.class, args);
    }
}

Application类中定义了一个main方法用来启动spring boot测试应用。数据库

Application类将被用于启动下面两个测试用例：express

单es集群测试用例
多es集群测试用例

3.单es集群配置和使用

3.1 配置单es集群

bboss es starter配置单集群能够采用properties文件也能够采用yml进行配置，两者任选其一。apache

3.1.1 application.properties

修改spring boot配置文件application.properties内容

##ES集群配置，支持x-pack和searchguard
spring.elasticsearch.bboss.elasticUser=elastic
spring.elasticsearch.bboss.elasticPassword=changeme


spring.elasticsearch.bboss.elasticsearch.rest.hostNames=10.21.20.168:9200
#spring.elasticsearch.bboss.elasticsearch.rest.hostNames=10.180.211.27:9280,10.180.211.27:9281,10.180.211.27:9282
##https配置，添加https://协议头
#spring.elasticsearch.bboss.default.elasticsearch.rest.hostNames=https://10.180.211.27:9280,https://10.180.211.27:9281,https://10.180.211.27:9282
spring.elasticsearch.bboss.elasticsearch.dateFormat=yyyy.MM.dd
spring.elasticsearch.bboss.elasticsearch.timeZone=Asia/Shanghai
spring.elasticsearch.bboss.elasticsearch.ttl=2d
#在控制台输出脚本调试开关showTemplate,false关闭，true打开，同时log4j至少是info级别
spring.elasticsearch.bboss.elasticsearch.showTemplate=true
spring.elasticsearch.bboss.elasticsearch.discoverHost=false
# dsl配置文件热加载扫描时间间隔，毫秒为单位，默认5秒扫描一次，<= 0时关闭扫描机制
spring.elasticsearch.bboss.dslfile.refreshInterval = -1

##es client http链接池配置
spring.elasticsearch.bboss.http.timeoutConnection = 5000
spring.elasticsearch.bboss.http.timeoutSocket = 5000
spring.elasticsearch.bboss.http.connectionRequestTimeout=5000
spring.elasticsearch.bboss.http.retryTime = 1
spring.elasticsearch.bboss.http.maxLineLength = -1
spring.elasticsearch.bboss.http.maxHeaderCount = 200
spring.elasticsearch.bboss.http.maxTotal = 400
spring.elasticsearch.bboss.http.defaultMaxPerRoute = 200
spring.elasticsearch.bboss.http.soReuseAddress = false
spring.elasticsearch.bboss.http.soKeepAlive = false
spring.elasticsearch.bboss.http.timeToLive = 3600000
spring.elasticsearch.bboss.http.keepAlive = 3600000
spring.elasticsearch.bboss.http.keystore =
spring.elasticsearch.bboss.http.keyPassword =
# ssl 主机名称校验，是否采用default配置，
# 若是指定为default，就采用DefaultHostnameVerifier,不然采用 SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER
spring.elasticsearch.bboss.http.hostnameVerifier =

## 数据库数据源配置，使用db-es数据导入功能时须要配置
#spring.elasticsearch.bboss.db.name = test
#spring.elasticsearch.bboss.db.user = root
#spring.elasticsearch.bboss.db.password = 123456
#spring.elasticsearch.bboss.db.driver = com.mysql.jdbc.Driver
#spring.elasticsearch.bboss.db.url = jdbc:mysql://localhost:3306/bboss
#spring.elasticsearch.bboss.db.usePool = false
#spring.elasticsearch.bboss.db.validateSQL = select 1

单ES集群配置项都是以spring.elasticsearch.bboss开头。

3.1.2 application.yml

若是采用application.yml配置，内容以下

spring:
  elasticsearch:
    bboss:
      elasticUser: elastic
      elasticPassword: changeme
      elasticsearch:
        rest:
          hostNames: 192.168.8.25:9200
          ##hostNames: 192.168.8.25:9200,192.168.8.26:9200,192.168.8.27:9200  ##集群地址配置
        dateFormat: yyyy.MM.dd
        timeZone: Asia/Shanghai
        ttl: 2d
        showTemplate: true
        discoverHost: false
      dslfile:
        refreshInterval: -1
      http:
         timeoutConnection: 5000
         timeoutSocket: 5000
         connectionRequestTimeout: 5000
         retryTime: 1
         maxLineLength: -1
         maxHeaderCount: 200
         maxTotal: 400
         defaultMaxPerRoute: 200
         soReuseAddress: false
         soKeepAlive: false
         timeToLive: 3600000
         keepAlive: 3600000
         keystore:
         keyPassword:
         hostnameVerifier:

3.2 单集群测试用例

编写es单集群测试用例BBossESStarterTestCase

/*
 * Copyright 1999-2018 Alibaba Group Holding Ltd.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.bboss.elasticsearchtest.springboot;


import org.frameworkset.elasticsearch.client.ClientInterface;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.junit4.SpringRunner;

/**
 * 单集群演示功能测试用例，spring boot配置项以spring.elasticsearch.bboss开头
 * 对应的配置文件为application.properties文件
 * @author  yinbp [122054810@qq.com]
 */
@RunWith(SpringRunner.class)
@SpringBootTest(classes = Application.class)
public class BBossESStarterTestCase {
	@Autowired
	private BBossESStarter bbossESStarter;
	@Autowired
	DocumentCRUD documentCRUD;

    @Test
    public void testBbossESStarter() throws Exception {
//        System.out.println(bbossESStarter);

		//验证环境,获取es状态
//		String response = serviceApiUtil.restClient().executeHttp("_cluster/state?pretty",ClientInterface.HTTP_GET);

//		System.out.println(response);
		//判断索引类型是否存在，false表示不存在，正常返回true表示存在
		boolean exist = bbossESStarter.getRestClient().existIndiceType("twitter","tweet");

		//判读索引是否存在，false表示不存在，正常返回true表示存在
		exist =  bbossESStarter.getRestClient().existIndice("twitter");

		exist =  bbossESStarter.getRestClient().existIndice("agentinfo");

    }

    @Test
	public void testCRUD() throws Exception {

		//删除/建立文档索引表
		documentCRUD.testCreateIndice();
		//添加/修改单个文档

		documentCRUD.testAddAndUpdateDocument();
		//批量添加文档
		documentCRUD.testBulkAddDocument();
		//检索文档
		documentCRUD.testSearch();
		//批量修改文档
		documentCRUD.testBulkUpdateDocument();

		//检索批量修改后的文档
		documentCRUD.testSearch();
		//带list复杂参数的文档检索操做
		documentCRUD.testSearchArray();
		//带from/size分页操做的文档检索操做
		documentCRUD.testPagineSearch();
		//带sourcefilter的文档检索操做
		documentCRUD.testSearchSourceFilter();

		documentCRUD.updateDemoIndice();
		documentCRUD.testBulkAddDocuments();
	}

	@Test
	public void testPerformaceCRUD() throws Exception {

		//删除/建立文档索引表
		documentCRUD.testCreateIndice();

		documentCRUD.testBulkAddDocuments();
	}

}

直接经过junit运行上述测试用例便可。

其中

BBossESStarter：由bboss提供，直接在代码中声明引用，并使用便可

DocumentCRUD:各类增删改查操做实例，在demo工程中提供

4.多ES集群测试用例

4.1 配置多es集群

bboss es starter配置多集群能够采用properties文件也能够采用yml进行配置，两者任选其一。

4.1.1properties配置

修改spring boot配置文件application-multi-datasource.properties，内容以下：

##多集群配置样例，若是须要作多集群配置，请将参照本文内容修改application.properties文件内容
spring.elasticsearch.bboss.default.name = default
##default集群配配置
spring.elasticsearch.bboss.default.elasticUser=elastic
spring.elasticsearch.bboss.default.elasticPassword=changeme


spring.elasticsearch.bboss.default.elasticsearch.rest.hostNames=10.21.20.168:9200
#spring.elasticsearch.bboss.default.elasticsearch.rest.hostNames=10.180.211.27:9280,10.180.211.27:9281,10.180.211.27:9282
spring.elasticsearch.bboss.default.elasticsearch.dateFormat=yyyy.MM.dd
spring.elasticsearch.bboss.default.elasticsearch.timeZone=Asia/Shanghai
spring.elasticsearch.bboss.default.elasticsearch.ttl=2d
#在控制台输出脚本调试开关showTemplate,false关闭，true打开，同时log4j至少是info级别
spring.elasticsearch.bboss.default.elasticsearch.showTemplate=true
spring.elasticsearch.bboss.default.elasticsearch.discoverHost=false

##default链接池配置
spring.elasticsearch.bboss.default.http.timeoutConnection = 5000
spring.elasticsearch.bboss.default.http.timeoutSocket = 5000
spring.elasticsearch.bboss.default.http.connectionRequestTimeout=5000
spring.elasticsearch.bboss.default.http.retryTime = 1
spring.elasticsearch.bboss.default.http.maxLineLength = -1
spring.elasticsearch.bboss.default.http.maxHeaderCount = 200
spring.elasticsearch.bboss.default.http.maxTotal = 400
spring.elasticsearch.bboss.default.http.defaultMaxPerRoute = 200
spring.elasticsearch.bboss.default.http.keystore =
spring.elasticsearch.bboss.default.http.keyPassword =
# ssl 主机名称校验，是否采用default配置，
# 若是指定为default，就采用DefaultHostnameVerifier,不然采用 SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER
spring.elasticsearch.bboss.default.http.hostnameVerifier =

##logs集群配置
spring.elasticsearch.bboss.logs.name = logs
spring.elasticsearch.bboss.logs.elasticUser=elastic
spring.elasticsearch.bboss.logs.elasticPassword=changeme


spring.elasticsearch.bboss.logs.elasticsearch.rest.hostNames=127.0.0.1:9200

#spring.elasticsearch.bboss.default.elasticsearch.rest.hostNames=10.180.211.27:9280,10.180.211.27:9281,10.180.211.27:9282
spring.elasticsearch.bboss.logs.elasticsearch.dateFormat=yyyy.MM.dd
spring.elasticsearch.bboss.logs.elasticsearch.timeZone=Asia/Shanghai
spring.elasticsearch.bboss.logs.elasticsearch.ttl=2d
#在控制台输出脚本调试开关showTemplate,false关闭，true打开，同时log4j至少是info级别
spring.elasticsearch.bboss.logs.elasticsearch.showTemplate=true
spring.elasticsearch.bboss.logs.elasticsearch.discoverHost=false

##logs集群对应的链接池配置
spring.elasticsearch.bboss.logs.http.timeoutConnection = 5000
spring.elasticsearch.bboss.logs.http.timeoutSocket = 5000
spring.elasticsearch.bboss.logs.http.connectionRequestTimeout=5000
spring.elasticsearch.bboss.logs.http.retryTime = 1
spring.elasticsearch.bboss.logs.http.maxLineLength = -1
spring.elasticsearch.bboss.logs.http.maxHeaderCount = 200
spring.elasticsearch.bboss.logs.http.maxTotal = 400
spring.elasticsearch.bboss.logs.http.defaultMaxPerRoute = 200
# https证书配置
spring.elasticsearch.bboss.logs.http.keystore =
spring.elasticsearch.bboss.logs.http.keyPassword =
# ssl 主机名称校验，是否采用default配置，
# 若是指定为default，就采用DefaultHostnameVerifier,不然采用 SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER
spring.elasticsearch.bboss.logs.http.hostnameVerifier =
# dsl配置文件热加载扫描时间间隔，毫秒为单位，默认5秒扫描一次，<= 0时关闭扫描机制
spring.elasticsearch.bboss.dslfile.refreshInterval = -1

配置说明：

上面配置了两个集群：default和logs

每一个集群配置项的前缀为：spring.elasticsearch.bboss.集群名字，其中的集群名字是一个自定义的逻辑名称，用来在client api中引用集群。

注意：这里的集群名字是根据集群的做用命名的名字，和Elasticsearch自己的cluster.name没有关系，可自行根据须要命名

default集群的配置项前缀为：

spring.elasticsearch.bboss.default

logs集群的配置项前缀为：

spring.elasticsearch.bboss.logs

同时每一个集群的配置项目里面必须包含name项目的配置

default集群name配置：

spring.elasticsearch.bboss.default.name = default

logs集群name配置：

##logs集群配置
spring.elasticsearch.bboss.logs.name = logs

4.1.2 yml配置（默认）

spring:
  elasticsearch:
    bboss:
      default:
        name: default
        elasticUser: elastic
        elasticPassword: changeme
        elasticsearch:
          rest:
            hostNames: 127.0.0.1:9200
            ##hostNames: 127.0.0.1:9200,127.0.0.1:9201,127.0.0.1:9202 
          dateFormat: yyyy.MM.dd
          timeZone: Asia/Shanghai
          ttl: 2d
          showTemplate: true
          discoverHost: false
        dslfile:
          refreshInterval: -1
        http:
           timeoutConnection: 5000
           timeoutSocket: 5000
           connectionRequestTimeout: 5000
           retryTime: 1
           maxLineLength: -1
           maxHeaderCount: 200
           maxTotal: 400
           defaultMaxPerRoute: 200
           soReuseAddress: false
           soKeepAlive: false
           timeToLive: 3600000
           keepAlive: 3600000
           keystore:
           keyPassword:
           hostnameVerifier:
      logs:
          name: logs
          elasticUser: elastic
          elasticPassword: changeme
          elasticsearch:
            rest:
              hostNames: 127.0.0.1:9200
            dateFormat: yyyy.MM.dd
            timeZone: Asia/Shanghai
            ttl: 2d
            showTemplate: true
            discoverHost: false
          dslfile:
            refreshInterval: -1
          http:
             timeoutConnection: 5000
             timeoutSocket: 5000
             connectionRequestTimeout: 5000
             retryTime: 1
             maxLineLength: -1
             maxHeaderCount: 200
             maxTotal: 400
             defaultMaxPerRoute: 200
             soReuseAddress: false
             soKeepAlive: false
             timeToLive: 3600000
             keepAlive: 3600000
             keystore:
             keyPassword:
             hostnameVerifier:

4.2 定义加载多es集群配置的spring boot Configuration类

新建类MultiESSTartConfigurer，声明分别对应两个不一样集群的bboss工厂组件

package org.bboss.elasticsearchtest.springboot;
/*
 *  Copyright 2008 biaoping.yin
 *
 *  Licensed under the Apache License, Version 2.0 (the "License");
 *  you may not use this file except in compliance with the License.
 *  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 *  Unless required by applicable law or agreed to in writing, software
 *  distributed under the License is distributed on an "AS IS" BASIS,
 *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 *  See the License for the specific language governing permissions and
 *  limitations under the License.
 */

import org.frameworkset.elasticsearch.ElasticSearchHelper;
import org.frameworkset.elasticsearch.boot.BBossESStarter;
import org.frameworkset.elasticsearch.client.ClientInterface;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.context.annotation.Profile;

/**
 * 配置多个es集群
 * 指定多es数据源profile：multi-datasource
 */
@Configuration
public class MultiESSTartConfigurer {
	@Primary
	@Bean(initMethod = "start")
	@ConfigurationProperties("spring.elasticsearch.bboss.default")
	public BBossESStarter bbossESStarterDefault(){
		return new BBossESStarter();

	}

	@Bean(initMethod = "start")
	@ConfigurationProperties("spring.elasticsearch.bboss.logs")
	public BBossESStarter bbossESStarterLogs(){
		return new BBossESStarter();

	}

}

说明：

MultiESSTartConfigurer经过如下两个方法分别加载default和logs两个es集群的配置

default集群配置加载

@Primary
@Bean(initMethod = "start")
@ConfigurationProperties("spring.elasticsearch.bboss.default")
public BBossESStarter bbossESStarterDefault()

logs集群配置加载

@Bean(initMethod = "start")
@ConfigurationProperties("spring.elasticsearch.bboss.logs")
public BBossESStarter bbossESStarterLogs()

BBossESStarter bbossESStarterDefault 对应spring.elasticsearch.bboss.default配置的elasticsearch集群

BBossESStarter bbossESStarterLogs 对应spring.elasticsearch.bboss.logs配置的elasticsearch集群

两个组件的声明都是必须的，在程序中只要用其中任意一个均可以获取到两个对应集群的ClientInterface组件，具体看后面的示例。

4.3 定义多es集群测试用例

多es集群测试用例MultiBBossESStartersTestCase

/*
 * Copyright 1999-2018 Alibaba Group Holding Ltd.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.bboss.elasticsearchtest.springboot;


import org.frameworkset.elasticsearch.client.ClientInterface;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.junit4.SpringRunner;

/**
 * 多集群演示功能测试用例，spring boot配置项以spring.elasticsearch.bboss.集群名称开头，例如：
 * spring.elasticsearch.bboss.default 默认es集群
 * spring.elasticsearch.bboss.logs  logs es集群
 * 两个集群经过 org.bboss.elasticsearchtest.springboot.MultiESSTartConfigurer加载
 * 对应的配置文件为application-multi-datasource.properties文件
 * 经过ActiveProfiles指定并激活多es集群配置：multi-datasource 
 * @author yinbp [122054810@qq.com]
 */
@RunWith(SpringRunner.class)
@SpringBootTest(classes = Application.class)
@ActiveProfiles("multi-datasource")
public class MultiBBossESStartersTestCase {
	@Autowired
	private BBossESStarter bbossESStarterDefault;
	@Autowired
	MultiESDocumentCRUD multiESDocumentCRUD;
    @Test
    public void testMultiBBossESStarters() throws Exception {

		//验证环境,获取es状态
//		String response = bbossESStarterDefault.getRestClient().executeHttp("_cluster/state?pretty",ClientInterface.HTTP_GET);
//		System.out.println(response);


		//判断索引类型是否存在，false表示不存在，正常返回true表示存在
		boolean exist = bbossESStarterDefault.getRestClient().existIndiceType("twitter","tweet");
		System.out.println("default twitter/tweet:"+exist);
		//获取logs对应的Elasticsearch集群客户端，并进行existIndiceType操做
		exist = bbossESStarterDefault.getRestClient("logs").existIndiceType("twitter","tweet");
		System.out.println("logs twitter/tweet:"+exist);
		//获取logs对应的Elasticsearch集群客户端，判读索引是否存在，false表示不存在，正常返回true表示存在
		exist = bbossESStarterDefault.getRestClient("logs").existIndice("twitter");
		System.out.println("logs  twitter:"+exist);
		//获取logs对应的Elasticsearch集群客户端，判断索引是否认义
		exist = bbossESStarterDefault.getRestClient("logs").existIndice("agentinfo");
		System.out.println("logs agentinfo:"+exist);
    }

	@Test
	public void testCRUD() throws Exception {

		//删除/建立文档索引表
		multiESDocumentCRUD.testCreateIndice();
		//添加/修改单个文档

		multiESDocumentCRUD.testAddAndUpdateDocument();
		//批量添加文档
		multiESDocumentCRUD.testBulkAddDocument();
		//检索文档
		multiESDocumentCRUD.testSearch();
		//批量修改文档
		multiESDocumentCRUD.testBulkUpdateDocument();

		//检索批量修改后的文档
		multiESDocumentCRUD.testSearch();
		//带list复杂参数的文档检索操做
		multiESDocumentCRUD.testSearchArray();
		//带from/size分页操做的文档检索操做
		multiESDocumentCRUD.testPagineSearch();
		//带sourcefilter的文档检索操做
		multiESDocumentCRUD.testSearchSourceFilter();

		multiESDocumentCRUD.updateDemoIndice();
		multiESDocumentCRUD.testBulkAddDocuments();
	}

	@Test
	public void testPerformaceCRUD() throws Exception {

		//删除/建立文档索引表
		multiESDocumentCRUD.testCreateIndice();

		multiESDocumentCRUD.testBulkAddDocuments();
	}
}

直接经过junit运行上述测试用例便可。

其中

BBossESStarter bbossESStarterDefault：实现由bboss提供，在直接在代码中MultiESSTartConfigurer 定义

MultiESDocumentCRUD:各类增删改查操做实例，在demo工程中提供

5.完整的demo工程

https://gitee.com/bboss/eshelloword-spring-boot-starter

https://github.com/bbossgroups/eshelloword-spring-boot-starter

6 开发交流

elasticsearch交流Q：166471282

elasticsearch：