Elasticsearch7.6开启xpack以及java代码配置客户端

最近这两天在研究es7.6的配置以及开启xpack后用java代码配置TransportClient、RestClient和JestClient，在这里记录一下。

1、安装Elasticsearch7.6

1. 首先下载es

   wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.1-linux-x86\_64.tar.gz

2. 下载后对文件进行解压，而后修改配置文件
   咱们暂时修改一下几个地方：

   cluster.name: es-7.6.1
   node.name: node-7.6.1
   network.host: 127.0.0.1
   http.port: 9200

   保存后，能够去启动es

   ./bin/elasticsearch

   若是遇到如下错误，须要配置一下他们三个里边任意一个属性便可：

   at least one of [discovery.seed_hosts,discovery.seed_providers,cluster.initial_master_nodes] must be configured

   这里咱们配置一下cluster.initial_master_nodes:

   cluster.initial_master_nodes: ["node-7.6.1"]

2、开启xpack，配置transport安全验证

1. 首先咱们先生成ca文件

   konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-certutil ca
    This tool assists you in the generation of X.509 certificates and certificate
    signing requests for use with SSL/TLS in the Elastic stack.
   
    The 'ca' mode generates a new 'certificate authority'
    This will create a new X.509 certificate and private key that can be used
    to sign certificate when running in 'cert' mode.
   
    Use the 'ca-dn' option if you wish to configure the 'distinguished name'
    of the certificate authority
   
    By default the 'ca' mode produces a single PKCS#12 output file which holds:
        *The CA certificate
        *The CA's private key
    If you elect to generate PEM format certificates (the -pem option), then the output will
    be a zip file containing individual files for the CA certificate and private key
    // 这里会提示咱们输入输出文件的名称，默认为elastic-stack-ca.p12,不更改直接回车便可
    Please enter the desired output file [elastic-stack-ca.p12]:
    // 输入之后使用elastic-stack-ca.p12文件的密码，若是不须要密码直接回车
    Enter password for elastic-stack-ca.p12 :

   通过此步骤后，会在当前目录下生成elastic-stack-ca.p12文件。

2. 而后咱们根据ca文件生成certificates文件

   konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
    This tool assists you in the generation of X.509 certificates and certificate
    signing requests for use with SSL/TLS in the Elastic stack.
   
    The 'cert' mode generates X.509 certificate and private keys.
        *By default, this generates a single certificate and key for use
           on a single instance.
        *The '-multiple' option will prompt you to enter details for multiple
           instances and will generate a certificate and key for each one
        *The '-in' option allows for the certificate generation to be automated by describing
           the details of each instance in a YAML file
   
        *An instance is any piece of the Elastic Stack that requires an SSL certificate.
          Depending on your configuration, Elasticsearch, Logstash, Kibana, and Beats
          may all require a certificate and private key.
        *The minimum required value for each instance is a name. This can simply be the
          hostname, which will be used as the Common Name of the certificate. A full
          distinguished name may also be used.
        *A filename value may be required for each instance. This is necessary when the
          name would result in an invalid file or directory name. The name provided here
          is used as the directory name (within the zip) and the prefix for the key and
          certificate files. The filename is required if you are prompted and the name
          is not displayed in the prompt.
        *IP addresses and DNS names are optional. Multiple values can be specified as a
          comma separated string. If no IP addresses or DNS names are provided, you may disable hostname verification in your SSL configuration.
        *All certificates generated by this tool will be signed by a certificate authority (CA).
        *The tool can automatically generate a new CA for you, or you can provide your own with the -ca or -ca-cert command line options.
    By default the 'cert' mode produces a single PKCS#12 output file which holds:
        *The instance certificate
        *The private key for the instance certificate
        *The CA certificate
    If you specify any of the following options:
        *-pem (PEM formatted output)
        *-keep-ca-key (retain generated CA key)
        *-multiple (generate multiple certificates)
        *-in (generate certificates from an input file)
    then the output will be be a zip file containing individual certificate/key files
    // 输入使用ca文件的密码，在上一步中设置，若是没有设置直接回车
    Enter password for CA (elastic-stack-ca.p12) :
    // 输入输出certificates文件的名称
    Please enter the desired output file [elastic-certificates.p12]:
    // 输入之后使用certificates文件的密码
    Enter password for elastic-certificates.p12 :
   
    Certificates written to /home/konghang/elasticsearch-7.6.1/elastic-certificates.p12
   
    This file should be properly secured as it contains the private key for
    your instance.
   
    This file is a self contained file and can be copied and used 'as is'
    For each Elastic product that you wish to configure, you should copy
    this '.p12' file to the relevant configuration directory
    and then follow the SSL configuration instructions in the product guide.
   
    For client applications, you may only need to copy the CA certificate and
    configure the client to trust this certificate.

   以后会在当前目录下生成certificates文件。

3. 启用xpack，配置transport的安全认证
   把第二节中生成的两个文件都移动到es目录的config配置目录中，而后修改配置文件，在配置文件末尾添加以下配置：

   xpack.security.enabled: true
   xpack.security.transport.ssl.enabled: true
   xpack.security.transport.ssl.verification_mode: certificate
   xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
   xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

   而后重启es便可。

4. 配置用户密码
   咱们开启xpack之后，访问es是须要用户名和密码的，因此咱们须要配置一下访问的密码，配置密码有两种方式，一种是自动生成的，自动生成各类用户的密码；另外一种是交互式的，咱们须要手动去输入每一个用户的密码。

   • 自动生成

     konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-setup-passwords auto

   • 手动生成

     konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-setup-passwords interactive
     Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_moni
     toring_user.
     You will be prompted to enter passwords as the process progresses.
     // 咱们输入y，而后输入每一个用户的密码
     Please confirm that you would like to continue [y/N]y
     Enter password for [elastic]:
     Reenter password for [elastic]:
     Enter password for [apm_system]:
     Reenter password for [apm_system]:
     Enter password for [kibana]:
     Reenter password for [kibana]:
     Enter password for [logstash_system]:
     Reenter password for [logstash_system]:
     Enter password for [beats_system]:
     Reenter password for [beats_system]:
     Enter password for [remote_monitoring_user]:
     Reenter password for [remote_monitoring_user]:
     Changed password for user [apm_system]
     Changed password for user [kibana]
     Changed password for user [logstash_system]
     Changed password for user [beats_system]
     Changed password for user [remote_monitoring_user]
     Changed password for user [elastic]

5. java配置TransportClient
   通过上边四步配置后，咱们就能够去用java实现TransportClient来实现查询了，详细的java代码以下：

   @Bean
    public TransportClient getTransportClient() throws UnknownHostException, URISyntaxException {
        logger.info("init TransportClient");
        String path = Paths.get(getClass().getClassLoader().getResource(elasticsearchProperties.getPkcsTransportFilePath()).toURI()).toString();
        TransportClient client = new PreBuiltXPackTransportClient(Settings.builder()
                .put("cluster.name", elasticsearchProperties.getClusterName())
                // missing authentication credentials for action
                .put("xpack.security.user", String.format("%s:%s", elasticsearchProperties.getUsername(), elasticsearchProperties.getPassword()))
                .put("xpack.security.transport.ssl.enabled", true)
                .put("xpack.security.transport.ssl.verification_mode", "certificate")
                .put("xpack.security.transport.ssl.keystore.path", path)
                .put("xpack.security.transport.ssl.truststore.path", path)
                .build()).addTransportAddress(new TransportAddress(InetAddress.getByName(elasticsearchProperties.getHost()), elasticsearchProperties.getTcpPort()));
        return client;
    }

   详细参考官方文档Java Client and security，不过因为TransportClient已通过时，不建议使用，咱们这里只是研究一下怎么可以成功配置它。

配置加密客户端HTTP通讯

该配置有两种模式，一种是pem模式，一种是pkcs模式。参考官方文档，经过命令bin/elasticsearch-certutil http

1. pem模式

   • 执行命令后会先询问是否生成csr文件，此时选y

     konghang@Surface-Pro7:~/elasticsearch-7.6.1$ bin/elasticsearch-certutil http
      ##Elasticsearch HTTP Certificate Utility
     
      The 'http' command guides you through the process of generating certificates
      for use on the HTTP (Rest) interface for Elasticsearch.
     
      This tool will ask you a number of questions in order to generate the right
      set of files for your needs.
     
      ##Do you wish to generate a Certificate Signing Request (CSR)?
     
      A CSR is used when you want your certificate to be created by an existing
      Certificate Authority (CA) that you do not control (that is, you don't have
      access to the keys for that CA).
     
      If you are in a corporate environment with a central security team, then you
      may have an existing Corporate CA that can generate your certificate for you.
      Infrastructure within your organisation may already be configured to trust this
      CA, so it may be easier for clients to connect to Elasticsearch if you use a
      CSR and send that request to the team that controls your CA.
     
      If you choose not to generate a CSR, this tool will generate a new certificate
      for you. That certificate will be signed by a CA under your control. This is a
      quick and easy way to secure your cluster with TLS, but you will need to
      configure all your clients to trust that custom CA.
     
      Generate a CSR? [y/N]y

   • 是否为每个节点生成certificates，选y

     ## Do you wish to generate one certificate per node?
     
      If you have multiple nodes in your cluster, then you may choose to generate a
      separate certificate for each of these nodes. Each certificate will have its
      own private key, and will be issued for a specific hostname or IP address.
     
      Alternatively, you may wish to generate a single certificate that is valid
      across all the hostnames or addresses in your cluster.
     
      If all of your nodes will be accessed through a single domain
      (e.g. node01.es.example.com, node02.es.example.com, etc) then you may find it
      simpler to generate one certificate with a wildcard hostname (*.es.example.com)
      and use that across all of your nodes.
     
      However, if you do not have a common domain name, and you expect to add
      additional nodes to your cluster in the future, then you should generate a
      certificate per node so that you can more easily generate new certificates when
      you provision new nodes.
     
      Generate a certificate per node? [y/N]y

   • 而后输入节点名称，下边的hostnames和ip addresses默认为空，enter进入下一步

     ## What is the name of node #1?
     You can use any descriptive name that you like, but we recommend using the name
     of the Elasticsearch node.
     node #1 name: node-7.6.1
     
     ## Which hostnames will be used to connect to node-7.6.1?
     Enter all the hostnames that you need, one per line.
     When you are done, press <ENTER> once more to move on to the next step.
     
     You did not enter any hostnames.
     Clients are likely to encounter TLS hostname verification errors if they
     connect to your cluster using a DNS name.
     Is this correct [Y/n]Y
     ## Which IP addresses will be used to connect to node-7.6.1?
     Enter all the IP addresses that you need, one per line.
     When you are done, press <ENTER> once more to move on to the next step.
     
     You did not enter any IP addresses.
     Is this correct [Y/n]Y

   • 而后是其余配置，根据须要进行配置，这里也要特别注意这个subject DN：CN=node-7，后续在java代码初始化RestClient的时候用到的域名要和这里相同，要否则会报错。

     ## Other certificate options
     
     The generated certificate will have the following additional configuration
     values. These values have been selected based on a combination of the
     information you have provided above and secure defaults. You should not need to
     change these values unless you have specific requirements.
     
     Key Name: node-7.6.1
     Subject DN: CN=node-7, DC=6, DC=1
     Key Size: 2048
     # 是否要更改这些选项，根据须要进行更改，必定要注意上边的CN值
     Do you wish to change any of these options? [y/N]N
     # 是否生成其余证书，这里根据须要选择
     Generate additional certificates? [Y/n]n

     若是到时候RestClient配置的域名和这里的CN值不一致，则会报如下错误IOException: Host name '127.0.0.1' does not match the certificate subject provided by the peer (CN=node7.example.com,DC-example, DC=com),若是在本地开发，记得改hosts。

   • 接下来根据须要设置密码，以及生成的zip文件名称

     ## What password do you want for your private key(s)?
     
     Your private key(s) will be stored as a PEM formatted file.
     We recommend that you protect your private keys with a password
     
     If you do not wish to use a password, simply press <enter> at the prompt below.
     # 若是须要密码，则输入密码，不须要直接enter下一步
     Provide a password for the private key:  [<ENTER> for none]
     
     ## Where should we save the generated files?
     
     A number of files will be generated including your private key(s),
     certificate request(s), and sample configuration options for Elastic Stack products.
     
     These files will be included in a single zip archive.
     # 默认给了文件名称，若是须要更改，则输入你想要的名称
     What filename should be used for the output zip file? [/home/konghang/elasticsearch-7.6.1/elasticsearch-ssl-http.zip]
     
     Zip file written to /home/konghang/elasticsearch-7.6.1/elasticsearch-ssl-http.zip

     压缩包的elasticsearch目录下有两个重要文件，一个csr格式，另外一个key格式。

   • 利用key文件生成ca文件（ca.crt）

     konghang@Surface-Pro7:~/elasticsearch-7.6.1/config$ openssl req -new -x509 -key http-node-7.6.1.key -out ca.crt -days 3650
     Can't load /home/konghang/.rnd into RNG
     140055252832704:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/home/konghang/.rnd
     You are about to be asked to enter information that will be incorporated
     into your certificate request.
     What you are about to enter is what is called a Distinguished Name or a DN.
     There are quite a few fields but you can leave some blank
     For some fields there will be a default value,
     If you enter '.', the field will be left blank.
     # 如下信息根据本身需求填写，暂时也没有发现有什么特殊需求，随便填，没有什么影响。若是有人知道这里有什么特殊用途能够告诉我
     -----
     Country Name (2 letter code) [AU]:zh
     State or Province Name (full name) [Some-State]:gd
     Locality Name (eg, city) []:gz
     Organization Name (eg, company) [Internet Widgits Pty Ltd]:snc
     Organizational Unit Name (eg, section) []:
     Common Name (e.g. server FQDN or YOUR name) []:node7.example.com
     Email Address []:

   • 而后再根据csr文件、ca文件、key文件生成须要的crt文件

     konghang@Surface-Pro7:~/elasticsearch-7.6.1/config$ openssl x509 -req -days 3650 -in http-node-7.6.1.csr -CA ca.crt -CAkey http-node-7.6.1.key -CAcreateserial -out http-node-7.6.1.crt
     Signature ok
     subject=DC = 1, DC = 6, CN = node7.example.com
     Getting CA Private Key

   • 而后移动http-node-7.6.1.crt和http-node-7.6.1.key到es的配置目录config下，而后配置es配置文件，添加如下配置

     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.certificate: http-node-7.6.1.crt
     xpack.security.http.ssl.key: http-node-7.6.1.key

     至此pem模式的es服务器端已经配置完成了，重启便可生效。

2. pkcs模式

   • 一样执行命令，只是在询问是否生成csr文件的时候，选择N

     ## Do you wish to generate a Certificate Signing Request (CSR)?
     
     A CSR is used when you want your certificate to be created by an existing
     Certificate Authority (CA) that you do not control (that is, you don't have
     access to the keys for that CA).
     
     If you are in a corporate environment with a central security team, then you
     may have an existing Corporate CA that can generate your certificate for you.
     Infrastructure within your organisation may already be configured to trust this
     CA, so it may be easier for clients to connect to Elasticsearch if you use a
     CSR and send that request to the team that controls your CA.
     
     If you choose not to generate a CSR, this tool will generate a new certificate
     for you. That certificate will be signed by a CA under your control. This is a
     quick and easy way to secure your cluster with TLS, but you will need to
     configure all your clients to trust that custom CA.
     
     Generate a CSR? [y/N]N

   • 而后就会让你输入一个现存在的ca文件路径，ca文件用咱们在配置TransportClient的时候生成的，输入后继续，输入ca文件的密码

     ## Do you have an existing Certificate Authority (CA) key-pair that you wish to use to sign your certificate?
     
     If you have an existing CA certificate and key, then you can use that CA to
     sign your new http certificate. This allows you to use the same CA across
     multiple Elasticsearch clusters which can make it easier to configure clients,
     and may be easier for you to manage.
     
     If you do not have an existing CA, one will be generated for you.
     # 选择使用现有的ca
     Use an existing CA? [y/N]y
     
     ## What is the path to your CA?
     
     Please enter the full pathname to the Certificate Authority that you wish to
     use for signing your new http certificate. This can be in PKCS#12 (.p12), JKS
     (.jks) or PEM (.crt, .key, .pem) format.
     # 输入咱们的ca文件位置
     CA Path: /home/konghang/elasticsearch-7.6.1/config/elastic-stack-ca.p12
     Reading a PKCS12 keystore requires a password.
     It is possible for the keystore's password to be blank,
     in which case you can simply press <ENTER> at the prompt
     # 若是ca有密码则输入密码，而后enter下一步
     Password for elastic-stack-ca.p12:

   • 输入认证文件的有效时间

     ## How long should your certificates be valid?
     
     Every certificate has an expiry date. When the expiry date is reached clients
     will stop trusting your certificate and TLS connections will fail.
     
     Best practice suggests that you should either:
     (a) set this to a short duration (90 - 120 days) and have automatic processes
     to generate a new certificate before the old one expires, or
     (b) set it to a longer duration (3 - 5 years) and then perform a manual update
     a few months before it expires.
     
     You may enter the validity period in years (e.g. 3Y), months (e.g. 18M), or days (e.g. 90D)
     # 输入须要的有效时间
     For how long should your certificate be valid? [5y]

   • 而后就和pem模式中从【是否为每个节点生成certificates】开始往下边3步，到【Other certificate options】配置CN。

   • 而后就是输入private key的密码，能够为空，而后就会生成一个名称为elasticsearch-ssl-http.zip的压缩包，里边的elasticsearch文件下生成一个名称为http.p12的文件

     ## What password do you want for your private key(s)?
     
     Your private key(s) will be stored in a PKCS#12 keystore file named "http.p12".
     This type of keystore is always password protected, but it is possible to use a
     blank password.
     
     If you wish to use a blank password, simply press <enter> at the prompt below.
     # 须要密码的根据本身需求进行配置
     Provide a password for the "http.p12" file:  [<ENTER> for none]
     
     ## Where should we save the generated files?
     
     A number of files will be generated including your private key(s),
     public certificate(s), and sample configuration options for Elastic Stack products.
     
     These files will be included in a single zip archive.
     # 生成的文件名称，自行配置
     What filename should be used for the output zip file? [/home/konghang/elasticsearch-7.6.1/elasticsearch-ssl-http.zip]
     
     Zip file written to /home/konghang/elasticsearch-7.6.1/elasticsearch-ssl-http.zip

     压缩包里的elasticsearch文件夹内在一个http.p12的文件,就是咱们须要的了。

   • 把http.p12移到es的配置目录config下，而后在配置文件末尾添加如下配置:

     xpack.security.http.ssl.enabled: true
     xpack.security.http.ssl.keystore.path: http.p12

     至此pkcs模式的es服务器端已经配置完成了

java代码实现RestCilent初始化

代码编写参考官方文档Encrypted communication。代码以下：

@Bean
// @ConditionalOnExpression("${es.properties.certificates_type}==1&&${es.properties.certificates_type:true}")
// @ConditionalOnExpression("elasticsearchProperties.getCertificatesType().equals('pkcs')")
@ConditionalOnExpression("'${es.properties.certificates_type}'.equals('pkcs')")
public RestClient getRestClient() throws CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
    logger.info("init RestClient, type:pkcs");
    RestClientBuilder builder = initRestClientBuilder("pkcs");
    RestClient restClient = builder.build();
    return restClient;
}

@Bean
@ConditionalOnExpression("'${es.properties.certificates_type}'.equals('pem')")
public RestClient getRestClientPem() throws CertificateException, NoSuchAlgorithmException, KeyStoreException, KeyManagementException {
    logger.info("init RestClient, type:pem");
    RestClientBuilder builder = initRestClientBuilder(elasticsearchProperties.getCertificatesType());
    RestClient restClient = builder.build();
    return restClient;
}

private RestClientBuilder initRestClientBuilder(String type) throws KeyStoreException, CertificateException, NoSuchAlgorithmException, KeyManagementException {
    CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
    credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(elasticsearchProperties.getUsername(), elasticsearchProperties.getPassword()));
    KeyStore keyStore = getKeyStoreByType(type);
    SSLContextBuilder sslContextBuilder = SSLContexts.custom().loadTrustMaterial(keyStore, (x509Certificates, s) -> true);
    final SSLContext sslContext = sslContextBuilder.build();
    RestClientBuilder builder = RestClient.builder(new HttpHost(elasticsearchProperties.getHost(), elasticsearchProperties.getHttpPort(), elasticsearchProperties.getSchema()));
    builder.setHttpClientConfigCallback(httpClientBuilder -> {
        httpClientBuilder.setSSLContext(sslContext);
        httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
        return httpClientBuilder;
    });
    return builder;
}

private KeyStore getKeyStoreByType(String type) throws KeyStoreException, CertificateException, NoSuchAlgorithmException {
    KeyStore keyStore = null;
    if (elasticsearchProperties.getCertificatesType().equalsIgnoreCase(type)){
        keyStore = KeyStore.getInstance("pkcs12");
        try (InputStream is = this.getClass().getClassLoader().getResourceAsStream(elasticsearchProperties.getPkcsClientFilePath())) {
            keyStore.load(is, "".toCharArray());
        } catch (IOException e) {
            e.printStackTrace();
        } catch (CertificateException e) {
            e.printStackTrace();
        } catch (NoSuchAlgorithmException e) {
            e.printStackTrace();
        }
    } else {
        CertificateFactory factory = CertificateFactory.getInstance("X.509");
        Certificate trustedCa;
        try (InputStream is = this.getClass().getClassLoader().getResourceAsStream(elasticsearchProperties.getPemFilePath())) {
            trustedCa = factory.generateCertificate(is);
            keyStore = KeyStore.getInstance("pkcs12");
            keyStore.load(null, null);
            keyStore.setCertificateEntry("ca", trustedCa);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    return keyStore;
}

完整代码github地址spring-boot-demo，eslasticsearch7.6模块。里边还包括JestClient的pkcs模式配置。

结尾

整个过程如上所述，但愿能够帮助一些朋友。若是以上过程当中有任何不对的地方欢迎在评论区指正，很是感谢。