【原创】大数据基础之ElasticSearch（1）简介、安装、使用

ElasticSearch 6.6.0

官方：https://www.elastic.co/

 

一 简介

ElasticSearch简单来讲是对lucene的分布式封装，增长了shard（每一个shard是一个子索引，也是一个lucene的index）和replica的概念；因此在ElasticSearch也能够见到lucene中的概念，好比index、document等。

 

Elasticsearch is a highly scalable open-source full-text search and analytics engine. It allows you to store, search, and analyze big volumes of data quickly and in near real time. It is generally used as the underlying engine/technology that powers applications that have complex search features and requirements.

es是一个可扩展、开源的全文检索和分析引擎。es能够近乎实时的存储、搜索和分析大规模数据；es一般做为底层的技术或者引擎使得应用能够实现复杂查询的需求和场景。

 

核心概念

1 集群概念

Cluster

A cluster is a collection of one or more nodes (servers) that together holds your entire data and provides federated indexing and search capabilities across all nodes. A cluster is identified by a unique name which by default is "elasticsearch". This name is important because a node can only be part of a cluster if the node is set up to join the cluster by its name.

Cluster（集群）由一个或多个Node组成，Cluster做为一个总体持有全部的数据，同时提供索引和查询功能；cluster有一个惟一的名字，默认是elasticsearch。

Node

A node is a single server that is part of your cluster, stores your data, and participates in the cluster’s indexing and search capabilities. Just like a cluster, a node is identified by a name which by default is a random Universally Unique IDentifier (UUID) that is assigned to the node at startup. You can define any node name you want if you do not want the default. This name is important for administration purposes where you want to identify which servers in your network correspond to which nodes in your Elasticsearch cluster.
A node can be configured to join a specific cluster by the cluster name. By default, each node is set up to join a cluster named elasticsearch which means that if you start up a number of nodes on your network and—assuming they can discover each other—they will all automatically form and join a single cluster named elasticsearch.

一个Node（节点）是一台服务器，做为集群的组成部分，存储数据同时参与集群的索引和查询功能；每一个Node也有一个惟一的名字，默认是UUID。
每一个Node均可以经过指定一个集群的名字来加入cluster；

2 索引概念

Index

An index is a collection of documents that have somewhat similar characteristics. An index is identified by a name (that must be all lowercase) and this name is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it.

一个Index是多个具备类似特征的document的集合；一个Index也有一个名字（全小写）。

Type（废弃）

A type used to be a logical category/partition of your index to allow you to store different types of documents in the same index. It is no longer possible to create multiple types in an index, and the whole concept of types will be removed in a later version.

一个type是一个逻辑概念，表示index中的category或者partition，将来会被废弃掉；

废弃进度：
Indices created in Elasticsearch 6.0.0 or later may only contain a single mapping type. Indices created in 5.x with multiple mapping types will continue to function as before in Elasticsearch 6.x. Types will be deprecated in APIs in Elasticsearch 7.0.0, and completely removed in 8.0.0.

废弃缘由：
Initially, we spoke about an “index” being similar to a “database” in an SQL database, and a “type” being equivalent to a “table”.
This was a bad analogy that led to incorrect assumptions. In an SQL database, tables are independent of each other. The columns in one table have no bearing on columns with the same name in another table. This is not the case for fields in a mapping type.

Document

A document is a basic unit of information that can be indexed. This document is expressed in JSON (JavaScript Object Notation) which is a ubiquitous internet data interchange format.

一个document是一个索引的基本信息单元，格式为json；

3 其余

Near Realtime (NRT)

Elasticsearch is a near-realtime search platform. What this means is there is a slight latency (normally one second) from the time you index a document until the time it becomes searchable.

es是一个近乎实时的搜索平台，近乎实时的含义是从插入document到能够搜索到document的延迟很小，一般是1s。

Shards

An index can potentially store a large amount of data that can exceed the hardware limits of a single node. For example, a single index of a billion documents taking up 1TB of disk space may not fit on the disk of a single node or may be too slow to serve search requests from a single node alone.
To solve this problem, Elasticsearch provides the ability to subdivide your index into multiple pieces called shards. When you create an index, you can simply define the number of shards that you want. Each shard is in itself a fully-functional and independent "index" that can be hosted on any node in the cluster.

Sharding is important for two primary reasons:

It allows you to horizontally split/scale your content volume
It allows you to distribute and parallelize operations across shards (potentially on multiple nodes) thus increasing performance/throughput

The mechanics of how a shard is distributed and also how its documents are aggregated back into search requests are completely managed by Elasticsearch and is transparent to you as the user.

一个索引若是很是大可能会超出单机硬件限制，es的解决方案是将索引划分为多个子索引，每一个子索引也叫shard（分片）。当建立索引的时候，能够指定shard的数量，每一个shard都是独立的索引；
shard很重要：1）水平扩展；2）分布式和并行操做；

Replicas

In a network/cloud environment where failures can be expected anytime, it is very useful and highly recommended to have a failover mechanism in case a shard/node somehow goes offline or disappears for whatever reason. To this end, Elasticsearch allows you to make one or more copies of your index’s shards into what are called replica shards, or replicas for short.

Replication is important for two primary reasons:

It provides high availability in case a shard/node fails. For this reason, it is important to note that a replica shard is never allocated on the same node as the original/primary shard that it was copied from.
It allows you to scale out your search volume/throughput since searches can be executed on all replicas in parallel.

网络环境中失败（硬件故障或网络故障）在所不免，因此failover就很重要；es容许你配置每一个shard有0个或多个备份，也叫replica（副本）；
replica很重要：1）高可用；2）并行操做；

Summary

To summarize, each index can be split into multiple shards. An index can also be replicated zero (meaning no replicas) or more times. Once replicated, each index will have primary shards (the original shards that were replicated from) and replica shards (the copies of the primary shards).

The number of shards and replicas can be defined per index at the time the index is created. After the index is created, you may also change the number of replicas dynamically anytime. You can change the number of shards for an existing index using the _shrink and _split APIs, however this is not a trivial task and pre-planning for the correct number of shards is the optimal approach.

By default, each index in Elasticsearch is allocated 5 primary shards and 1 replica which means that if you have at least two nodes in your cluster, your index will have 5 primary shards and another 5 replica shards (1 complete replica) for a total of 10 shards per index.

每一个index能够被拆分为多个shard，每一个索引均可以被复制0或屡次；一旦被复制，每一个索引都会有primary shard（主分片，被复制数据）和replica shard（复制分片，从主分片复制数据）；
shard和replica的数量能够在建立索引的时候配置，后续也能够修改；
默认的shard数量是5，默认的replica数量是1；

Each Elasticsearch shard is a Lucene index. There is a maximum number of documents you can have in a single Lucene index. As of LUCENE-5843, the limit is 2,147,483,519 (= Integer.MAX_VALUE - 128) documents. You can monitor shard sizes using the _cat/shards API.

每一个es的shard都是一个lucene的index，lucene的index中有document数量限制，若是超出数量，es须要增长更多的分片；

 

二 安装

1 docker安装

# docker run elasticsearch

2 ambari安装

详见：http://www.javashuo.com/article/p-ugvdlakh-dz.html

3 手工tar安装 

# curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.2.tar.gz
# tar -xvf elasticsearch-6.6.2.tar.gz
# cd elasticsearch-6.6.2

配置文件

config/elasticsearch.yml

能够配置集群名称，数据目录（和hdfs同样，能够配置多个硬盘提高性能）等；

启动

# bin/elasticsearch

4 手工yum安装

# yum install elasticsearch

配置文件

/etc/elasticsearch/elasticsearch.yml

启动

# service elasticsearch start

 

启动以后访问

# curl http://$es_server:9200

{
"name" : "y8NAQ9f",
"cluster_name" : "es_jdc",
"cluster_uuid" : "fZU17hfjTKO2IDUwnGmbEg",
"version" : {
"number" : "6.6.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "a9861f4",
"build_date" : "2019-01-24T11:27:09.439740Z",
"build_snapshot" : false,
"lucene_version" : "7.6.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

若是启动报错

ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

当前配置

# sysctl -a|grep vm.max_map_count
vm.max_map_count = 65530

修改方法

sysctl -w vm.max_map_count=262144

or

echo 'vm.max_map_count=262144' >> /etc/sysctl.conf

 

重要配置：https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html

三 使用

Elasticsearch uses standard RESTful APIs and JSON. We also build and maintain clients in many languages such as Java, Python, .NET, SQL, and PHP. 

其中rest api方式，详见 http://www.javashuo.com/article/p-atxrhcbc-kz.html

 

在hadoop和es之间进行数据迁移，参考：https://www.elastic.co/products/hadoop