[Elasticsearch]4.可伸缩性解密：集群、节点和分片

可伸缩性解密：集群、节点和分片

更新连载中…请关注

Scalability and resilience: clusters,nodes, and shard

Elasticsearch支持根据须要进行扩缩容.这得益于Elasticsearch是原生支持分布式的.能够经过往机器中添加服务器（节点）的方式扩大集群容量从而存储更多数据.Elasticsearch会自动的均一些数据和计算任务给新加入的数据.甚至不须要应用程序参与，Elasticsearch彻底知道该怎么把数据均衡到多个节点而且提供良好的可伸缩性和高可用性.集群的节点越多这种操做越顺滑越无感. 就是这么丝滑，堪比丝袜!

Elasticsearch is built to be always available and to scale with your needs. It does this by being distributed by nature. You can add servers (nodes) to a cluster to increase capacity and Elasticsearch automatically distributes your data and query load across all of the available nodes. No need to overhaul your application, Elasticsearch knows how to balance multi-node clusters to provide scale and high availability. The more nodes, the merrier.

为何能这么丝滑呢?用的什么配方?爆开揉碎一探究竟，在Elasticsearch的内部索引实际上是包含一个或多个物理存储分片的逻辑组。每一个分片都有它本身的索引数据.Elasticsearch就是经过把索引中的文档划分到多个分片（shard）中，再把划分的分片分到多个节点来实现索引的分布式存储的.当机器发生扩容或缩容时，Elasticsearch自动迁移从新均衡分片，这样就能保证在磁盘损坏或经过新加服务器（节点）扩容时依然能够正常提供服务.

How does this work? Under the covers, an Elasticsearch index is really just a logical grouping of one or more physical shards, where each shard is actually a self-contained index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple nodes, Elasticsearch can ensure redundancy, which both protects against hardware failures and increases query capacity as nodes are added to a cluster. As the cluster grows (or shrinks), Elasticsearch automatically migrates shards to rebalance the cluster.

等等！好像哪里不对,扩容好说，磁盘损坏了上面的分片不都坏了,还能提供正常服务?按读书少，你可不要骗俺啊?

这就要说下分片的类型了，其实有俩种类型的分片: 主分片和副分片(备用分片).在索引中的每一个文档隶属于一个主分片.副分片就是主分片的备份.

副分片:

其实我就是传说中的备胎，咿呀咿呀哟!

主分片和副分片一般是不在一个磁盘上的，当发生磁盘损坏时，磁盘上的主分片对应的副分片也就转正了，这样就解释了为何当磁盘损坏时Elasticsearch仍能提供服务了.另外备胎，不！是副分片也能提供读服务，这样就提升了集群的读取文档的吞吐量.(由于同时能够有多台机器提供文档读取服务)

There are two types of shards: primaries and replicas. Each document in an index belongs to one primary shard. A replica shard is a copy of a primary shard. Replicas provide redundant copies of your data to protect against hardware failure and increase capacity to serve read requests like searching or retrieving a document.

索引的主分片数量须要在建立索引的时候指定，建立后就不能修改了。但副分片的数量在索引建立后仍是能够修改地.并且修改副分片数量不会影响正在执行的索引和查询操做. 这就是备胎的分量啊!

The number of primary shards in an index is fixed at the time that an index is created, but the number of replica shards can be changed at any time, without interrupting indexing or query operations.

分片如此重要，怎么设置分片大小和数量呢?

It depands:

设置索引的分片大小和主分片数量时，有些咱们须要权衡取舍地方:

分片越多维护这些分片的开销就越大，分片越大再进行数据均衡时须要移动的数据也越多耗费的时间也越多.

鱼和熊掌不可兼得

There are a number of performance considerations and trade offs with respect to shard size and the number of primary shards configured for an index. The more shards, the more overhead there is simply in maintaining those indices. The larger the shard size, the longer it takes to move shards around when Elasticsearch needs to rebalance a cluster.

查询的分片越小查询每一个分片用的时间越少，但也意味须要更多的查询，查询越多消耗也越大，因此有时候分区大点分区数量少点查询可能反而更快.总之一句话，你本身看着办吧.

Querying lots of small shards makes the processing per shard faster, but more queries means more overhead, so querying a smaller number of larger shards might be faster. In short…it depends.

这里给出点建议，仅供参考:

• 控制分片大小在GB到数十GB.对于时序数据一般能够控制20GB到40GB.

• Aim to keep the average shard size between a few GB and a few tens of GB. For use cases with time-based data, it is common to see shards in the 20GB to 40GB range.

• 避免分片过多，一个节点能够容纳的分片数与可用堆空间成正比.通常来讲，每GB堆空间的分片数不该大于20.

• Avoid the gazillion shards problem. The number of shards a node can hold is proportional to the available heap space. As a general rule, the number of shards per GB of heap space should be less than 20.

最好的肯定分区大小和数量的方式就是在你的应用场景下使用数据和查询测试下.

The best way to determine the optimal configuration for your use case is through testing with your own data and queries.

容灾

In case of disaster

为了保证良好的性能，集群中的节点须要放在同一个网络环境中,由于跨数据中心移动迁移分片会耗费比较长的时间.可是为了高可用，咱们又不能把全部鸡蛋都放在一个篮子里.怎么作到当一个数据中心发生重大故障另外一个数据中心可以及时的接管，甚至让用户都感受不到发生了故障呢?可使用跨集群副本功能也就是CCR.

For performance reasons, the nodes within a cluster need to be on the same network. Balancing shards in a cluster across nodes in different data centers simply takes too long. But high-availability architectures demand that you avoid putting all of your eggs in one basket. In the event of a major outage in one location, servers in another location need to be able to take over. Seamlessly. The answer? Cross-cluster replication (CCR).

跨集群副本能够自动热备份主集群的索引数据到第二集群.若是主集群挂了，第二集群继续对外提供服务.第二集群也能够提供只读的服务这样就能够优选与用户距离近的集群提供服务了.

CCR provides a way to automatically synchronize indices from your primary cluster to a secondary remote cluster that can serve as a hot backup. If the primary cluster fails, the secondary cluster can take over. You can also use CCR to create secondary clusters to serve read requests in geo-proximity to your users.

跨集群副本是主从机制.在主集群上的是主，处理全部的写请求。在第二集群上的是从，只提供读服务(只处理读请求).

Cross-cluster replication is active-passive. The index on the primary cluster is the active leader index and handles all write requests. Indices replicated to secondary clusters are read-only followers.

运维

Care And feeding

像其它企业应用系统同样，咱们也须要对Elasticsearch进行安全管理和监控.能够经过Kibana提供的控制中心管理、监控Elasticsearch.好比数据回滚、管理索引生命周期啊什么的.

As with any enterprise system, you need tools to secure, manage, and monitor your Elasticsearch clusters. Security, monitoring, and administrative features that are integrated into Elasticsearch enable you to use Kibana as a control center for managing a cluster. Features like data rollups and index lifecycle management help you intelligently manage your data over time