ES索引和快照的自动化管理方案

时间 2021-04-27

标签 html python docker json vim bash 微信 session app dom 栏目测试自动化繁體版

原文原文链接

说明

因为业务的须要，咱们ES使用的是Amazon Elasticsearch Service 7.4，为了配合开发同窗的使用和节省部门没必要要的开支，咱们将按期去备份索引快照至S3中，同时删除ES对应的索引数据。html

需求

咱们须要按期备份必定周期（好比：一周以前）的索引快照至S3，删除Elasticsearch Service中对应索引数据。同时，若有须要还要能够恢复备份的索引数据。整个流程执行成功后，要有微信或其余途径信息提醒。python

流程图

ES需求处理流程图：docker

curator工具

curator与es版本的兼容性

The current version of Curator is 5.8.3，详见传送门json

curator可执行的操做

curator容许对索引和快照执行许多不一样的操做，包括：vim

从别名添加或删除索引（或二者！）
更改分片路由分配
关闭索引
建立索引
删除索引
删除快照
打开被关闭的索引
对索引执行forcemerge段合并操做
reindex索引，包括来自远程集群的索引
更改索引的每一个分片的副本数
rollover索引
生成索引的快照（备份）
还原快照

安装配置curator

curator安装方式有多种，好比：yum/apt-get、pip、docker等，这里咱们选择经常使用的pip。bash

pip安装curator

# 安装必要的基础包
yum install -y vim python-pip

# 安装虚拟环境
pip install virtualenvwrapper
# 配置虚拟环境，在/etc/profile添加：
### virtualenv start ###
#设置virtualenv的统一管理目录
export WORKON_HOME=~/Envs
#添加virtualenvwrapper的参数，生成干净隔绝的环境
#export VIRTUALENVWRAPPER_VIRTUALENV_ARGS='--no-site-packages'
#指定python解释器
#export VIRTUALENVWRAPPER_PYTHON=/opt/python36/bin/python3.6
#执行virtualenvwrapper安装脚本
export VIRTUALENVWRAPPER_SCRIPT=/usr/bin/virtualenvwrapper.sh
source /usr/bin/virtualenvwrapper_lazy.sh
### virtualenv end ###
# 刷新配置文件
source !$
# 建立管理es的虚拟环境
mkvirtualenv es-snapshot
# 查看刚建立的虚拟环境
lsvirtualenv
# 进入虚拟环境
workon es-snapshot

# 在es-snapshot虚拟环境中安装
pip install elasticsearch-curator

命令行下单次执行

# 查看当前es的全部索引的详细信息，默认host：127.0.0.1，默认port：9200
curator_cli --host 127.0.0.1 --port 9200 show_indices --verbose

配置config.yml

# Rmember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  #es集群地址
  hosts: http://your-domain.com
  #es端口
  port: your-port
  url_prefix:
  use_ssl: False
  # aws区域，如ap-south-1
  aws_region: xxxxx
  aws_sign_request: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  #日志级别
  loglevel: INFO
  #日志存放路径
  logfile: /var/log/cur-run.log
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

配置action.yml

actions:
  1:
    # 备份7天前的索引快照
    action: snapshot
    description: >-
      Snapshot sdk_|game_ prefixed indices older than 7 day (based on index
      creation_date) with the default snapshot name pattern of
      'es-%Y%m%d%H%M%S'.  Wait for the snapshot to complete.  Do not skip
      the repository filesystem access check.  Use the other options to create
      the snapshot.
    options:
      # s3仓库名称，可经过脚本生成
      repository: "es_backup_\
                   "
      # Leaving name blank will result in the default 'curator-%Y%m%d%H%M%S'
      name: es-%Y%m%d%H%M%S
      ignore_unavailable: False
      include_global_state: True
      partial: True
      wait_for_completion: True
      skip_repo_fs_check: True
      ignore_empty_list: True
      continue_if_exception: False
      disable_action: False
    filters:
      - filtertype: pattern
        kind: regex
        # 匹配"logstash-"的索引
        value: 'logstash-'
      - filtertype: age
        source: creation_date
        direction: older
        unit: days
        # 7天以前的索引
        unit_count: 7
  2:
    # 关闭7天前以logstash-为前缀的索引:
    action: close
    description: >-
      Close indices older than 7 days (based on index name), for dtlog-
      prefixed indices.
    options:
      delete_aliases: False
      timeout_override:
      continue_if_exception: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^logstash-'
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
  3:
    # 删除7天前的索引
    action: delete_indices
    description: >-
      Delete metric indices older than 7 days (based on index name), for logstash-2021.04.10
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
    filters:
    - filtertype: pattern
      kind: prefix
      # 匹配"logstash-"的索引
      value: logstash-
    - filtertype: age
      # 这里根据索引name来匹配，还能够根据字段等，详见官方文档
      source: name
      direction: older
      # 用于匹配和提取索引或快照名称中的时间戳
      timestring: '%Y.%m.%d'
      unit: days
      # 7天以前的索引
      unit_count: 7

配置action顺序：7天前索引作快照 --> 关闭7天前索引 --> 删除7天前索引 --> 保留7天内的索引，若有须要可把7天前的快照恢复当前es中。微信

action.yml配置中：session

# s3仓库名称，可经过脚本生成
      repository: "es_backup_\
                   "

之因此这样写，是由于执行python register-repo.py会获得两个值：带有时间戳仓库的后缀好比es_backup_20210424150533，另外一个值是时间戳并把它写入time_save.txt。sed '/es_backup_/r time_save.txt' action_temp.yml -i将得到的时间戳传进action_temp.yml中。app

注意：actions: 后面的，依次类推：dom

2：执行操做
3：执行操做
4：执行操做
N：执行操做

建立S3仓库

在执行curator以前，咱们须要建立s3仓库，须要配置IAM role访问Elasticsearch Service权限，详见AWS Elasticsearch Service 创建snapshot

详见以下脚本：

# cat register-repo.py

import boto3
import requests
from requests_aws4auth import AWS4Auth
import time

def create_s3_register(timeup):
   host = 'https://your-aws-es-domain.com/' # include https:// and trailing /
   region = 'ap-south-1' # e.g. us-west-1
   service = 'es'
   credentials = boto3.Session().get_credentials()
   awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
   # Register repository
   path = '_snapshot/'+'es_backup_'+timeup # the Elasticsearch API endpoint
   url = host + path

   payload = {
     "type": "s3",
     "settings": {
       "bucket": "your-s3-bucket",
       "region": "ap-south-1",
       "role_arn": "arn:aws:iam::1234567890:role/your-role-name"
     }
   }

   headers = {"Content-Type": "application/json"}

   r = requests.put(url, auth=awsauth, json=payload, headers=headers)

   print(r.status_code)
   print(r.text)

def var_save(timeup,filename,mode='w'):
    file = open(filename,mode)
    file.write('              '+timeup+'\\'+'\n')
    file.close

if __name__=="__main__":
    time = time.strftime('%Y%m%d%H%M%S',time.localtime(time.time()))
    create_s3_register(time)
    var_save(time,'time_save.txt')

建立执行脚本

该脚本将完成，索引快照备份、索引关闭和索引删除，最后微信信息通知，详见：

#!/bin/bash
#author: tengfei.wu
#email: tengfei.wu@domain.com
#date:2021/04/25
#version: 2

# Create the S3 repository
python register-repo.py
# Get the name of the warehouse
cp action.yml action_temp.yml
sed '/es_backup_/r  time_save.txt' action_temp.yml -i
# Perform ES index shutdown, backup, and deletion
#curator  --config config.yml action_temp.yml --dry-run
curator  --config config.yml action_temp.yml
rm -rf action_temp.yml
# WeChat alarm
content='
【AI测试环境】-- ES操做通知

详情信息: "ES快照备份、索引关闭和索引删除"
操做细节:   
  索引快照: "7天前索引"
  索引关闭: "7天前索引"
  索引删除: "7天前索引"
  状态:  SUCCESS
  报警建立方式: "自动脚本对接"
  当前索引: "保留最近一周的索引"'

curl http://x.x.x.x:4567/send -d "tos=your-IM&content=${content}"

定时执行

# logstash-日志备份，每周日am 9:30
30 9 * * 0  cd /root/Envs/es-snapshot/bin && source ./activate && cd /root/Envs/es-snapshot && (/bin/bash ccc) && deactivate

手动执行

workon es-snapshot && cd /root/Envs/es-snapshot/
sh start_es_backup.sh > /dev/null 2>&1 &

检查执行结果

其余的操做

索引恢复

# cat action_restore.yml
actions:
  1:
    action: restore
    description: >-
      Restore all indices in the most recent snapshot with state SUCCESS.  Wait
      for the restore to complete before continuing.  Do not skip the repository
      filesystem access check.  Use the other options to define the index/shard
      settings for the restore.
    options:
      repository: es_backup_20210425054626
      name: 
      indices: 
      wait_for_completion: True
      #max_wait: 3600
      #wait_interval: 10
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:

查看当前索引状态：curator_cli --host your-domain.es.amazonaws.com --port your-port show_indices --verbose

索引打开

# cat action_open.yml
actions:
  1:
    action: open
    description: "open selected indices"
    options:
      continue_if_exception: False
      timeout_override: 300
    filters:
    - filtertype: pattern
      kind: regex
      value: '^logstash-'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7

仓库快照删除和删除空仓库

# action_delete_snapshot.yml

# 删除快照配置示例
actions:
  1:
    action: delete_snapshots
    description: "Delete selected snapshots from 'repository'"
    options:
      repository: es_backup_20210424150533
      retry_interval: 120
      retry_count: 3
      timeout_override: 3600
    filters:
    - filtertype: state
      state: SUCCESS
      exclude:

注意：

上面的action_delete_snapshot.yml配置只是清空了es_backup_20210424150533仓库中的快照内容，仓库并无删除，删除空仓库：DELETE /_snapshot/es_backup_20210424150533