Deploy the manual snapshot acquisition mechanism with Cloudformation at once with Amazon Elasticsearch Service (AES). And it is set to automatically delete old snapshots using Curator. It's surprisingly difficult just to take a snapshot.
The Python script is based on what is in this AWS documentation.
https://docs.aws.amazon.com/ja_jp/elasticsearch-service/latest/developerguide/curator.html
Make this. The script uses what is in the AWS documentation almost as it is, so I think that it is a level that you can understand immediately if you read it. There are also comments.
In July 2019, automatic snapshots were enhanced as follows, and hourly snapshots are now stored in S3 for 14 days.
https://aws.amazon.com/jp/about-aws/whats-new/2019/07/amazon-elasticsearch-service-increases-data-protection-with-automated-hourly-snapshots-at-no-extra-charge/
This alone is very convenient because even when data is lost due to a simultaneous failure of multiple nodes, the situation can be restored to the situation one hour before the failure.
However, automatic snapshots are taken into a managed S3 bucket, so everything is deleted when the domain is deleted.
There are also restrictions such as not being able to restore a snapshot to another domain, such as when you want to make a duplicate of a domain.
I think there are quite a few needs to use Kibana definitions and Index maps in different domains. For that purpose, it is good to take a manual snapshot in parallel with the automatic snapshot.
There is only a proven service, and there is official information about this area.
https://docs.aws.amazon.com/ja_jp/elasticsearch-service/latest/developerguide/curator.html
However, even if you copy this to Lambda as it is, it will not work. The library used in the Python sample script needs to be set separately so that it can be imported by Lambda.
In addition, IAM and other settings are required to perform manual snapshots. There is a manual here, but if you're not used to it, you won't know what to do.
https://docs.aws.amazon.com/ja_jp/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html
In this article, the goal is to create an environment in which this program (a slightly modified version) works.
This is the premise of the operation performed in this article.
At a minimum, a Linux or Mac machine with the following installed: (By the way, I'm working on wsl2)
--Amazon Elasticsearch Service has been deployed with VPC access type
Download the following repositories from Github.
$ git clone [email protected]:yomon8/aes-snapshot-sample.git
$ cd aes-snapshot-sample/
https://github.com/yomon8/aes-snapshot-sample
Enter the variables in the configuration file.
$ cp settings.sh{.sample,}
$ vim settings.sh
There are the following items, so set them according to your environment.
#Cloudformation Stack name
readonly STACK_NAME=es-snapshot-sample
#Bucket for uploading Cloudformation packages
readonly STACK_S3_BUCKET=bucket-name
readonly STACK_S3_PREFIX=es-snapshot-sample
#AES host name (AES is created on the assumption of VPC access)
readonly AES_HOST=vpc-xxxxxxxxxxxxxxxxxxxxxx.ap-northeast-1.es.amazonaws.com
#Specify Subnet that can access AES
readonly LAMBDA_SUBNET_ID=subnet-xxxxxxxxxxxxxxxxxxxxxx
#Specify SG that can access AES
readonly LAMBDA_SECURITY_GROUP_ID=sg-xxxxxxxxxxxxxxxxxxx
#The name of the AES snapshot storage location
readonly SNAPSHOT_REPOSITORY_NAME=your-repo
#Prefix with the name of the AES snapshot
readonly SNAPSHOT_PREFIX=your-snapshot
After that, hit the script for deployment and you're done. The first time it takes a long time to download the Layer software. It takes less than 10 minutes in my environment.
$ ./deploy.sh
~ Omitted ~
When I run deploy.sh
, the following message is displayed at the end.
1.Execute the following command once at the beginning.
aws lambda invoke --function-name aes-snapshot-sample-AESRegistSnapshotRepositoryFun-1SHIY9NWSSH8B:live /dev/null
As the message says, when you execute the command, Lambda for repository registration will be executed.
$ aws lambda invoke --function-name aes-snapshot-sample-AESRegistSnapshotRepositoryFun-1SHIY9NWSSH8B:live /dev/null
{
"StatusCode": 200,
"ExecutedVersion": "1"
}
Let's check if the repository is registered. A repository called my-repo
is registered.
GET _cat/repositories?v
id type
cs-automated s3
my-repo s3
At this point, I just registered the repository, so there is no snapshot. Even if you hit the following API, the snapshot should not be displayed.
GET _cat/snapshots/my-repo
You should see another message in deploy.sh
to take a snapshot.
2.To take a snapshot manually, run the following command.
In addition, this Lambda is also set to schedule execution.
aws lambda invoke --function-name aes-snapshot-sample-AESRotateSnapshotFunction-1GB4JKXMNZ8KQ:live /dev/null
You can get a backup by executing the command displayed here.
$ aws lambda invoke --function-name aes-snapshot-sample-AESRotateSnapshotFunction-1GB4JKXMNZ8KQ:live /dev/null
{
"StatusCode": 200,
"ExecutedVersion": "1"
}
GET _cat/snapshots/my-repo?v
id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
my-snapshot-2019-11-13-12-35 SUCCESS 1573647456 12:17:36 1573647456 12:17:36 459ms 1 1 0 1
You can confirm that the snapshot data is also included in S3.
In addition, since the backup is set to start for a long time with CloudWatch Event, it will work once a day without executing the command as described above.
You can adjust the schedule in the following part of template.yaml
. Other than the schedule, I just brought the script, so please adjust and use it.
Events:
Rule:
Type: Schedule
Properties:
Schedule: cron(5 16 * * ? *)
You can see what you are doing by looking at the contents of deploy.sh
.
elasticsearch and Curator are downloaded with Docker, built and uploaded as Lambda Layer.
#!/bin/bash
#Read configuration file
source ./settings.sh
#Download the required Python library as a Lambda Layer
docker run --rm -v $(pwd)/layer/python:/python python:3.7.5-alpine pip install -t /python requests-aws4auth elasticsearch elasticsearch-curator
#Deploy with Cloudformation
aws cloudformation package --template-file ./template.yaml --output-template-file ./package.yaml \
--s3-bucket ${STACK_S3_BUCKET} \
--s3-prefix ${STACK_S3_PREFIX}
aws cloudformation deploy --template-file ./package.yaml --capabilities CAPABILITY_IAM \
--stack-name ${STACK_NAME} \
--parameter-overrides \
AESHost=${AES_HOST} \
LambdaSubnetId=${LAMBDA_SUBNET_ID} \
LambdaSecurityGroupId=${LAMBDA_SECURITY_GROUP_ID} \
SnapshotRepositoryName=${SNAPSHOT_REPOSITORY_NAME} \
SnapshotPrefix=${SNAPSHOT_PREFIX}
#Message display
echo ""
echo "1.Execute the following command once at the beginning."
echo aws lambda invoke --function-name $(aws cloudformation describe-stacks --stack-name ${STACK_NAME} --query 'Stacks[].Outputs[?OutputKey == `RegistSnapshotFunctionName`].OutputValue' --output text):live /dev/null
echo ""
echo "2.To take a snapshot manually, run the following command."
echo "In addition, this Lambda is also set to schedule execution."
echo aws lambda invoke --function-name $(aws cloudformation describe-stacks --stack-name ${STACK_NAME} --query 'Stacks[].Outputs[?OutputKey == `RotateSnapshotFunctionName`].OutputValue' --output text):live /dev/null
echo ""
If you register S3 where snapshots are stored as a repository in the same way, you can easily restore it by following the procedure linked below.
https://docs.aws.amazon.com/ja_jp/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html#es-managedomains-snapshot-restore
https://github.com/elastic/curator
Often, you want to restore something other than Kibana. You can restore non-Kibana indexes by doing the following: You can restore only a specific index by changing the parameters of indices.
POST _snapshot/Repository name/Snapshot name/_restore
{"indices": "*,-.kibana_1"}
https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
https://docs.aws.amazon.com/ja_jp/elasticsearch-service/latest/developerguide/es-managedomains-snapshots.html#es-managedomains-snapshot-restore
Recommended Posts