This time, I will introduce how to make an automatic backup of Google Cloud Platform Datastore.
--Something exists in Datastore
Go to GoogleCloudPlatformStorage (https://console.cloud.google.com/projectselector2/storage/browser?supportedpurview=project) and select your project
Click Create Bucket to create a bucket
--Put a name in the bucket --Let's enter the name in lowercase alphanumeric characters (this time we will do it with backup_qiita) --Selection of data storage location --Region: Specific geographical location --Multi-region: A large geographic location that includes two or more geographic locations such as the United States --Dual-region: Specific geographic location indicating a pair, such as Finland or the Netherlands
Press [Create] when the settings are complete. A bucket for backups has been created like this. Next, set up to put the backup in the bucket
Here, https://cloud.google.com/datastore/docs/schedule-export?hl=ja I will add it with Yaml and Python while referring to it. First, create ** [app.yaml] ** in Yaml. (If it already exists, create a folder and create it in it) ← I put it in the backup folder like this
The app.yaml here is prepared for Python settings.
app.yaml
runtime: python27
api_version: 1
threadsafe: true
service: cloud-datastore-admin
libraries:
- name: webapp2
version: "latest"
handlers:
- url: /cloud-datastore-export
script: cloud_datastore_admin.app
login: admin
Next, let's prepare ** [cloud_datastore_admin.py] ** in Python.
cloud_datastore_admin.py
import datetime
import httplib
import json
import logging
import webapp2
from google.appengine.api import app_identity
from google.appengine.api import urlfetch
class Export(webapp2.RequestHandler):
def get(self):
access_token, _ = app_identity.get_access_token(
'https://www.googleapis.com/auth/datastore')
app_id = app_identity.get_application_id()
timestamp = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
output_url_prefix = self.request.get('output_url_prefix')
assert output_url_prefix and output_url_prefix.startswith('gs://')
if '/' not in output_url_prefix[5:]:
# Only a bucket name has been provided - no prefix or trailing slash
output_url_prefix += '/' + timestamp
else:
output_url_prefix += timestamp
entity_filter = {
'kinds': self.request.get_all('kind'),
'namespace_ids': self.request.get_all('namespace_id')
}
request = {
'project_id': app_id,
'output_url_prefix': output_url_prefix,
'entity_filter': entity_filter
}
headers = {
'Content-Type': 'application/json',
'Authorization': 'Bearer ' + access_token
}
url = 'https://datastore.googleapis.com/v1/projects/%s:export' % app_id
try:
result = urlfetch.fetch(
url=url,
payload=json.dumps(request),
method=urlfetch.POST,
deadline=60,
headers=headers)
if result.status_code == httplib.OK:
logging.info(result.content)
elif result.status_code >= 500:
logging.error(result.content)
else:
logging.warning(result.content)
self.response.status_int = result.status_code
except urlfetch.Error:
logging.exception('Failed to initiate export.')
self.response.status_int = httplib.INTERNAL_SERVER_ERROR
app = webapp2.WSGIApplication(
[
('/cloud-datastore-export', Export),
], debug=True)
If you can input like this, let's deploy
gcloud config set project [Project name]
gcloud app deploy
If you can deploy it successfully, you are ready for automatic backup. We will automate it next.
See also https://cloud.google.com/datastore/docs/schedule-export?hl=ja I will proceed to
First to set up a cron job that calls the GCP schedule-datastore-exports Create ** [cron.yaml] **.
cron.yaml
cron:
- description: "Explanatory text"
url: /cloud-datastore-export?output_url_prefix=gs://[Bucket name]
target: cloud-datastore-admin
timezone: Asia/Tokyo
schedule: every 12 hours
If this is okay, let's deploy
gcloud app deploy cron.yaml
Use kind if you want to export only certain types of entities when backing up
url: /cloud-datastore-export?output_url_prefix=gs://[Bucket name]&kind=[Entity type name]
It is also possible to specify multiple specific ones
url: /cloud-datastore-export?output_url_prefix=gs://[Bucket name]&kind=[Entity type name]&kind=[Entity type name]
If you want to export if the namespace exists for a particular entity
url: /cloud-datastore-export?output_url_prefix=gs://[Bucket name]&kind=[Entity type name]&kind=[Entity type name]
You can also make detailed settings by entering
Let's specify the time to execute the backup in ** [schedule] ** Currently, it is set to perform backup once every 12 hours, but if it is once a day
schedule: every 24 hours
If you want to perform a backup every day at 00:00
schedule: every day 00:00
Can be set widely For more information [https://cloud.google.com/appengine/docs/flexible/java/scheduling-jobs-with-cron-yaml?hl=ja#defining_the_cron_job_schedule](https://cloud.google.com/appengine/ docs / flexible / java / scheduling-jobs-with-cron-yaml? hl = ja #defining_the_cron_job_schedule) for details. By the way, if you want to set the reference time for backup to Japan time
timezone: Asia/Tokyo
If so, it will be done in Japan time.
If the App Engine cron job is displayed like this, cron is working well
Now let's move it with [Run Now] Now!
Wow, Elatter (´ ゚ д ゚ `) In such a case, let's see the error in [View] of the log.
When I displayed the log, it was 403.
When you open it, it says "The caller does not have permission" ... In other words, "I don't have the authority to import and export."
Let's go give authority
Open IAM in IAN and admin Edit [Project ID]@appspot.gserviceaccount.com in it Press [Add another role] Select [Cloud Datastore Import / Export Administrator] in Datastore and save!
So let's go back to running again If you press [Run Now] again ... It's done! !!
Now it will take the data every 12 hours!
How was everyone, did it work? I'm wondering if this commentary went well (´ ゚ д ゚ `)
But if you back up the data in the Datastore like this You can erase it by human error or deal with any troubles. I don't think there is any loss in doing it
It's been a long time, but thank you! !!
https://cloud.google.com/storage/docs/locations?_ga=2.5981299.-176719181.1568084299
https://cloud.google.com/datastore/docs/schedule-export?hl=ja
https://cloud.google.com/appengine/docs/flexible/java/scheduling-jobs-with-cron-yaml?hl=ja#defining_the_cron_job_schedule
Recommended Posts