Automatically stop the VM when the machine learning process running on GCP is finished

table of contents

1. Overview
2. Install Google Cloud SDK on the VM that performs machine learning processing
3. Google Cloud Platform authentication
4. Create a script to stop when processing is finished on the gcloud command line
5. Notify with Slack & stop VM when processing is completed with gcloud command line

1. Overview

I wrote an article [Qiita] Notify me with Slack when the machine learning process running on GCP is finished. Isn't it sometimes better to stop the VM automatically when sleeping in the middle of the night? So, I will describe how to do it.

Don't forget to save the model (joblib, pickle) and save the result (csv export) so that the calculation is not wasted.

2. Install Google Cloud SDK on the VM that performs machine learning processing

As some of you may have noticed here, when the process is over The strategy is to issue the glcoud command yourself to stop your VM.

Download and install Google Cloud SDK

Terminal

curl https://sdk.cloud.google.com | bash

Pass the path of Google Cloud SDK

At the time of installation, if you do not specify Path, execute the following If Path is specified, change the part of .bashrc

Terminal

source .bashrc

Check if the Google Cloud SDK path is in place

Terminal

gcloud -v

Execution example: It is OK if the result is returned like this

3. Google Cloud Platform authentication

Execute initialization command

Authenticate the target project name and Gmail

Terminal

gcloud init

4. Create a script to stop when processing is finished on the gcloud command line

[Qiita] Try creating, stopping, resuming, and destroying a GCP Compute Engine instance with commands It seems that you can stop with gcloud compute instances stop <instance name>, so Put this into Python processing

In the case of Jupterlab, just add "!" To the beginning of the target command.

For Jupyterlab

#Some kind of execution process
! gcloud compute instances stop <Instance name>

Execution example: Since the execution image is hard to boil in the stop processing, it is an example of displaying the instance list.

5. Notify with Slack & stop VM when processing is completed with gcloud command line

[Qiita] Notify me with Slack when the machine learning process running on GCP is finished In accordance with the Slack notification written above, we will also test to stop the VM

python

#Import of required module
import slackweb
import datetime

#Classify Slack notifications
class Slack_notice():
    start = 0

    def __init__(self):
        self.slack = slackweb.Slack(url="URL of Slack's Incoming Webhook") #Rewrite it to your own URL

    def begin(self):
        self.start = datetime.datetime.now()
        self.slack.notify(text='[Notice] ' + str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M")) + '\n Processing has started!')

    def end(self):
        elapsed_time = datetime.datetime.now() - self.start
        self.slack.notify(text='[Notice] ' + str(datetime.datetime.now().strftime("%Y-%m-%d %H:%M")) + '\n Processing is finished! \n Time required'+ str(elapsed_time) + 'is\n Stop the VM.')

# Slack_notice class call
slack_notice = Slack_notice()

#Pre-execution notification
slack_notice.begin()

#The process you want to execute-------------------------------------------------------------------------
#This time, for the sake of clarity, I just printed every second.
from time import sleep

for num in range(10):
    print(num)
    sleep(1)
# -------------------------------------------------------------------------------------

#Pre-execution notification
slack_notice.end()

#VM stop
! gcloud compute instances stop <Instance name>

Execution example

It was stopped safely