Create a GCE instance from a GCR Docker image using terraform

Various memos when using terraform

What is terraform

Terraform to understand in 10 minutes

The point is, when creating a cloud environment, instead of creating a procedure manual and creating it manually, It seems to be a tool that tries to do all that with code. (If you can manage it with code, it may be easier to manage the version of the environment itself, or you can eliminate personality.)

Purpose (in my case)

I tested the scraping script with Colaboratory and For launching a server quickly when you want to run it all night

environment

cloudshell Terraform is installed in the default cloudshell docker image, so you don't need to create a custom image

Premise

Docker image pushed to GCR

(Reference) Dockerfile for python scraping

Use selenium and beautiful soup to put scraping results into a pandas dataframe Dockerfile that can be used when you want to insert from pandas to BigQuery

GCP JSON key file and scraping python file must be in the same directory as Dockerfile

Dockerfile


FROM python:3

ARG project_dir=/selenium/
ARG credential_json=<JSON key file name>

ADD requirements.txt $project_dir
ADD $credential_json $project_dir
ADD <Scraping python file> $project_dir
ENV GOOGLE_APPLICATION_CREDENTIALS ./$credential_json 

WORKDIR $project_dir

RUN apt-get update
RUN apt-get install -y vim less python3-selenium

RUN pip install -r requirements.txt

CMD ["/bin/bash"]

requirements.txt


selenium==3.141.0
requests==2.23.0
pandas==1.0.4
bs4==0.0.1
google-cloud-bigquery==1.21.0
pandas-gbq==0.11.0
lxml==4.5.1

Create a terraform configuration file (tf file)

Create a file (tf file) that defines resources in an appropriate directory

main.tf The main file that sets things up for GCE

main.tf


provider "google" {
    credentials = "${file("<JSON key file>")}"
	project     = "${var.project_name}"
    region      = "${var.region}"
  }

data "google_compute_network" "default" {
  name = "default"
}
  
resource "google_compute_instance" "apps-gcp-terraform" {
  name         = "selenium-docker"
  machine_type = "n1-standard-1"
  zone         = "${var.zone}"
  
  boot_disk {
	auto_delete = true
    initialize_params {
      image = "${var.boot_image_name}"
      type  = "pd-standard"
    }
  }

  metadata = {
    gce-container-declaration = "${var.docker_declaration}"
  }
  
  network_interface {
	network       = "default"
	access_config {
    }
  }

  service_account {
    email = "<Service account>"
    scopes = ["cloud-platform"]
  }
}

variables.tf File that handles parameters

When creating a GCE instance from Docker image, use a boot image called container-optimized image You can check the current container-optimized image version with the following command.

gcloud compute images list --project cos-cloud --no-standard-images

variables.tf


variable "project_name" {
  type    = string
  default = "<Project ID>"
}

variable "region" {
  type    = string
  default = "us-central1"
}

variable "zone" {
  type    = string
  default = "us-central1-a"
}

variable "boot_image_name" {
  type    = string
  default = "projects/cos-cloud/global/images/cos-stable-81-12871-119-0"
}

variable "docker_declaration" {
  type    = string
  default = "spec:\n  containers:\n    - name: selenium-docker\n      image: '<docker image>'\n      stdin: false\n      tty: true\n  restartPolicy: Always\n"
}

Create a GCE instance on terraform

Execute the following command in the directory where the tf file is located

  1. terraform init Initialize terraform If successful, it seems that a .terraform directory will be created in the executed directory and necessary plugins will be installed.

When such a message appears Terraform has initialized, but configuration upgrades may be needed.

terraform 0.12upgrade ↑ It seems that it will update the tf file to the latest version nicely

  1. terraform plan Check if the tf file is correct If an error occurs, fix the tf file

  2. terraform apply Up to gcp You will be asked if you want to execute it on the way, so enter yes

Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve.

Enter a value: yes

google_compute_instance.apps-gcp-terraform: Creating... google_compute_instance.apps-gcp-terraform: Still creating... [10s elapsed] google_compute_instance.apps-gcp-terraform: Creation complete after 13s

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

If it looks like ↑, it's OK

Then enter the created GCE instance

docker container ls 

Then the container is running, so

docker container exec -it container ID/bin/bash 

Enter the container with commands such as

nohup python scraping python file> log.txt 2>&1 & 

Run the program

Delete the instance

terrafrom destroy Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm.

Enter a value:yes

Will be deleted by

Addictive point

Attach a service account

Error: Cannot get auth token: Metadata server responded with status 404 When when It seems that you can not get tokens from GCE's metadata server,

So, if you execute the following command in the instance,

curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
-H "Metadata-Flavor: Google"

↓ message appears Service account not enabled on this instance

So, when I added the service account attachment to main.tf, it was OK ↓

main.This part of tf


service_account {
    email = "<Service account>"
    scopes = ["cloud-platform"]
  }

reference

terraform documentation Create and configure an instance Use variables in Terraform GCP Infrastructure Construction Automation Road part1 ~ Terraform Introduction ~ Deploy a web app on Google Compute Engine with Terraform. Terraform 0.12 has been released, so I upgraded it [Try building a GCP environment with Terraform](https://techblog.gmo-ap.jp/2017/11/16/terraform%E3%81%A7gcp%E7%92%B0%E5%A2%83% E3% 82% 92% E6% A7% 8B% E7% AF% 89% E3% 81% 97% E3% 81% A6% E3% 81% BF% E3% 82% 8B /) scopes param for service_account of google_compute_instance should be optional

Recommended Posts

Create a GCE instance from a GCR Docker image using terraform
Generate a Docker image using Fabric
Create a dataframe from excel using pandas
Create a phylogenetic tree from Biopyton using ClustalW2
[AWS Lambda] Create a deployment package using the Docker image of Amazon Linux
Try using Jupyter's Docker image
How to create an instance of a particular class from dict using __new__ () in python
Create a gadget-like transparent background image type window using wxpython
Create an instance of a predefined class from a string in Python
Create an API that returns data from a model using turicreate
Do a search by image from the camera roll using Pythonista3
Create a Docker container image with JRE8 / JDK8 on Amazon Linux
Make a Santa classifier from a Santa image
Create a python GUI using tkinter
Build a go environment using Docker
Create a pandas Dataframe from a string.
Create a nested dictionary using defaultdict
Create a CRUD API using FastAPI
Create a C wrapper using Boost.Python
Create multiple line charts from a data frame at once using Matplotlib
Create a tool to automatically furigana with html using Mecab from Python3
Create a stack with a queue and a queue with a stack (from LetCode / Implement Stack using Queues, Implement Queue using Stacks)
How to create a clone from Github
Create a dummy image with Python + PIL.
Create a graph using the Sympy module
[Python] Create a Batch environment using AWS-CDK
Create a standings from JFL match results
How to create a repository from media
Video acquisition / image shooting from a webcam
[Golang] Create docker image with Github Actions
Create a web service with Docker + Flask
I made a Docker image that can call FBX SDK Python from Node.js
Create a setting in terraform to send a message from AWS Lambda Python3.8 to Slack