EC2 provisioning with Vagrant + Jupyter (IPython Notebook) on Docker

Overview

(First half) EC2 provisioning using Vagrant (Second half) Start Jupyter (IPython Notebook) using Docker

I will write to Qiita for the first time. During the New Year holidays, I tried Deep learning from scratch and, hopefully, study TensorFlow, and study environment construction as well. , It is an introduction that also serves as a memorandum. Since the first and second half of this time are independent, I think it is possible to refer to only one of them.

:)

:(

Construction procedure

As a premise

Is required.

I think this will be helpful.

(First half) EC2 provisioning using Vagrant

Refer to HashiCorp's GitHub page: Vagrant AWS Provider to configure the settings.

First, install the plug-in to use AWS.

$ vagrant plugin install vagrant-aws

It is a BOX equivalent to a machine image, but since a dummy box is open to the public for EC2, I will use this. (Officially, the box name is dummy, but it's a bit vague, so I changed it to ec2.)

$ vagrant box add ec2 https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

Next, move to the working directory and create a Vagrantfile.

$ mkdir jupyter_ec2
$ cd jupyter_ec2
$ vim Vagrantfile

Edit the Vagrantfile and set various parameters. This time, we have specified booting with a VPC created in advance.

There seem to be quite a few other parameters, and if you want to specify them, you can use the README or AWS API documentation. I think it is better to refer to).

Vagrantfile


Vagrant.configure("2") do |config|
  config.vm.box = "ec2" # dummy box name

  config.vm.provider :aws do |aws, override|
    #AWS Credentials
    aws.aws_dir = ENV['HOME'] + "/.aws/"
    aws.aws_profile = "YOUR_PROFILE"
    aws.keypair_name = "YOUR_AWS_ACCESS_KEY"
    
    #SSH credentials
    override.ssh.username = "ec2-user"
    override.ssh.private_key_path = "SSH_KEY_PATH"

    #VPC settings
    aws.region = 'ap-northeast-1'
    aws.availability_zone = 'ap-northeast-1a'
    aws.subnet_id = 'subnet-XXXXXXXX'
    aws.security_groups = 'sg-XXXXXXXX'

    #EC2 settings
    aws.ami = "ami-9f0c67f8" # Amazon Linux 2016.09.01
    aws.instance_type = 'm4.large'
    aws.elastic_ip = true
    aws.associate_public_ip = true
    aws.tags = { 'Name' => 'NAME_TAGS_VALUE' }

    #EBS settings
    aws.block_device_mapping = [
      {
        'DeviceName' => "/dev/xvda",
        'Ebs.VolumeSize' => 50,
        'Ebs.DeleteOnTermination' => true,
        'Ebs.VolumeType' => 'standard',
      }
    ]
  end
end

After editing, start EC2.

$ vagrant up --provider=aws

When it doesn't work,

$ export VAGRANT_LOG=DEBUG

Then re-execute vagrant up and it will display a detailed error message.

If you want to undo the error message ʻexport VAGRANT_LOG = WARN `

Make sure that the security group SSH is also open properly. If the upper limit is not relaxed, the upper limit of the number of EIPs tends to be caught.

When it finishes starting

$ vagrant ssh

Check if you can SSH login with.

$ vagrant halt

You can stop the instance with, so when you are not working, stop it as appropriate to save money.

(Second half) Start Jupyter (IPython Notebook) using Docker

First, install Docker on EC2 by referring to Installing Docker.

$ sudo yum update -y
$ sudo yum install -y docker

Start the docker daemon so that it will start automatically even after the instance is restarted.

$ sudo service docker start
$ sudo chkconfig docker on

Add ec2-user to the docker group so that you can add docker commands without sudo privileges.

$ sudo usermod -a -G docker ec2-user

Once you exit and log in again, docker will start without sudo.

Next, describe the Dockerfile.

$ vim Dockerfile

This time, for the sake of simplicity, we will install TensorFlow based on the image of anaconda. In CMD, specify that jupyter should be started at docker run.

FROM continuumio/anaconda:latest

RUN mkdir /opt/notebooks
RUN pip install tensorflow
# Another installations comes here.

CMD ["/opt/conda/bin/jupyter", "notebook", "--notebook-dir=/opt/notebooks", "--ip='*'", "--port=8888", "--no-browser"]

After editing, create a container image from the Dockerfile.

$ docker build -t myjupyter .

Finally start the container. Specify volume and mount it so that the notebook can be made persistent and can be seen from the host side. Also, the --restart option causes the docker daemon to automatically start the container even after the instance is stopped / started.

$ mkdir notebooks
$ docker run -p 8888:8888 -v /home/ec2-user/notebooks:/opt/notebooks --restart=always myjupyter

After allowing TCP port 8888 communication in the security group, access [Elastic IP]: 8888 with a browser to confirm that Jupyter has started.


Only when studying with this, learn on the browser with vagrant up, stop charging with vagrant halt when quitting, build from the same Dockerfile when you want to try locally, easy porting if you get ipythonnb via Vagrant It became an environment ...! It should be, so I will correct it as appropriate while studying books.

Recommended Posts

EC2 provisioning with Vagrant + Jupyter (IPython Notebook) on Docker
Graph drawing with jupyter (ipython notebook) + matplotlib + vagrant
Play with Jupyter Notebook (IPython Notebook)
Run IPython Notebook on Docker
Formatting with autopep8 on Jupyter notebook
Use apache Spark with jupyter notebook (IPython notebook)
Use Jupyter Lab and Jupyter Notebook with EC2
Try SVM with scikit-learn on Jupyter Notebook
Rich cell output with Jupyter Notebook (IPython)
How to debug with Jupyter or iPython Notebook
Analytical environment construction with Docker (jupyter notebook + PostgreSQL)
Enable Jupyter Notebook with conda on remote server
Try Apache Spark on Jupyter Notebook (on local Docker
R environment construction with Jupyter (formerly IPython notebook) (on OS X El Capitan 10.11.3)
Virtual environment construction with Docker + Flask (Python) + Jupyter notebook
Monitor the training model with TensorBord on Jupyter Notebook
Set up Docker on Oracle Linux (7.x) with Vagrant
[Windows] [Python3] Install python3 and Jupyter Notebook (formerly ipython notebook) on Windows
Pepper-kun remote control environment construction with Docker + IPython Notebook
Parallel computing with iPython notebook
Using Graphviz with Jupyter Notebook
Use pip with Jupyter Notebook
Displaying strings on IPython Notebook
High charts on Jupyter notebook
Use Cython with Jupyter Notebook
Run Apache-Spark with IPython Notebook
Run Jupyter Notebook on windows
Graph drawing with IPython Notebook
Use Bokeh with IPython Notebook
Unable to display tensorboard in jupyter notebook on docker (solved)
Write charts in real time with Matplotlib on Jupyter notebook
Try clustering with a mixed Gaussian model on a Jupyter Notebook
Allow external connections with jupyter notebook
R & D life with iPython notebook
Visualize decision trees with jupyter notebook
Make a sound with Jupyter notebook
Resolve garbled Japanese characters in matplotlib of Jupyter Notebook on Docker
How to use jupyter notebook without polluting your environment with Docker
Run azure ML on jupyter notebook
Use markdown with jupyter notebook (with shortcut)
Add more kernels with Jupyter Notebook
Build IPython Notebook environment with boot2docker
Convenient analysis with Pandas + Jupyter notebook
Install Anaconda on Mac and upload Jupyter (IPython) notebook to Anaconda Cloud
Try running Jupyter Notebook on Mac
I wanted to use jupyter notebook with docker in pip environment (opticspy)
Use Jupyter Notebook with Visual Studio Code on Windows 10 + Python + Poetry + pyenv-win
How to set up a jupyter notebook on ssh destination (AWS EC2)
Settings when reading S3 files with pandas from Jupyter Notebook on AWS
Links for handling jupyter notebooks on Docker on Ubuntu on VirtualBox on macOS with SSH
Build Jupyter Lab (Python) environment with Docker
IPython Notebook is now rendered on GitHub!
I want to blog with Jupyter Notebook
Make Jupyter Notebook a service on CentOS
Start jupyter notebook on GPU server (remote server)
Clone the github repository on jupyter notebook
How to use jupyter notebook with ABCI
GPU check of PC on jupyter notebook
Display histogram / scatter plot on Jupyter Notebook
Linking python and JavaScript with jupyter notebook
"LIVE" HTML presentation with IPython 3.0.0-dev, IPython Notebook