"Deep Learning from scratch" self-study memo (No. 13) Try using Google Colaboratory

While reading "Deep Learning from scratch" (written by Yasuki Saito, published by O'Reilly Japan), I will make a note of the sites I referred to. Part 12 ← → Part 14

I've been learning by running Jupyter Lab on my computer, but memory errors started to occur frequently and I couldn't proceed.

so

I'm going to try Google's Colaboratory.

Colaboratory

There doesn't seem to be an English word for Colaboratory. That's the name Google gave to its service, Collaboration collaborate + laboratory laboratory Is it a coined word from? If it is co + laboratory, is it okay to read it in collaboration? Emphasize the meaning of collaboration If you think of collabora (te) + (labora) story, you can read it as collaboration, but l will increase by one.

Many posts on Qiita

Information about colaboratory on Qiita

So, I will go little by little.

After all, I'm worried about the structure of folders and files on the drive

So far, I've tried various things, and when I run a program, I refer to many libraries, input files, and pkl files. So, in order to create and reference them on google drive, you have to first understand the structure of folders and files and how to specify them.

Pass through

For the time being, it seems that there is no problem if you mount the drive and then pass the path to the folder where the libraries and files are located.

#Drive mount
from google.colab import drive
drive.mount('/content/drive')

It seems that you can name the drive part of / content / drive yourself. Does this command give you a name for using Google Drive in your scripts? So, under this drive, there is a folder My Drive as your own area, and under that you can create a folder for script execution, Notebook (~ .ipynb), etc.

This means that all resources available in Colaboratory are under / content / drivename / My Drive. However, since the current directory is / content, you need to be careful when specifying a file with a relative path. Must be'drive name / My Drive / filename'. Of course, you can also change the current drive with the os.chdir () instruction. Changing the current drive only works within that notebook and doesn't seem to affect other notebooks.

I created a folder Colab Notebooks to separate it from other files, and deep_learning under it as a folder to put the example programs of this book. I also created common and dataset folders that I often use in the programs in this book, and placed the downloaded programs and data in them.

Then, pass the path so that the programs under this folder can be referenced from anywhere.

#Add path
import sys
sys.path.append('/content/drive/My Drive/Colab Notebooks/deep_learning/common')
sys.path.append('/content/drive/My Drive/Colab Notebooks/deep_learning/dataset')

The program that refers to these is modified as follows.

#I used to do this when specifying modules in the folder dataset in JupyterLab
from dataset.mnist import load_mnist

#This is fine if you pass through the pass
from mnist import load_mnist

There are many programs that reference the common library, so it seems quite difficult to fix this.

I tried various related functions

import os
print(os.getcwd())  #Current directory
print(os.pardir) #Parent directory
print(os.path.dirname(os.path.abspath(__file__))) #Should return the directory where this script file is located, but in interactive mode it gives an error

/content .. --------------------------------------------------------------------------- NameError Traceback (most recent call last) in () 2 print (os.getcwd ()) #current directory 3 print(os.pardir) ----> 4 print(os.path.dirname(os.path.abspath(_file_))) 5 NameError: name '_file_' is not defined

The interactive mode of Google Colaboratory seems to be running in the current directory / content. Running the above script in a different folder will have the same result, the current directory will be / content. Also, it seems that \ __file \ __ cannot be used in interactive mode. Since the current directory is always / content, it doesn't make sense to use it.

However, it seems that \ __file \ __ can be used in the script saved with a file name. The following script works fine and loads a file in the same folder as the script.

# coding: utf-8
import sys, os
sys.path.append(os.pardir)  #Settings for importing files in the parent directory
import numpy as np
from mnist import load_mnist
import matplotlib.pyplot as plt

def showImg(x):
    example = x.reshape((28, 28))
    plt.figure()
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(example, cmap=plt.cm.binary)
    plt.show()
    return

(x_train, t_train), (x_test, t_test) = load_mnist(flatten=True, normalize=False)

img = x_train[1]
label = t_train[1]
print(img.shape)  # (784,)
img = img.reshape(28, 28)  #Transform the shape to the original image size
print(img.shape)  # (28, 28)
print(label)  # 0
showImg(img)

(784,) (28, 28) 0

In the imported mnist.py, the file is specified like this

mnist.Script setting mnist data directory with py


dataset_dir = os.path.dirname(os.path.abspath(__file__))
save_file = dataset_dir + "/mnist.pkl"

You can read mnist.pkl from the folder'/ content / drive / My Drive / Colab Notebooks / deep_learning / dataset' where mnist.py is located.

When specifying a file in interactive mode, it seems safer to specify an absolute path

# coding: utf-8
import matplotlib.pyplot as plt
from matplotlib.image import imread

img = imread('/content/drive/My Drive/Colab Notebooks/deep_learning/dataset/lena.png') #Loading images
plt.imshow(img)

plt.show()

It seems to work somehow, so after actually running the program in the book and checking it, I'm going to try to identify the dog and cat photos that were too heavy to move.

Part 12 ← → Part 14 Click here for the table of contents of the memo Unreadable Glossary

Referenced site

Import .py files on Google Drive with Google Colaboratory Find out why you can't get the directory name of a running script using file Welcome to Colaboratory Deep Learning Practice Tips on Colaboratory

Recommended Posts

"Deep Learning from scratch" self-study memo (No. 13) Try using Google Colaboratory
"Deep Learning from scratch" Self-study memo (No. 11) CNN
"Deep Learning from scratch" Self-study memo (No. 19) Data Augmentation
"Deep Learning from scratch 2" Self-study memo (No. 21) Chapters 3 and 4
"Deep Learning from scratch" Self-study memo (No. 14) Run the program in Chapter 4 on Google Colaboratory
"Deep Learning from scratch" self-study memo (No. 18) One! Meow! Grad-CAM!
"Deep Learning from scratch" Self-study memo (Part 12) Deep learning
"Deep Learning from scratch" self-study memo (No. 15) TensorFlow beginner tutorial
"Deep Learning from scratch" Self-study memo (No. 10-2) Initial value of weight
"Deep Learning from scratch" self-study memo (unreadable glossary)
"Deep Learning from scratch" Self-study memo (9) MultiLayerNet class
"Deep Learning from scratch" Self-study memo (10) MultiLayerNet class
"Deep Learning from scratch" Self-study memo (No. 16) I tried to build SimpleConvNet with Keras
"Deep Learning from scratch" Self-study memo (No. 17) I tried to build DeepConvNet with Keras
[Learning memo] Deep Learning made from scratch [Chapter 7]
Deep learning / Deep learning made from scratch Chapter 6 Memo
[Learning memo] Deep Learning made from scratch [Chapter 5]
[Learning memo] Deep Learning made from scratch [Chapter 6]
[Learning memo] Deep Learning made from scratch [~ Chapter 4]
A memo when executing the deep learning sample code created from scratch with Google Colaboratory
Deep Learning from scratch
Deep Learning from scratch Chapter 2 Perceptron (reading memo)
[Learning memo] Deep Learning from scratch ~ Implementation of Dropout ~
Deep Learning from scratch 1-3 chapters
Deep Learning / Deep Learning from Zero 2 Chapter 4 Memo
Deep Learning / Deep Learning from Zero Chapter 3 Memo
Deep learning from scratch (cost calculation)
Deep Learning / Deep Learning from Zero 2 Chapter 7 Memo
Deep Learning / Deep Learning from Zero 2 Chapter 8 Memo
Deep Learning / Deep Learning from Zero Chapter 5 Memo
Deep Learning / Deep Learning from Zero Chapter 4 Memo
Deep Learning / Deep Learning from Zero 2 Chapter 3 Memo
Deep Learning memos made from scratch
Deep Learning / Deep Learning from Zero 2 Chapter 6 Memo
"Deep Learning from scratch" Self-study memo (Part 8) I drew the graph in Chapter 6 with matplotlib
Why ModuleNotFoundError: No module named'dataset.mnist' appears in "Deep Learning from scratch".
Deep learning from scratch (forward propagation edition)
"Deep Learning from scratch" in Haskell (unfinished)
[Windows 10] "Deep Learning from scratch" environment construction
Learning record of reading "Deep Learning from scratch"
[Deep Learning from scratch] About hyperparameter optimization
Deep Learning from scratch ① Chapter 6 "Techniques related to learning"
Good book "Deep Learning from scratch" on GitHub
Python vs Ruby "Deep Learning from scratch" Summary
[Deep Learning from scratch] Initial value of neural network weight using sigmoid function
[Deep Learning from scratch] I implemented the Affine layer
Django memo # 1 from scratch
Application of Deep Learning 2 made from scratch Spam filter
[Deep Learning from scratch] I tried to explain Dropout
[Deep Learning from scratch] Initial value of neural network weight when using Relu function
[Deep Learning from scratch] Implementation of Momentum method and AdaGrad method
Try to build a deep learning / neural network with scratch
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 5
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 2
Create an environment for "Deep Learning from scratch" with Docker
An amateur stumbled in Deep Learning from scratch Note: Chapter 3
An amateur stumbled in Deep Learning from scratch Note: Chapter 7
An amateur stumbled in Deep Learning from scratch Note: Chapter 5
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 7
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 1
Making from scratch Deep Learning ❷ An amateur stumbled Note: Chapter 4