PyTorch Super Introduction PyTorch Basics

Introduction

This series is written as a summary of learning for PyTorch, Python's machine learning framework. This time, I wrote a summary of the basics of PyTorch. It may be difficult to read as a reading material because I try to write the main points rather than to convey it, but I hope that you can use it for understanding the points.

What is PyTorch?

As mentioned earlier, PyTorch is Python's machine learning framework. Speaking of machine learning frameworks, there are TensorFlow developed by Google, Keras or chainer. Personal subjectivity is included, but if you roughly divide these, it seems like the following.

Framework Major features
TensorFlow, Keras For industry
PyTorch, Chainer For research (highly customizable)

Also, I think most people will come from TensorFlow when they want to start machine learning, but a slightly unusual rule that defines and executes the data structure `define and run``` separately. Because of this, I think that some people may not be familiar with it and fade out. On the other hand, PyTorch and Chainer use `define by run```, so you can code it in the same way as the Python code you normally code, so it's familiar (although I like it). Is a feature.

However, the mechanism of `` `define and run```, which is a little difficult to get along with, seems to have merits such as easy optimization and faster calculation speed by the amount defined earlier. There are advantages and disadvantages.

Now let's take a look at the contents of PyTorch.

PyTorch configuration

PyTorch has the following structure.

Contents Description
torch This package contains Tensors and various math functions in the main namespace.It imitates the structure of NumPy
torch.autograd Contains functions for automatic differentiation. Automatic differentiation on/Enable context manager to control off_grad/no_Includes grad and Function, which is a base class used when defining a unique differentiable function.
torch.nn Various data structures and layers for building neural networks are defined. For example, activation functions such as Convolution, LSTM, and ReLU and loss functions such as MSELoss are also included.
torch.optim A parameter optimization algorithm centered on Stochastic Gradient Descent (SGD) is implemented
torch.utils.data Contains utility functions for creating mini-batch when turning SGD iterative calculations
torch.onnx ONNX(Open Neural Network Exchange)[^1]Used to export the model in the format of.

[^ 1]: Format for sharing models between various deep learning frameworks

Tensor generation and conversion

Generation of tensor

There are several ways to generate a tensor.

Generation of tensor


import numpy as np
import torch

#Created by passing a nested list
t = torch.tensor([[1, 2], [3, 4]])

#Create Tensor on GPU by specifying device
t = torch.tensor([[1, 2], [3, 4]], device = "cuda:0")

#Create a double-precision tensor by specifying dtype
t = torch.tensor([[1, 2], [3, 4]], dtype = "torch.float64")

# 0~One-dimensional Tensor initialized with a number of 9
t = torch.arange(0, 10)

#Create a 100x100 Tensor with all values 0 and transfer it to the GPU with the to method
t = torch.zeros(100,100).to("cuda:0")

#100x100 random tensor
t = torch.random(100, 100)

How to convert Tensor

Tensor can be easily converted to NumPy's ndarray. However, the Tensor on the GPU cannot be converted as it is, and must be moved to the CPU once.

tensor conversion


#Convert to ndarray using numpy method
t = torch.tensor([[1, 2], [3, 4]])
nd_arr = t.numpy()

#Move Tensor on GPU to CPU
t = torch.tensor([[1, 2], [3, 4]], device = "cuda: 0")
t_cpu = t.to("cpu")
nd_arr = t_cpu.numpy()

#Continue to fill in the above
t = torch.tensor([[1, 2], [3, 4]], device = "cuda: 0")
nd_arr = t.to("cpu").numpy()

Tensor indexing operation

Like ndarray, Tensor also supports indexing operation [^ 2]. The methods that can be specified are as follows.

[^ 2]: To get or change the value of an array by specifying a subscript

[^ 3]: An array of the same size as the original array, with each element in the array set to True / False.

Tensor indexing operation



t = torch.tensor([[1, 2, 3], [4, 5, 6]])

#Specified by scalar subscript
t[0, 2] # tensor(3)

#Specified by slice
t[:, :2] # tensor([[1, 2], [4, 5]])

#Specified in a list of subscripts
t[:, [1, 2]]  # tensor([[2, 3], [5, 6]])

#Use mask array to select only parts larger than 3
t[t > 3] # tensor([4, 5, 6])

# [0, 1]Replace element with 100
t[0, 1] = 100  # tensor([[1, 100, 3], [4, 5, 6]])

#Bulk assignment using slices
t[:, 1] = 200  # tensor([[1, 200, 3], [4, 200, 6]])

#Replace only elements with specific conditions using a mask array
t[t > 10] = 20  # tensor([[1, 2, 3], [4, 5, 6]])

Tensor arithmetic

Tensor can perform four arithmetic operations, mathematical functions, linear algebra calculations, and more. Although it can be done with ndarray, linear algebra calculations such as matrix multiplication and singular value decomposition [^ 4] have better performance than using NumPy / SciPy especially for large-scale data because GPU can be used. There are many.

[^ 4]: Singular Value Decomposition (SVD) is a calculation often used in linear algebra, which decomposes matrix A into the product of three matrices like USV.
U and V are orthogonal matrices, and S is a square matrix with only diagonal components, which are used when solving the least squares method and for approximating and compressing matrices.

Tensor's four arithmetic operations

The availability of Tensor's four arithmetic operations is as follows.

Tensor Tensor(Same type)
Tensor Tensor(Different type)
Tensor Python scalar
Tensor ndarray

It should be noted that the four arithmetic operations between tensors must have the same type. When performing four-rule operations, the dimensions are automatically complemented by vector-scalar operations and matrix-vector operations (Broadcast. user / basics.broadcasting.html)).

Tensor's four arithmetic operations


#vector
v = torch.tensor([1, 2, 3])
w = torch.tensor([4, 5, 6])

#queue
m = torch.tensor([[0, 1, 2], [10, 20, 30]])
n = torch.tensor([[3, 4, 5], [40, 50, 60]])

#vector-scalar
v_pl = v + 10  # tensor([11, 12, 13])
v_mi = v - 10  # tensor([-9, -8, -7])
v_mu = v * 10  # tensor([10, 20, 30])
v_di = v / 10  # tensor([0, 0, 0])

#vector-vector
v_pl = v + w  # tensor([5, 7, 9])
v_mi = v - w  # tensor([-3, -3, -3])
v_mu = v * w  # tensor([4, 10, 18])
v_di = v / w  # tensor([0, 0, 0])

#Matrix and vector
v_pl = m + v  # tensor([[ 1,  3,  5], [11, 22, 33]])
v_mi = m - v  # tensor([[-1, -1, -1], [ 9, 18, 27]])
v_mu = m * v  # tensor([[ 0,  2,  6], [10, 40, 90]])
v_di = m / v  # tensor([[ 0,  0,  0], [10, 10, 10]])

#Matrix and matrix
v_pl = m + n  # tensor([[3, 5, 7], [50, 70, 90]])
v_mi = m - n  # tensor([[-3, -3, -3], [-30, -30, -30]])
v_mu = m * n  # tensor([[0, 4, 10], [400, 1000, 1800]])
v_di = m / n  # tensor([[0, 0, 0], [0, 0, 0]])


There is a place where the output becomes 0 when dividing. As you can see, even if the result of the operation shows a value after the decimal point, the type of tensor is int. If you want to avoid it, specify float etc. in advance.

Mathematical functions available in PyTorch

PyTorch provides various mathematical functions for Tensor.

** Mathematical functions that affect each value of Tensor **

Function name Overview
abs Absolute value
sin sine
cos cosine
exp Exponential
log Logarithmic function
sqrt square root

** Aggregate function **

function Overview
sum Sum of values in tensor
max Maximum value in tensor
min Minimum value in tensor
mean Mean of values in tensor
std Standard deviation of values in tensor

Linear algebra operators

function Overview
dot Inner product of vectors
mv Product of matrix and vector
mm Matrix-matrix product
matmul Automatically select dot, mv, mm depending on the type of argument and execute
gesv Solving simultaneous equations by LU decomposition
eig, symeig Eigenvalue decomposition. symeig is for the target matrix
svd Singular value decomposition

Let's see what the result will be (I haven't shown it because the result is quite long).

Linear algebra arithmetic


m = torch.randn(100, 10)  #Create 100x10 matrix test data
v = torch.randn(10)

#inner product
torch.dot(v, v)

#Product of matrix and vector
torch.mv(m, v)

#Matrix product
torch.mm(m.t(), m)

#Singular value decomposition
u, s, v = torch.svd(m)

Other frequently used functions

Function name Overview
view Change the dimension of the tensor
cat, stack Join Tensors together
transpose Swap dimensions

Automatic differentiation

If you set `requires_grad``` of Tensor to True```, the flag for automatic differentiation will be enabled. This flag is enabled for all parameters and data when dealing with neural networks. By stacking various operations on a Tensor with `` requires_grad``` enabled, a calculation graph is constructed, and by calling the backward method, it is automatically differentiated from that information.

in conclusion

This time, I mainly summarized Tensor, which is indispensable for handling PyTorch configuration and machine learning. Thank you to everyone who read to the end. From the next time onward, we will build neural networks little by little, so if you are interested, please take a look.

reference

Morisebashi (2018) "Can be used in the field! Introduction to PyTorch Development Creating a Deep Learning Model and Implementing it in an Application

Recommended Posts

PyTorch Super Introduction PyTorch Basics
pytorch super introduction
[Super Introduction to Machine Learning] Learn Pytorch tutorials
Introduction to Lightning pytorch
bottle super elementary introduction
[Super Introduction to Machine Learning] Learn Pytorch tutorials
PyTorch introduction (virtual environment)
Introduction to PyTorch (1) Automatic differentiation
A super introduction to Linux
[Super Basics] About jupyter Notebook
Super introduction to machine learning
Introduction
[Details (?)] Introduction to pytorch ~ CNN CIFAR10 ~
[PyTorch] Introduction to document classification using BERT
[Python] Introduction to CNN with Pytorch MNIST
[Introduction to Pytorch] I played with sinGAN ♬
Basics of PyTorch (1) -How to use Tensor-
[Introduction to Data Scientists] Basics of Python ♬