This series is written as a summary of learning for PyTorch, Python's machine learning framework. This time, I wrote a summary of the basics of PyTorch. It may be difficult to read as a reading material because I try to write the main points rather than to convey it, but I hope that you can use it for understanding the points.
As mentioned earlier, PyTorch is Python's machine learning framework. Speaking of machine learning frameworks, there are TensorFlow developed by Google, Keras or chainer. Personal subjectivity is included, but if you roughly divide these, it seems like the following.
| Framework | Major features |
|---|---|
| TensorFlow, Keras | For industry |
| PyTorch, Chainer | For research (highly customizable) |
Also, I think most people will come from TensorFlow when they want to start machine learning, but a slightly unusual rule that defines and executes the data structure `define and run``` separately. Because of this, I think that some people may not be familiar with it and fade out. On the other hand, PyTorch and Chainer use `define by run```, so you can code it in the same way as the Python code you normally code, so it's familiar (although I like it). Is a feature.
However, the mechanism of `` `define and run```, which is a little difficult to get along with, seems to have merits such as easy optimization and faster calculation speed by the amount defined earlier. There are advantages and disadvantages.
Now let's take a look at the contents of PyTorch.
PyTorch has the following structure.
| Contents | Description |
|---|---|
| torch | This package contains Tensors and various math functions in the main namespace.It imitates the structure of NumPy |
| torch.autograd | Contains functions for automatic differentiation. Automatic differentiation on/Enable context manager to control off_grad/no_Includes grad and Function, which is a base class used when defining a unique differentiable function. |
| torch.nn | Various data structures and layers for building neural networks are defined. For example, activation functions such as Convolution, LSTM, and ReLU and loss functions such as MSELoss are also included. |
| torch.optim | A parameter optimization algorithm centered on Stochastic Gradient Descent (SGD) is implemented |
| torch.utils.data | Contains utility functions for creating mini-batch when turning SGD iterative calculations |
| torch.onnx | ONNX(Open Neural Network Exchange)[^1]Used to export the model in the format of. |
[^ 1]: Format for sharing models between various deep learning frameworks
There are several ways to generate a tensor.
Generation of tensor
import numpy as np
import torch
#Created by passing a nested list
t = torch.tensor([[1, 2], [3, 4]])
#Create Tensor on GPU by specifying device
t = torch.tensor([[1, 2], [3, 4]], device = "cuda:0")
#Create a double-precision tensor by specifying dtype
t = torch.tensor([[1, 2], [3, 4]], dtype = "torch.float64")
# 0~One-dimensional Tensor initialized with a number of 9
t = torch.arange(0, 10)
#Create a 100x100 Tensor with all values 0 and transfer it to the GPU with the to method
t = torch.zeros(100,100).to("cuda:0")
#100x100 random tensor
t = torch.random(100, 100)
Tensor can be easily converted to NumPy's ndarray. However, the Tensor on the GPU cannot be converted as it is, and must be moved to the CPU once.
tensor conversion
#Convert to ndarray using numpy method
t = torch.tensor([[1, 2], [3, 4]])
nd_arr = t.numpy()
#Move Tensor on GPU to CPU
t = torch.tensor([[1, 2], [3, 4]], device = "cuda: 0")
t_cpu = t.to("cpu")
nd_arr = t_cpu.numpy()
#Continue to fill in the above
t = torch.tensor([[1, 2], [3, 4]], device = "cuda: 0")
nd_arr = t.to("cpu").numpy()
Like ndarray, Tensor also supports indexing operation [^ 2]. The methods that can be specified are as follows.
[^ 2]: To get or change the value of an array by specifying a subscript
[^ 3]: An array of the same size as the original array, with each element in the array set to True / False.
Tensor indexing operation
t = torch.tensor([[1, 2, 3], [4, 5, 6]])
#Specified by scalar subscript
t[0, 2] # tensor(3)
#Specified by slice
t[:, :2] # tensor([[1, 2], [4, 5]])
#Specified in a list of subscripts
t[:, [1, 2]] # tensor([[2, 3], [5, 6]])
#Use mask array to select only parts larger than 3
t[t > 3] # tensor([4, 5, 6])
# [0, 1]Replace element with 100
t[0, 1] = 100 # tensor([[1, 100, 3], [4, 5, 6]])
#Bulk assignment using slices
t[:, 1] = 200 # tensor([[1, 200, 3], [4, 200, 6]])
#Replace only elements with specific conditions using a mask array
t[t > 10] = 20 # tensor([[1, 2, 3], [4, 5, 6]])
Tensor can perform four arithmetic operations, mathematical functions, linear algebra calculations, and more. Although it can be done with ndarray, linear algebra calculations such as matrix multiplication and singular value decomposition [^ 4] have better performance than using NumPy / SciPy especially for large-scale data because GPU can be used. There are many.
[^ 4]: Singular Value Decomposition (SVD) is a calculation often used in linear algebra, which decomposes matrix A into the product of three matrices like USV.
U and V are orthogonal matrices, and S is a square matrix with only diagonal components, which are used when solving the least squares method and for approximating and compressing matrices.
The availability of Tensor's four arithmetic operations is as follows.
| Tensor | Tensor(Same type) | 〇 |
| Tensor | Tensor(Different type) | ✕ |
| Tensor | Python scalar | 〇 |
| Tensor | ndarray | ✕ |
It should be noted that the four arithmetic operations between tensors must have the same type. When performing four-rule operations, the dimensions are automatically complemented by vector-scalar operations and matrix-vector operations (Broadcast. user / basics.broadcasting.html)).
Tensor's four arithmetic operations
#vector
v = torch.tensor([1, 2, 3])
w = torch.tensor([4, 5, 6])
#queue
m = torch.tensor([[0, 1, 2], [10, 20, 30]])
n = torch.tensor([[3, 4, 5], [40, 50, 60]])
#vector-scalar
v_pl = v + 10 # tensor([11, 12, 13])
v_mi = v - 10 # tensor([-9, -8, -7])
v_mu = v * 10 # tensor([10, 20, 30])
v_di = v / 10 # tensor([0, 0, 0])
#vector-vector
v_pl = v + w # tensor([5, 7, 9])
v_mi = v - w # tensor([-3, -3, -3])
v_mu = v * w # tensor([4, 10, 18])
v_di = v / w # tensor([0, 0, 0])
#Matrix and vector
v_pl = m + v # tensor([[ 1, 3, 5], [11, 22, 33]])
v_mi = m - v # tensor([[-1, -1, -1], [ 9, 18, 27]])
v_mu = m * v # tensor([[ 0, 2, 6], [10, 40, 90]])
v_di = m / v # tensor([[ 0, 0, 0], [10, 10, 10]])
#Matrix and matrix
v_pl = m + n # tensor([[3, 5, 7], [50, 70, 90]])
v_mi = m - n # tensor([[-3, -3, -3], [-30, -30, -30]])
v_mu = m * n # tensor([[0, 4, 10], [400, 1000, 1800]])
v_di = m / n # tensor([[0, 0, 0], [0, 0, 0]])
There is a place where the output becomes 0 when dividing. As you can see, even if the result of the operation shows a value after the decimal point, the type of tensor is int. If you want to avoid it, specify float etc. in advance.
PyTorch provides various mathematical functions for Tensor.
** Mathematical functions that affect each value of Tensor **
| Function name | Overview |
|---|---|
| abs | Absolute value |
| sin | sine |
| cos | cosine |
| exp | Exponential |
| log | Logarithmic function |
| sqrt | square root |
** Aggregate function **
| function | Overview |
|---|---|
| sum | Sum of values in tensor |
| max | Maximum value in tensor |
| min | Minimum value in tensor |
| mean | Mean of values in tensor |
| std | Standard deviation of values in tensor |
| function | Overview |
|---|---|
| dot | Inner product of vectors |
| mv | Product of matrix and vector |
| mm | Matrix-matrix product |
| matmul | Automatically select dot, mv, mm depending on the type of argument and execute |
| gesv | Solving simultaneous equations by LU decomposition |
| eig, symeig | Eigenvalue decomposition. symeig is for the target matrix |
| svd | Singular value decomposition |
Let's see what the result will be (I haven't shown it because the result is quite long).
Linear algebra arithmetic
m = torch.randn(100, 10) #Create 100x10 matrix test data
v = torch.randn(10)
#inner product
torch.dot(v, v)
#Product of matrix and vector
torch.mv(m, v)
#Matrix product
torch.mm(m.t(), m)
#Singular value decomposition
u, s, v = torch.svd(m)
| Function name | Overview |
|---|---|
| view | Change the dimension of the tensor |
| cat, stack | Join Tensors together |
| transpose | Swap dimensions |
If you set `requires_grad``` of Tensor to True```, the flag for automatic differentiation will be enabled. This flag is enabled for all parameters and data when dealing with neural networks. By stacking various operations on a Tensor with `` requires_grad``` enabled, a calculation graph is constructed, and by calling the backward method, it is automatically differentiated from that information.
This time, I mainly summarized Tensor, which is indispensable for handling PyTorch configuration and machine learning. Thank you to everyone who read to the end. From the next time onward, we will build neural networks little by little, so if you are interested, please take a look.
Morisebashi (2018) "Can be used in the field! Introduction to PyTorch Development Creating a Deep Learning Model and Implementing it in an Application ”
Recommended Posts