Notes on optimization using Pytorch

Since the function-based tensorflow2 is difficult to use, we are migrating to Pytorch. This article is for nostalgia who wants to create their own loss functions and optimization methods, not modern children who use packages as they are. Fossils that say "gradients are calculated by hand" are not the target.

Data generation

Properly generated

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch.autograd import Variable
import torch.optim as optim

N = 100
a_data =  - 1
b_data = 1
sigma_data = 0.1
x_data = 2 * np.random.rand(N)
y_data = a_data * x_data + b_data + sigma_data * np.random.randn(N)

sample.png

When using an optimized package

For twisted people who want to create their own Loss function.

#Variable definition
a = Variable(torch.randn(1), requires_grad=True)
b = Variable(torch.randn(1), requires_grad=True)

#Data conversion
x_train = torch.tensor(x_data)
y_train = torch.tensor(y_data)

#Optimizing function settings
optimizer = optim.Adam([a, b])

#Number of repetitions
epoch = 4000

loss_epoch = np.zeros(epoch)

for i in range(epoch):
    #Gradient initialization used in optimizer
    optimizer.zero_grad()
    #Linear model
    y_hat = a * x_train + b
    #Calculation of loss function
    loss = (y_train - y_hat).norm()
    loss_epoch[i] = loss.item()
    #Gradient setting
    loss.backward()
    #Perform optimization
    optimizer.step()

Adam_loss.png Adam.png

When using only the gradient

For those who don't want to use Optimizer and have more twists. Optimized using the gradient method.

#Parameter preparation
a = torch.randn(1,requires_grad=True)
b = torch.randn(1,requires_grad=True)

#Data conversion
x_train = torch.tensor(x_data)
y_train = torch.tensor(y_data)

#Learning rate
eta = 0.001

#Number of repetitions
epoch = 4000
loss_epoch = np.zeros(epoch)
for i in range(epoch):
    #Gradient recording start
    a.requires_grad_(True)
    b.requires_grad_(True)
    #Prediction and calculation of loss function
    y_hat = a * x_train + b
    loss = (y_train - y_hat).norm()
    loss_epoch[i] = loss.item()
    #Gradient setting
    loss.backward()
    #Gradient recording stop
    a.requires_grad_(False)
    b.requires_grad_(False)

    #Update with gradient
    a = a - eta * a.grad
    b = b - eta * b.grad

original_loss.png original.png

Summary

It seems easy to use once you get used to the slope recording and stopping parts.

Code details

https://github.com/yuji0001/2020Introduction_of_Pytorch

Reference Pytorch tutorials (here).

Author Yuji Okamoto [email protected]

Recommended Posts

Notes on optimization using Pytorch
Notes on using Alembic
[Django] Notes on using django-debug-toolbar
Notes on using MeCab from Python
Notes on using post-receive and post-merge
Notes on installing Python using PyEnv
Notes on using rstrip with python.
Notes on using matplotlib on the server
(Beginner) Notes on using pyenv on Mac
Notes for using OpenCV on Windows10 Python 3.8.3.
Notes on Flask
Notes on implementing APNs tests using Pytest
Notes on using OpenCL on Linux on the RX6800
Notes about pytorch
Notes on using code formatter in Python
Notes on using dict in python [Competition Pro]
Notes for using TensorFlow on Bash on Ubuntu on Windows
[Python] Notes on accelerating genetic algorithms using multiprocessing
Notes on neural networks
Celery notes on Django
Notes on installing PycURL
Notes using Python subprocesses
Notes on SciPy.linalg functions
Minimum notes when using Python on Mac (Homebrew edition)
Notes on tf.function and Tracing
Notes on installing dlib on mac
Notes on python's sqlite3 module
Try function optimization using Hyperopt
Notes on * args and ** kargs
Notes on defining PySide slots (2)
Try using OpenCV on Windows
Notes on pyenv and Atom
Notes on defining PySide slots
[Python] Notes on data analysis
Notes on installing Python on Mac
Notes on studying multidimensional scaling
How to know the number of GPUs from python ~ Notes on using multiprocessing with pytorch ~
Process on GPU using chainer.cuda.elementwise
Notes on installing pipenv on Mac
Broadcast on LINE using python
Notes on installing Anaconda 3 on Windows
Notes on imshow () in OpenCV
Notes on installing Python on CentOS
Small speedup when using pytorch
Solve multivariable optimization problems using sagemath
Notes on Python and dictionary types
Notes on package management with conda
Try using Pillow on iPython (Part 1)
Thorough commentary on pyTorch optim SGD
Introducing Python using pyenv on Ubuntu 20.04
Preparing python using vscode on ubuntu
[Golang] Notes on frequently used functions
Notes on how to use pywinauto
Install Pytorch on Blender 2.90 python on Windows
Notes on how to use featuretools
Try using ArUco on Raspberry Pi
Find golf expectations using dynamic optimization
Study on Tokyo Rent Using Python (3-2)
Try using Pillow on iPython (Part 3)
Using a serial console on Ubuntu 20.04
Notes on accessing dashDB from python