Notes on optimization using Pytorch

Since the function-based tensorflow2 is difficult to use, we are migrating to Pytorch. This article is for nostalgia who wants to create their own loss functions and optimization methods, not modern children who use packages as they are. Fossils that say "gradients are calculated by hand" are not the target.

Data generation

Properly generated

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch.autograd import Variable
import torch.optim as optim

N = 100
a_data =  - 1
b_data = 1
sigma_data = 0.1
x_data = 2 * np.random.rand(N)
y_data = a_data * x_data + b_data + sigma_data * np.random.randn(N)

When using an optimized package

For twisted people who want to create their own Loss function.

#Variable definition
a = Variable(torch.randn(1), requires_grad=True)
b = Variable(torch.randn(1), requires_grad=True)

#Data conversion
x_train = torch.tensor(x_data)
y_train = torch.tensor(y_data)

#Optimizing function settings
optimizer = optim.Adam([a, b])

#Number of repetitions
epoch = 4000

loss_epoch = np.zeros(epoch)

for i in range(epoch):
    #Gradient initialization used in optimizer
    optimizer.zero_grad()
    #Linear model
    y_hat = a * x_train + b
    #Calculation of loss function
    loss = (y_train - y_hat).norm()
    loss_epoch[i] = loss.item()
    #Gradient setting
    loss.backward()
    #Perform optimization
    optimizer.step()

When using only the gradient

For those who don't want to use Optimizer and have more twists. Optimized using the gradient method.

#Parameter preparation
a = torch.randn(1,requires_grad=True)
b = torch.randn(1,requires_grad=True)

#Data conversion
x_train = torch.tensor(x_data)
y_train = torch.tensor(y_data)

#Learning rate
eta = 0.001

#Number of repetitions
epoch = 4000
loss_epoch = np.zeros(epoch)
for i in range(epoch):
    #Gradient recording start
    a.requires_grad_(True)
    b.requires_grad_(True)
    #Prediction and calculation of loss function
    y_hat = a * x_train + b
    loss = (y_train - y_hat).norm()
    loss_epoch[i] = loss.item()
    #Gradient setting
    loss.backward()
    #Gradient recording stop
    a.requires_grad_(False)
    b.requires_grad_(False)

    #Update with gradient
    a = a - eta * a.grad
    b = b - eta * b.grad

Summary

It seems easy to use once you get used to the slope recording and stopping parts.

Code details

https://github.com/yuji0001/2020Introduction_of_Pytorch

Reference Pytorch tutorials (here).

Author Yuji Okamoto [email protected]