Sine wave prediction (regression) with Pytorch

Overview

Hello. I am a new employee of a certain IT company. Here, using Pytorch, an open source machine learning library, I described a regression model learning program for sine wave prediction as a sample. (It's not a time series forecast ... I'd like to do a time series forecast soon.)

By moving chainer to pytorch, It was late, but I wrote it to catch up this time. It also describes the points that made me feel different.

It's like the pytorch migration of the following article I wrote earlier. [Now, let's carefully learn the sine wave with chainer] 1

If you have any suggestions or questions, please do not hesitate to leave them in the comments section.

Whole code

The code is on GitHub below. kazu-ojisan/NN-pytorch_PredictSinWave

environment

macOS Catalina 10.15.3 conda 4.7.12 python 3.7.6 (Create a virtual environment with conda) pytorch 1.4.0

Installation of pytorch is killed in seconds. If you select your own environment on the official website below, It will display the installation command. [Pytorch -Official Site-] 3

Each parameter

• Input ：0〜2π
• Output ：sin⁡(Input) --Number of learning: 200 --Batch size: 10 (mini batch method) --Number of training data: 1000 --Number of test data: 200

Model structure

--Intermediate layer: 2 layers (number of units: 10) --Activation function: ReLU

Implementation

Below are the URLs that I referred to for implementation. -[Practice Pytorch] 4 -[Solving regression problems with a neural network implemented in PyTroch] 5

Module used

The pytorch module is OK if at least the following is imported.

test.py

import numpy as np      #Array
import time             #time
from matplotlib import pyplot as plt #Graph
import os               #To create a folder

# pytorch
import torch as T
import torch.nn as nn           #layer configuration
import torch.nn.functional as F #Activation function
from torch import optim         #Optimization function

data set

A simple y = sin (x) dataset (x, y).

python

# y=sin(x)Create N datasets of
def get_data(N, Nte):
    x = np.linspace(0, 2 * np.pi, N+Nte)
    #Divide into training data and test data
    ram = np.random.permutation(N+Nte)
    x_train = np.sort(x[ram[:N]])
    x_test = np.sort(x[ram[N:]])

    t_train = np.sin(x_train)
    t_test = np.sin(x_test)

    return x_train, t_train, x_test, t_test

Neural Network structure, forward propagation, etc.

The only difference from chainer is that it replaces the module without any particular difference. The data type handled by pytorch is "Tensor type", so conversion is necessary. (Valiable type in chainer)

.py

class SIN_NN(nn.Module):
    def __init__(self, h_units, act):
        super(SIN_NN, self).__init__()
        self.l1=nn.Linear(1, h_units[0])
        self.l2=nn.Linear(h_units[0], h_units[1])
        self.l3=nn.Linear(h_units[1], 1)

        if act == "relu":
            self.act = F.relu
        elif act == "sig":
            self.act = F.sigmoid

    def __call__(self, x, t):
        x = T.from_numpy(x.astype(np.float32).reshape(x.shape[0],1))
        t = T.from_numpy(t.astype(np.float32).reshape(t.shape[0],1))
        y = self.forward(x)
        return y, t

    def forward(self, x):
        h = self.act(self.l1(x))
        h = self.act(self.l2(h))
        h = self.l3(h)

        return h

Learning

I felt the difference from Chainer (more details later) (1) model.parameter () is required as the first argument of the optimization function (2) Since MSE is defined in Class, it cannot be used unless an instance is created. ③ Switching between learning mode and test mode by "model.train ()" and "model.eval ()" (It doesn't matter if you don't have it this time.) ④ The extension of model is ".pt" or ".pth"

.py

def training(N, Nte, bs, n_epoch, h_units, act):

    #Get dataset
    x_train, t_train, x_test, t_test = get_data(N, Nte)
    x_test_torch = T.from_numpy(x_test.astype(np.float32).reshape(x_test.shape[0],1))
    t_test_torch = T.from_numpy(t_test.astype(np.float32).reshape(t_test.shape[0],1))

    #Model setup
    model = SIN_NN(h_units, act)
    optimizer = optim.Adam(model.parameters())
    MSE = nn.MSELoss()

    #loss storage array
    tr_loss = []
    te_loss = []

    #Create directory
    if os.path.exists("Results/{}/Pred".format(act)) == False:
        os.makedirs("Results/{}/Pred".format(act))

    #Measure time
    start_time = time.time()
    print("START")

    #Loop for the number of learnings
    for epoch in range(1, n_epoch + 1):
        model.train()
        perm = np.random.permutation(N)
        sum_loss = 0
        for i in range(0, N, bs):
            x_batch = x_train[perm[i:i + bs]]
            t_batch = t_train[perm[i:i + bs]]

            optimizer.zero_grad()
            y_batch, t_batch = model(x_batch, t_batch)
            loss = MSE(y_batch, t_batch)
            loss.backward()
            optimizer.step()
            sum_loss += loss.data * bs

        #Calculate the average of learning errors
        ave_loss = sum_loss / N
        tr_loss.append(ave_loss)

        #Test error
        model.eval()
        y_test_torch = model.forward(x_test_torch)
        loss = MSE(y_test_torch, t_test_torch)
        te_loss.append(loss.data)

    #Save trained model
    T.save(model, "Results/model.pt")

Details: What I felt was different from Chainer

(1) model.parameters () is required as the first argument of the optimization function

model.parameters () stores weights and the number of units, which are model information. Official documentation: [model.parameters ()] 9 If you try to output according to the document, it will be output as follows.

$ python exportModelParam.py 
<class 'torch.Tensor'> torch.Size([10, 1])
<class 'torch.Tensor'> torch.Size([10])
<class 'torch.Tensor'> torch.Size([10, 10])
<class 'torch.Tensor'> torch.Size([10])
<class 'torch.Tensor'> torch.Size([1, 10])
<class 'torch.Tensor'> torch.Size([1])

(2) Since MSE is defined in Class, it cannot be used unless an instance is created.

The official documentation for Pytorch's "MSE Loss" and Chainer's "mean_squared_error" makes sense. Pytorch:Pytorch -SOURCE CODE FOR TORCH.NN.MODULES.LOSS-

MSELoss.py

class MSELoss(_Loss):

Chainer:Chainer -mean_squared_error.py-

mean_squared_error.py

def mean_squared_error(x0, x1):

③ Switching between learning mode and test mode by "model.train ()" and "model.eval ()"

If you put it in test mode with .eval, Dropout and Batch Normalization will be disabled and it will be a test specification. This code doesn't use Dropout or Batch Normalization, so it works the same without it. However, if you plan to use pytorch in the future, I think it's better to add a habit of writing.

④ The extension of model is ".pt" or ".pth"

See below. Tutorials > Saving and Loading Models

result

Error graph

Test data prediction graph epoch:20 epoch:200

Trouble Shooting --Are you not learning? --I forgot to initialize the gradient (optimizer.zero_grad ()) --You cannot learn unless you initialize the gradient for each mini-batch.

Reference URL

-[Pytorch -Official Site-] 3 -[Practice Pytorch] 4 -[Solving regression problems with a neural network implemented in PyTroch] 5 -[Now, let's carefully learn the sine wave with chainer] 1

Whole code

The code is on GitHub below. kazu-ojisan/NN-pytorch_PredictSinWave

At the end

As you can see, it was pretty similar to chainer. I felt that Pytorch was well-equipped. If you have any suggestions, please do not hesitate to leave them in the comments section as they are still lacking in understanding.