Write your own activation function with Pytorch (hard sigmoid)

Porting from Chainer to Pytorch

I'm doing deep learning in my research, but the other day I learned that the Chainer update was coming to an end, so I decided to change the framework to Pytorch as well as the developer. As a starting point, I decided to port from the existing Chainer program to Pytorch.

Basically, all I had to do was change the name of the function, but on the way I noticed that Pytorch doesn't have HardSigmoid. So let's make it ourselves.

I actually wrote it

... but it's written in the official reference, so I just did it almost exactly. --> https://pytorch.org/docs/master/autograd.html

python


class MyHardSigmoid(torch.autograd.Function):
    @staticmethod
     def forward(ctx, i):
         ctx.save_for_backward(i)
         result = (0.2 * i + 0.5).clamp(min=0.0, max=1.0)
         return result

     @staticmethod
     def backward(ctx, grad_output):
         grad_input = grad_output.clone()
         result, = ctx.saved_tensors
         grad_input *= 0.2
         grad_input[result < -2.5] = 0
         grad_input[result > -2.5] = 0
         return grad_input

If you do not write @staticmethod, a warning will appear. The official one is an exponential function, but we will change it to a hard sigmoid.

First, forward () is forward-propagated. hard sigmoid () has the following formula, so I wrote it so.

h(x) = \left\{
\begin{array}{ll}
0 & (x \lt -2.5) \\
0.2x + 0.5 & (-2.5 \leq x \leq 2.5)  \\
1 & (2.5 \lt x)
\end{array}
\right.

Then with backward (), this writes backpropagation. The differential coefficient is as follows.

\frac{\partial h(x)}{\partial x} = \left\{
\begin{array}{ll}
0 & (x \lt -2.5) \\
0.2 & (-2.5 \leq x \leq 2.5)  \\
0 & (2.5 \lt x)
\end{array}
\right.

And finally apply this to the model. (The contents of the model are appropriate.)

model.py


import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        hard_sigmoid = MyHardSigmoid.apply
        return hard_sigmoid(self.conv2(x))

This is perfect! !! ... should be

Recommended Posts

Write your own activation function with Pytorch (hard sigmoid)
Define your own distance function with k-means of scikit-learn
Solve your own maze with Q-learning
Train UGATIT with your own dataset
Solve your own maze with DQN
Your own Twitter client made with Django
[Reinforcement learning] DQN with your own library
Create your own DNS server with Twisted
Create your own Composite Value with SQLAlchemy
To import your own module with jupyter
Publish your own Python library with Homebrew
Reproduce thermography-like color change with sigmoid function
Try to make your own AWS-SDK with bash
Make your own module quickly with setuptools (python)
Train Stanford NER Tagger with your own data
Make your own music player with Bottle0.13 + jPlayer2.5!
Steps to install your own library with pip