Visualize activation functions side by side

(Addition)

Added the explanation of the code.

I used to be a beginner. Although I was self-taught, I managed to understand it. When I was just starting out, I added a commentary that I wanted to hear like this. I think this article will probably be read by beginners and intermediates. At first, it was difficult for me to study by myself, and I couldn't understand it well, and sometimes I was frustrated. I wrote that I could help even a little in such a case.

1.First of all

An activation function has been proposed in deep learning and is still being researched. I tried to display each side by side. I often use ReLU exclusively, but when I thought about it, I thought I couldn't remember it exactly, so I listed it.

2. Try to line up

I implemented it using Jupyter Notebook. Each activation function can be displayed side by side.

%matplotlib inline
#Module import
import numpy as np
import matplotlib.pyplot as plt

#Graph settings
n_cols = 4
n_rows = 2
plt.figure(figsize=(20,8))

#Generate value of x
x = np.arange(-10, 10, 0.01)

#List of each function to display
activation_function_list = [ 'Indentity', 'Step', 'Sigmoid', 'tanh', 'ReLU', 'Leaky ReLU','Swish', 'Mish']

#graph display
def graph(x,y):
    plt.subplot(n_rows, n_cols, i+1)
    plt.grid()
    plt.title(f)
    plt.ylim(-1, 1)
    plt.xlim(-5, 5)
    plt.plot(x,y)
    
#Calculation of each function
for i, f in enumerate(activation_function_list):
    #Identity function
    if f == 'Indentity':
        y = x
    #Step function
    elif f == 'Step':
        y = np.where(x<=0, 0, 1)
    #Sigmoid function
    elif f == 'Sigmoid':
        y = 1./(1. + np.exp(-x))
    #tanh function
    elif f == 'tanh':
        y = (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(-x))
    #ReLU function
    elif f == 'ReLU':
        y = np.maximum(0,x)
    #Leaky ReLU function
    elif f == 'Leaky ReLU':
        y = np.where(x > 0, x , 0.01 * x)
    #Swish function
    elif f == 'Swish':
        y = ｘ*1./(1. + np.exp(-x))
    #mish function
    elif f == 'Mish':
        x_sp = np.log(1+np.exp(x))     #Softplus function
        y = x * ((np.exp(x_sp ) - np.exp(-x_sp)) / (np.exp(x_sp) + np.exp(-x_sp)))    
        
    #Creating a graph
    graph(x,y)
        
#Drawing a graph
plt.show()

#Save graph
fig.savefig('activation_function_list.png')

3. Output function

The graph created by matplotlip at the end of the code is saved as a png file. In the above case, this file will be saved in the same directory where the notebook file is located. Swish and Mish are new activation functions.

In the future, I would like to verify the difference in performance between these activation functions.

This is the end.

4. Let's explain the above code.

From here, I will explain the code written above. I wrote the commentary I wanted to hear when I was a beginner. I think that it is not enough for intermediate and advanced users, but if you are interested, please take a look.

4-1. Graph display

%matplotlib inline

It is drawn in the Notebook by writing % matplotlib inline.

4-2. Module import

The code below will import two modules, NumPy and Matplotlib.

#Module import
import numpy as np
import matplotlib.pyplot as plt

Let's check where this module is imported from. When I run the following code on my notebook

import numpy as np
print(np.__file__)

output /Users/username/.pyenv/versions/3.6.2/lib/python3.6/site-packages/numpy/__init__.py

This output depends on each environment, but what you can see from this output result is that numpy installed on the local PC is referenced. You can read more about __init__.py in @ PYTHONISTA's article. How to write Python __init __.py

Since as is added when importing, you can refer to numpy with np after this.

import numpy
print(numpy.__file__)

Even so, you can do the same.

4-3. Graph settings

Here, the graph that displays the activation functions side by side is set. I am using the subplot described below to set the graphs to be arranged horizontally 4 horizontally and 2 vertically. The figure is made with the size specified by figsize.

At the end of the code, the list is saved as a png file, but the graph placed in the figure becomes one file.

#Graph settings
n_cols = 4
n_rows = 2
plt.figure(figsize=(20,8))

4-4. X value generation

#Generate x-axis values
x = np.arange(-10, 10, 0.01)

Let's try one here.

If you press the [shift] + [tab] keys while the cursor is in () like this, the details will be displayed as shown below.

From here you can see what the parameters that arrange takes as arguments. If x = np.arange (-10, 10, 0.1), start is -10, stop is 10, step is 0.1, and so on. A list of numbers from -10 to 10 will be returned in 0.1 increments. Let's check it.

print('List of x:',x)
print('x length:',len(x))
print('type of x', type(x))

List of x: [-10. -9.99 -9.98 ... 9.97 9.98 9.99] Length of x: 2000 type of x <class'numpy.ndarray'> 2000 numbers have been generated. You can see that the 10 specified for stop is not included.

4-5. List of each function to display

Here, we are making a list of functions that we want to list.

activation_function_list = [ 'Indentity', 'Step', 'Sigmoid', 'tanh', 'ReLU', 'Leaky ReLU','Swish', 'Mish']

4-6. Graph display

This part defines the graph drawing function. This function is called in the process of 4-7 to make a graph. The i part in this shows the location of the graph when arranging in 4x2. From top left to right 1 2 3 4 5 6 7 8 It will be. I put i in the third argument of plt.subplot. This means that i when you call the graph function specifies the location of the graph itself.

4-7 retrieves the index and activation function names from the list.

#graph display
def graph(x, y):
    #The index starts from 0, but the location of the graph is specified from 1, so
    #Add 1 here and 4-We are calling the graph function defined in 6.
    plt.subplot(n_rows, n_cols, i+1)
    plt.grid()   #Enable grid lines in the graph
    plt.title(f)    #Set function name to graph title
    plt.ylim(-1, 1)    #Specify the y-axis range to display
    plt.xlim(-5, 5)    #Specify the x-axis range to display
    plt.plot(x, y)    #Plot the x and y values

4-7 Calculation of each function

It's a bit long, so I'll add a comment in the code. The details of each function are too much to cover here, so I will omit them in this article.

#Calculation of each function
#Use the enumerate function to activate_function_Extracting indexes and list values from list.
#i is the index and f is the function name.
#i is entered in order from 0.
for i, f in enumerate(activation_function_list):
    #The value of y is calculated based on the extracted function name.
    #The value of x is 2,Since there were 000, the value of y is also 2.,000 have been generated.
    if f == 'Indentity':    #Identity function
        y = x
    elif f == 'Step':    #Step function
        y = np.where(x<=0, 0, 1)
    elif f == 'Sigmoid':    #Sigmoid function
        y = 1./(1. + np.exp(-x))
    elif f == 'tanh':    #Hyperbolic tangent function
        y = (np.exp(x) - np.exp(-x)) / (np.exp(x) + np.exp(-x))
    elif f == 'ReLU':    #ReLU function
        y = np.maximum(0,x)
    elif f == 'Leaky ReLU':    #Leaky ReLU function
        y = np.where(x > 0, x , 0.01 * x)
    elif f == 'Swish':    #Swish function
        y = ｘ*1./(1. + np.exp(-x))
    elif f == 'Mish':    #Swish function
        x_sp = np.log(1+np.exp(x))    #softplus function
        y = x * ((np.exp(x_sp ) - np.exp(-x_sp )) / (np.exp(x_sp ) + np.exp(-x_sp )))    
        
    #Drawing a graph
    #Pass y calculated using x as an argument of the graph function.
    graph(x,y)

I'm using the ʻenumerate` function to get the index and value from the list. To find out what value is being obtained, execute the following code

for i, f in enumerate(activation_function_list):
    print('index:', i , 'value:', f)

output index: 0 value: Indentity index: 1 value: Step index: 2 value: Sigmoid index: 3 value: tanh index: 4 value: ReLU index: 5 value: Leaky ReLU index: 6 value: Swish index: 7 value: Mish You can see that the indexes and values can be retrieved in order.

4-8 Drawing and saving graphs

The graph is drawn and saved in the following two lines. The png file to be saved is saved in the folder containing the notebook file you are using.

#Drawing a graph
plt.show()

#Save graph
fig.savefig('activation_function_list.png')

If there are any mistakes or mistakes, I would appreciate it if you could let me know.