Implement FReLU with tf.keras

Overview

I will introduce the implementation of a new activation function called FReLU in keras.

For more information about FReLU + implementation in pytorch, please read the following article. It is explained very carefully. Reference: Birth & explanation of new activation function "FReLU"!

This article is about implementing FReLU in tf.keras. With Depthwise Conv2D and Lambda, you can implement it quickly.

Definition

y= \max(x,\mathbb{T}(x))

$ \ mathbb {T} (\ cdot) $ is DepthwiseConv2D.

Premise

tensorflow 2.3.0

Implementation

FReLU.py


from tensorflow.keras.layers import DepthwiseConv2D,BatchNormalization
import tensorflow as tf
from tensorflow.keras.layers import Lambda
from tensorflow.keras import backend as K

#Mathematical symbol max()Definition of
def max_unit(args):
    inputs , depthconv_output = args
    return tf.maximum(inputs, depthconv_output)

def FReLU(inputs, kernel_size = 3):
    #T(x)Part of
    x = DepthwiseConv2D(kernel_size, strides=(1, 1), padding='same')(inputs)
    x = BatchNormalization()(x)
    #Calculate tensor shape for Lambda
    x_shape = K.int_shape(x)
    #max(x, T(x))Part of
    x = Lambda(max_unit, output_shape=(x_shape[1], x_shape[2], x_shape[3]))([inputs, x])
    return x

At the end

The implementation itself is easy, isn't it? I will try to implement the Layer version when I have time. It's nice because the name will appear when that person summarizes. If you have any questions or concerns, please leave a comment.

Recommended Posts

Implement FReLU with tf.keras
Implement login function with django-allauth
Implement subcommands with Python's argparse
Implement PyTorch + GPU with Docker
[QtDesigner] Implement WebView with PyQt5
Implement blockchain with about 60 lines
Ensure reproducibility with tf.keras in Tensorflow 2.3
Easily implement subcommands with python click
Implement Keras LSTM feedforward with numpy
Implement "Data Visualization Design # 2" with matplotlib
I tried to implement Autoencoder with TensorFlow
[Logistic regression] Implement k-validation with stats models
Implement large size stamp function with discordBOT
Nowadays, implement DQN (complete version) with Tensorflow
Implement a minimal self-made estimator with scikit-learn
I tried to implement CVAE with PyTorch
Easily implement ItemView incremental search with PySide
Implement DeepChem's GraphPoolLayer with PyTorch's custom layer
Easily implement login authentication function with Laravel
Quickly implement S3 compatible storage with python-flask
Implement a model with state and behavior