Multivariate LSTM and data preprocessing in TensorFlow 2.x

Introduction

I haven't found many commentary articles on implementing multivariate LSTMs for time series data in TensorFlow 2.x, so I will write them. This article is recommended for the following people.

• People who want to implement (multivariate) LSTM for time series data in TensorFlow
• People who want to know the format and preprocessing of data to be input to LSTM

The code and data used are published on the GitHub repository. The code is in Jupyter Notebook format, and it can be run immediately on Colab, so please refer to it (please also star if you like).

Tasks to be tackled this time

This time, I would like to perform ** "Prediction of average cloud cover" ** in Tokyo from meteorological data such as temperature.

data

This time, as data used for forecasting, we downloaded ** "Average temperature, average relative humidity, total sunshine duration, average wind speed, average cloud cover" for the past 10 years in Tokyo from Japan Meteorological Agency HP Did. This data is also located in the GitHub repository (https://github.com/ishikawa08/tf_multi_LSTM/tree/main/data).

The code to read the data is below.

#Library import
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
%matplotlib inline
import seaborn as sns
import time

#Data read
df = pd.read_csv('data/tokyo_weather.csv')
df['date'] = pd.to_datetime(df['date'])
# df.set_index('date', inplace=True)  #When index is date
df

Looking at the actual data, it looks like the following. Of course, it's all numerical data. Next, we will process this data.

Data preprocessing

When training an LSTM with multivariate data, the data must be in the format ** [number of samples, number of lookups, number of variables] ** (** super important! **). Here, the number of lookups means ** "how many times of the past data should be regarded as one data" **. For example, in the figure below, the number of lookups is ** "3" ** (because the data for 3 weeks is regarded as one data).

The code that actually creates the dataset looks like this:

#Lookback number
look_back = 25
#The number of data
sample_size = len(df) - look_back
#Period used for forecasting
past_size = int(sample_size*0.8)
future_size = sample_size - past_size +1

#Functions that create datasets
def make_dataset(raw_data, look_back=25):
    _X = []
    _y = []

    for i in range(len(raw_data) - look_back):
        _X.append(raw_data[i : i + look_back])
        _y.append(raw_data[i + look_back])
    _X = np.array(_X).reshape(len(_X), look_back, 1)
    _y = np.array(_y).reshape(len(_y), 1)

    return _X, _y

Create a dataset using the created function. It also normalizes the data. We did not use wind speed data this time.

from sklearn import preprocessing

columns = list(df.columns)
del columns[0]

#Normalize to a minimum of 0 and a maximum of 1
Xs = []
for i in range(len(columns)):
    Xs.append(preprocessing.minmax_scale(df[columns[i]]))
Xs = np.array(Xs)

#Create each numerical data
X_tmpr, y_tmpr = make_dataset(Xs[0], look_back=look_back)
X_humid, y_humid = make_dataset(Xs[1], look_back=look_back)
X_dlh, y_dlh = make_dataset(Xs[2], look_back=look_back)
X_prec, y_prec = make_dataset(Xs[3], look_back=look_back)
X_cloud, y_cloud = make_dataset(Xs[4], look_back=look_back)

#Combine each data to support multivariate LSTMs
X_con = np.concatenate([X_tmpr, X_humid, X_dlh, X_prec, X_cloud], axis=2)

X = X_con
y = y_cloud

#Divide the data into past (used for training) and future (used for future prediction)
X_past = X[:past_size]
X_future = X[past_size-1:]
y_past = y[:past_size]
y_future = y[past_size-1:]

#Define training data
X_train = X_past
y_train = y_past

Multivariate LSTM model

Now, let's actually learn and predict with LSTM using the created data.

Creating a model

Make an LSTM model. Defining it with a function is also convenient when loading a trained model. If you are used to it, you can use the class.

import tensorflow as tf
from tensorflow.keras.layers import Input, LSTM, Dense, BatchNormalization
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import SGD, Adam

#Functions that create LSTM models
def create_LSTM_model():
    input = Input(shape=(np.array(X_train).shape[1], np.array(X_train).shape[2]))
    x = LSTM(64, return_sequences=True)(input)
    x = BatchNormalization()(x)
    x = LSTM(64)(x)
    output = Dense(1, activation='relu')(x)
    model = Model(input, output)
    return model

model = create_LSTM_model()
model.summary()
model.compile(optimizer=Adam(learning_rate=0.0001), loss='mean_squared_error')

Model learning

Now let's actually train the model.

history = model.fit(X_train, y_train, epochs=200, batch_size=64, verbose=1)

Confirmation of forecast results

Let the model predict when the training is over.

predictions = model.predict(X_past)
future_predictions = model.predict(X_future)

Let's display the prediction result.

plt.figure(figsize=(18, 9))
plt.plot(df['date'][look_back:], y, color="b", label="true_cloud_cover")
plt.plot(df['date'][look_back:look_back + past_size], predictions, color="r", linestyle="dashed", label="prediction")
plt.plot(df['date'][-future_size:], future_predictions, color="g", linestyle="dashed", label="future_predisction")
plt.legend()
plt.show()

Below is a diagram of the forecast results. It seems that the future prediction part (green line) that is not used for training has ended with reasonable accuracy. Well, let's forgive this time in such a place.

in conclusion

We implemented and explained multivariate LSTM for time series data in TensorFlow 2.x. There are some parts that are not separated from the training data and the test data, but I hope it will be helpful.