I tried to organize the evaluation indexes used in machine learning (regression model)

Introduction

This time, I tried to organize the evaluation index (regression model) of machine learning. We also summarized the points when using each evaluation index.

1. Coefficient of determination

The coefficient of determination $ R ^ 2 $ evaluates the goodness of the estimated regression model. The formula is as follows.

\begin{eqnarray}
R^2 = 1 - \sum \frac{(y_i - \hat{y_i})^2}{(y_i - \bar{y})^2}
\end{eqnarray}

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ \ bar {y} $: Average of measured values

item Overview
Evaluation criteria ・ The closer it is to 1, the better the value
Interpretation ・ Evaluate the goodness of the regression model
Characteristic ・ As the number of explanatory variables increases, it approaches 1.
・ Non-linear model cannot be evaluated

The python code is below.

#Library for score calculation
from sklearn.metrics import r2_score

#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]

#Calculate the value of the coefficient of determination R2
r2_score(true, pred)
  1. MAE MAE (Mean Absolute Error) evaluates the magnitude of the error in the predicted value. The formula is as follows.
\begin{eqnarray}
MAE = \frac{1}{N} \sum|y_i - \hat{y_i}|
\end{eqnarray}

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ N $: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation ・ The size of the average deviation (error) between the predicted value and the measured value
Characteristic ・ We do not attach much importance to cases that are largely off-prediction.
・ If a good model is selected based on MAE, the maximum error tends to increase.

The python code is below.

#Library for score calculation
from sklearn.metrics import mean_absolute_error

#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]

#Calculate the value of MAE
mean_absolute_error(true, pred)
  1. MSE In MSE (Mean Squared Error), large errors are emphasized and evaluated. The formula is as follows.
\begin{eqnarray}
MSE = \frac{1}{N} \sum(y_i - \hat{y_i})^2
\end{eqnarray}

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ N $: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation ・予測値と実測値のズレの大きさとInterpretationできる
-Similar to MAE, but different from the actual mean error like MAE
Characteristic ・ We attach great importance to cases that are largely out of the forecast.
・ MSE tends to increase significantly if the forecast is largely missed.

The python code is below.

#Library for score calculation
from sklearn.metrics import mean_squared_error

#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]

#Calculate the value of MSE
mean_squared_error(true, pred)
  1. RMSE In RMSE (Root Mean Squared Error), large errors are emphasized and evaluated. The formula is as follows.
\begin{eqnarray}
RMSE = \sqrt{\frac{1}{N} \sum(y_i - \hat{y_i})^2}
\end{eqnarray}

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ N $: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation ・予測値と実測値のズレの大きさとInterpretationできる
-Similar to MAE, but different from the actual mean error like MAE
Characteristic ・ We attach great importance to cases that are largely out of the forecast.
・ If the forecast is largely missed, the RMSE tends to increase significantly.

The python code is below.

#Library for score calculation
import numpy as np
from sklearn.metrics import mean_squared_error

#true is the true value (measured value), pred is the predicted value
true = [1.0, 1.5, 2.0, 1.2, 1.8]
pred = [0.8, 1.5, 1.8, 1.3, 2.0]

#Calculate RMSE value
np.sqrt(mean_squared_error(true, pred))
  1. MAPE MAPE (Mean Absolute Percentage Error) evaluates the magnitude of the prediction error per measured value. The formula is as follows.
\begin{eqnarray}
MAPE = \frac{1}{N} \sum|\frac{y_i - \hat{y_i}}{y_i}|
\end{eqnarray}

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ N $: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation -Ratio of the average deviation (error) of the predicted value to the size of the measured value
Characteristic ・ Emphasis is placed on the size of the ratio of prediction error to the measured value.
・ Cannot be used in cases where the measured value is 0

The python code is below.

#Library for score calculation
import numpy as np

#true is the true value (measured value), pred is the predicted value
true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])

#Calculate the value of MAPE
np.mean(np.abs((true - pred) / true)) * 100
  1. SMAPE SMAPE (Symmetric Mean Absolute Percentage Error) evaluates the magnitude of the prediction error per measured value. The formula is as follows.
\begin{eqnarray}
SMAPE = \frac{100}{N} \sum|\frac{2(y_i - \hat{y_i})}{(|y_i|+|\hat{y_i}|)}|
\end{eqnarray}

$ y_i $: Measured value of $ i $ th sample $ \ hat {y_i} $: Predicted value of $ i $ th sample $ N $: Number of samples

item Overview
Evaluation criteria ・ The closer to 0, the better the value
Interpretation -Ratio of the average deviation (error) of the predicted value to the size of the measured value
Characteristic ・ Emphasis is placed on the size of the ratio of prediction error to the measured value.
・ Unlike MAPE, it can be used even when the measured value is 0.

The python code is below.

#Library for score calculation
import numpy as np

#true is the true value (measured value), pred is the predicted value
true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])

#Calculate SMAPE value
100/len(true) * np.sum(2 * np.abs(pred - true) / (np.abs(pred) + np.abs(true)))

Script summary

Organize the scripts so far.

#Library for score calculation
import numpy as np
import pandas as pd
from sklearn.metrics import r2_score
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error

def calculate_scores(true, pred):
    """Calculate all metrics

    Parameters
    ----------
    true : np.array
Measured value
    pred : np.array
Predicted value

    Returns
    -------
    scores : pd.DataFrame
Results of summarizing each evaluation index
    """
    scores = pd.DataFrame({'R2': r2_score(true, pred),
                          'MAE': mean_absolute_error(true, pred),
                          'MSE': mean_squared_error(true, pred),
                          'RMSE': np.sqrt(mean_squared_error(true, pred)),
                          'MAPE': mape(true, pred),
                          'SMAPE': smape(true, pred)},
                          index = ['scores'])
    return scores

def mape(true, pred):  
    """Calculate MAPE

    Parameters
    ----------
    true : np.array
Measured value
    pred : np.array
Predicted value

    Returns
    -------
    np.array :mape calculation result
    """
    return np.mean(np.abs((true - pred) / true)) * 100

#SMAPE calculation
def smape(true, pred):
    """Calculate SMAPE

    Parameters
    ----------
    true : np.array
Measured value
    pred : np.array
Predicted value

    Returns
    -------
    np.array :Calculation of smape
    """
    return 100/len(true) * np.sum(2 * np.abs(pred - true) / (np.abs(pred) + np.abs(true)))

def main():
    true = np.array([1.0, 1.5, 2.0, 1.2, 1.8])
    pred = np.array([0.8, 1.5, 1.8, 1.3, 2.0])

    scores = calculate_scores(true, pred)

    print(scores)

if __name__ == "__main__":
    main()
R2 MAE MSE RMSE MAPE SMAPE
scores 0.808824 0.14 0.026 0.161245 9.888889 10.254971

finally

Thank you for reading to the end. If you have a request for correction, we would appreciate it if you could contact us.

Recommended Posts

I tried to organize the evaluation indexes used in machine learning (regression model)
I tried to visualize the model with the low-code machine learning library "PyCaret"
I tried to compress the image using machine learning
I tried to predict the change in snowfall for 2 years by machine learning
Try to evaluate the performance of machine learning / regression model
I tried to summarize the code often used in Pandas
I tried to summarize the commands often used in business
[Machine learning] I tried to summarize the theory of Adaboost
I tried calling the prediction API of the machine learning model from WordPress
I tried to classify guitar chords in real time using machine learning
(Machine learning) I tried to understand Bayesian linear regression carefully with implementation.
I implemented the VGG16 model in Keras and tried to identify CIFAR10
I tried to train the RWA (Recurrent Weighted Average) model in Keras
I tried to implement TOPIC MODEL in Python
Summary of evaluation functions used in machine learning
(Machine learning) I tried to understand the EM algorithm in a mixed Gaussian distribution carefully with implementation.
I tried to understand the learning function in the neural network carefully without using the machine learning library (second half).
I tried to predict the presence or absence of snow by machine learning.
I tried to implement various methods for machine learning (prediction model) using scikit-learn.
I tried to process and transform the image and expand the data for machine learning
I tried to move machine learning (ObjectDetection) with TouchDesigner
I tried to graph the packages installed in Python
Attempt to include machine learning model in python package
I tried to organize SVM.
I tried "Lobe" which can easily train the machine learning model published by Microsoft.
Used in machine learning EDA
Try to evaluate the performance of machine learning / classification model
I tried machine learning to convert sentences into XX style
I tried to illustrate the time and time in C language
I tried to implement the mail sending function in Python
[TF] I tried to visualize the learning result using Tensorboard
I tried to divide with a deep learning language model
I tried to compare the accuracy of machine learning models using kaggle as a theme.
Matching app I tried to take statistics of strong people & tried to create a machine learning model
[Deep Learning from scratch] I tried to explain the gradient confirmation in an easy-to-understand manner.
I tried to verify the yin and yang classification of Hololive members by machine learning
I tried to understand it carefully while implementing the algorithm Adaboost in machine learning (+ I deepened my understanding of array calculation)
I tried machine learning with liblinear
Classification and regression in machine learning
I tried to organize about MCMC.
I tried to move the ball
I tried to estimate the interval.
I tried to describe the traffic in real time with WebSocket
[Updated as appropriate] I tried to organize the basic visualization methods
I tried to process the image in "sketch style" with OpenCV
I tried to summarize the commands used by beginner engineers today
I tried to process the image in "pencil style" with OpenCV
I tried to summarize the frequently used implementation method of pytest-mock
Machine learning model management to avoid quarreling with the business side
[Machine learning] I tried to do something like passing an image
People memorize learned knowledge in the brain, how to memorize learned knowledge in machine learning
I tried to make Othello AI with tensorflow without understanding the theory of machine learning ~ Introduction ~
I tried to understand supervised learning of machine learning in an easy-to-understand manner even for server engineers 1
[Qiita API] [Statistics • Machine learning] I tried to summarize and analyze the articles posted so far.
I tried to understand supervised learning of machine learning in an easy-to-understand manner even for server engineers 2
I tried to make Othello AI with tensorflow without understanding the theory of machine learning ~ Implementation ~
I tried to implement PLSA in Python
<Course> Machine Learning Chapter 3: Logistic Regression Model
I tried to summarize the umask command
I tried to implement permutation in Python
I tried to summarize the methods that are often used when implementing basic algo in Quantx Factory