Regression with linear model

Hello everyone. I am now [Machine learning starting with Python](https://www.amazon.co.jp/Python%E3%81%A7%E3%81%AF%E3%81%98%E3%82%81%E3 % 82% 8B% E6% A9% 9F% E6% A2% B0% E5% AD% A6% E7% BF% 92-% E2% 80% 95scikit-learn% E3% 81% A7% E5% AD% A6% E3% 81% B6% E7% 89% B9% E5% BE% B4% E9% 87% 8F% E3% 82% A8% E3% 83% B3% E3% 82% B8% E3% 83% 8B% E3% 82% A2% E3% 83% AA% E3% 83% B3% E3% 82% B0% E3% 81% A8% E6% A9% 9F% E6% A2% B0% E5% AD% A6% E7% BF% I am studying at 92% E3% 81% AE% E5% 9F% BA% E7% A4% 8E-Andreas-C-Muller / dp / 4873117984).

An interesting concept called linear regression has emerged, so I think I should summarize it a little.

What is regression in the first place?

Regression is the application of the model Y = f (X) to data when Y is a continuous value in statistics. In other words, fit the model between the dependent variable (objective variable) Y and the independent variable (explanatory variable) X of the continuous scale. If X is one-dimensional, it is called simple regression, and if X is two-dimensional or more, it is called multiple regression. Wikipedia

In other words, the purpose of regression is to land on ** continuous value prediction **. For example ... Predicting a person's ** annual income (objective variable) ** from educational background, age, and address (explanatory variable). Predict the ** yield (objective variable) ** of a corn farmer from the previous year's yield, weather, and number of employees (explanatory variable).

So what is regression with a linear model?

Regression with a linear model literally uses a linear function to predict the objective variable.

A general prediction formula by a linear model in a regression problem is

y = w[0] \times x[0] + w[1] \times x[1] + \dots + w[p] \times x[p] + b

Can be expressed as.

Here, x [0] ... x [p] indicates the ** features ** of one data point, w and b are the ** parameters ** of the trained model, and y is * from the model. * Forecast **.

In other words, when the optimized w and b are obtained (learned) from the training data and a new x [0] ... x [p] is entered, It comes down to outputting ** y ** as accurate as possible.

Here, there are various algorithms for regression using a linear model. The difference between these models lies in the method of finding (learning) ** w, b ** and the method of living the complexity of the model.

Today I would like to touch on the simplest and most classical linear regression method, ** usually least squares (OLS) **.

Usually least squares (OLS)

Let's consider one explanatory variable for simplicity. In other words, the formula is

y = wx + b

In this linear regression, w and b are calculated so that the mean square error between y and the predicted value is minimized in the training data. It is difficult to understand from the text, so if you think about it with an image, it is as follows.

In other words, find ** w ** and ** b ** so that the sum of squares of the length of the blue line (error) is as small as possible, and the result is a red straight line.

If you write in mathematical formulas

Mean squared error=\frac{1}{n}\sum_{i=1}^{n} (y_i - y'_i)^2,  (y_i:Value in i-th training data, y'_i:i-th predicted value)

Select w and b so that this mean square error is small. This is OLS. It's easy and beautiful.

Sample code

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

import mglearn

X, y =mglearn.datasets.make_wave(n_samples=60)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

lr = LinearRegression().fit(X_train, y_train)

print("lr.coef_(Slope OR weight) : {}".format(lr.coef_))
print("lr.intercept_(Intercept) : {}".format(lr.intercept_))

print("Training set score: {:.2f}".format(lr.score(X_train, y_train)))
print("test set score: {:.2f}".format(lr.score(X_test, y_test)))

Click here for results

lr.coef_(Slope OR weight) : [0.39390555]
lr.intercept_(Intercept) : -0.031804343026759746
Training set score: 0.67
test set score: 0.66

Let's plot this straight line.

However, the forecast is not very good at 66%. Perhaps this is underfitting.

Next time, we will talk about different regression models, Ridge and Lasso. Have a nice night, everyone. good night.