[Machine learning] Understanding linear simple regression from both scikit-learn and mathematics
1. Purpose
If you want to try machine learning, anyone can use scikit-learn etc. to implement it relatively easily. However, in order to achieve results at work or to improve your level ** You can see that the explanation "I don't know the background, but I got this result" is obviously weak **.
In this article, ** 2 to 3 are aimed at "the theory is good, so try using scikit-learn first", and 4 and later are aimed at "understanding the background from mathematics" **.
- I am from a private liberal arts school, so I am not good at mathematics. I tried to explain it so that it is easy to understand even for those who are not good at math as much as possible.
2/5 postscript
- A similar article has been posted in Logistic Regression Ver, so please read it as well. [[Machine learning] Understanding logistic regression from both scikit-learn and mathematics] (https://qiita.com/Hawaii/items/ee2a0687ca451fe213be)
3/1 postscript
・ 3. Linear regression with scikit-learn → (4) Model construction → (iv) Supplement was added.
2. What is linear (simple) regression?
(1) What is regression?
** Predict numbers. ** In machine learning, there are other "classifications", but if you want to predict numerical values such as "●● yen" and "△ Kg", you can think of using regression.
(2) What is linear regression?
There may be some misunderstandings,
"What you want ($ = y
I think it's hard to understand, so I'll give you a concrete example.
Specific example
You are a self-employed ice cream shop, and you strongly want to be able to predict the sales of ice cream in your store ** in order to stabilize the sales prospects. I will.
You desperately wonder what is affecting your store's ice cream sales, and realize that the hotter the temperature, the more ice cream sells, and the cooler the temperature, the less ice cream sells. I did.
So, if you try to illustrate "temperature ($ = x
Next, let's use scikit-learn to build a machine learning model that seeks ice cream sales from temperature.
3. Linear regression with scikit-learn
(1) Import of required libraries
Import the following required to perform linear regression.
from sklearn.linear_model import LinearRegression
(2) Data preparation
Set the temperature and ice cream sales as data as shown below.
- For example, in the following, sales will be 300,000 yen on a day when the temperature is 8 °, and 350,000 yen on a day when the temperature is 10 °.
data = pd.DataFrame({
"temprature(=x)":[8,10,6,15,12,16,20,13,24,26,12,18,19,16,20,23,26,28],
"sales(=y)":[30,35,28,38,35,40,60,34,63,65,38,40,41,43,42,55,65,69]
})
(3) Try to illustrate (important)
Let's illustrate the temperature and ice cream sales. Even if you use linear regression if it is not in a linear relationship, the accuracy will be very poor because the original data is not linear. Instead of using scikit-learn all at once, try to illustrate any data.
plt.scatter(data["temprature(=x)"],data["sales(=y)"])
plt.xlabel("temprature(°)")
plt.ylabel("sales")
plt.grid(which='major',color='black',linestyle=':')
Approximately, there seems to be a linear relationship between temperature ($ = x
(4) Model construction
(I) Data shaping
First of all, we will arrange the shape of the data to build the model.
x = data["temprature(=x)"].values
y = data["sales(=y)"].values
X = x.reshape(-1,1)
Since this is not an article on python grammar, I will omit the details, but I will arrange x and y into a form for linear regression with scikit-learn.
- I think this is a code that cannot be written unless you understand it to some extent, so I would like to summarize it somewhere.
(Ii) Model construction
It's finally the model building code.
regr = LinearRegression(fit_intercept = True)
regr.fit(X,y)
It may be out of place, but if it's a simple model, that's it. We will create a linear regression model for a variable called regr! It is an image of doing something like a declaration and letting the regr fit (= learn) the prepared X and y in the next line.
(Iii) Try to get the slope and intercept of the straight line
As described in "2. What is linear (simple) regression?" (2), the scikit-learn up to this point is used to find $ a $ and $ b $ of $ y = ax + b $, and forecast sales from the temperature. I am looking for the formula of the straight line to be done behind the scenes. If you leave it as it is, you will not realize it, so let's actually set the slope.
a = regr.coef_ #Find the slope
b = regr.intercept_ #Find the intercept
print(a)
print(b)
You should see a as [1.92602996] and b as [12.226591760299613]. In other words, the straight line is $ y (= sales) = 1.92602996 * x (= temperature) + 12.226591760299613 $ and scikit-learn asked for it.
(Iv) Supplement
If you just want to build a model, (iii) is enough, but there are other things as follows. Reference: https://pythondatascience.plavox.info/scikit-learn/%E7%B7%9A%E5%BD%A2%E5%9B%9E%E5%B8%B0
◆ Display the parameters used in model construction This time I just set fit_intercept to True, but there are other parameters that can be set and you can see how they are set now.
regr.get_params()
Then {'copy_X': True,'fit_intercept': True,'n_jobs': None,'normalize': False} will be displayed.
-Copy_X: Select whether to execute after duplicating the data in memory. (Default value: True) -Fit_intercept: When set to False, the calculation to obtain the intercept ($ b $ in this case) is not included, so it is used when handling data in which the objective variable always passes through the origin. (Default value: True) -N_jbobs: The number of jobs used for calculation. If set to -1, it will be calculated using all CPUs. (Default value: 1) -Normalize: When set to True, the explanatory variables are prenormalized. (Default value: False)
◆ Display coefficient of determination The coefficient of determination is a measure of how well the model fits the actual data, in the range 0 to 1.
regr.score(X,y)
◆ Evaluation of error Since the amount of description will be large, I will not describe it again, but the following will be helpful. https://pythondatascience.plavox.info/scikit-learn/%E5%9B%9E%E5%B8%B0%E3%83%A2%E3%83%87%E3%83%AB%E3%81%AE%E8%A9%95%E4%BE%A1
(5) Illustrate the constructed model
Now, let's illustrate this straight line in the scatter plot above.
#Straight line formula
y_est_sklearn = regr.intercept_ + regr.coef_[0] * x
#Original temperature and sales plot
plt.scatter(x, y, marker='o')
#Original temperature and straight line formula of prediction
plt.plot(x, y_est_sklearn, linestyle=':', color='green')
#Detailed settings in the figure
plt.grid(which='major',color='black',linestyle=':')
plt.grid(which='minor',color='black',linestyle=':')
plt.xlabel("temprature(°)")
plt.ylabel("sales")
In this way, let's be aware of what scikit-learn is doing and what it is connected to.
(6) In the real world ...
It doesn't make sense to finish making a model. In the real world, it is necessary to use this straight line forecast model to forecast future sales. You looked at the weather forecast for the next four days and made a note of the temperature. Store it in a variable called z as shown below.
z = pd.DataFrame([10,25,24,22])
What I want to do is to apply the above future temperature forecast to the straight line formula obtained by scikit-learn earlier and forecast sales.
regr.predict(z)
If you do this, you will see the result as "([31.48689139, 60.37734082, 58.45131086, 54.59925094])". In other words, tomorrow the temperature is 10 °, so sales will be about 315,000 yen, and the day after tomorrow, the temperature will be 25 °, so sales will be about 603,000 yen. If you can get a forecast of the temperature for the next month, you will have a rough idea of sales and your goal will be achieved.
There are many other details, but I think it's good to try implementing orthodox linear regression first.
4. Understanding linear (simple) regression from mathematics
By the way, up to 3, I implemented the flow of calculating $ a $ and $ b $ of $ y = ax + b $ using scikit-learn → illustration → forecasting sales from the temperature for the next 4 days. .. Here, I would like to clarify ** how the "calculate $ a $ and $ b $ of $ y = ax + b $" in this flow is ** mathematically calculated. I will.
- If you do not need this knowledge at present, you can skip it.
(1) Prerequisite knowledge
a. Basic differentiation
y = x^When 2 is differentiated by x, y'=2x\\
y = x^2 +When 4 is differentiated by x, y'=2x\\
y = (-3x + 2)^When 2 is differentiated by x, y' = 2(-3x +2)(-3)
b. Meaning of Σ (sigma) Means sum
(2) Mathematical understanding
(I) What you are doing to get a and b of y = ax + b
I will repost the table I mentioned earlier.
As shown below, I want to draw a "good-looking straight line" for predicting temperature and sales, that is, to determine the slope and intercept a and b.
How do you decide a and b for that?
Now look at the two straight lines below. Which is the straight line, green or orange, that is more likely to predict the relationship between temperature and sales?
Obviously, you can see that orange (a = 1.92, b = 12.2) is more likely to represent the relationship between actual temperature and sales than green (a = 2.0, b = 30.0). I think it can be said that the orange straight line is "because the distance between the straight line and the actual blue point is closer" **.
In other words, scikit-learn wants a and b that are "good straight lines" where the distance between the straight line and the blue point is the shortest.
This method of finding a and b such that "the distance between the straight line and the blue point is the closest" is called the "least squares method".
(Ii) Least squares method
Let's chew a little more. The "distance between the straight line and the blue point" described in (i) can be written as follows.
Actual coordinates (red dot in the above figure): $ (x_1, y_1) $ Coordinates predicted by $ y = ax + b $: Expressed as $ (x_1, ax_1 + b) $.
The error between these two y-coordinates (= the difference between the forecast and the actual sales) can be expressed as $ {y_1- (ax_1 + b)} $.
- Refers to the green distance in the above figure.
This $ {y_1- (ax_1 + b)} $ is just the error between the forecast and the actual sales of one red dot, so add them up for all the dots and set a and b so that this error is as small as possible. Find (= find a and b from the calculation so that the difference between the prediction and the actual is as small as possible).
In addition, if the difference is taken purely, the plus and minus will be canceled out, so in general, the value obtained by squaring the error is calculated so as to be as small as possible. This idea is called the least squares method.
- For example, suppose that the error is 5 at one point and -5 at another point. If you add these two, it will be 0, but you can imagine that it is not a perfect straight line without any error. Therefore, the square of the error, in this case 25 and 25, is added and the squared error is considered to be 50.
(Iii) Try to solve the least squares method
◆ Setting the error function
Assuming that the sum of the errors of the predicted values calculated from all the actual points and the straight lines is "E", E can be expressed as follows.
- Because I am a beginner, I didn't know how to add {}, so I ended up with [] ... I'm sorry.
E = \sum_{i=1}^n [{y_i - (ax_i + b)}]^{2}
This is the square of the error between the actual value $ y_i $ and the predicted value $ (ax_i + b) $, added from the first point to the nth (substantially all).
◆ To minimize the error function
To consider the minimum of $ E $, let's represent the form of the function of $ E $.
- The shape of the graph below is not accurate, so please take it as an image.
In general, $ E $ is represented as above, and you can see that $ E $ is minimized around the red dot. Then, what is this red point? ** "Differentiation of $ E $ by $ a $ makes it 0, and even if $ E $ is differentiated by $ b $, it becomes 0" ** is. Differentiation means "slope", so the point where the slope is 0 when viewed from the axis side of $ a $ and the slope is 0 when viewed from the axis side of $ b $ is the red point. ..
- If you go deeper, you will get a complicated background such as convex optimization, so I will limit it to this explanation.
◆ Let's actually calculate
If possible, please bring a piece of paper and a pen and move your hands. [Differentiate with $ a $] The $ ∂ $ used below is read as "Dell" to mean differentiating (there are several ways to read it).
↓ The following is a supplement to the formula ① ↓
\begin{align}
\frac{∂E}{∂a} &= \frac{∂}{∂a} \sum_{i=1}^n (y_i - ax_i - b)^{2}・ ・ ①\\
&= \sum_{i=1}^n 2*(y_i - ax_i - b)*(-x_i)・ ・ ②\\
&= \sum_{i=1}^n -2x_i(y_i - ax_i -b)・ ・ ③\\
&= \sum_{i=1}^n -2x_iy_i + \sum_{i=1}^n 2ax_i^2 + \sum_{i=1}^n 2x_ib ... ④\\
\end{align}
・ When ① is differentiated, it becomes ② (=, so $ ∂ $, which means to differentiate, disappears in ②) ・ ③ is just a slight transformation of ②, and ④ is decomposed by $ Σ $ for each character of ③. I'm looking for a point that differentiates with $ a $ and becomes 0, so let's solve it with ④ = 0.
\begin{align}
- \sum_{i=1}^n x_iy_i + a\sum_{i=1}^nx_i^2 + b \sum_{i=1}^n x_i =0 ・ ・ ⑤\\
- \bar{xy} + a\bar{x^2} + b \bar{x} =0 ・ ・ ⑥\\
\end{align}
Since ⑤ is ④ = 0, it is the formula obtained by dividing the coefficient 2 attached to the formula ④ by both sides. ⑥ is the formula of ⑤ divided by n on both sides. In ⑤, data is added from the first to n for each $ Σ $, so if you divide by n, the average will come out. To be more specific, the first $ Σ $ ($-\ sum_ {i = 1} ^ n x_iy_i $) is the sum of $ xy $ from the first to the nth. In other words, if you divide this by $ n $, it will be the average of the whole, so it can be expressed as $-\ bar {xy} $.
[Differentiate with $ b $] Similarly, $ b $ is also differentiated.
\begin{align}
\frac{∂E}{∂b} &= \frac{∂}{∂b} \sum_{i=1}^n (y_i - ax_i - b)^{2}・ ・ [1]\\
&= \sum_{i=1}^n 2*(y_i - ax_i - b)*(-1)・ ・ [2]\\
&= \sum_{i=1}^n -2(y_i - ax_i -b)・ ・ [3]\\
&= \sum_{i=1}^n-2y_i + \sum_{i=1}^n 2ax_i + \sum_{i=1}^n 2b ... [4]\\
\end{align}
What you are doing in [1] to [4] is basically the same as ① to ④, which are differentiated by $ a $, and correspond. Similarly, let's solve [5] and [6] together with ⑤ and ⑥.
\begin{align}
- \sum_{i=1}^ny_i + a\sum_{i=1}^nx_i + b =0 ... [5]\\
- \bar{y} + a\bar{x} + b =0 ... [6]\\
\end{align}
[Solving simultaneous equations] ⑥ and [6] are reprinted with some transformations.
a\bar{x^2} + b \bar{x} = \bar{xy}・ ・ ⑥'\\
a\bar{x} + b = \bar{y}・ ・ 【6'】
To solve these two simultaneous equations (to eliminate = $ b $), multiply [6'] by $ \ bar {x} $.
a\bar{x^2} + b \bar{x} = \bar{xy}・ ・ ⑥'\\
a\bar{x}^2 + b\bar{x} = \bar{x}\bar{y}・ ・ 【6''】
Here are two things to keep in mind that are easy to make mistakes.
・ "$ A \ bar {x ^ 2}
⑥-[6''] will result in the following.
a\bar{x^2} - a\bar{x}^2 = \bar{xy} - \bar{x}\bar{y}
Solving this for $ a $
a = \frac{\bar{xy} - \bar{x}\bar{y}}{\bar{x^2} - \bar{x}^2}・ ・[A]
Finally, we will solve about $ b $. From [6], $ b = \ bar {y} --a \ bar {x} $, so if you substitute [A],
b = \bar{y} - \frac{\bar{xy} - \bar{x}\bar{y}}{\bar{x^2} - \bar{x}^2}\bar{x}・ ・[B]
From [A] and [B], I was able to get the $ a $ and $ b $ I wanted.
◆ The straight line formula you wanted to find
Now that we have $ a $ and $ b $, we can express the "best" straight line formula that minimizes $ E $ (= minimum error) as follows. The original straight line formula is $ y = ax + b $, so
y = \frac{\bar{xy} - \bar{x}\bar{y}}{\bar{x^2} - \bar{x}^2}x +( \bar{y} - \frac{\bar{xy} - \bar{x}\bar{y}}{\bar{x^2} - \bar{x}^2}\bar{x})
I was able to express it!
The point is that you could calculate this by hand, it's not amazing, but the above formula can be calculated only with the data you have now (in this example, temperature and sales data) ** That is the point.
With scikit-learn, it calculates in one shot, but I think it is very important to understand that this kind of calculation is done behind the scenes. At first, it took me a long time to understand the sequence of steps up to this point. It may be difficult at first, but I hope everyone can move their hands together.
◆ Slight development
Hand-calculated $ y = \ frac {\ bar {xy}-\ bar {x} \ bar {y}} {\ bar {x ^ 2}-\ bar {x} ^ 2} x + (\ bar {y}-\ frac {\ bar {xy}-\ bar {x} \ bar {y}} {\ bar {x ^ 2}-\ bar {x} ^ 2} \ bar {x}) $ , For some of the formulas that appear in this, the following holds.
Premise
$ \ bar {xy}-\ bar {x} \ bar {y} = σ_ {xy} $ * From the covariance formula
$ \ bar {x ^ 2}-\ bar {x} ^ 2 = σ_x ^ 2 $ * From the variance formula
Conclusion
y = \frac{σ_{xy}}{σ_x^2}x + (\bar{y} - \frac{σ_{xy}}{σ_x^2}\bar{x})
It can also be written as above.
5. Summary
How was it? My thought is, "I can't interpret the extremely complicated code from the beginning, so I don't care about the accuracy, so I'll try to implement a basic series of steps with scikit-learn etc." I think it's very important.
However, once I get used to it, I feel that it is very important to understand from a mathematical background how they work behind the scenes.
I think there are many contents that are difficult to understand, but I hope it helps to deepen my understanding.
- In the future, I would like to post similar articles for multiple regression and logistic regression.