Variance, statistics up to standard deviation
First, create two types of data with different distributions.
Data generation
import numpy as np
np.random.seed(seed=32)
groupA = np.random.normal(100, 20, 100000) #Average 100,Randomly generate 100000 values with a standard deviation of 20
groupB = np.random.normal(100, 50, 100000) #Average 100,Randomly generate 100000 values with a standard deviation of 50
print("groupA sample: {}".format(groupA[0:5]))
print("groupA sample: {}".format(groupB[0:5]))
groupA sample: [ 93.02211098 119.67406867 111.61845661 101.40568882 115.55065353]
groupA sample: [ 122.71129387 134.76068077 88.52182502 92.52887435 107.2470646 ]
average
First of all, looking at the average, as shown below, the values of both groups are around 100 on average (it is natural because 100 is specified on average).
mean
meanA, meanB = np.mean(groupA), np.mean(groupB)
print("group A average= {},group B average= {}".format(meanA, meanB))
group A average= 100.0881959959255,group B average= 100.13663565328969
However, when writing the histogram as follows, the distribution is clearly different. Group B has a wider mountain hem than group A and feels gentle (it is natural because the standard deviation value is different).
import seaborn as sns
sns.distplot(groupA, bins=100, label='groupA', kde=False)
sns.distplot(groupB, bins=100, label='groupB', kde=False)
plt.legend()
plt.show()
In this way, when comparing only the average, Group A and Group B seem to be equivalent groups, but the information that group B has a larger variation in value than group A is discarded. The summary statistics that express these variations in numbers are the variance and standard deviation.
- Since the mean, median, mode, and quartile are summary statistics, the variance and standard deviation are not the only names.
Distributed
The variance can be calculated with the following formula
S^2 = \frac{1}{n}{\sum_{i=1}^n(x_i-\bar{x})^2}
In other words, (each value-mean value) squared is added together and divided by the number of data.
By subtracting the average value from each value, it seems to be an index showing how much it deviates from the average, but it may be a negative value, so it is squared.
After that, by adding them up and dividing by the number of data, the degree of variation in the data can be expressed.
By the way, the result of each value-mean value is the deviation, and the value obtained by squaring it and adding them all is called the sum of squared deviations.
Writing in python looks like this.
deviation
groupA[0] - groupA.mean()
Deviation square sum
s = np.sum((xi - groupA.mean())**2 for xi in groupA)
print(s)
40178555.707617663
Distributed
var = sum / len(groupA)
print(var)
401.78555707618096
groupA,B Try to get the variance of both
a = ((groupA - groupA.mean())**2).sum()/len(groupA)
b = ((groupB - groupB.mean())**2).sum()/len(groupB)
print("Distribution of groupA:{:.2f}\distribution of ngroupB:{:.2f}".format(a, b))
Distribution of groupA:401.79
groupB distribution:2496.21
So, we can see the variance of each, group B has a larger variance than A, and the numerical values are not concentrated near the mean value. In other words, it can be seen that there are variations.
standard deviation
I found that B was more variable, but I'm not sure if it was said that the variance was 2496, which was a set of numbers with an average of around 100 for both group A and group B. In such a case, it is better to give the standard deviation.
S=\sqrt{S^2}
The standard deviation is the square root of the variance. Since it was squared when calculating the variance, it returns to the original dimension by taking the square root here. Therefore, it becomes easier to understand how much the numbers vary.
standard deviation
print("standard deviation of groupA:{:.2f}\standard deviation of ngroupB:{:.2f}".format(math.sqrt(a), math.sqrt(b)))
standard deviation of groupA:20.04
group B standard deviation:49.96
Finally
numpy is so convenient that you can easily calculate the mean, variance, and standard deviation.
mean = groupA.mean()
var = groupA.var()
std = groupA.std()
print("average:{:.2f}Distributed:{:.2f}standard deviation:{:.2f}".format(mean, var, std))
average:100.09 Dispersion:401.79 standard deviation:20.04
Recommended Posts