[Statistics for programmers] Lorenz curve and Gini coefficient
table of contents
Statistics for Programmers-Table of Contents
Overview
It is assumed that you have already read the following articles. [Statistics for programmers] Frequency distribution and histogram
What is the Lorenz curve?
It is a graph created to check income and population bias.
As an example, we will investigate the population bias of villages in a region. The total number of villages is 100. The population of each village is as follows. It's hard to see, but the total population of all villages is listed.
population = [
124, 151, 102, 189, 160, 145, 120, 132, 135, 159, 114, 175, 171, 124, 154,
177, 152, 120, 144, 121, 113, 163, 186, 196, 183, 105, 130, 149, 130, 123,
175, 143, 186, 182, 184, 174, 134, 158, 196, 109,
216, 285, 209, 288, 276, 281, 283, 200, 262,
267, 235, 206, 245, 232, 299, 249, 295, 232, 206, 237,
369,
487, 450, 487,
597, 548,
682,
712, 706, 755, 700, 709, 747, 773, 796, 739, 716, 756, 767, 752, 728, 750,
829, 875, 845, 881, 865, 804, 845, 890, 872, 833, 874, 845, 859, 837, 847, 811, 893, 807
]
To find the Lorenz curve
The Lorenz curve requires two types of cumulative relative frequencies on the X-axis and Y-axis.
- Cumulative relative frequency of frequency of each class
- Cumulative relative frequency of the total of the values belonging to each class
I will explain each of them.
1. Cumulative relative frequency of frequency of each class
The first is to calculate the cumulative relative frequency, where the class is the population and the frequency is the number of villages.
Calculate in the same way as described in this article. [Statistics for programmers] Frequency distribution and histogram
| class | Class value | frequency | Relative frequency | Cumulative relative frequency |
|---|---|---|---|---|
| Over 100 people-Less than 200 people | 150 | 40 | 0.40 | 0.40 |
| Over 200 people-Less than 300 people | 250 | 20 | 0.20 | 0.60 |
| Over 300 people-Less than 400 people | 350 | 1 | 0.01 | 0.61 |
| Over 400 people-Less than 500 people | 450 | 3 | 0.03 | 0.64 |
| Over 500 people-Less than 600 people | 550 | 2 | 0.02 | 0.66 |
| Over 600 people-Less than 700 people | 650 | 1 | 0.01 | 0.67 |
| Over 700 people-Less than 800 people | 750 | 15 | 0.15 | 0.82 |
| Over 800 people-Less than 900 people | 850 | 18 | 0.18 | 1.00 |
2. Cumulative relative frequency of the total of the values belonging to each class
The second is to calculate the cumulative relative frequency, where the class is the population and the frequency is the total number of villages for each class.
To get the total number of villages for each class, calculate based on the value of the list of variables named population above. For example, the total number of "400 or more-less than 500" is 1424 as shown below.
1424 = 487 + 450 + 487
Also, since the total population of all villages is 41029 (the sum of all population values), the relative frequency is In the case of "400 or more-less than 500", it will be 0.03 (rounded down to the third decimal place).
0.03 = \frac{1424}{41029}
Each class is calculated in this way, and the cumulative relative frequency of the number of people is calculated.
Frequency distribution table
It is a frequency distribution table that summarizes the first and second mentioned above. (Relative frequency and cumulative relative frequency are rounded down to the third decimal place)
| class | Class value | frequency | Relative frequency | Cumulative relative frequency of frequency | Total number of people (frequency) | Relative frequency of the number of people | Cumulative relative frequency of the number of people |
|---|---|---|---|---|---|---|---|
| Over 100 people-Less than 200 people | 150 | 40 | 0.40 | 0.40 | 5988 | 0.14 | 0.15 |
| Over 200 people-Less than 300 people | 250 | 20 | 0.20 | 0.60 | 5003 | 0.12 | 0.27 |
| Over 300 people-Less than 400 people | 350 | 1 | 0.01 | 0.61 | 369 | 0.01 | 0.28 |
| Over 400 people-Less than 500 people | 450 | 3 | 0.03 | 0.64 | 1424 | 0.03 | 0.31 |
| Over 500 people-Less than 600 people | 550 | 2 | 0.02 | 0.66 | 1145 | 0.02 | 0.34 |
| Over 600 people-Less than 700 people | 650 | 1 | 0.01 | 0.67 | 682 | 0.01 | 0.36 |
| Over 700 people-Less than 800 people | 750 | 15 | 0.15 | 0.82 | 11106 | 0.27 | 0.63 |
| Over 800 people-Less than 900 people | 850 | 18 | 0.18 | 1.00 | 15312 | 0.37 | 1.00 |
The Lorenz curve is a graph with the "cumulative relative frequency of frequency" on the x-axis and the "cumulative relative frequency of the number of people" on the y-axis.
Draw a Lorenz curve
Graph the above data using matplotlib.
Reference: Draw graph with jupyter (ipython notebook) + matplotlib + vagrant
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
fig, ax = plt.subplots()
#Abscissa,Cumulative relative frequency of frequency
left = np.array([
0,
0.40,
0.60,
0.61,
0.64,
0.66,
0.67,
0.82,
1.00,
])
#Vertical coordinates,Cumulative relative frequency of the number of people
height = np.array([
0,
0.15,
0.27,
0.28,
0.31,
0.34,
0.36,
0.63,
1.00
])
ax.plot(left, height, marker='o')
#Graph title
plt.title('Lorenz Curve')
#X-axis title
plt.xlabel('villages')
#Y-axis title
plt.ylabel('population')
left = np.array([0, 0.2, 0.4, 0.6, 0.8, 1.0])
height = np.array([0, 0.2, 0.4, 0.6, 0.8, 1.0])
ax.plot(left, height, linestyle='dashed', color='black')
#X-axis label
label = [0, 0.2, 0.4, 0.6, 0.8, 1.0]
#X-axis location to hit the label
ax.set_xticks(label)
#Set label on X axis
ax.set_xticklabels(label)
#Location of the Y-axis to hit the label
ax.set_yticks(label)
#Set label on Y axis
ax.set_yticklabels(label)
#drawing
plt.show()
This blue curve is the Lorenz curve.
What you can see by looking at the Lorenz curve
I will explain how to look at the Lorenz curve created above.
Let's look at the coordinates of the part drawn with the red line. What we can see from this point is that about 65% of the villages are only about 36% of the total population. This means that the remaining 64% of the population is biased towards some other village. Also, the larger the bulge of the curve, the more biased it is.
Gini coefficient
The numerical value of this bias is called the Gini coefficient. Regarding how to calculate the Gini coefficient, first, the broken line in the above graph is called the "perfect equality line". The Gini coefficient is obtained by doubling the area between this "perfect equality line" and the Lorenz curve.
The Gini coefficient is a value from 0 to 1, and the closer it is to 1, the greater the bias, and the closer it is to 0, the less the bias.
By the way, when the Gini coefficient is 0, the Lorenz curve overlaps the perfect equality line.
Calculate the Gini coefficient
Actually calculate the Gini coefficient. Here, for ease of calculation, the following values are used instead of the above example.
X axis
0, 0.4, 0.8, 1
Y axis
0, 0.2, 0.6, 1.0
From this value, the following Lorenz curve is created.
The Gini coefficient can be obtained by dividing this graph as shown below and subtracting the area of parts 1, 2 and 3 in the figure from the triangle with the hypotenuse as the hypotenuse and doubling it.
First, find the area of the triangle with the hypotenuse of the perfect equality line.
0.5 = 1\times1\div2
Find the area of part 1. Since it is a right triangle with a base of 0.4 and a height of 0.2, it can be calculated as follows.
0.04 = 0.4\times0.2\div2
Find the area of part 2. When rotated to the right, it becomes a trapezoid with an upper base of 0.2, a lower base of 0.6, and a height of 0.4, which can be calculated as follows.
0.16 = (0.2+0.6)\times0.4\div2
Find the area of part 3. If you rotate it to the right, it will become a trapezoid with an upper base of 0.6, a lower base of 1, and a height of 0.2, so you can find it below.
0.16 = (0.6+1)\times0.2\div2
If you subtract these three values from the area of the triangle whose hypotenuse is the perfect equality line and double them, you will get the Gini coefficient, so 0.28 is the Gini coefficient as shown in the formula below.
0.28 = (0.5 - 0.04 - 0.16 - 0.16)\times2
that's all
reference
-Statistics WEB --Lorenz curve -Lorenz curve and Gini coefficient-Mathematics learned with concrete examples -Calculate the Gini coefficient with Python
Recommended Posts