Intuitive understanding of Jensen's inequality

This is an article that tries to intuitively understand Jensen's inequality regarding random variables using examples.

If $ x $ is a random variable and $ p (x) $ is a probability density function of x, the expected value $ E [x] $ is

E[x] = \int xp(x)dx

It is expressed as. At this time, for ** upwardly convex function </ u> ** $ f (x) $,

f(E[x]) \ge E[f(x)]

Is called the Jensen's inequality. This proof has already been explained in various places (for example, here), so it is omitted here. I will.

To intuitively understand this inequality $ f (E [x]) \ ge E [f (x)] $, let's graphically represent an example using random numbers.

First, let's assume that x is a random variable that follows a normal distribution, and create a random number generated from it. Also, convert that x with an upwardly convex function of $ f (x) =-x ^ 2 + 10 $. The histogram at the top of the graph below is the distribution of x that follows a normal distribution, and the histogram on the right is the distribution that $ x ^ 2 $ follows. In other words, Jensen's inequality is ** greener than the ** red circle ** below (after taking the expected value, that is, taking the average of the histogram above and then converting with $ f (x) $). It shows that it is larger than the circle ** (converted with $ f (x) $ and then take the expected value, that is, the average value of the histogram on the right).

jensen_explanation.png

The following is an animation of the average of the normal distribution, which is the distribution of x, shifted. In each case you can see that the green circle is below the red circle.

jensen.gif

What is this inequality good for? ?? ??

Jensen's inequality

f(E[x]) \ge E[f(x)]

Wants to maximize $ f (E [x]) $, but when it is unclear what this function is, $ E [f (x)] $ if it can be calculated. Since E [f (x)] $ can be treated as the lower limit of $ f (E [x]) $, by maximizing the computable $ E [f (x)] $, the original target $ It is possible to maximize f (E [x]) $.

Often used is because $ \ log (\ cdot) $ is an upwardly convex function

log \int p(x)f(x)dx \ge \int p(x) log f(x)dx

Like, $ \ log (\ cdot) $ is put in the integral so that it can be calculated.

It's a little confusing, but here's the animation for $ \ log (\ cdot) $. If you look at the histogram on the right side, it is distorted downwards, so you can feel that the average value shifts downwards. It can be seen that the green circle is below the red circle by that amount. jensen_log.gif

reference

1.8 Random variable inequalities 1 http://mcm-www.jwu.ac.jp/~konno/pdf/statg-1-8r.pdf

Python code that describes the graph used in this article https://github.com/matsuken92/Qiita_Contents/blob/master/General/Jensens-inequality.ipynb

Recommended Posts

Intuitive understanding of Jensen's inequality
elasticsearch_dsl Memorandum of Understanding
Complete understanding of numpy.pad functions
Full understanding of Python debugging
Understanding memo of collective intelligence programming
[Python] Understanding the potential_field_planning of Python Robotics