Take your own peak memory usage on Linux & Python

Conclusion

I searched a lot, but I couldn't find a better way. Since it uses the / proc file system, it only works on supported platforms (such as Linux).

import os
import re

def peak_memory():
    pid = os.getpid()
    with open(f'/proc/{pid}/status') as f:
        # "VmHWM:    862168 kB"Extract lines in the format of
        for line in f:
            if not line.startswith('VmHWM:'):
                continue
            return int(re.search('[0-9]+', line)[0])
    raise ValueError('Not Found')

Why peak usage?

Inside a certain function, a phenomenon occurred in which the memory usage temporarily increased.

If you want to profile the function and find out where the memory usage is rising, you should use memory-profiler. This time, I knew where it would increase, and I couldn't help it, but I wanted to find out how much it would increase, so I needed to know the peak usage.

What happens if you don't use peak usage?

You can get the current memory usage with psutil. See this.

However, it is not a peak, so for example, in the case of such a code

Take current memory usage

import psutil

def f():
    a = [0] * 20000000

print(psutil.Process().memory_full_info())
f()
print(psutil.Process().memory_full_info())

The result is as follows.

Not much different

pfullmem(rss=12406784, vms=18395136, shared=6369280, text=4096, lib=0, data=6225920, dirty=0, uss=8134656, pss=9319424, swap=0)
pfullmem(rss=12619776, vms=18354176, shared=6385664, text=4096, lib=0, data=6184960, dirty=0, uss=8253440, pss=9438208, swap=0)

I'm using a lot of memory inside f (), but I don't use it outside, so I can't tell from the outside.

Take it in f

import psutil

def f():
    a = [0] * 20000000
    print('inner: ', psutil.Process().memory_full_info())

print('before:', psutil.Process().memory_full_info())
f()
print('after: ', psutil.Process().memory_full_info())

In this case, it looks like this:

I use a lot only in the middle

before: pfullmem(rss=12476416, vms=18395136, shared=6443008, text=4096, lib=0, data=6225920, dirty=0, uss=8179712, pss=9407488, swap=0)
inner:  pfullmem(rss=172519424, vms=178356224, shared=6520832, text=4096, lib=0, data=166187008, dirty=0, uss=168300544, pss=169528320, swap=0)
after:  pfullmem(rss=12754944, vms=18354176, shared=6520832, text=4096, lib=0, data=6184960, dirty=0, uss=8298496, pss=9526272, swap=0)

If you could put code that pinpoints memory usage where you're likely to be using memory, that's fine, but I didn't want to.

What happened with this method?

Take peak usage

import os
import re

def peak_memory():
    pid = os.getpid()
    with open(f'/proc/{pid}/status') as f:
        for line in f:
            if not line.startswith('VmHWM:'):
                continue
            return int(re.search('[0-9]+', line)[0])
    raise ValueError('Not Found')

def f():
    a = [0] * 20000000

print('before:', peak_memory(), 'KB')
f()
print('after: ', peak_memory(), 'KB')

As a result, it became like this.

Is increasing after

before: 9072 KB
after:  165532 KB

I'm happy.

References

Measure the memory consumption of Linux programs

About the code

Feel free to use the code in this article for Python scripts that you or your company is developing.