The story that Apache dealt with because it was down at AH00144

At midnight, I was informed that the production system was down. The BtoB service website could not be opened. I shared the information because I urgently dealt with the case that I had to solve it by the business hour of the next day by struggling alone.

environment

A website that runs on Ubuntu 16.04 on Amazon EC2.

$ cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.3 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.3 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

$ apache2 -v
Server version: Apache/2.4.18 (Ubuntu)

Error log

When I logged in to the server, Apache was dead with an error. When you see the string ** Fatal error **, you'll instantly get sick.

$ tail /var/log/apache2/error.log
[Tue Aug 25 20:51:54.215281 2020] [mpm_prefork:emerg] [pid 930](43)Identifier removed: AH00144: couldn't grab the accept mutex
[Tue Aug 25 20:51:54.215369 2020] [mpm_prefork:emerg] [pid 929](43)Identifier removed: AH00144: couldn't grab the accept mutex
[Tue Aug 25 20:51:54.216849 2020] [mpm_prefork:emerg] [pid 933](43)Identifier removed: AH00144: couldn't grab the accept mutex
[Tue Aug 25 20:51:54.217182 2020] [mpm_prefork:emerg] [pid 931](43)Identifier removed: AH00144: couldn't grab the accept mutex
[Tue Aug 25 20:51:54.218350 2020] [mpm_prefork:emerg] [pid 928](43)Identifier removed: AH00144: couldn't grab the accept mutex
[Tue Aug 25 20:51:54.218757 2020] [mpm_prefork:emerg] [pid 1336](43)Identifier removed: AH00144: couldn't grab the accept mutex
[Tue Aug 25 20:51:55.040404 2020] [core:alert] [pid 1287] AH00050: Child 928 returned a Fatal error... Apache is exiting!
[Tue Aug 25 20:58:05.444208 2020] [mpm_prefork:emerg] [pid 1333](22)Invalid argument: AH00146: couldn't release the accept mutex

ʻAH00144: couldn't grab the accept mutex` is output for the number of processes.

For the time being, when I googled with ** AH00144 **, https://hirose31.hatenablog.jp/entry/2020/03/10/190010 I found an excellent blog! Even in this environment, due to various reasons, Apache was started as a non-system user (uid is 1000 or more). The reproduction method also matches perfectly.

Coping

The procedure is simple.

/etc/apache2/apache2.conf

#Mutex file:${APACHE_LOCK_DIR} default

I just uncommented and rebooted. You may want to create a file like /etc/apache2/conf-available/mutex-file.conf.

Why are you curious about this time? However, I would like to see the situation for a while.