It is said that the netstat command became faster after a slight modification of the Linux kernel and the netstat command.
--The netstat command refers to a special file [^ about-procfs] called / proc / net / tcp to get information about TCP sessions from the kernel.
--In IPv4, it is divided into / proc / net / tcp, and in IPv6, it is divided into / proc / net / tcp6. To do
[^ about-procfs]: I won't go into detail about procfs itself here. See man proc etc. as appropriate.
--Procfs internally uses a mechanism called the seq file system
--The seq file system is summarized here About the seq file system --Qiita
--When you execute a read system call for a special file that uses the seq file system, it goes through a buffer with the same capacity as the value of PAGE_SIZE (4096 bytes in many environments) normally reserved on the kernel side. , Read Copy data to the buffer area on the user space side specified in the system call
--In other words, no matter how large the capacity is specified in the buffer area on the user space side ([^ about-read] in the second and third arguments of the read system call), data can only be read with a capacity of 4096 bytes or less. Means
[^ about-read]: See man 2 read etc. as appropriate for the arguments of the read system call.
--In an environment that handles a large number of high-frequency TCP connections, the capacity of / proc / net / tcp can be several MiByte, but the read system call can only read data with a capacity of 4096Byte or less, and / Hundreds of read system calls are required to read the entire contents of proc / net / tcp
--In / proc / net / tcp, if one TCP session = 1 line is 150Byte and 10,000 sessions are held (including TIME_WAIT), the file size will be around 1.5MiByte and read system call will be 300. Will be executed more than once
--Actually, it is rounded to a well-separated capacity in units of lines and can read only a little less than 4096 bytes, so the number of times will increase further.
--If you modify the part that processes / proc / net / tcp in the Linux kernel and the netstat command to increase the amount of data read by one read system call, the number of read system call calls decreases. And the netstat command is faster
--The subsequent operations, program creation, and operation check were performed in the latest environment of CentOS 7 at the time of writing this article.
$ uname -r
3.10.0-693.17.1.el7.x86_64
$ rpm -q kernel
kernel-3.10.0-693.11.6.el7.x86_64
kernel-3.10.0-693.17.1.el7.x86_64
$ rpm -q net-tools
net-tools-2.0-0.22.20131004git.el7.x86_64
$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
$ LANG=C lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 2
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
Stepping: 2
CPU MHz: 2394.456
BogoMIPS: 4788.91
Virtualization: VT-x
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 30720K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm tpr_shadow vnmi ept vpid fsgsbase bmi1 avx2 smep bmi2 erms xsaveopt
$ free -m
total used free shared buff/cache available
Mem: 7966 416 6876 8 673 7205
Swap: 0 0 0
--Install and configure various items required to build the RPM package
sudo useradd mockbuild
sudo yum -y install rpm-build
sudo yum -y groups install "Development Tools"
--Install the packages required at build time
yumdownloader --source kernel
rpm -ivh kernel-3.10.0-693.17.1.el7.centos.plus.src.rpm
sudo yum -y install m4 gcc xmlto asciidoc hmaccalc python-devel newt-devel 'perl(ExtUtils::Embed)' pesign elfutils-devel zlib-devel binutils-devel bison audit-libs-devel java-devel openssl-devel numactl-devel pciutils-devel ncurses-devel
--Create a patch with the following contents as ~ / rpmbuild / SOURCES / proc-net-tcp.patch
--- a/net/ipv4/tcp_ipv4.c 2018-01-14 15:02:54.000000000 +0000
+++ b/net/ipv4/tcp_ipv4.c 2018-02-10 06:08:39.043951339 +0000
@@ -2382,10 +2382,21 @@
return 0;
}
+static ssize_t tcp_seq_read(struct file *file, char __user *buf, size_t size, loff_t *ppos)
+{
+ struct seq_file *m = file->private_data;
+ if (!m->buf) {
+ m->buf = kmalloc( m->size = 16 * PAGE_SIZE, GFP_KERNEL);
+ if (!m->buf)
+ return -ENOMEM;
+ }
+ return seq_read(file, buf, size, ppos);
+}
+
static const struct file_operations tcp_afinfo_seq_fops = {
.owner = THIS_MODULE,
.open = tcp_seq_open,
- .read = seq_read,
+ .read = tcp_seq_read,
.llseek = seq_lseek,
.release = seq_release_net
};
--- a/net/ipv6/tcp_ipv6.c 2018-01-14 15:02:54.000000000 +0000
+++ b/net/ipv6/tcp_ipv6.c 2018-02-10 06:10:39.910935928 +0000
@@ -1812,10 +1812,21 @@
return 0;
}
+static ssize_t tcp_seq_read(struct file *file, char __user *buf, size_t size, loff_t *ppos)
+{
+ struct seq_file *m = file->private_data;
+ if (!m->buf) {
+ m->buf = kmalloc( m->size = 16 * PAGE_SIZE, GFP_KERNEL);
+ if (!m->buf)
+ return -ENOMEM;
+ }
+ return seq_read(file, buf, size, ppos);
+}
+
static const struct file_operations tcp6_afinfo_seq_fops = {
.owner = THIS_MODULE,
.open = tcp_seq_open,
- .read = seq_read,
+ .read = tcp_seq_read,
.llseek = seq_lseek,
.release = seq_release_net
};
--Modified SPEC file to apply the above patch at build time
$ diff ~/rpmbuild/SPECS/kernel.spec.org ~/rpmbuild/SPECS/kernel.spec
8c8
< %define buildid .centos.plus
---
> %define buildid .procnettcp
463a464,465
> Patch50000: proc-net-tcp.patch
>
832a835,836
>
> ApplyPatch proc-net-tcp.patch
――The time required in the environment used this time is over 1.5 hours
rpmbuild -ba ~/rpmbuild/SPECS/kernel.spec
find ~/rpmbuild/ -name "*.rpm"
--After installing the RPM package obtained by building and rebooting, you can now read to / proc / net / tcp in 64KiByte increments.
--For the cat command that reads the contents of / proc / net / tcp, use the strace command to observe the read system call.
$ uname -r
3.10.0-693.17.1.el7.procnettcp.x86_64
$ LANG=C strace -e open,read cat /proc/net/tcp|wc -l
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832
open("/proc/net/tcp", O_RDONLY) = 3
read(3, " sl local_address rem_address "..., 65536) = 11700
read(3, "", 65536) = 0
78
+++ exited with 0 +++
--The result of executing the same command under the environment of the Linux kernel before modification is as follows. --Read You can see that even though the same 64KiByte is specified as the third argument of the system call, less than 4096Byte of data can be read.
$ uname -r
3.10.0-693.17.1.el7.x86_64
$ LANG=C strace -e open,read cat /proc/net/tcp|wc -l
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832
open("/proc/net/tcp", O_RDONLY) = 3
read(3, " sl local_address rem_address "..., 65536) = 4050
read(3, " 26: 0502010A:861A 10813FA8:005"..., 65536) = 1650
read(3, "", 65536) = 0
38
+++ exited with 0 +++
--Observe the read system call with the strace command for the netstat command in the modified kernel environment. --Since only 4096Byte is passed when reading, it has little effect.
$ uname -r
3.10.0-693.17.1.el7.procnettcp.x86_64
$ LANG=C strace -e open,read netstat -atn |wc -l
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300j\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\16\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0m\0\0\0\0\0\0"..., 832) = 832
open("/proc/net/tcp", O_RDONLY) = 3
read(3, " sl local_address rem_address "..., 4096) = 4096
read(3, "00000000 03:00001608 00000000 "..., 4096) = 4096
read(3, " 3 ffff880287e55d00 "..., 4096) = 4096
read(3, " \n", 4096) = 12
read(3, "", 4096) = 0
open("/proc/net/tcp6", O_RDONLY) = 3
read(3, " sl local_address "..., 4096) = 853
read(3, "", 4096) = 0
+++ exited with 0 +++
87
--About the netstat command in the kernel environment before modification, the state of the read system call with the strace command is as follows --You can see that only 4050Byte can be read for the buffer size 4096Byte specified in the read system call.
$ uname -r
3.10.0-693.17.1.el7.x86_64
$ LANG=C strace -e open,read netstat -atn |wc -l
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300j\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\16\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0m\0\0\0\0\0\0"..., 832) = 832
open("/proc/net/tcp", O_RDONLY) = 3
read(3, " sl local_address rem_address "..., 4096) = 4050
read(3, " 26: 0502010A:870C 10813FA8:005"..., 4096) = 4050
read(3, " 53: 0502010A:E488 19E57328:01B"..., 4096) = 900
read(3, "", 4096) = 0
open("/proc/net/tcp6", O_RDONLY) = 3
read(3, " sl local_address "..., 4096) = 676
read(3, "", 4096) = 0
+++ exited with 0 +++
64
yumdownloader --source net-tools
rpm -ivh net-tools-2.0-0.22.20131004git.el7.src.rpm
sudo yum -y install libselinux-devel
--Create a patch with the following contents as ~ / rpmbuild / SOURCES / net-tools-netstat-proc-net-tcp-buffer-size.patch
--Changed to allocate buffer area with 16 times the page size
--- a/lib/proc.c 2013-09-30 08:07:32.000000000 +0000
+++ b/lib/proc.c 2018-02-11 04:48:40.959339035 +0000
@@ -88,9 +88,9 @@
if (!buffer) {
pagesz = getpagesize();
- buffer = malloc(pagesz);
+ buffer = malloc(16*pagesz);
}
- setvbuf(fd, buffer, _IOFBF, pagesz);
+ setvbuf(fd, buffer, _IOFBF, 16*pagesz);
return fd;
}
--Modified SPEC file to apply the above patch at build time
$ diff ~/rpmbuild/SPECS/net-tools.spec.org ~/rpmbuild/SPECS/net-tools.spec
67a68,69
> Patch100: net-tools-netstat-proc-net-tcp-buffer-size.patch
>
104a107,108
>
> %patch100 -p1 -b .proc-net-tcp
--Do not install the RPM package obtained by building for comparison with the netstat command before modification.
rpmbuild -bb ~/rpmbuild/SPECS/net-tools.spec
ls -l ~/rpmbuild/BUILD/net-tools/netstat
--Observe the read system call with the strace command for the netstat command before modification.
--Data is read by 4096 bytes for read after opening / proc / net / tcp.
$ LANG=C strace -e open,read netstat -atn |wc -l
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300j\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\16\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0m\0\0\0\0\0\0"..., 832) = 832
open("/proc/net/tcp", O_RDONLY) = 3
read(3, " sl local_address rem_address "..., 4096) = 4096
read(3, "00000000 03:0000022B 00000000 "..., 4096) = 4096
read(3, " 3 ffff8802c43a7d00 "..., 4096) = 3508
read(3, "", 4096) = 0
open("/proc/net/tcp6", O_RDONLY) = 3
read(3, " sl local_address "..., 4096) = 853
read(3, "", 4096) = 0
+++ exited with 0 +++
83
--Observe the read system call with the strace command for the modified netstat command. --65536 = 64KiByte is set as the third argument of the read system call, and a capacity larger than 4096Byte can be read by one read system call.
$ LANG=C strace -e open,read ~/rpmbuild/BUILD/net-tools-2.0/netstat -atn |wc -l
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300j\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0"..., 832) = 832
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\16\0\0\0\0\0\0"..., 832) = 832
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0m\0\0\0\0\0\0"..., 832) = 832
open("/proc/net/tcp", O_RDONLY) = 3
read(3, " sl local_address rem_address "..., 65536) = 11700
read(3, "", 65536) = 0
open("/proc/net/tcp6", O_RDONLY) = 3
read(3, " sl local_address "..., 65536) = 853
read(3, "", 65536) = 0
+++ exited with 0 +++
83
--Use Apache and the benchmark tool ab command that comes with Apache to increase the number of TCP sessions --Compare the behavior of the netstat command before and after modification while executing the one-liner that installs and starts Apache and repeatedly executes the ab command
sudo yum -y install httpd
sudo systemctl start httpd.service
while : ; do ab -n 100000 -c 1000 127.0.0.1/test; done
--In the measurement with the time command, it took more than 300 milliseconds with the conventional netstat command when the number of TCP sessions was over 16000, but with the modified netstat command, it took less than half the time to 100 milliseconds. It became so --Especially system time is significantly reduced
$ time netstat -atn |wc -l
16528
real 0m0.355s
user 0m0.110s
sys 0m0.254s
$ time ~/rpmbuild/BUILD/net-tools-2.0/netstat -atn |wc -l
16202
real 0m0.133s
user 0m0.115s
sys 0m0.027s
--By the way, the ss command, which is often used for the same purpose as the netstat command, usually has a different mechanism from the reference of [^ ss-using-procfs] / proc / net / tcp (uses a special socket". The number of TCP sessions is acquired by netlink "), and processing can be performed faster than the above modified netstat command.
[^ ss-using-procfs]: If the acquisition of information by netlink fails, the information will be acquired using procfs.
$ time ss -atn |wc -l
15922
real 0m0.046s
user 0m0.033s
sys 0m0.021s
Considering that the netstat command or net-tools is being regarded as obsolete, it is better to use the ss command obediently.
Recommended Posts