Docker volume performance tuning

-Volumes mounted with Docker Desktop for Mac are said to have low access performance. -It can be improved by using the mount option introduced from Docker 17.03.

Why it's slow in the first place

Since Linux is based on VFS between the host and the container, there is no overhead due to file synchronization. On macOS there is significant overhead for full consistency between the host and the container.

Does it mean that it only takes time to synchronize when updating a file, and it does not take time to read it?

Tuning policy

Being perfectly consistent is slow with overhead. There are many cases where consistency is not necessary. Therefore, performance is improved by sacrificing consistency.

Mount settings

By specifying the mounting method when specifying the mount with Docker, you can decide whether to take consistency or performance.

· Consistent: Fully synchronized between host and container. -Cached: When the host is updated, it is delayed and reflected in the container. -Delegated: Delayed when updating the container and reflected on the host

It can be used by adding: to the end of the -v option container path and adding the above parameters.

Example


$ docker run -v /host/hoge:/container/hoge:cached sample
It is also possible to specify differently for multiple volumes

$ docker run -v /host/1:/container/1:cached -v /host/2:/container/2:delegated sample

delegated The performance goes up the most. It seems good to use it for things that change unilaterally from the container side without changing the host. Even if the file status of the container is set to positive and the container is modified, it will be reflected on the host with a delay. Suitable for temporary files that are ok even if damaged and builds that can be regenerated. If you mount the location mounted by delegated with cached or consistent, the location will follow the respective specifications. Even if the change is made on the host side, the change may disappear due to synchronization on the container side.

cached Make the host positive. Host writes are immediately synced to the container, but container changes are delayed and synced to the host. If you mount the part mounted by cached with consistent, the behavior will be consistent.

consistent The host and container are perfectly synchronized. This specification by default.

Performance measurement

Mount your Laravel project and change the mount settings to measure performance.

First of all, the state where artisan serve is started directly locally. Handles 29 requests in 1 second.

$ ab -c 5 -n 25 http://127.0.0.1:8000/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient).....done


Server Software:        
Server Hostname:        127.0.0.1
Server Port:            8000

Document Path:          /
Document Length:        17473 bytes

Concurrency Level:      5
Time taken for tests:   0.855 seconds
Complete requests:      25
Failed requests:        0
Total transferred:      465475 bytes
HTML transferred:       436825 bytes
Requests per second:    29.22 [#/sec](mean)
Time per request:       171.097 [ms](mean)
Time per request:       34.219 [ms](mean, across all concurrent requests)
Transfer rate:          531.35 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    43  148  35.2    161     179
Waiting:       41  147  35.4    159     177
Total:         43  149  35.2    161     179

Percentage of the requests served within a certain time (ms)
  50%    161
  66%    162
  75%    164
  80%    166
  90%    172
  95%    174
  98%    179
  99%    179
 100%    179 (longest request)

If you mount it on the container side with/app,

$ tree -L 2
.
├── Dockerfile
└── laravel
    ├── README.md
    ├── app
    ├── artisan
    ├── bootstrap
    ├── composer.json
    ├── composer.lock
    ├── config
    ├── database
    ├── docker-compose.yml
    ├── package.json
    ├── phpunit.xml
    ├── public
    ├── resources
    ├── routes
    ├── server.php
    ├── storage
    ├── tests
    ├── vendor
    └── webpack.mix.js
FROM centos:centos8

RUN dnf module install -y php:7.4

WORKDIR /app
CMD [ "php", "artisan", "serve", "--port=80", "--host=0.0.0.0" ]

Mount

$ docker run -it --rm -v $(pwd)/laravel:/app/ -p 8080:80 ro_test 

Performance when 5 people access 5 times in this state for a total of 25

$ ab -c 5 -n 25 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        17473 bytes

Concurrency Level:      5
Time taken for tests:   8.982 seconds
Complete requests:      25
Failed requests:        0
Total transferred:      465475 bytes
HTML transferred:       436825 bytes
Requests per second:    2.78 [#/sec](mean)
Time per request:       1796.408 [ms](mean)
Time per request:       359.282 [ms](mean, across all concurrent requests)
Transfer rate:          50.61 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   373 1578 436.6   1755    1863
Waiting:      373 1578 436.7   1755    1862
Total:        373 1578 436.6   1755    1863

Percentage of the requests served within a certain time (ms)
  50%   1752
  66%   1769
  75%   1778
  80%   1804
  90%   1810
  95%   1825
  98%   1863
  99%   1863
 100%   1863 (longest request)

Performance cached

Since it is cached, performance should improve a little

$ docker run -it --rm -v $(pwd)/laravel:/app/:cached -p 8080:80 ro_test 

measurement. It doesn't change much. .. ..

$ ab -c 5 -n 25 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        17473 bytes

Concurrency Level:      5
Time taken for tests:   8.948 seconds
Complete requests:      25
Failed requests:        0
Total transferred:      465475 bytes
HTML transferred:       436825 bytes
Requests per second:    2.79 [#/sec](mean)
Time per request:       1789.501 [ms](mean)
Time per request:       357.900 [ms](mean, across all concurrent requests)
Transfer rate:          50.80 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   353 1571 435.7   1750    1813
Waiting:      353 1571 435.8   1749    1813
Total:        353 1571 435.7   1750    1814

Percentage of the requests served within a certain time (ms)
  50%   1749
  66%   1762
  75%   1777
  80%   1791
  90%   1795
  95%   1812
  98%   1814
  99%   1814
 100%   1814 (longest request)

Performance delegated

$ docker run -it --rm -v $(pwd)/laravel:/app/:delegated -p 8080:80 ro_test 

measurement. This doesn't change much either. It's rather late.

$ ab -c 5 -n 25 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        17473 bytes

Concurrency Level:      5
Time taken for tests:   9.187 seconds
Complete requests:      25
Failed requests:        0
Total transferred:      465475 bytes
HTML transferred:       436825 bytes
Requests per second:    2.72 [#/sec](mean)
Time per request:       1837.459 [ms](mean)
Time per request:       367.492 [ms](mean, across all concurrent requests)
Transfer rate:          49.48 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:   391 1582 403.0   1740    1843
Waiting:      391 1581 402.9   1739    1843
Total:        391 1582 403.0   1740    1843

Percentage of the requests served within a certain time (ms)
  50%   1739
  66%   1753
  75%   1765
  80%   1770
  90%   1841
  95%   1842
  98%   1843
  99%   1843
 100%   1843 (longest request)

Put the code inside the container

FROM centos:centos8

RUN dnf module install -y php:7.4

COPY ./laravel /app

WORKDIR /app
CMD ["php", "artisan", "serve", "--host=0.0.0.0"]

measurement. This is the fastest way to use a container. Handles 15 requests per second.

ab -c 5 -n 25 http://localhost:8080/  
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient).....done


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        17473 bytes

Concurrency Level:      5
Time taken for tests:   1.682 seconds
Complete requests:      25
Failed requests:        0
Total transferred:      465475 bytes
HTML transferred:       436825 bytes
Requests per second:    14.86 [#/sec](mean)
Time per request:       336.380 [ms](mean)
Time per request:       67.276 [ms](mean, across all concurrent requests)
Transfer rate:          270.27 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    94  295  70.8    316     355
Waiting:       93  294  71.0    315     354
Total:         94  295  70.7    316     355

Percentage of the requests served within a certain time (ms)
  50%    315
  66%    328
  75%    331
  80%    333
  90%    345
  95%    348
  98%    355
  99%    355
 100%    355 (longest request)

Write performance

Check performance when writing files

When measured locally

$ time (for i in $(seq 1 10000);do echo $i >> a.txt; done)
========================
Program : ( for i in $(seq 1 10000); do; echo $i >> a.txt; done; )
CPU     : 88%
user    : 0.110s
system  : 0.550s
total   : 0.746s
========================

When mounted consistently. 0.220s when writing in a container. very late. 0.119s when written on the host. I thought it would take about the same, but the container is slower.

$ docker run -it --rm -v $(pwd)/fuga:/fuga centos:centos8

[root@29e3d05941ce fuga]# time (for i in $(seq 1 10000);do echo $i >> a.txt; done)

real	0m11.249s
user	0m0.220s
sys	0m0.700s

$ time (for i in $(seq 1 10000);do echo $i >> b.txt; done)

========================
Program : ( for i in $(seq 1 10000); do; echo $i >> b.txt; done; )
CPU     : 88%
user    : 0.119s
system  : 0.608s
total   : 0.817s
========================

Write performance cached

$ docker run -it --rm -v $(pwd)/fuga:/fuga:cached centos:centos8

[root@53b5e0d31c60 fuga]# time (for i in $(seq 1 10000);do echo $i >> aa.txt; done)

real	0m13.132s
user	0m0.271s
sys	0m0.804s

↓ Host side. It's a little faster than when it's consistent

time (for i in $(seq 1 10000);do echo $i >> ab.txt; done)


========================
Program : ( for i in $(seq 1 10000); do; echo $i >> ab.txt; done; )
CPU     : 89%
user    : 0.119s
system  : 0.597s
total   : 0.800s
========================

Write performance delegated

It's faster than cached.

$ docker run -it --rm -v $(pwd)/fuga:/fuga:delegated centos:centos8
[root@e8334765081e fuga]# time (for i in $(seq 1 10000);do echo $i >> ddd.txt; done)

real	0m11.714s
user	0m0.191s
sys	0m0.780s

↓ Host side. Slower than cched ...?

$ time (for i in $(seq 1 10000);do echo $i >> affff.txt; done)

========================
Program : ( for i in $(seq 1 10000); do; echo $i >> affff.txt; done; )
CPU     : 87%
user    : 0.123s
system  : 0.616s
total   : 0.849s
========================

reference https://docs.docker.jp/docker-for-mac/osxfs-caching.html

Recommended Posts

Docker volume performance tuning
docker volume checked
Java app performance tuning
[WIP] Use NFS for Docker Volume
Microservices With Docker and Cloud Performance
docker