When I tried to support IPv6 easily with Docker-proxy, I couldn't do it before I knew it.

(Memorandum for myself)

Thing you want to do

I want to use Docker proxy on a dual stack Docker host to provide IPv6 services in an IPv4 single stack Docker container.

Common environment

Ubuntu 20.04

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"

# uname -a
Linux server 5.8.0-36-generic #40~20.04.1-Ubuntu SMP Wed Jan 6 10:15:55 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Expected behavior

Start an appropriate Docker container with docker-compose as shown below, and perform port forwarding with ports.

version: '3'
services:
  grafana:
    image: grafana/grafana
    ports:
    - 3000:3000

After starting the container, the docker-proxy process that performs the corresponding port forwarding is started at the same time.

root      120970  0.0  0.0 475320  3912 ?        Sl   18:27   0:00 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 3000 -container-ip 172.18.0.4 -container-port 3000

--IPv4 traffic The following DNAT rules will appear in iptables, so they will be processed before reaching the docker-proxy process.

# iptables -t nat -L DOCKER
Chain DOCKER (2 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere
DNAT       tcp  --  anywhere             anywhere             tcp dpt:3000 to:172.18.0.3:3000

By the way, the return packet is IP masqueraded by POSTROUTING.

# iptables -t nat -L POSTROUTING 

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
MASQUERADE  all  --  172.17.0.0/16        anywhere
MASQUERADE  all  --  172.18.0.0/16        anywhere
MASQUERADE  tcp  --  172.18.0.3           172.18.0.3           tcp dpt:3000
MASQUERADE  tcp  --  172.18.0.4           172.18.0.4           tcp dpt:8888

--IPv6 traffic In the case of IPv6, unlike IPv4, DNAT rules are not inserted, and the above docker-proxy process performs conversion at the L4 level.

Actual behavior

So far

Confirmed version: Docker version 19.03.14, build 5eb3275d40

It can be seen that Type is IPv6 and listens on both IPv4/IPv6.

reference: A trap that can be accessed with IPv4 even if the address port LISTENed by netstat or lsof is displayed as IPv6

# lsof -i:3000
COMMAND      PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
docker-pr 120970 root    4u  IPv6 1065660      0t0  TCP *:3000 (LISTEN)

Recently

Confirmed version: Docker version 20.10.2, build 2291f61

Type is IPv4, and only listens on IPv4. ** Therefore, the service cannot be provided by IPv6 **

# lsof -i:3000
COMMAND     PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
docker-pr 14845 root    4u  IPv4 6231154      0t0  TCP *:3000 (LISTEN)

Workaround

If you set the Docker version to less than 19.0 (probably unverified), it will return to the previous behavior. Refer to the official documentation and install older versions of docker-ce and docker-ce-cli.

Install Docker Engine

--Check the version of docker-ce that can be installed

# apt-cache madison docker-ce
 docker-ce | 5:20.10.2~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
 docker-ce | 5:20.10.1~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
 docker-ce | 5:20.10.0~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages
 docker-ce | 5:19.03.14~3-0~ubuntu-focal | https://download.docker.com/linux/ubuntu focal/stable amd64 Packages

--Uninstall new docker-ce

# apt-get remove docker docker-engine docker.io containerd runc

--Install old docker-ce, docker-ce-cli

# apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io

Impressions and future

We haven't investigated how such a specification change was made, but ** it's really troublesome **

In the first place, IPv4 is converted by DNAT of iptables, so proxy is not required, so why does docker-proxy listen with TYPE: IPv4 ... What if listen with TYPE: IPv6? Can provide services with IPv4 ...