Linux Policy Base Routing
When building EC2 on AWS, packets may not reach EC2 with multiple NICs in the following situations, so this is a memorandum. As a configuration, the EC2 instance has multiple NICs, each belonging to a different subnet.
(Quoted from the figure) http://blog.serverworks.co.jp/tech/2018/02/16/rhel-pbr-setting/
The routing table (main) has the following situation. The default GW is a router on the subnet (10.0.1.0/24).
[root@ip-10-0-1-5 etc]# ip route show table main
10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.5
10.0.2.0/24 dev eth1 proto kernel scope link src 10.0.2.5
default via 10.0.1.1 dev eth0
In this situation, I tried to ping the 10.0.0.2.x NIC (eth1) from the NW belonging to 10.0.1.0/24, but there is no return packet. As a result of investigating why, PING has arrived at the EC2, but when returning the return packet, it is sending a packet from the default GW of the EC2, and as a result it does not reach the source. It was a thing.
For the same communication, communication in which the receiving NIC and the returning NIC are different is called asymmetric routing. In Linux, from some version, the following kernel parameters are set to drop packets if the NIC that communicates back is different from the received NIC. This setting is enabled at the packet destination, and only the return communication is filtered.
Return route filter
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
There is a way to disable the above, but it is a security setting for the entire OS to prevent IP spoofing. In order to avoid this, there is also a function called Policy Base Routing, so implement it.
Policy Base Routing
The standard routing table routes packets to a specific router or default gateway by looking at the destination address. On the other hand, Policy Base Routing (PBR) is a function that determines the routing by looking at the source IP address and TCP / UDP port. Also, prioritize PBR as a rule over normal routing.
Therefore, in this case, it can be concluded that the PBR should be set so that packets with a source IP address of 10.0.2.0/24 as a reference will be skipped to 10.0.2.1. (Although it is the source IP address mentioned here, it refers to the part of the return packet of PING. As you can see by tcpdump, the return packet is 10.0.2.x ==> 10.0.1.x.)
--Since this is the current rule, set the priority to a smaller value than "32766: from all lookup main". The lower the priority, the higher the priority rule
[root@ip-10-0-1-5 etc]# ip rule show
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
--Create route table rules for PBR
temporary
ip route flush table 2000
ip route add table 2000 to 10.0.2.0/24 dev eth1
ip route add table 2000 to default via 10.0.2.1 dev eth1
ip rule add from 10.0.2.0/24 table 2000 priority 200
Permanent
cat << EOF > /etc/sysconfig/network-scripts/pbr02-eth1
from 10.0.2.0/24 table 2000 priority 200
EOF
cat << EOF > /etc/sysconfig/network-scripts/route-eth1
10.0.2.0/24 dev eth1 table pbr02
default via 10.0.2.1 dev eth1 table 2000
EOF
--Setting confirmation system
[root@ip-10-0-1-5 network-scripts]# ip rule show
0: from all lookup local
200: from 10.0.2.0/24 lookup pbr02
32766: from all lookup main
32767: from all lookup default
[root@ip-10-0-1-5 network-scripts]# ip route show table 2000
10.0.2.0/24 dev eth1 scope link
default via 10.0.2.1 dev eth1
Reference
https://milestone-of-se.nesuke.com/nw-basic/routing/policy-based-routing/