Incorrect pod IPs for the pods that run in pod network

[svalleru]

Hi, we are currently running flannel v0.22.3 (with --ip-masq=true) and seeing incorrect pod IPs for the pods that run in pod network.

For example, here you can see the server pod on node1.foo.com receiving the request from client pod that is on node2.foo.com but logs the client pod's IP as 100.96.125.1 instead of 100.101.114.41

❯ k get po pod1 -o wide
NAME   READY   STATUS    RESTARTS   AGE     IP              NODE                              NOMINATED NODE   READINESS GATES
pod1   1/1     Running   0          2m45s   100.96.125.82   node1.foo.com   <none>           <none>

❯ k exec -it pod1 -- /bin/bash
pod1:/root$ hostname -i
100.96.125.82
pod1:/root$ python3
python3     python3.12
pod1:/root$ python3 -m http.server
Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ...


100.96.125.1 - - [06/Jan/2025 23:09:55] "GET / HTTP/1.1" 200 -


❯ k get po pod2 -o wide
NAME   READY   STATUS    RESTARTS        AGE     IP               NODE                              NOMINATED NODE   READINESS GATES
pod2   1/1     Running   1 (2m41s ago)   5m48s   100.101.114.41   node2.foo.com   <none>           <none>

❯ k exec -it pod2 -- /bin/bash
pod2:/root$ hostname -i
100.101.114.41

pod2:/root$ curl -I http://100.96.125.82:8000/
HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.12.3
Date: Mon, 06 Jan 2025 23:11:06 GMT
Content-type: text/html; charset=utf-8
Content-Length: 304

debug info for node1.foo.com (has PodCIDR 100.96.125.0/25):

$ iptables -t nat -nvL FLANNEL-POSTRTG --line-numbers
Chain FLANNEL-POSTRTG (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1      48M 3377M MASQUERADE  all  --  *      *       100.96.0.0/12       !224.0.0.0/4          /* flanneld masq */ random-fully
2    4435K  267M MASQUERADE  all  --  *      *      !100.96.0.0/12        100.96.0.0/12        /* flanneld masq */ random-fully
3        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x4000/0x4000 /* flanneld masq */
4        0     0 RETURN     all  --  *      *       100.96.125.0/25      100.96.0.0/12        /* flanneld masq */
5        0     0 RETURN     all  --  *      *       100.96.0.0/12        100.96.125.0/25      /* flanneld masq */
6        0     0 RETURN     all  --  *      *      !100.96.0.0/12        100.96.125.0/25      /* flanneld masq */
7        0     0 MASQUERADE  all  --  *      *       100.96.0.0/12       !224.0.0.0/4          /* flanneld masq */
8        0     0 MASQUERADE  all  --  *      *      !100.96.0.0/12        100.96.0.0/12        /* flanneld masq */

debug info for node1.foo.com (has PodCIDR 100.101.114.0/25):

$ sudo iptables -t nat -nvL FLANNEL-POSTRTG --line-numbers
Chain FLANNEL-POSTRTG (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x4000/0x4000 /* flanneld masq */
2     256K   15M RETURN     all  --  *      *       100.101.114.0/25     100.96.0.0/12        /* flanneld masq */
3     2647  159K RETURN     all  --  *      *       100.96.0.0/12        100.101.114.0/25     /* flanneld masq */
4        0     0 RETURN     all  --  *      *      !100.96.0.0/12        100.101.114.0/25     /* flanneld masq */
5    94020 8342K MASQUERADE  all  --  *      *       100.96.0.0/12       !224.0.0.0/4          /* flanneld masq */ random-fully
6        0     0 MASQUERADE  all  --  *      *      !100.96.0.0/12        100.96.0.0/12        /* flanneld masq */ random-fully

Any ideas on what might be causing this?

[rbrtbnfgl]

It's strange that you are having multiple MASQUERADE rules on the first node. How did you setup flannel? Did you clean the iptables rules when you started kubernetes?

[svalleru]

flannel along with other node components were installed after a fresh reimage. Node also get reimaged periodically to keep the OS up to date. Also, flannel was upgraded form 0.0.36 to 0.22.3 - not sure if this major bump would result in redundant iptable rules

[rbrtbnfgl]

There are surely some changes o how Flannel creates the rules between the two versions. You can try to restart the node to clean the old rules.