using multiple tables to route ips from TWO ISPs

Question

I need to configure my router (Ubuntu Server 24/ NNFTables) so that computers connected to the lan1 and lan2 interfaces can browse the internet using the wan1 or wan2 interfaces, depending on the subnet IP.

I receive internet through 2 different providers.

ISP 1: Bloco ip: 111.11.111.0/28

ISP 2: Bloco ip: 222.22.222.8/29

config netplan yaml:

network:
    ethernets:
        wan1:
            addresses:
            - 111.11.111.2/28
            - 111.11.111.3/28
            - 111.11.111.4/28
            - 111.11.111.5/28
            - 111.11.111.6/28
            - 111.11.111.7/28
            - 111.11.111.8/28
            - 111.11.111.9/28
            - 111.11.111.10/28
            - 111.11.111.11/28
            - 111.11.111.12/28
            - 111.11.111.13/28
            - 111.11.111.14/28
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
                search: []
            routes:
            -   to: default
                via: 111.11.111.1
        wan2:
            addresses:
            - 222.22.222.10/29
            - 222.22.222.11/29
            - 222.22.222.12/29
            - 222.22.222.13/29
            - 222.22.222.14/29
            nameservers:
                addresses:
                - 8.8.8.8
                - 8.8.4.4
                search: []
            routes:
            -   to: default
                via: 222.22.222.9                
        lan1:
            addresses:
            - 192.168.10.1/24
            nameservers:
                addresses: []
                search: []
        lan2:
            addresses:
            - 192.168.20.1/24
            nameservers:
                addresses: []
                search: []
    version: 2`

usertest@router1:~$ ip route
default via 111.11.111.1 dev wan1 proto static 
192.168.10.0/24 dev enp4s0 proto kernel scope link src 192.168.10.1 
192.168.20.0/24 dev enp5s0 proto kernel scope link src 192.168.20.1 
111.11.111.0/28 dev enp1s5 proto kernel scope link src 111.11.111.2 
222.22.222.8/29 dev enp8s0 proto kernel scope link src 222.22.222.9 

Subnet stations and their respective public IPs that we like to browse:

192.168.10.101 ---> 111.11.111.11
192.168.10.102 ---> 111.11.111.12
192.168.10.201 ---> 222.22.222.11
192.168.10.202 ---> 222.22.222.12
192.168.20.10 ---> 222.22.222.10

I saw that I need to work with multiple routing tables using IP ROUTE, but all the configurations I tried failed...

Does anyone know how I can solve this?

I've already tried some nftables/ip route configurations... but I couldn't make it work... I've messed up so much that I don't even know everything I tried... Now the configuration looks like this:

$ ip route
default via 111.11.111.9 dev wan1 
192.168.10.0/24 dev lan1 proto kernel scope link src 192.168.10.1 
192.168.20.0/24 dev lan2 proto kernel scope link src 192.168.20.1 
111.11.111.0/28 dev wan1 proto kernel scope link src 111.11.111.2 
222.22.222.8/29 dev wan2 proto kernel scope link src 111.11.111.10

$ ip route show table 100
default via 111.11.111.1 dev wan1 proto static 

$ ip route show table 200
default via 222.22.222.9 dev wan2 proto static 

$ ip rule show
0:  from all lookup local
32764:  from 222.22.222.8/29 lookup 200 proto static
32765:  from 111.11.111.0/28 lookup 100 proto static
32766:  from all lookup main
32767:  from all lookup default

Answer 1

The routing configuration you show in your question looks like a substantial part of the solution. In order to test things out, I put together a simulation of your environment using Kathara; you can find my complete test environment here. The test topology looks like this:

The goal is that we can reach remotehost from nodes on the right (node00 through node11), and each node will use the appropriate outbound path from router.

The router node in this diagram corresponds to the system you are trying to configure. It has the following interface configuration:

161: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 111.11.111.2/28 scope global eth0
       valid_lft forever preferred_lft forever
    inet 111.11.111.11/28 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet 111.11.111.12/28 scope global secondary eth0
       valid_lft forever preferred_lft forever
169: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 222.22.222.10/29 scope global eth1
       valid_lft forever preferred_lft forever
    inet 222.22.222.11/29 scope global secondary eth1
       valid_lft forever preferred_lft forever
    inet 222.22.222.12/29 scope global secondary eth1
       valid_lft forever preferred_lft forever
170: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.10.1/24 scope global eth2
       valid_lft forever preferred_lft forever
171: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    inet 192.168.20.1/24 scope global eth3
       valid_lft forever preferred_lft forever

The following default routing table:

root@router:/# ip route
default via 111.11.111.1 dev eth0
111.11.111.0/28 dev eth0 proto kernel scope link src 111.11.111.2
192.168.10.0/24 dev eth2 proto kernel scope link src 192.168.10.1
192.168.20.0/24 dev eth3 proto kernel scope link src 192.168.20.1
222.22.222.8/29 dev eth1 proto kernel scope link src 222.22.222.10

The following policy routing rules:

root@router:/# ip rule show
0:      from all lookup local
32764:  from 192.168.20.0/24 lookup 222
32765:  from 192.168.10.0/24 lookup 111
32766:  from all lookup main
32767:  from all lookup default

In table 111:

root@router:/# ip route show table 111
default via 111.11.111.1 dev eth0

In table 222:

root@router:/# ip route show table 222
default via 222.22.222.9 dev eth1

In order to achieve your NAT goals, the router has the following nft ruleset:

table ip nat {
        chain router_masquerade {
                type nat hook postrouting priority srcnat; policy accept;
                ip saddr 192.168.10.101 counter snat to 111.11.111.11
                ip saddr 192.168.10.102 counter snat to 111.11.111.12
                ip saddr 192.168.20.201 counter snat to 222.22.222.11
                ip saddr 192.168.20.202 counter snat to 222.22.222.12
                oifname "eth0" counter masquerade
                oifname "eth1" counter masquerade
        }
}

This applies the static NAT entries you list in your question, and applies regular masquerading to anything on lan1 or lan2 without an explicit entry.

---

With this configuration in place, if I run a tcpdump on remotehost, like this:

tcpdump -nn -i eth0

And then attempt to ping -c1 remotehost from node00, we see:

19:19:00.808202 IP 111.11.111.11 > 100.64.64.100: ICMP echo request, id 31, seq 1, length 64
19:19:00.808236 IP 100.64.64.100 > 111.11.111.11: ICMP echo reply, id 31, seq 1, length 64

The path from node00 to remotehost (a) has the expected source address and (b) is using the expected path. Similarly, if we try the same thing from node01, we see:

19:19:58.209656 IP 111.11.111.12 > 100.64.64.100: ICMP echo request, id 33, seq 1, length 64
19:19:58.209717 IP 100.64.64.100 > 111.11.111.12: ICMP echo reply, id 33, seq 1, length 64

Again, it's using the expected path, and we see that our externally visible address is 111.11.111.12, as expected from our NAT mappings.

If we attempt similar operations from nodes on lan2, we see from node10:

19:21:08.356388 IP 222.22.222.11 > 100.64.64.100: ICMP echo request, id 35, seq 1, length 64
19:21:08.356433 IP 100.64.64.100 > 222.22.222.11: ICMP echo reply, id 35, seq 1, length 64

And from node11:

19:21:30.824730 IP 222.22.222.12 > 100.64.64.100: ICMP echo request, id 38, seq 1, length 64
19:21:30.824799 IP 100.64.64.100 > 222.22.222.12: ICMP echo reply, id 38, seq 1, length 64

As far as I can tell, this behavior meets all of your goals.