I would like to use port-based routing on my Linux local gateway. Here is my network diagram :
192.168.42.148/24 192.168.42.1/24 192.168.44.2 192.168.44.1
+--------------------+ +----------------------------+ +----------+
| Workstation (eth0)-|----------------|-(em0) Local_GW (tun0)-|------------| VPN_GW |
+--------------------+ | (rl0) | +----------+
+-------------|--------------+
10.133.8.79/21 |
|
10.133.15.254/21 |
+--------------------+
| Provider_GW |
+--------------------+
Provider_GW only allows 21,110,143,554,587,993,995,5222,6666:6669 as outgoing ports for packets bound for the Internet. Local_GW is connected to VPN_GW through a UDP SSL VPN (OpenVPN)
Therefore, I would like to use :
- 10.133.15.254 as gateway for tcp packets whose outgoing port is in the list 21,110,143,554,587,993,995,5222,6666:6669
- 192.168.44.1 as gateway for tcp packets with any other outgoing port
Here are the steps I follow :
#!/bin/bash
## Flush FIP (Forbidden Internet Ports) table
ip route flush table FIP
## Copy all routes from main table except the default one
ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table FIP $ROUTE; done
## Add default route
ip route add default via 192.168.44.1 table FIP
## Flush iptables PREROUTING chain in table mangle
iptables -t mangle -F PREROUTING
## Create iptables rules for packet marking
iptables -t mangle -N MFIP
### Only mark packets bound for the Internet
iptables -t mangle -A MFIP --dst 192.168.0.0/16 --jump RETURN
iptables -t mangle -A MFIP --dst 172.16.0.0/12 --jump RETURN
iptables -t mangle -A MFIP --dst 10.0.0.0/8 --jump RETURN
iptables -t mangle -A MFIP --dst 169.254.0.0/16 --jump RETURN
iptables -t mangle -A MFIP --jump MARK --set-mark 1
iptables -t mangle -A PREROUTING -i em0 -p tcp -m multiport ! --dports 21,110,143,554,587,993,995,5222,6666:6669 --jump MFIP
## Delete old routing rule if it exist
ip rule del fwmark 1
## Create new routing rule
ip rule add fwmark 1 table FIP
## Zero Counters (useful for debugging)
iptables -t mangle -Z
I can't connect to an SSH server from Workstation :
fabien@Workstation:~$ ssh [email protected]
ssh: connect to host sdf.org port 22: Connection timed out
However I can see that packets are marked and routed as expected :
(On Local_GW) :
fabien@Local_GW:~$ sudo tcpdump -fi em0 tcp port 22 and host not 192.168.42.1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:07:32.065071 IP LoopbackMarvin.local.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:35.064024 IP LoopbackMarvin.local.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:41.060423 IP LoopbackMarvin.local.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
Local_GW indeed receives the SYN packets from workstation
fabien@Local_GW:~$ sudo iptables -t mangle -vL -n
Chain PREROUTING (policy ACCEPT 7046 packets, 829K bytes)
pkts bytes target prot opt in out source destination
558 43499 MFIP tcp -- em0 * 0.0.0.0/0 0.0.0.0/0 multiport dports ! 21,110,143,554,587,993,995,5222,6666:6669
Chain INPUT (policy ACCEPT 5690 packets, 713K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 220 packets, 49131 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 540 packets, 352K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 767 packets, 402K bytes)
pkts bytes target prot opt in out source destination
Chain MFIP (1 references)
pkts bytes target prot opt in out source destination
555 43319 RETURN all -- * * 0.0.0.0/0 192.168.0.0/16
0 0 RETURN all -- * * 0.0.0.0/0 172.16.0.0/12
0 0 RETURN all -- * * 0.0.0.0/0 10.0.0.0/8
0 0 RETURN all -- * * 0.0.0.0/0 169.254.0.0/16
3 180 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK xset 0x1/0xffffffff
Apparently, SYN packets from Workstation are correctly marked by iptables.
fabien@Local_GW:~$ sudo tcpdump -fi tun0 tcp port 22
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
13:07:32.065153 IP 192.168.42.148.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:32.487217 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:35.064062 IP 192.168.42.148.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:35.341717 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:35.510967 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:41.060459 IP 192.168.42.148.52540 > 192.94.73.15.ssh: Flags [S], seq 3448990172 [...]
13:07:41.336589 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:41.411675 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
13:07:53.411081 IP 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855 [...]
The SYN packet from Workstation is correctly routed to VPN_GW and it even receives and answer (from the sdf.org ssh server): 192.94.73.15.ssh > 192.168.42.148.52540: Flags [S.], seq 2834611855. However, Local_GW doesn't route this response packet back to Workstation, Why ? I must be missing something obvious but I can't understand what.
Thank you very much for reading.
If you want the exact details regarding my configuration, here are the (very) verbose outputs from iptables and ip. Sorry if it makes my question very long, I try to be as precise as possible.
Firewall:
fabien@Local_GW:~$ sudo iptables -t nat -vL -n
Chain PREROUTING (policy ACCEPT 145K packets, 10M bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 1934 packets, 137K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 1938 packets, 137K bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * rl0 192.168.44.0/24 0.0.0.0/0
668 43570 MASQUERADE all -- * rl0 192.168.42.0/24 0.0.0.0/0
fabien@Local_GW:~$ sudo iptables -vL -n
Chain INPUT (policy ACCEPT 132K packets, 15M bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 34345 packets, 24M bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 15490 packets, 7741K bytes)
pkts bytes target prot opt in out source destination
Routing tables:
fabien@Local_GW:~$ sudo ip route show
192.168.46.1 dev tun1 proto kernel scope link src 192.168.46.3
192.168.44.1 dev tun0 proto kernel scope link src 192.168.44.2
192.168.46.0/24 via 192.168.46.1 dev tun1
192.168.42.0/24 dev em0 proto kernel scope link src 192.168.42.1
192.168.43.0/24 via 192.168.44.1 dev tun0
10.133.8.0/21 dev rl0 proto kernel scope link src 10.133.8.79
169.254.0.0/16 dev em0 scope link metric 1000
default via 10.133.15.254 dev rl0 metric 100
fabien@Local_GW:~$ sudo ip route show table FIP
192.168.44.1 dev tun0 proto kernel scope link src 192.168.44.2
192.168.46.1 dev tun1 proto kernel scope link src 192.168.46.3
192.168.46.0/24 via 192.168.46.1 dev tun1
192.168.42.0/24 dev em0 proto kernel scope link src 192.168.42.1
192.168.43.0/24 via 192.168.44.1 dev tun0
10.133.8.0/21 dev rl0 proto kernel scope link src 10.133.8.79
169.254.0.0/16 dev em0 scope link metric 1000
default via 192.168.44.1 dev tun0
fabien@Zaphod:~$ ip rule show
0: from all lookup local
32765: from all fwmark 0x1 lookup FIP
32766: from all lookup main
32767: from all lookup default
