How to route network traffic for private link? [SOLVED]

[dufeu]

I can't seem to find an example for this case. What I want to do is have two PCs directly connected with 10Ge {SPF+} as well as keep the current 1Ge connections to my LAN for each PC.

PC1 is my primary workstation while PC2 is my NAS. The NAS offers both nfs and samba based shares across my network. Ny BluRay reader/writer is on PC1. I both read/restore from the BD reader to the NAS and burn from the NAS to the BD writer. The current 1Ge connections across my switch limits me to a max throughput of real life measured 900Mbps. The BD r/w unit maxes at 300Mbps.

Both PCs are connected to the same switch. The LAN segment is 192.168.1.0/24, PC1's 1Ge nic is 192.168.1.200 and PC2's 1Ge nic is 192.168.1.202.

I do very large data transfers between PC1 and PC2 on a regular basis. What I want to do is keep all traffic between PC1 and PC2 off my LAN altogether.

I've recently added Mellanox 10Ge cards to each PC and the cards seem to work fine.

I have some experience routing two different subnets with their own respective routers through a two nic box, but this case is not that. It's also been many years {I stopped building my own routers} since I've worked with iptables.

I believe I need to set on both PCs:

[eccerr0r]

If both machines are connected to a switch, the hardware switch will prevent traffic from permeating the rest of the network, unless someone somehow sets the switch to promiscuous mode.

If you actually want 11Gbps (the original 1Gb and the new 10Gb) then this will require some trickery as you'd need to bond the network adapters.

If you just want to use the 10Gb link for just the two machines together, easiest way is to just make the 10Gb cards on their own subnet and just use that subnet when you need that speed boost, else use the 1Gb network address.

e.g.
Box1:
eth1g: 192.168.1.201/24 def route 192.168.1.1
eth10g: 192.168.2.2/24
Box2:
eth1g: 192.168.1.202/24 def route 192.168.1.1
eth10g: 192.168.2.3/24

Note there is no default route for 10g. Thus no general traffic will be sent out that port. Also you don't need a router on the 192.168.2.x net, it's just one fast subnet. Unfortunately you will need to treat the new ethernet as a different host, but that's no big deal since you only have two machines (one machine - the server) that needs to keep two sets of configuration (like two exports, two sets of samba rules) - one for the fast port and one for the slow port.

You don't want to forward... not sure how that helps you. You might well ditch the 1Gb card and share everything on the same 10Gb switch and let the switch handle things - this is the easiest solution. The iptables you present have a problem when a remote machine tries contacting one of these machines and gets the wrong ethercard, and that packet gets dropped... If you were experimenting with this and ended up with the problem in https://forums.gentoo.org/viewtopic-t-1071888-highlight-.html , that would explain the problems you see in the other thread, bad iptables rules definitely would cause that weird behavior.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching?

[dufeu]