| View previous topic :: View next topic |
| Author |
Message |
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
 Posted: Fri Mar 20, 2009 5:31 am Post subject: kvm + br0 + tap0 + eth0 = virtual hub? [yep!] |
 |
|
I've been banging my head against this trying to do something that I thought would be fairly common. I'm trying to get a kvm virtual machine (windows-xp) to use its host as a virtual hub to the outside world. I want this vm to be in the same subnet as the host as if it had it's own nic and connection to the outside world. It needs to get a dhcp address from an external server as if it were just another drop on the network switch. Meanwhile the host needs to maintain its network identity. I don't care about filtering with either iptables or ebtables (yet).
br0 consists of eth0 and tap0 (a tuntap tap device) in promiscuous mode with no ip. br0 is set to 192.168.11.12 while the windows vm thinks it's supposed to be 192.168.11.200
I'm thinking arp replies back to the vm are getting thrown on the floor because I see its broadcasts coming out of tap0 in wireshark but nothing going back. ping of br0 itself from the vm yields an arp request from tap0 asking for who owns 192.168.11.12 but nothing else. Ping from br0 to 192.168.11.200 works. ping to an outside ip yields a similar arp request on tap0 but with no reply
It looks like some mac shenanigans with ebtables is needed, but all of the docs out there are ugly. Everybody assumes your setup in bridging docs consists of three ethernet devices where eth0 is outside and an eth1 and eth2 are bridging a lan segment. Not what I need.
Current software on amd64 stable.
kernel 2.6.27-gentoo-r8
kvm-84 (unmasked from ~amd64)
ebtables-2.0.8.2-r2
bridge-utils-1.4
iptables-1.4.2-r2
shorewall-common-4.0.15
shorewall-perl-4.0.15
/proc/sys/net/ipv4/ip_forward is set to 1
/proc/sys/net/bridge/* entries are all set to 1
Even though I have the filtering stuff installed, I'm totally passthru at the moment:
| Code: | # iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
# ebtables -L; ebtables -t nat -L; ebtables -t broute -L
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
Bridge table: nat
Bridge chain: PREROUTING, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
Bridge chain: POSTROUTING, entries: 0, policy: ACCEPT
Bridge table: broute
Bridge chain: BROUTING, entries: 0, policy: ACCEPT
|
I've played with vmware server and the bridge vmnet setup before so I know this is doable. I would use their vmnet stuff for this if I knew how to snake my way into their taps.
Last edited by vaxbrat on Tue Mar 24, 2009 4:22 am; edited 1 time in total |
|
| Back to top |
|
 |
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Fri Mar 20, 2009 2:59 pm Post subject: host route |
|
|
Adding a host route on the host system allows the outside world to ping in. The outside world's arp request gets over to tap0 fine and the vm responds back so that the pings can then happen:
| Code: | route add -host 192.168.11.200 dev tap0
|
|
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Tue Mar 24, 2009 2:44 am Post subject: Correctly add the host route |
|
|
The previous rock didn't quite get things done. This route statement on the host will finally allow transparent two way traffic in and out of the guest. I still need to try this in a dhcp setup where the guest gets its address from another server.
| Code: | | route add -host 192.168.11.200 gw 192.168.11.12 dev br0 |
This whole process has caused much more hair pulling than I had originally bargained for  |
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Tue Mar 24, 2009 4:24 am Post subject: one other thing |
|
|
| It always helps to start from a fresh boot after you have been chasing your tail around and around. |
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Sun Apr 05, 2009 12:29 am Post subject: still no joy at work |
|
|
| I have a HP desktop at work with the VPro chipset. I have yet to duplicate at work what I had working at home with the Amd Phenom and host route. Not sure if there is something screwed up with the bridge or because of the fact that I'm playing with dhcp from an outside server, but I have yet to successfully ping inside and out of the guest. |
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Sun Apr 05, 2009 12:32 am Post subject: trying a sanity check |
|
|
| I put a Fedora 10 install on the work box in a double boot and then will try a vmware server 2 with its bridge scheme to see what happens. It was interesting to see that adding a vmware server 2 setup to the gentoo side still didn't get the networking to work. There must be something funny in the kernel with iptables/ebtables and whatever vmware wants to do. |
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Fri Apr 10, 2009 3:44 am Post subject: mac address confusion |
|
|
One thing that I just noticed that was screwing me up is the conception that I would have the same mac address on the hypervisor's tap interface as what the guest sees on its virtual nic. I had one system where both were the same and was unable to ping back into the guest from an external host or to ping anything but the hypervisor from the guest. Another system where the two were different allowed me all the way in and out.
Thus, the /etc/conf.d/net has
| Code: | | mac_tap0="52:54:00:12:34:56" |
while the kvm command line has the snippet
| Code: | | kvm -net nic,macaddr=00:1d:92:ab:3f:79 |
One of those is the mac range assigned to the realtek nic that the guest thinks it sees while the other is a range I had seen from kvm examples out there
I'm putting both in the /etc/ethers but wonder if I really need this file now
| Code: | 00:1d:92:ab:3f:79 192.168.11.202
52:54:00:12:34:58 192.168.11.202 |
|
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Sat Apr 11, 2009 2:08 am Post subject: /etc/ethers not needed |
|
|
I fixed a work box today to go back to using kvm after working with a vmware server 2 set up. The inside/outside mac address scheme for the tap is the key. /etc/ethers is not needed. I don't think /etc/sysctl.conf is crucial either but making sure that ip_forward is set with that is probably a good thing anyway:
| Code: | # /etc/sysctl.conf
#
# For more information on how this file works, please see
# the manpages sysctl(8) and sysctl.conf(5).
#
# In order for this file to work properly, you must first
# enable 'Sysctl support' in the kernel.
#
# Look in /proc/sys/ for all the things you can setup.
#
# enables packet forwarding
net.ipv4.ip_forward = 1
# enables IP dynaddr
net.ipv4.ip_dynaddr = 1
# Disable ECN
#net.ipv4.tcp_ecn = 0
# Enables source route verification
net.ipv4.conf.default.rp_filter = 1
# Enable reverse path
net.ipv4.conf.all.rp_filter = 1
# Enable SYN cookies (yum!)
# http://cr.yp.to/syncookies.html
#net.ipv4.tcp_syncookies = 1
# Disable source route
#net.ipv4.conf.all.accept_source_route = 0
#net.ipv4.conf.default.accept_source_route = 0
# Disable redirects
#net.ipv4.conf.all.accept_redirects = 0
#net.ipv4.conf.default.accept_redirects = 0
# Disable secure redirects
#net.ipv4.conf.all.secure_redirects = 0
#net.ipv4.conf.default.secure_redirects = 0
# Ignore ICMP broadcasts
#net.ipv4.icmp_echo_ignore_broadcasts = 1
# Disables the magic-sysrq key
#kernel.sysrq = 0
# When the kernel panics, automatically reboot in 3 seconds
#kernel.panic = 3
# Allow for more PIDs (cool factor!); may break some programs
#kernel.pid_max = 999999
# You should compile nfsd into the kernel or add it
# to modules.autoload for this to work properly
# TCP Port for lock manager
#fs.nfs.nlm_tcpport = 0
# UDP Port for lock manager
#fs.nfs.nlm_udpport = 0
|
If you make edits and want the running system to change do this afterwards:
It would be interesting to see whether the vmware "vmnet" network devices as well as its hidden virtual bridge device to your nic (ie eth0) could co-exist with a br0-eth0-tap0 style bridge on the same nic. While doing the vmware testing, I had the /etc/conf.d/net files swapped so that only eth0 was up and running. |
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Sat Apr 11, 2009 2:12 am Post subject: host route is not needed either |
|
|
| I should note that the host route doesn't need to be added to the scheme either. That windows guest at work uses dhcp from an external host server and works fine without a host route. In fact, the corporate lan is based on a class B block which is subnetted into class C pools. Random assignment gave my hypervisor an address from one class C block while the guest got one from another. That didn't impact the bridge pasthru. |
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
|
| Back to top |
|
|
jago25_98
Apprentice
Joined: 23 Aug 2002
Posts: 180
|
| Posted: Sun May 24, 2009 12:37 am Post subject: |
|
|
I was kind of hoping it would be possible to just link all interfaces into a bridge and then just firewall the bridge - much simpler!
Personally all I need to do is join Tun/Tap to eth0(internet side).
My config file worked initially but then it stopped after about 5 hours and I've no idea why:
| Code: | # Used by ifup(8) and ifdown(8). See the interfaces(5) manpage or
# /usr/share/doc/ifupdown/examples for more information.
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
# Uncomment this and configure after the system has booted for the first time
auto br0
iface br0 inet static
pre-up ifconfig eth0 up
address myIP.to.internet.side
netmask 255.255.255.0
gateway 67.23.25.1
dns-nameservers 72.3.128.240 72.3.128.241
bridge_ports eth0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off
auto eth1
iface eth1 inet static
address myIP.to.slicehost.side.i.think
netmask 255.255.224.0
auto tap0
iface tap0 inet static
pre-up tunctl -u j -t tap0
address 10.0.0.1
netmask 255.0.0.0
#auto br0
# iface br0 inet dhcp
# bridge_ports eth0 eth1 tun0 tun1
up route add -net 10.176.0.0 netmask 255.248.0.0 gw 10.176.96.1
down route add -net 10.176.0.0 netmask 255.248.0.0 gw 10.176.96.1
up route add -net 10.191.192.0 netmask 255.255.192.0 gw 10.176.96.1
down route add -net 10.191.192.0 netmask 255.255.192.0 gw 10.176.96.1 |
edit: looks like gentoo doesn't use the same network config file as me anymore, it's been a while |
|
| Back to top |
|
|
vaxbrat
l33t
Joined: 05 Oct 2005
Posts: 731
Location: DC Burbs
|
| Posted: Tue May 26, 2009 1:48 pm Post subject: device issue? |
|
|
I've had some of these bridges up for days at a time so not sure why you are losing it after 5 hours.
Also if you bridge everything to eth0, you should be able to put firewall rules on eth0 to have global effect. Likewise, you should be able to put individual rules at the taps to customize for each guest. |
|
| Back to top |
|
|
|
Networking & Security
