| View previous topic :: View next topic |
| Author |
Message |
regox
Tux's lil' helper
Joined: 12 Sep 2021
Posts: 79
|
 Posted: Fri Apr 15, 2022 2:04 pm Post subject: [SOLVED] Network configuration for KVM with nftables |
 |
|
I need help manually configuring the network for my QEMU/KVM guests. I would like to have two groups: VMs that can connect to the internet and VMs that are isolated, each with their own nftables rules.
I use libvirtd and virt-manager GUI to manage my virtual machines. It seems like the network configuration does not work out of the box because I use nftables instead of iptables.
Also I don't use firewalld or systemd, like most of the online tutorials do that I found. Actually I prefer setting up the network manually and learning along the way.
I imagine something like this:
| Code: |
┌────────────────────┐
│ Internet │
│ │
└──────────┬─────────┘
▲ │
│ │
│ ▼
┌────────┴───────────┐
│ Router │
│ DNS server │
│ DHCP server │
│ │
│ 192.168.1.1 │
└──────────┬─────────┘
▲ │
│ │ Local subnet
│ │ 192.168.1.0/24
│ │
│ ▼
┌┼─┬┐ wlp0s20f3
┌────────────────┼┼┼┼┼──────────────────────────────────────────────┐
│ └┴─┼┘ Gentoo host │
│ ▲ │ 192.168.1.100 │
│ ───┼─┼─── Existing nftables firewall │
│ ┌──────────────┼─┼──────────────────────────────────────┐ │
│ │ │ │ │ │
│ │ │ ▼ ┌──────────────────────────┐ │ │
│ │ ┌┴──┐ │ │ QEMU/KVM Guest │ │ │
│ │ virbr0 │ ├──┼──►│ │ │ │
│ │ 192.168.2.1 │ │ │ │ │ │ │
│ │ │ │◄─┼───┤ │ │ │
│ │ └───┘ │ │ │ │ │
│ │ │ └──────────────────────────┘ │ │
│ │ nftables │ │
│ │ firewall │ │
│ │ Subnet for VMs │ │
│ │ 192.168.2.0/24 │ │
│ └───────────────────────────────────────────────────────┘ │
│ │
│ ┌───────────────────────────────────────────────────────┐ │
│ │ virbr1 │ │
│ │ ┌───────────────┐ ┌────┐ ┌─────────────────┐ │ │
│ │ │ QEMU/KVM ├───►│ ├───►│ QEMU/KVM │ │ │
│ │ │ Guest 1 │ │ │ │ Guest 2 │ │ │
│ │ │ │◄───┤ │◄───┤ │ │ │
│ │ └───────────────┘ └────┘ └─────────────────┘ │ │
│ │ Isolated subnet, 192.168.3.0/24 │ │
│ └───────────────────────────────────────────────────────┘ │
│ │
└───────────────────────────────────────────────────────────────────┘
|
libvirtd created the "default" virbr0 bridge on my system. For the guests, I selected this bridge in "NAT" mode in virt-manager.
I am not sure if this is correct or if I have to select something else. I am also confused how I need to configure NAT with nftables.
I tried
| Code: |
table inet nat {
chain nat {
type nat hook postrouting priority 100; policy accept;
ip saddr 192.168.2.0/24 counter snat to 192.168.1.100;
}
}
|
but that does not work (guest can not access internet); I think I have not entirely understood how NAT works. As far as I understand,
it allows to translate between two subnets. But do I have to specify a special port for this (on my host)?
I can see in wireshark that my guests communicates with the bridge (which confuses me, because in my understanding, bridging happens only in layer 2...)
I would very much appreciate if someone could share their configuration or give me some ideas on how I could set this up. I looked around in the QEMU, KVM and Nftables wikis, but I couldn't find anything that would help me.
Thanks in advance.
Last edited by regox on Sun Apr 17, 2022 4:02 pm; edited 2 times in total |
|
| Back to top |
|
 |
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 4045
Location: Rasi, Finland
|
| Posted: Fri Apr 15, 2022 3:45 pm Post subject: Re: Network configuration for KVM with nftables |
|
|
| regox wrote: | | It seems like the network configuration does not work out of the box because I use nftables instead of iptables. |
As far as I know libvirt specifically prefers to use libnft via firewalld. Make sure you have enabled firewalld USE flag for app-emulation/libvirt and nftables use flag for net-firewall/firewalld. You may also need to enable nftables use flag for net-misc/networkmanager too. And of course have kernel compiled for nftables support (which I assume you have already).
Lastly, remember to set | Code: | | net.ipv4.ip_forward = 1 |
in sysctl.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
regox
Tux's lil' helper
Joined: 12 Sep 2021
Posts: 79
|
| Posted: Fri Apr 15, 2022 4:21 pm Post subject: Re: Network configuration for KVM with nftables |
|
|
Thank you for your reply.
| Zucca wrote: | | regox wrote: | | It seems like the network configuration does not work out of the box because I use nftables instead of iptables. |
As far as I know libvirt specifically prefers to use libnft via firewalld. Make sure you have enabled firewalld USE flag for app-emulation/libvirt and nftables use flag for net-firewall/firewalld. You may also need to enable nftables use flag for net-misc/networkmanager too. And of course have kernel compiled for nftables support (which I assume you have already). |
Okay, I understand that firewalld is needed so that libvirtd can manage the network "itself". But is it not possible to do this without firewalld, with a manual configuration?
I have my (host) nftables rules set up without it and I am happy the way it currently is, I am concerned that firewalld may interfere with the current setup. I am also not happy with the fact that firewalld would create a kind of redundancy, since I already use nft to set my rules, I don't need another management tool just for VMs; or am I mistaken what firewalld actually does?
| Zucca wrote: |
Lastly, remember to set | Code: | | net.ipv4.ip_forward = 1 |
in sysctl. |
Sorry I didn't mention. It is set correctly. |
|
| Back to top |
|
|
Zucca
Moderator
Joined: 14 Jun 2007
Posts: 4045
Location: Rasi, Finland
|
| Posted: Fri Apr 15, 2022 7:27 pm Post subject: |
|
|
Firewalld may indeed interfere with your custom configs. I know there was an ability to insert "raw" iptables rules before. At that time nftables "raw" rules were not a thing. I don't know the current state of things. This whole virt-manager, firewalld and networkmanager combination is kind of a RedHat walled garden. :( I'm actually quite fine with firewalld, but not with networkmanager.
If I had my server running I'd give you my nft configs.
I'll try to look for them in another place.
EDIT: Maybe see this post for NAT.
_________________
..: Zucca :..
| My gentoo installs: | init=/sbin/openrc-init
-systemd -logind -elogind seatd |
| Quote: | | I am NaN! I am a man! |
|
|
| Back to top |
|
|
regox
Tux's lil' helper
Joined: 12 Sep 2021
Posts: 79
|
| Posted: Sun Apr 17, 2022 12:45 pm Post subject: |
|
|
Okay, I have made some progress. I decided to focus on the bridge with internet access ("virbr0" in my previous post).
I use this script to set up the bridge: | Code: |
BR_NAME_VM=vm-bridge
# Create bridge device
ip link add name $BR_NAME_VM type bridge
# Set state up
ip link set $BR_NAME_VM up
# Associate IP address to bridge device
ip addr add 192.168.2.1/24 brd + dev $BR_NAME_VM
# Delete the default route
ip route delete 192.168.2.0/24
# Add route
ip route add 192.168.2.0/24 dev $BR_NAME_VM proto dhcp
|
This leads to this:
| Code: |
16: vm-bridge: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 1e:b9:a5:35:78:81 brd ff:ff:ff:ff:ff:ff
inet 192.168.2.1/24 brd 192.168.2.255 scope global vm-bridge
valid_lft forever preferred_lft forever
|
and these routes:
| Code: |
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 192.168.236.229 0.0.0.0 UG 600 0 0 wlp0s20f3
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 vm-bridge
192.168.236.0 0.0.0.0 255.255.255.0 U 600 0 0 wlp0s20f3
|
Please excuse me that the local IPs do not match my initial post. I currently use my Android phone as an access point for my laptop, so I can't change the IPs.
Then, I started another instance of dnsmasq, for DNS+DHCP (besides the one for the host): | Code: |
interface=vm-bridge
bind-dynamic
no-poll
no-resolv
dhcp-range=192.168.2.2,192.168.2.255
server=127.0.0.1#8934
|
On my hosts 127.0.0.1:8934 there is an upstream DNS resolver listening.
My hosts nftables config is this: | Code: |
table ip nat {
chain prerouting {
type nat hook prerouting priority 0; policy accept;
log prefix "prerouting: " flags all
}
chain postrouting {
type nat hook postrouting priority 100; policy accept;
oifname "vm-bridge" masquerade
log prefix "postrouting: " flags all
}
}
table inet filter { # handle 6
chain input { # handle 1
type filter hook input priority filter; policy drop;
ct state { established, related } accept # handle 5
ct state invalid drop # handle 6
iifname "lo" accept # handle 7
iifname "vm-bridge" accept
...
}
chain forward { # handle 2
type filter hook forward priority filter; policy drop;
...
iifname "vm-bridge" accept
oifname "vm-bridge" accept
...
}
chain output { # handle 3
type filter hook output priority filter; policy drop;
oifname "vm-bridge" accept
...
}
}
|
With this, my Ubuntu QEMU/KVM guest can successfully get an IP, Gateway and DNS information over DHCP:
| Code: |
IPv4: 192.168.2.225
Gateway: 192.168.2.1
DNS: 192.168.2.1
|
DNS works from within the guest (verified with manual queries using dig). From the host, I can also ping the guest and vice versa. So far so good.
The problem is that the guest can not connect to the internet (ping, curl or Firefox). If I let the guest continuously ping a public IP, I can check with wireshark that those packets are on vm-bridge, but they never leave my physical interface, wlp0s20f3. In dmesg, there is this:
| Code: |
Apr 17 14:16:32 thinkpad kernel: prerouting: IN=vm-bridge OUT= MACSRC=52:54:00:81:13:bf MACDST=1e:b9:a5:35:78:81 MACPROTO=0800 SRC=192.168.2.225 DST=216.58.212.174 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9948 DF PROTO=ICMP TYPE=8 CODE=0 ID=10 SEQ=1105
|
but no postrouting (?)
So there is still a problem with NAT and/or the routing table. Can anyone help? |
|
| Back to top |
|
|
regox
Tux's lil' helper
Joined: 12 Sep 2021
Posts: 79
|
| Posted: Sun Apr 17, 2022 4:02 pm Post subject: |
|
|
I think it should be like this:
| Code: |
table ip nat {
chain postrouting {
type nat hook postrouting priority 100;
policy accept;
ip saddr 192.168.2.0/24 masquerade
}
} |
That seems to work for now. |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Mon Sep 12, 2022 6:49 pm Post subject: |
|
|
I'm trying to manually setup nftables for qemu just like you
About the
> On my hosts 127.0.0.1:8934 there is an upstream DNS resolver listening
do you need to set this beforehand? I'm just using NetworkManager, and never tweaked anything manually.
Also, I'm wondering: are all of the steps (creating bridge interfaces, IPs, etc) "safe"? I'm concerned with messing up my setup beyond my fixing capabilities (I don't really know much about any of this) 
Cheers. |
|
| Back to top |
|
|
regox
Tux's lil' helper
Joined: 12 Sep 2021
Posts: 79
|
| Posted: Tue Sep 13, 2022 10:44 am Post subject: |
|
|
| mani001 wrote: |
do you need to set this beforehand? I'm just using NetworkManager, and never tweaked anything manually.
|
Yes. I use stubby for DNS-over-TLS which I set up to listen on localhost:8934. In /etc/NetworkManager/conf.d/20-dnsmasq.conf I have disabled NetworkManager's dnsmasq with
Instead I use a standalone instance of dnsmasq which gets started automatically by OpenRC (default runlevel). This instance is listening on the normal DNS port 53 and using localhost:8934 as an upstream server.
If you don't want or need DNS-over-TLS, you could change localhost:8934 to localhost:53 in the dnsmasq configuration for the virtual machines. This way, this instance will use your host's dnsmasq instance as an upstream server (which should have been started by NetworkManager by default I think).
| mani001 wrote: |
Also, I'm wondering: are all of the steps (creating bridge interfaces, IPs, etc) "safe"? I'm concerned with messing up my setup beyond my fixing capabilities (I don't really know much about any of this)
|
I think it's pretty safe because creating bridges and assigning IPs is not permanent. After a reboot, your configuration will be lost. I re-run my script every time I start my KVMs. I would make backups of dnsmasq's and NetworkManager's configuration files, though.
Let me know if this works.
Cheers, regox |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Sat Sep 17, 2022 3:30 pm Post subject: |
|
|
Thanks for your help.
A couple of quick questions before stirring up things:
- do you make a new interface, vm-bridge and just ignore the virbr0 that is (automatically) created by libvirtd?
| Quote: | | Then, I started another instance of dnsmasq, for DNS+DHCP (besides the one for the host) |
How do you start another instance? Manually (if so, how?)? Openrc? Right now, I'm using Networkmanager to start the one used by the host.
Cheers. |
|
| Back to top |
|
|
regox
Tux's lil' helper
Joined: 12 Sep 2021
Posts: 79
|
| Posted: Mon Sep 19, 2022 11:46 am Post subject: |
|
|
| mani001 wrote: |
- do you make a new interface, vm-bridge and just ignore the virbr0 that is (automatically) created by libvirtd?
|
Yes.
| mani001 wrote: |
| Quote: | | Then, I started another instance of dnsmasq, for DNS+DHCP (besides the one for the host) |
How do you start another instance? Manually (if so, how?)? Openrc? Right now, I'm using Networkmanager to start the one used by the host.
|
Manually with
| Code: |
sudo dnsmasq -C vm-bridge.conf
|
where vm-bridge.conf contains the configuration I posted beforehand.
Cheers, regox |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Sat Sep 24, 2022 9:00 am Post subject: |
|
|
Thanks for all the tips, but I'm still struggling with this
In particular, when I try to run another instance of dnsmasq for the new interface, I go from
| Code: | root@cochi ~ $ss -alpn src 127.0.0.1
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=370,fd=4))
tcp LISTEN 0 0 127.0.0.1:8384 0.0.0.0:* users:(("syncthing",pid=427,fd=16))
tcp LISTEN 0 0 127.0.0.1:631 0.0.0.0:* users:(("cupsd",pid=296,fd=8))
tcp LISTEN 0 0 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=370,fd=5)) |
to
| Code: | root@cochi ~ $ss -alpn src 127.0.0.1
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=3757,fd=8))
udp UNCONN 0 0 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=370,fd=4))
tcp LISTEN 0 0 127.0.0.1:8384 0.0.0.0:* users:(("syncthing",pid=427,fd=16))
tcp LISTEN 0 0 127.0.0.1:631 0.0.0.0:* users:(("cupsd",pid=296,fd=8))
tcp LISTEN 0 0 127.0.0.1:53 0.0.0.0:* users:(("dnsmasq",pid=370,fd=5)) |
i.e., a new instance of dnsmasq is listening (but only for UDP) at port 53 
Looking at this post https://unix.stackexchange.com/questions/690523/dnsmasq-failed-to-create-listening-socket-for-127-0-0-1-address-already-in-use I thought it was because the systemd-started instance of dnsmasq already handles every interface. I tried setting
| Code: | | except-interface=vm-bridge |
in /etc/dnsmasq.conf, but it didn't help (same result). Any clue? |
|
| Back to top |
|
|
regox
Tux's lil' helper
Joined: 12 Sep 2021
Posts: 79
|
| Posted: Sun Sep 25, 2022 9:14 am Post subject: |
|
|
| mani001 wrote: | | in /etc/dnsmasq.conf |
Are you sure NetworkManager uses this file? I think /etc/dnsmasq.conf is just the default config if you start dnsmasq manually or via OpenRC.
Instead, I would try to change /etc/NetworkManager/dnsmasq.q/dnsmasq.conf.
Maybe you also want to open up a new thread so more people can see it (and link it here). |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Sun Sep 25, 2022 12:22 pm Post subject: |
|
|
Hi,
OK, I think I got confused with all the dnsmasq stuff. Since I don't need stubby, (as you said) I can just setup localhost:53 in the dnsmasq configuration for the virtual machines.
Now, when I use your nft configuration, my internet connection is blocked because of the *chain output* part. I guess it was to be expected since the policy is "drop", and when I comment that out, it's fine. I take it, you fix that in the "...". Anyway, even If switch the policy to accept, DNS is not working from within the VM (there is internet connection, though, since I can ping google or bing).
In virt-manager, you just choose "Bridge device" as "Network source" and then fill in "vm-bridge" as the "Device name", right? Right now I have
| Code: | <interface type="bridge">
<mac address="52:54:00:5d:d5:75"/>
<source bridge="vm-bridge"/>
<target dev="vnet1"/>
<model type="e1000e"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface> |
Cheers. |
|
| Back to top |
|
|
regox
Tux's lil' helper
Joined: 12 Sep 2021
Posts: 79
|
| Posted: Sun Sep 25, 2022 2:07 pm Post subject: |
|
|
| mani001 wrote: |
Now, when I use your nft configuration, my internet connection is blocked because of the *chain output* part. I guess it was to be expected since the policy is "drop", and when I comment that out, it's fine. I take it, you fix that in the "...".
|
Correct. In the "..." I specify all ports I need.
| mani001 wrote: |
Anyway, even If switch the policy to accept, DNS is not working from within the VM (there is internet connection, though, since I can ping google or bing).
|
Is DHCP working? Maybe you can try to manually set a static IP, gateway and DNS inside the VM.
| mani001 wrote: |
In virt-manager, you just choose "Bridge device" as "Network source" and then fill in "vm-bridge" as the "Device name", right? Right now I have
| Code: | <interface type="bridge">
<mac address="52:54:00:5d:d5:75"/>
<source bridge="vm-bridge"/>
<target dev="vnet1"/>
<model type="e1000e"/>
<alias name="net0"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface> |
|
Yes.
By the way, some helpful tools to debug:
- net-analyzer/wireshark: you can set it up to listen on vm-bridge, so you exactly know which packets are being sent
- dig command: Test your dnsmasq servers, with dig @127.0.0.1 -p 53 gentoo.org (test both host and within VM)
- you can use additional "log" statements in the nftables chains so packets get logged to dmesg
Cheers, regox |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Sun Sep 25, 2022 8:18 pm Post subject: |
|
|
Using dig before starting libvirtd and booting the VM I get
| Code: | manu@cochi ~ dig @127.0.0.1 -p 53 gentoo.org
; <<>> DiG 9.16.30 <<>> @127.0.0.1 -p 53 gentoo.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39366
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;gentoo.org. IN A
;; ANSWER SECTION:
gentoo.org. 473 IN A 151.101.2.137
gentoo.org. 473 IN A 151.101.66.137
gentoo.org. 473 IN A 151.101.130.137
gentoo.org. 473 IN A 151.101.194.137
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Sep 25 22:02:05 CEST 2022
;; MSG SIZE rcvd: 103 |
which looks good. However, after starting libvirtd and booting the VM I get
| Code: | manu@cochi ~ dig @127.0.0.1 -p 53 gentoo.org
; <<>> DiG 9.16.30 <<>> @127.0.0.1 -p 53 gentoo.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 48282
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; EDE: 14 (Not Ready)
;; QUESTION SECTION:
;gentoo.org. IN A
;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Sep 25 22:06:54 CEST 2022
;; MSG SIZE rcvd: 45 |
i.e., the server refuses to answer, and I can see tons of related messages in Wireshark, e.g.,
| Code: | 728 350.895281544 192.168.2.203 192.168.2.1 DNS 70 Standard query 0x826a A g.live.com
729 350.895343193 192.168.2.1 192.168.2.203 DNS 70 Standard query response 0x826a Refused A g.live.com |
Could it be due to the iptables rules that libvirt adds (I'm hunting high and low here...)
| Code: | root@cochi qemu_nftables $iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
LIBVIRT_INP all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT)
target prot opt source destination
LIBVIRT_FWX all -- 0.0.0.0/0 0.0.0.0/0
LIBVIRT_FWI all -- 0.0.0.0/0 0.0.0.0/0
LIBVIRT_FWO all -- 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
LIBVIRT_OUT all -- 0.0.0.0/0 0.0.0.0/0
Chain LIBVIRT_FWI (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWO (1 references)
target prot opt source destination
ACCEPT all -- 192.168.122.0/24 0.0.0.0/0
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWX (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain LIBVIRT_INP (1 references)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
Chain LIBVIRT_OUT (1 references)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:68 |
PD: dig and wireshark are pretty cool tools!! |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5625
Location: Bavaria
|
| Posted: Sun Sep 25, 2022 8:58 pm Post subject: |
|
|
| mani001 wrote: | | Could it be due to the iptables rules that libvirt adds (I'm hunting high and low here...) |
With "iptables -L -n" we dont see your device. We would need "iptables -L -v -n" |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Sun Sep 25, 2022 9:54 pm Post subject: |
|
|
Good point...It seems that libvirtd is only tweaking stuff for virbr0 (which should be fine, I guess)
| Code: | root@cochi qemu_nftables $iptables -L -nv
Chain INPUT (policy ACCEPT 205K packets, 92M bytes)
pkts bytes target prot opt in out source destination
205K 92M LIBVIRT_INP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 2 packets, 128 bytes)
pkts bytes target prot opt in out source destination
2 128 LIBVIRT_FWX all -- * * 0.0.0.0/0 0.0.0.0/0
2 128 LIBVIRT_FWI all -- * * 0.0.0.0/0 0.0.0.0/0
2 128 LIBVIRT_FWO all -- * * 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 278K packets, 226M bytes)
pkts bytes target prot opt in out source destination
278K 226M LIBVIRT_OUT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain LIBVIRT_FWI (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
0 0 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWO (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Chain LIBVIRT_FWX (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0
Chain LIBVIRT_INP (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67
0 0 ACCEPT tcp -- virbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
Chain LIBVIRT_OUT (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:53
0 0 ACCEPT tcp -- * virbr0 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
0 0 ACCEPT udp -- * virbr0 0.0.0.0/0 0.0.0.0/0 udp dpt:68
0 0 ACCEPT tcp -- * virbr0 0.0.0.0/0 0.0.0.0/0 tcp dpt:68 |
|
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5625
Location: Bavaria
|
| Posted: Sun Sep 25, 2022 10:15 pm Post subject: |
|
|
| mani001 wrote: | | Good point...It seems that libvirtd is only tweaking stuff for virbr0 (which should be fine, I guess) |
Yes ... BUT ... Only to be safe I have to tell you that your host is not protected by your firewall, because default policy/action for INPUT and OUTPUT is ACCEPT (so, both Chains "LIBVIRT_INP" and "LIBVIRT_OUT" are senseless, because after these accepts all other traffic is allowed also). |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Mon Sep 26, 2022 7:09 am Post subject: |
|
|
I'm not actually using iptables, and my rules are in nftables. As far as I know both firewalls act on the system so I'm fine as long as I have set up rules in nftables...right?  When I switched from iptables to nftables (about 1 year ago), I couldn't find any way to "turn off" iptables (I guess I should disable it in the kernel...maybe?). I just posted the iptables rules because libvirtd make changes there which I don't really understand (and have no control over). |
|
| Back to top |
|
|
pietinger
Moderator
Joined: 17 Oct 2006
Posts: 5625
Location: Bavaria
|
| Posted: Mon Sep 26, 2022 10:12 am Post subject: |
|
|
| mani001 wrote: | | I just posted the iptables rules because libvirtd make changes there which I don't really understand (and have no control over). |
I dont know libvirt but I cant believe that it is not able to support nftables also. (The rules from libvirt are ok).
If you want disable iptables you can remove its start at boot-time (OpenRC: It is /etc/init.d/iptables). Which doesnt help if some systems sets up some ipatbles commands - yes, in this case you must remove kernel modules. But a better way would be to tell libvirt doing nftables commands. |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Mon Sep 26, 2022 11:28 am Post subject: |
|
|
| Quote: | | I dont know libvirt but I cant believe that it is not able to support nftables also. (The rules from libvirt are ok). |
I know...it's kind of shocking since I wouldn't say nftables is nowadays "cutting edge" :-/ One is proably expected to let firewalld manage everything as suggested above. Also, my first attemp was to let libvirtd deal with it using "xtables-nft-multi" iptables implementation (eselect iptables...), which I though was some kind of adapter. It didn't work.
| Quote: | | If you want disable iptables you can remove its start at boot-time (OpenRC: It is /etc/init.d/iptables). Which doesnt help if some systems sets up some ipatbles commands - yes, in this case you must remove kernel modules. But a better way would be to tell libvirt doing nftables commands. |
I don't know anymore about openrc, but I think the only thing the equivalent systemd service does is to restore the saved rules (meaning, if you disable it, you just have iptables working with the default, i.e., none, rules). I guess dropping support in the kernel must indeed be the way. |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Mon Sep 26, 2022 7:00 pm Post subject: |
|
|
An update...I was wrong about the REFUSED errors in dig: they have nothing to do with libvirtd but start to happen when I start another instance of dnsmasq with the above setup replacing the port, i.e.,
| Code: | interface=vm-bridge
bind-dynamic
no-poll
no-resolv
dhcp-range=192.168.2.2,192.168.2.255
server=127.0.0.1#53 |
I guess dig is asking this new instance of dnsmasq, which only listens to interface vm-bridge. I think my problem is I don't get how dnsmasq works... For instance, shouldn't I tell dnsmasq to listen in a different port (e.g., port=54) so that I don't have Networkmanager's and my own instance both listening on port 53 (or setup Networkmanager to start its instance at some other port)? |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
| Posted: Sat Oct 01, 2022 8:09 am Post subject: |
|
|
I simplified the dnsmasq thing and now I just start a single dnsmasq instance (through systemd) with the following parameters in /etc/dnsmasq.conf:
| Code: | port=5028 # standard port 5353 was already in use
bind-dynamic
interface=vm-bridge
dhcp-range=192.168.2.2,192.168.2.255 |
(I don't know whether it is to be expected but this dnsmasq server, despite the interface=vm-bridge, still answers requests that I write (dig) in any terminal)
The result is apparently the same as before: the VM has internet connection (I can ping sites) but no DNS server. One thing I don't get is: if the VM got hold of an IP in the above range (served by dnsmasq), that must mean that it has access to the dnsmasq instance (right?)...but then, why is not using it also for DNS?
In the Windows VM, I checked the network properties, and I found indeed that the DNS server is missing. I took a screenshot (it's in Spanish, sorry): https://ibb.co/vDnS3VZ
Any other clue? |
|
| Back to top |
|
|
mani001
Guru
Joined: 04 Dec 2004
Posts: 487
Location: Oleiros
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
