| View previous topic :: View next topic |
| Author |
Message |
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9882
Location: almost Mile High in the USA
|
 Posted: Tue Sep 29, 2015 3:30 pm Post subject: NFS speed over virtual machines/qemu |
 |
|
I wonder what people are seeing for NFS transaction speeds to virtual machines on the same host.
Setup: Core2 Quad Q9550S with 6GB RAM (DDR2/PC5300)
QEMU/KVM on host
Server disks is LVM on RAID5. All partitions except boot on LVM.
NFS server running on host to serve out LVM partitions
Networking: Software bridge, QEMU network via macvtap/macvlan. virtual machine using NFS.
Problem: NFS is extremely slow. For small files I'm seemingly getting IOPS of like 5 or so. When I get queued SMTP from remote hosts on my virtual machine, the IOPS is so slow that it can timeout when sendmail runs out of queue slots.
Interactive performance of shells is also very slow due to file open/close/lock speeds. I think after the files get opened/closed the performance is OK.
Any ideas how to speed things up? Should it be this slow? What kinds of speeds are you seeing? Perhaps LVM or RAID5 is what's causing issues?
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
 |
szatox
Advocate
Joined: 27 Aug 2013
Posts: 3493
|
| Posted: Tue Sep 29, 2015 6:02 pm Post subject: |
|
|
| Quote: | | For small files I'm seemingly getting IOPS of like 5 or so. |
How you measure it so I could repeat it?
If it's getting slower for small files, I'd look at NFS itself first.
/etc/conf.d/nfs, the following line
# Number of servers to be started up by default
OPTS_RPC_NFSD="8"
This is the first thing that might need a big increase.
Another thing, is your IO blocking or parallel on the client side? (Does initializing session have a big impact?)
| Quote: | starting machine testmachine with command
qemu-system-x86_64 -name testmachine -enable-kvm -m 2048 -smp cores=2 -net nic,model=rtl8139,macaddr=DE:AD:BE:EF:8F:2E,vlan=1 -net tap,ifname=testmachine-1,script=no,downscript=no,vlan=1
|
Well, it's an ancient image build without virtio, so it's pretty obvious where's the bottleneck, but let's see it anyway.
running "tar -c / | dd of=/dev/null" on qemu with squashfs over nfs root, so unfortunately it's one big file, but still:
iftop at host:
| Code: |
TX: cum: 1.31GB peak: 76.1Mb rates: 74.6Mb 73.2Mb 73.2Mb
RX: 250MB 14.2Mb 13.9Mb 13.6Mb 13.6Mb
TOTAL: 1.56GB 90.3Mb 88.6Mb 86.8Mb 86.8Mb |
a bit later
| Code: | TX: cum: 4.31GB peak: 76.3Mb rates: 55.9Mb 40.5Mb 49.5Mb
RX: 816MB 14.2Mb 8.55Mb 6.20Mb 8.08Mb
TOTAL: 5.11GB 90.6Mb 64.5Mb 46.7Mb 57.6Mb
|
and it keeps 2 nfsd bussy:
| Code: | PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12588 me 20 0 2761248 2.051g 34536 S 150.7 28.1 24:49.99 qemu-system-x86
4743 root 20 0 0 0 0 S 16.9 0.0 3:05.92 nfsd
4742 root 20 0 0 0 0 R 9.9 0.0 1:44.86 nfsd
7357 me 20 0 1369448 474248 92620 S 7.9 6.2 124:05.33 seamonkey
4174 root 20 0 305836 63568 34856 S 5.9 0.8 63:32.69 X |
On the other hand, running tar against /usr/portage (without distfiles) gives ~4Mbps while running separate tar for every directory inside /usr/portage (skipping distfiles) gives between 15Mb and 40Mb |
|
| Back to top |
|
|
eccerr0r
Watchman
Joined: 01 Jul 2004
Posts: 9882
Location: almost Mile High in the USA
|
| Posted: Tue Sep 29, 2015 6:21 pm Post subject: |
|
|
I suppose the perceived 5 IOPS more like a guess - I should look for a better benchmark but it needs to also test flocks. The impact is very visible at least to processing sendmail requests. I think it takes at least a minute to process a single mail from sendmail -> procmail -> write to disk, and if another sendmail request comes it, they seem to clash with each other, almost exponentially increasing time taken to complete. Come on, this is a small text email, shouldn't take a minute to procmail and write to disk... sometimes I get large batches of emails and sendmail seems to livelock - and I stop getting any mail.
I think just one of the nfsd's on the host/VM server is in D wait most of the time, but others are sleeping. It's clearly waiting on disk resources.
_________________
Intel Core i7 2700K/Radeon R7 250/24GB DDR3/256GB SSD
What am I supposed watching? |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
Networking & Security
