IO hang for 60-90 seconds deleting many small files [solved]

lyz · n00b Joined: 17 Feb 2003 Posts: 14

Can't seem to fix this. Bear with me as this will be a lot of information.

The bottom line is that when deleting a unpacked kernel with rm -rf the system just hangs for 60-90 seconds.

I have two xfs filesystems that this has been tested on. Both of them are inside of LVM.

One is on an ssd:
/dev/mapper/ssdvg0-usr on /usr type xfs (rw,noatime,nodiratime,discard,delaylog,logbufs=8,logbsize=256k,nobarrier)
One is on software raid consisting of two hard drives:
/dev/mapper/vg-home on /home type xfs (rw,noatime,nodiratime,delaylog,logbufs=8,logbsize=32k)

Most of the mount options listed are from reading up on this for a few hours.

I have changed the scheduler on the ssd to deadline.

The filesystem on the ssd is newer than the one on the disks:

roarinelk · Guru Joined: 04 Mar 2004 Posts: 520

can you test each disk individually (i.e. without LVM) ? So we could narrow it down
to either the SSD firmware being buggy (NCQ), a bug in LVM, or buggy SATA controller (many are
known to have subtle problems with NCQ).

I'm no expert but judging from the backtraces it seems that command submission to
one of the drives is stuck. So either buggy firmware on one of the disks or buggy
SATA controller/driver or both).

lyz · n00b Joined: 17 Feb 2003 Posts: 14

I am able to test the two hard disks without LVM, but not the SSD. When just mounted alone, the disks fly.

/dev/sda2 on /media/temp/sda type xfs (rw,noatime,nodiratime,delaylog,logbufs=8,logbsize=256k)
/dev/sdb2 on /media/temp/sdb type xfs (rw,noatime,nodiratime,delaylog,logbufs=8,logbsize=256k)

core24 sda # time rm -rf linux-3.2.6-gentoo/
real 0m1.452s
user 0m0.000s
sys 0m1.420s

core24 sdb # time rm -rf linux-3.2.6-gentoo/

real 0m1.690s
user 0m0.060s
sys 0m1.490s

My home mount is much slower. It is striped raid and LVM.
core24 lyz # pwd
/home/lyz
core24 lyz # time rm -rf linux-3.2.6-gentoo/

real 0m6.631s
user 0m0.000s
sys 0m1.670s

The ssd time is terrible and has the freezes. There isn't a way I can test this outside of LVM without a few resize commands, I'm hoping that I don't have to do that.

/dev/mapper/ssdvg0-usr on /usr type xfs (rw,noatime,nodiratime,discard,delaylog,logbufs=8,logbsize=256k,nobarrier)

core24 usr # time rm -rf linux-3.2.6-gentoo/
real 4m5.474s
user 0m0.040s
sys 0m2.000s

When a freeze occurs, I cannot save a document to /home. That's one of the things that is throwing me off. Gonna reboot and try without the discard option.

lyz · n00b Joined: 17 Feb 2003 Posts: 14

It was discard!

/dev/mapper/ssdvg0-usr on /usr type xfs (rw,noatime,nodiratime,delaylog,logbufs=8,logbsize=256k,nobarrier)

core24 src # time rm -rf linux-3.2.6-gentoo2/

real 0m1.702s
user 0m0.050s
sys 0m1.570s

I just the question now what is the impact of turning the discard mount option off? I read a few things on it. Isn't is supposed to make things faster?

invasivenorman · n00b Joined: 06 Feb 2012 Posts: 28 Location: ::1

From reading around a bit, it isn't well-tested at this point. Also, your SSD has to support TRIM for discard to make any sense.
_________________
gurgle

lyz · n00b Joined: 17 Feb 2003 Posts: 14

Thanks for looking it up. I didn't mention the type of SSD. It is a PP120GS25SSDR. It does come with trim support.

lyz · n00b Joined: 17 Feb 2003 Posts: 14

I did a test on ext4. discard works fine. By process of elimination, that would mean the problem is xfs combined with discard. It looks like

lyz · n00b Joined: 17 Feb 2003 Posts: 14

The result from a discussion on their mailing list is, essentially, don't mount a drive using discard. Run fstrim using cron instead.