[Solved] NFS server goes down like clockwork, every 15 mins

TJNII · l33t Joined: 09 Nov 2003 Posts: 648 Location: for(;;);

Tonight I made the mistake of trying to upgrade my system. Now NFS (1.0.12) doesn't work right. The server will run for 15 minutes (exactly. I timed it.) and then go belly up. Mounts on client machines returr the "Stale NFS file handle" error. Restarting the NFS server corrects this, with no action needed on the client.

I haven't changed anything from my old version (1.0.4, I think. Stable version from early this year), other than update nfs-utils. I've tried no_subtree_check in my exports file and -o rsize=8192,wsize=8192,tcp,hard,intr on the clients. No_subtree_check made it last a little longer, client options didn't have any effect (other than making the client apps wig out less).

I don't see anything usefull in the logs, though I'm not sure exactly which logfile NFS dumps to. /usr/sbin/rpc.nfsd doesn't have a verbose option.

Help?

turtles · Veteran Joined: 31 Dec 2004 Posts: 1696

I am no expert on this but have you recompiled your kernel? IF yes > Can you revert you your old kernel version?
EDIT: Post

embobo · Guru Joined: 19 May 2003 Posts: 311

I've had similar problems. I traced the problem to kernel 2.6.22. Until today, I've had to use 2.6.21. Now I'm trying 2.6.22-gentoo-r4.

From the client try

turtles · Veteran Joined: 31 Dec 2004 Posts: 1696

Here is a similar problem with a work around

embobo · Guru Joined: 19 May 2003 Posts: 311

TJNII · l33t Joined: 09 Nov 2003 Posts: 648 Location: for(;;);

I haven't touched the kernel in months. Unless there is some quirk in 1.0.12 and the 2.6.16 kernels I doubt that's the problem.

Everything else works OK on the system, and I can get the fault time to vary with different options, so I don't think its failing hardware. (Though I've been wrong before.) Only nfs seems to have a problem, unlike that other linked thread. If it is some wierd combination issue it is going to be stupid hard to find.

I've turned off all the use flags and I have conservative compiler flags. I don't know what else to try.

TJNII · l33t Joined: 09 Nov 2003 Posts: 648 Location: for(;;);

Last night I did the 15 minute restart and the client didn't start working again. It had done this before, but infrequently. So instead of /etc/init.d/nfs restart, which I had been doing, I did /etc/init.d/nfs stop. While the restarts had been reporting success, this reported fail. a ps -ec revealed a couple rpc processes that wouldn't stop, including rpc.gssd. Thus, rpc.gssd had been running since I compiled out Kerberos a few days and many nfs restarts before. I killed all the rpc processes still running, restarted portmap for good measure, and started nfs again. Now it seems to be working.

I have a feeling, though I can't prove it now, that the root cause was Kerberos. I had a Kerberos use flag set from when I fiddled with it a long time ago, but I never set up Kerberos properly. Until this update I don't think nfs tried to use Kerberos, so I never had a problem.