View previous topic :: View next topic |
Author |
Message |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Sun Jul 07, 2024 2:09 am Post subject: [SOLVED] NFS mounts at boot on WAN, but not LAN |
|
|
I have a gentoo desktop configured to mount an NFS directory from a server at boot. Previously, it was at a remote location, and correctly mounted the directory over the WAN. Recently, I moved it to the same LAN as the server, and now the directory doesn't mount at boot. However, Once the desktop boots up, I can manually mount them without a problem. Interestingly, this problem persists regardless if /etc/fstab uses the WAN or LAN address of the server.
Here's the fstab config:
Code: | <IP>:/shared /shared nfs bg,timeo=14,_netdev,cto,async,noatime,rsize=32768,wsize=32768 0 0 |
Any ideas?
Last edited by jyoung on Fri Aug 16, 2024 5:05 pm; edited 1 time in total |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Hu Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
Joined: 06 Mar 2007 Posts: 22375
|
Posted: Sun Jul 07, 2024 12:32 pm Post subject: |
|
|
When it fails to mount at boot on the LAN, does it attempt to mount and get an error or does it skip attempting to mount? Understanding whether it tries and fails, or chooses not to try at all, will influence where we go next. On the assumption it tries and fails, what messages does it print? You used _netdev, which is good. Does the output indicate that the relevant networking is up before you see the output of the attempted mount? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Sun Jul 07, 2024 9:39 pm Post subject: |
|
|
It looks like it attempts the mount, but is unable, since I get this warning
Code: | WARNING: netmount will start when net.enp4s0 has started |
That sounds like it's trying to mount the NFS directory before the network is up. That's weird, since earlier in the boot messages there's a line
Code: | Bringing up network interface lo ..." |
And no error message follows. Also, I have no difficulty logging in remotely within seconds of a reboot, so presumably the network is up pretty fast. But, maybe not fast enough for netmount? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Mon Jul 22, 2024 4:48 pm Post subject: |
|
|
Hi Folks, I'm still searching for a solution to this. The first thing I tried was to increase timeo from 14 to 30, since that might help if the network was slow to come up. That didn't seem likely, since I'm able to ssh into this computer pretty fast, but worth trying. That didn't help. Then, I tried simplifying the entry in /etc/fstab. Nothing in the old entry looked problematic, but it's probably best to keep things simple while debugging.
Code: | <IP>:/shared /shared nfs rw,_netdev 0 0 |
No improvement. And one more item, I noticed a clock drift on the computer that's having the problem. I setup ntpd, and now it's fully synced, but still no improvement.
Hu, I like your idea of trying asses whether it's getting an error or skipping the mount, but I'm not sure how to diagnose this issue. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Mon Jul 22, 2024 4:54 pm Post subject: |
|
|
And one more possible clue: I've noticed that if I create a file on the shared directory using the machine that's having the problem, it doesn't appear right away to other machines on the network. The lag is long, maybe 30s to a minute. I'm not sure if this related, but this second issue seems new too. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
freke Veteran
![Veteran Veteran](/images/ranks/rank_rect_5_vet.gif)
Joined: 23 Jan 2003 Posts: 1018 Location: Somewhere in Denmark
|
Posted: Wed Jul 24, 2024 6:39 pm Post subject: |
|
|
jyoung wrote: | It looks like it attempts the mount, but is unable, since I get this warning
Code: | WARNING: netmount will start when net.enp4s0 has started |
That sounds like it's trying to mount the NFS directory before the network is up. That's weird, since earlier in the boot messages there's a line
Code: | Bringing up network interface lo ..." |
And no error message follows. Also, I have no difficulty logging in remotely within seconds of a reboot, so presumably the network is up pretty fast. But, maybe not fast enough for netmount? |
If you edit /etc/conf.d/netmount to need net.enp4s0 ie. Code: | rc_need="net.enp4s0" | in said file?
(and restart net/reboot) |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
gentoo_ram Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 25 Oct 2007 Posts: 495 Location: San Diego, California USA
|
Posted: Wed Jul 24, 2024 11:29 pm Post subject: |
|
|
This answer assumes OpenRC startup. If you are systemd, nevermind.
How is the variable "rc_depend_strict" set in /etc/rc.conf?
If it's YES then I don't think any of the network services will start until all network interfaces are up. If it isn't then I think the network services will start immediately since the loopback interface always starts up.
It sounds like maybe you really want YES for that setting. That's how I have it set.
Or maybe you have you rc_parallel set? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Sat Jul 27, 2024 6:23 pm Post subject: |
|
|
Thanks, this system is using OpenRC.
I tried both of those changes, "rc_depend_strict" /etc/rc.conf and rc_need="net.enp4s0" in /etc/conf.d/netmount, and neither worked, either separately or in combination. Both seem sensible. Are there any other diagnostics we could try? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Hu Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
Joined: 06 Mar 2007 Posts: 22375
|
Posted: Sat Jul 27, 2024 7:00 pm Post subject: |
|
|
You showed earlier that netmount will wait for the network to be available. This seems like a good thing. What happens when netmount ceases waiting and tries the mount? Does it ever try the mount? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Sat Jul 27, 2024 8:10 pm Post subject: |
|
|
Is there a log I could check? /var/log/messages doesn't seem to mention anything about an attempted mount. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
Hu Administrator
![Administrator Administrator](/images/ranks/rank-admin.gif)
Joined: 06 Mar 2007 Posts: 22375
|
Posted: Sat Jul 27, 2024 9:20 pm Post subject: |
|
|
Relevant output should be on the console. You might also be able to capture it with the openrc boot logging facility. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Sun Jul 28, 2024 3:21 pm Post subject: |
|
|
Here's a chunk from /var/log/rc.log that contains messages about enp4s0 and netmount.
Code: | * Bringing up interface enp4s0
* Caching network module dependencies
* Starting ifplugd on enp4s0 ...
[ ok ]
* Backgrounding ...
* WARNING: net.enp4s0 has started, but is inactive
* Starting rpcbind ...
[ ok ]
* Starting NFS statd ...
[ ok ]
* Setting up RPC pipefs ...
[ ok ]
* Starting idmapd ...
[ ok ]
* Starting NFS sm-notify ...
[ ok ]
* WARNING: netmount will start when net.enp4s0 has started
|
This message seems weird "net.enp4s0 has started, but is inactive". Is there a lag in getting a network connection? Is this normal?
This message is also interesting: "* WARNING: netmount will start when net.enp4s0 has started". That seems weird, since net.enp4s0 has already started (though it's "inactive"). But, even if that's correct, then when does netmount get started? Maybe never? Or, since netmount is the service, maybe it's starting, but not mounting this specific shared folder? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
grknight Retired Dev
![Retired Dev Retired Dev](/images/ranks/rank-retired.gif)
Joined: 20 Feb 2015 Posts: 1832
|
Posted: Sun Jul 28, 2024 3:51 pm Post subject: |
|
|
jyoung wrote: | This message seems weird "net.enp4s0 has started, but is inactive". Is there a lag in getting a network connection? Is this normal?
This message is also interesting: "* WARNING: netmount will start when net.enp4s0 has started". That seems weird, since net.enp4s0 has already started (though it's "inactive"). But, even if that's correct, then when does netmount get started? Maybe never? Or, since netmount is the service, maybe it's starting, but not mounting this specific shared folder? |
This is because ifplugd is installed. If enp4s0 is always "plugged" by a cable, set modules_enp4s0="!ifplugd" in /etc/conf.d/net to disable this behavior. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Tue Jul 30, 2024 3:30 pm Post subject: |
|
|
I noticed one more clue: The shared folder mounts at boot correct *if* the machine's local network isn't connected to the WAN. (I found this out only because my internet was out yesterday). That seems so weird, since both the server and the host are on the local network. How could that affect this problem?
grknight, could you say a bit more? Which behavior would be disabled by modules_enp4s0="!ifplugd"? |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
grknight Retired Dev
![Retired Dev Retired Dev](/images/ranks/rank-retired.gif)
Joined: 20 Feb 2015 Posts: 1832
|
Posted: Tue Jul 30, 2024 10:45 pm Post subject: |
|
|
jyoung wrote: | grknight, could you say a bit more? Which behavior would be disabled by modules_enp4s0="!ifplugd"? |
Specifically this: Code: | * Starting ifplugd on enp4s0 ...
[ ok ]
* Backgrounding ...
* WARNING: net.enp4s0 has started, but is inactive |
|
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Wed Jul 31, 2024 5:05 pm Post subject: |
|
|
Interesting, /etc/conf.d/net did not exist, so I created it and added
Code: | modules_enp4s0="!ifplugd" |
I found that there is now no mention of enp4s0 in /var/log/rc.log. However, the messages about NFS have changed
Code: | * Starting NFS statd ...
[ ok ]
* Setting up RPC pipefs ...
[ ok ]
* Starting idmapd ...
[ ok ]
* Starting NFS sm-notify ...
[ ok ]
* Mounting network filesystems ...
mount.nfs: Network is unreachable for <IP>:/shared on /shared
* Could not mount all network filesystems
[ !! ]
|
Sadly, the effect is the same, /shared does not mount at boot, but can be mounted after the fact. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Fri Aug 09, 2024 3:31 pm Post subject: |
|
|
Hi Folks, I'm still trying to trouble-shoot this item. I added these two lines to /etc/rc.conf
Code: | rc_verbose=yes
rc_netmount_before="net.enp4s0" |
I also added dhcpcd to the default runlevel. I'm actually not sure why it wasn't there before. None of these affected the issue. However, the verbose output did reveal a weird set of errors. Here's the full /var/log/rc.log from a boot-up:
Code: | rc default logging started at Fri Aug 9 10:22:36 2024
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/sysklogd start
* Starting sysklogd ...
* start-stop-daemon: fopen `/run/sysklogd.pid': No such file or directory
* Detaching to start `/usr/sbin/syslogd' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/cronie start
* Starting cronie ...
* start-stop-daemon: fopen `/run/crond.pid': No such file or directory
* Detaching to start `/usr/sbin/crond' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/dhcpcd start
* Starting DHCP Client Daemon ...
* start-stop-daemon: fopen `/var/run/dhcpcd/pid': No such file or directory
* Detaching to start `/sbin/dhcpcd' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/dbus start
* Starting dbus ...
* start-stop-daemon: fopen `/run/dbus.pid': No such file or directory
* Detaching to start `/usr/bin/dbus-daemon' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/elogind start
* Starting elogind ...
* start-stop-daemon: fopen `/run/elogind.pid': No such file or directory
* Detaching to start `/lib64/elogind/elogind' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/rpcbind start
* Starting rpcbind ...
* Detaching to start `/sbin/rpcbind' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/rpc.statd start
* Starting NFS statd ...
* Detaching to start `/sbin/rpc.statd' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/rpc.pipefs start
* Setting up RPC pipefs ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/rpc.idmapd start
* Starting idmapd ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/nfsclient start
* Starting NFS sm-notify ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/netmount start
* Mounting network filesystems ...
mount.nfs: Network is unreachable for <IP>:/shared on /shared
* Could not mount all network filesystems
[ !! ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/ntpd start
* Starting ntpd ...
* start-stop-daemon: fopen `/var/run/ntpd.pid': No such file or directory
* Detaching to start `/usr/sbin/ntpd' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/sshd start
* Starting sshd ...
* start-stop-daemon: fopen `/run/sshd.pid': No such file or directory
* Detaching to start `/usr/sbin/sshd' ...
[ ok ]
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/display-manager-setup start
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/display-manager start
* Starting display-manager ...
* start-stop-daemon: fopen `/run/lxdm.pid': No such file or directory
* Detaching to start `/usr/sbin/lxdm-binary' ...
[ ok ]
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/xdm start
* Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/local start
* Starting local ...
* Executing "/etc/local.d/ntp.start" ...
[ ok ]
[ ok ]
rc default logging stopped at Fri Aug 9 10:22:42 2024
|
Apparently, none of the services can find their .pid file in /run. I'm not sure if that's related to this problem, but that seems kind of serious. I checked after the boot-up, and /run exists, is owned by root, and is rwx for root. It's also populated with files, some .pid files.
Because of my job's workflow, this computer will be moving back outside the local network again today or tomorrow. The last time it was outside the LAN, this problem didn't manifest. It would be kind of good if it went away again, but it might limit the ability to debug this issue. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
grknight Retired Dev
![Retired Dev Retired Dev](/images/ranks/rank-retired.gif)
Joined: 20 Feb 2015 Posts: 1832
|
Posted: Fri Aug 09, 2024 5:21 pm Post subject: |
|
|
jyoung wrote: | Apparently, none of the services can find their .pid file in /run. I'm not sure if that's related to this problem, but that seems kind of serious. |
PID files do not exist before a service's daemon starts. This is normal behavior.
If you want to ensure that a certain IP is available, configure /etc/conf.d/net-online and add rc_need="network-online" to /etc/conf.d/netmount |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
jyoung Guru
![Guru Guru](/images/ranks/rank_rect_3.gif)
Joined: 20 Mar 2007 Posts: 456
|
Posted: Fri Aug 16, 2024 5:05 pm Post subject: |
|
|
grknight, that did the trick. I've rebooted several times to eliminate a false positive, and this solution is holding.
So, it seems like the network interface came up before netmount, or netmout waited as it should, but the specific IP wasn't available right away. Does that sound right? I'm going to mark this thread as SOLVED, but anyone wants to add closing thoughts it might help future readers. |
|
Back to top |
|
![](templates/gentoo/images/spacer.gif) |
|