Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[SOLVED] NFS mounts at boot on WAN, but not LAN
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Networking & Security
View previous topic :: View next topic  
Author Message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Sun Jul 07, 2024 2:09 am    Post subject: [SOLVED] NFS mounts at boot on WAN, but not LAN Reply with quote

I have a gentoo desktop configured to mount an NFS directory from a server at boot. Previously, it was at a remote location, and correctly mounted the directory over the WAN. Recently, I moved it to the same LAN as the server, and now the directory doesn't mount at boot. However, Once the desktop boots up, I can manually mount them without a problem. Interestingly, this problem persists regardless if /etc/fstab uses the WAN or LAN address of the server.

Here's the fstab config:


Code:
<IP>:/shared    /shared nfs bg,timeo=14,_netdev,cto,async,noatime,rsize=32768,wsize=32768 0 0


Any ideas?


Last edited by jyoung on Fri Aug 16, 2024 5:05 pm; edited 1 time in total
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22366

PostPosted: Sun Jul 07, 2024 12:32 pm    Post subject: Reply with quote

When it fails to mount at boot on the LAN, does it attempt to mount and get an error or does it skip attempting to mount? Understanding whether it tries and fails, or chooses not to try at all, will influence where we go next. On the assumption it tries and fails, what messages does it print? You used _netdev, which is good. Does the output indicate that the relevant networking is up before you see the output of the attempted mount?
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Sun Jul 07, 2024 9:39 pm    Post subject: Reply with quote

It looks like it attempts the mount, but is unable, since I get this warning

Code:
WARNING: netmount will start when net.enp4s0 has started


That sounds like it's trying to mount the NFS directory before the network is up. That's weird, since earlier in the boot messages there's a line

Code:
Bringing up network interface lo ..."


And no error message follows. Also, I have no difficulty logging in remotely within seconds of a reboot, so presumably the network is up pretty fast. But, maybe not fast enough for netmount?
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Mon Jul 22, 2024 4:48 pm    Post subject: Reply with quote

Hi Folks, I'm still searching for a solution to this. The first thing I tried was to increase timeo from 14 to 30, since that might help if the network was slow to come up. That didn't seem likely, since I'm able to ssh into this computer pretty fast, but worth trying. That didn't help. Then, I tried simplifying the entry in /etc/fstab. Nothing in the old entry looked problematic, but it's probably best to keep things simple while debugging.

Code:
<IP>:/shared    /shared nfs rw,_netdev 0 0


No improvement. And one more item, I noticed a clock drift on the computer that's having the problem. I setup ntpd, and now it's fully synced, but still no improvement.

Hu, I like your idea of trying asses whether it's getting an error or skipping the mount, but I'm not sure how to diagnose this issue.
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Mon Jul 22, 2024 4:54 pm    Post subject: Reply with quote

And one more possible clue: I've noticed that if I create a file on the shared directory using the machine that's having the problem, it doesn't appear right away to other machines on the network. The lag is long, maybe 30s to a minute. I'm not sure if this related, but this second issue seems new too.
Back to top
View user's profile Send private message
freke
Veteran
Veteran


Joined: 23 Jan 2003
Posts: 1017
Location: Somewhere in Denmark

PostPosted: Wed Jul 24, 2024 6:39 pm    Post subject: Reply with quote

jyoung wrote:
It looks like it attempts the mount, but is unable, since I get this warning

Code:
WARNING: netmount will start when net.enp4s0 has started


That sounds like it's trying to mount the NFS directory before the network is up. That's weird, since earlier in the boot messages there's a line

Code:
Bringing up network interface lo ..."


And no error message follows. Also, I have no difficulty logging in remotely within seconds of a reboot, so presumably the network is up pretty fast. But, maybe not fast enough for netmount?


If you edit /etc/conf.d/netmount to need net.enp4s0 ie.
Code:
rc_need="net.enp4s0"
in said file?
(and restart net/reboot)
Back to top
View user's profile Send private message
gentoo_ram
Guru
Guru


Joined: 25 Oct 2007
Posts: 495
Location: San Diego, California USA

PostPosted: Wed Jul 24, 2024 11:29 pm    Post subject: Reply with quote

This answer assumes OpenRC startup. If you are systemd, nevermind.

How is the variable "rc_depend_strict" set in /etc/rc.conf?

If it's YES then I don't think any of the network services will start until all network interfaces are up. If it isn't then I think the network services will start immediately since the loopback interface always starts up.

It sounds like maybe you really want YES for that setting. That's how I have it set.

Or maybe you have you rc_parallel set?
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Sat Jul 27, 2024 6:23 pm    Post subject: Reply with quote

Thanks, this system is using OpenRC.

I tried both of those changes, "rc_depend_strict" /etc/rc.conf and rc_need="net.enp4s0" in /etc/conf.d/netmount, and neither worked, either separately or in combination. Both seem sensible. Are there any other diagnostics we could try?
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22366

PostPosted: Sat Jul 27, 2024 7:00 pm    Post subject: Reply with quote

You showed earlier that netmount will wait for the network to be available. This seems like a good thing. What happens when netmount ceases waiting and tries the mount? Does it ever try the mount?
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Sat Jul 27, 2024 8:10 pm    Post subject: Reply with quote

Is there a log I could check? /var/log/messages doesn't seem to mention anything about an attempted mount.
Back to top
View user's profile Send private message
Hu
Administrator
Administrator


Joined: 06 Mar 2007
Posts: 22366

PostPosted: Sat Jul 27, 2024 9:20 pm    Post subject: Reply with quote

Relevant output should be on the console. You might also be able to capture it with the openrc boot logging facility.
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Sun Jul 28, 2024 3:21 pm    Post subject: Reply with quote

Here's a chunk from /var/log/rc.log that contains messages about enp4s0 and netmount.
Code:
 * Bringing up interface enp4s0
 *   Caching network module dependencies
 *   Starting ifplugd on enp4s0 ...
 [ ok ]
 *     Backgrounding ...
 * WARNING: net.enp4s0 has started, but is inactive
 * Starting rpcbind ...
 [ ok ]
 * Starting NFS statd ...
 [ ok ]
 * Setting up RPC pipefs ...
 [ ok ]
 * Starting idmapd ...
 [ ok ]
 * Starting NFS sm-notify ...
 [ ok ]
 * WARNING: netmount will start when net.enp4s0 has started


This message seems weird "net.enp4s0 has started, but is inactive". Is there a lag in getting a network connection? Is this normal?

This message is also interesting: "* WARNING: netmount will start when net.enp4s0 has started". That seems weird, since net.enp4s0 has already started (though it's "inactive"). But, even if that's correct, then when does netmount get started? Maybe never? Or, since netmount is the service, maybe it's starting, but not mounting this specific shared folder?
Back to top
View user's profile Send private message
grknight
Retired Dev
Retired Dev


Joined: 20 Feb 2015
Posts: 1832

PostPosted: Sun Jul 28, 2024 3:51 pm    Post subject: Reply with quote

jyoung wrote:
This message seems weird "net.enp4s0 has started, but is inactive". Is there a lag in getting a network connection? Is this normal?

This message is also interesting: "* WARNING: netmount will start when net.enp4s0 has started". That seems weird, since net.enp4s0 has already started (though it's "inactive"). But, even if that's correct, then when does netmount get started? Maybe never? Or, since netmount is the service, maybe it's starting, but not mounting this specific shared folder?

This is because ifplugd is installed. If enp4s0 is always "plugged" by a cable, set modules_enp4s0="!ifplugd" in /etc/conf.d/net to disable this behavior.
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Tue Jul 30, 2024 3:30 pm    Post subject: Reply with quote

I noticed one more clue: The shared folder mounts at boot correct *if* the machine's local network isn't connected to the WAN. (I found this out only because my internet was out yesterday). That seems so weird, since both the server and the host are on the local network. How could that affect this problem?

grknight, could you say a bit more? Which behavior would be disabled by modules_enp4s0="!ifplugd"?
Back to top
View user's profile Send private message
grknight
Retired Dev
Retired Dev


Joined: 20 Feb 2015
Posts: 1832

PostPosted: Tue Jul 30, 2024 10:45 pm    Post subject: Reply with quote

jyoung wrote:
grknight, could you say a bit more? Which behavior would be disabled by modules_enp4s0="!ifplugd"?

Specifically this:
Code:
 *   Starting ifplugd on enp4s0 ...
 [ ok ]
 *     Backgrounding ...
 * WARNING: net.enp4s0 has started, but is inactive
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Wed Jul 31, 2024 5:05 pm    Post subject: Reply with quote

Interesting, /etc/conf.d/net did not exist, so I created it and added

Code:
modules_enp4s0="!ifplugd"


I found that there is now no mention of enp4s0 in /var/log/rc.log. However, the messages about NFS have changed

Code:
 * Starting NFS statd ...
 [ ok ]
 * Setting up RPC pipefs ...
 [ ok ]
 * Starting idmapd ...
 [ ok ]
 * Starting NFS sm-notify ...
 [ ok ]
 * Mounting network filesystems ...
mount.nfs: Network is unreachable for <IP>:/shared on /shared
 * Could not mount all network filesystems
 [ !! ]


Sadly, the effect is the same, /shared does not mount at boot, but can be mounted after the fact.
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Fri Aug 09, 2024 3:31 pm    Post subject: Reply with quote

Hi Folks, I'm still trying to trouble-shoot this item. I added these two lines to /etc/rc.conf

Code:
rc_verbose=yes
rc_netmount_before="net.enp4s0"


I also added dhcpcd to the default runlevel. I'm actually not sure why it wasn't there before. None of these affected the issue. However, the verbose output did reveal a weird set of errors. Here's the full /var/log/rc.log from a boot-up:

Code:
rc default logging started at Fri Aug  9 10:22:36 2024

 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/sysklogd start
 * Starting sysklogd ...
 * start-stop-daemon: fopen `/run/sysklogd.pid': No such file or directory
 * Detaching to start `/usr/sbin/syslogd' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/cronie start
 * Starting cronie ...
 * start-stop-daemon: fopen `/run/crond.pid': No such file or directory
 * Detaching to start `/usr/sbin/crond' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/dhcpcd start
 * Starting DHCP Client Daemon ...
 * start-stop-daemon: fopen `/var/run/dhcpcd/pid': No such file or directory
 * Detaching to start `/sbin/dhcpcd' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/dbus start
 * Starting dbus ...
 * start-stop-daemon: fopen `/run/dbus.pid': No such file or directory
 * Detaching to start `/usr/bin/dbus-daemon' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/elogind start
 * Starting elogind ...
 * start-stop-daemon: fopen `/run/elogind.pid': No such file or directory
 * Detaching to start `/lib64/elogind/elogind' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/rpcbind start
 * Starting rpcbind ...
 * Detaching to start `/sbin/rpcbind' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/rpc.statd start
 * Starting NFS statd ...
 * Detaching to start `/sbin/rpc.statd' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/rpc.pipefs start
 * Setting up RPC pipefs ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/rpc.idmapd start
 * Starting idmapd ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/nfsclient start
 * Starting NFS sm-notify ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/netmount start
 * Mounting network filesystems ...
mount.nfs: Network is unreachable for <IP>:/shared on /shared
 * Could not mount all network filesystems
 [ !! ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/ntpd start
 * Starting ntpd ...
 * start-stop-daemon: fopen `/var/run/ntpd.pid': No such file or directory
 * Detaching to start `/usr/sbin/ntpd' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/sshd start
 * Starting sshd ...
 * start-stop-daemon: fopen `/run/sshd.pid': No such file or directory
 * Detaching to start `/usr/sbin/sshd' ...
 [ ok ]
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/display-manager-setup start
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/display-manager start
 * Starting display-manager ...
 * start-stop-daemon: fopen `/run/lxdm.pid': No such file or directory
 * Detaching to start `/usr/sbin/lxdm-binary' ...
 [ ok ]
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/xdm start
 * Executing: /lib/rc/sh/openrc-run.sh /lib/rc/sh/openrc-run.sh /etc/init.d/local start
 * Starting local ...
 *   Executing "/etc/local.d/ntp.start" ...
 [ ok ]
 [ ok ]

rc default logging stopped at Fri Aug  9 10:22:42 2024


Apparently, none of the services can find their .pid file in /run. I'm not sure if that's related to this problem, but that seems kind of serious. I checked after the boot-up, and /run exists, is owned by root, and is rwx for root. It's also populated with files, some .pid files.

Because of my job's workflow, this computer will be moving back outside the local network again today or tomorrow. The last time it was outside the LAN, this problem didn't manifest. It would be kind of good if it went away again, but it might limit the ability to debug this issue.
Back to top
View user's profile Send private message
grknight
Retired Dev
Retired Dev


Joined: 20 Feb 2015
Posts: 1832

PostPosted: Fri Aug 09, 2024 5:21 pm    Post subject: Reply with quote

jyoung wrote:
Apparently, none of the services can find their .pid file in /run. I'm not sure if that's related to this problem, but that seems kind of serious.

PID files do not exist before a service's daemon starts. This is normal behavior.

If you want to ensure that a certain IP is available, configure /etc/conf.d/net-online and add rc_need="network-online" to /etc/conf.d/netmount
Back to top
View user's profile Send private message
jyoung
Guru
Guru


Joined: 20 Mar 2007
Posts: 456

PostPosted: Fri Aug 16, 2024 5:05 pm    Post subject: Reply with quote

grknight, that did the trick. I've rebooted several times to eliminate a false positive, and this solution is holding.

So, it seems like the network interface came up before netmount, or netmout waited as it should, but the specific IP wasn't available right away. Does that sound right? I'm going to mark this thread as SOLVED, but anyone wants to add closing thoughts it might help future readers.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Networking & Security All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum