One NFS file system, multiple servers with multiple disks

wildhorse

I am looking for a solution that allows me to serve one file system via one NFS mount point. The disks (IDE) are located on multiple Gentoo machines. Each machine is serving multiple disks.

It seems that Multicast NFS (MNFS) can do what I need, but I am not sure if MNFS is available on Gentoo.

As an alternative solution I could use one physical server and let this server import multiple NFS file systems from the other machines and the re-export the file systems all-in-one as one NFS file system. But I am sure the performance of this solution would be terrible.

I can also imagine to set up a connection from one central NFS server to the disk servers via iSCSI, but do not know if that is fully supported (on Gentoo and wrt my IDE disks) and what the performance might be. The problem with the server-in-the-middle solution is latency.

I can do anything on the servers. I can add another machine as central server. I can detach the networks and put a bridge inbetween. I also have my own DNS Bind server (which does not run Gentoo Linux). Anything on my servers and my local network is possible. But I cannot change the clients. Only mounting a single NFS file system is allowed on the clients.

Any suggestions?

SnEptUne · l33t Joined: 23 Aug 2004 Posts: 656

How about using the nohide option in NFS?

wildhorse · Posted: Fri Feb 01, 2008 7:06 pm Post subject:

But exports' nohide option does not hide the fact that the clients would still be dealing with multiple servers. Distribution of the data over all disk servers is not addressed with this option, or at least not with this option alone.

SnEptUne · l33t Joined: 23 Aug 2004 Posts: 656

wildhorse · Posted: Fri Feb 01, 2008 10:15 pm Post subject:

N clients
1 common NFS file system seen by each of the N clients
M disk servers, each with multiple disks

I can easily combine all disks on 1 disk server to 1 file system (e.g. RAID, LVM). That is fine if I have only 1 NFS server. But I have too many servers and I do not want the users to manage the allocation of the disks. The file system should distribute the data onto the disk servers and disks, and the users will see only one huge NFS file system.

SnEptUne · l33t Joined: 23 Aug 2004 Posts: 656

Why do clients need to manage allocation of disks? I still don't quite understand what you are trying to do.

From your descriptions, you want clients to access files from multiple disk servers, but you can only install NFS server on one system? Or is it in the case there clients cannot have access to the disk servers and they can only update to the NFS servers? Is NFS a requirement? How will the clients access the remote file system? Why is nohide not a solution? Could you be more specific instead of giving vague description? I can't read your mind.
_________________
"There will be more joy in heaven over the tear-bathed face of a repentant sinner than over the white robes of a hundred just men." (LM, 114)

wildhorse · Posted: Fri Feb 01, 2008 11:15 pm Post subject:

The problem is writing data. The exports nohide option only works well for reading data.
If you write data from any of the N clients onto the NFS file system, then someone needs to make the decision onto which of the M file systems the data go. And that should be done on the server site, not the client site, in particular not by the users on the clients.

robdd · Posted: Sat Feb 02, 2008 2:57 am Post subject:

Hi wildhorse - I'm trying to narrow down the possible solutions by thinking about your problem from first principles...

SnEptUne · l33t Joined: 23 Aug 2004 Posts: 656

I see. I am not sure how nohide works, but NFS wasn't made to be recursive by design for performance reasons anyway.

How about openAFS? It also caches data on clients, and only updates through venus and vice when neccessary.
_________________
"There will be more joy in heaven over the tear-bathed face of a repentant sinner than over the white robes of a hundred just men." (LM, 114)

misterbob05 · Tux's lil' helper Joined: 28 Apr 2007 Posts: 90

one idea i had was making a parent directory and mounting all the shares in that directory and then mounting that parent directory on all the client machines???

i havent tried it or know that theory behine nfs or anything as i use samba

sorry if it has already been thought of and shot down

robdd · Posted: Sat Feb 02, 2008 4:53 am Post subject:

Sorry SnEptUne - I didn't read your earlier post carefully :oops:

wildhorse · Posted: Mon Feb 04, 2008 9:03 pm Post subject:

Hi Everybody, thanks for your comments!
It seems the idea of redistributing the files via an NFS server in the middle may not work. There are the inode problem, latency and most important the matter of distributing data onto the actual disk servers for write access. AFS might be able to cope with this configuration, but I cannot use AFS. And the USB solution would require additional investment into new hardware for a lot of disks and it would be too slow anyway.

How about Network Block Devices (NBD)? Has anybody ever set up a big NFS server with lots of NBDs? I could imagine that with read-ahead and write-through caching performance may not be that bad. It seems that RAID with NBD works. What I like to know is if there is someone who has actually experience with such a configuration?

your_WooDness · Tux's lil' helper Joined: 25 Oct 2007 Posts: 77

Hi there,

this sounds adventurous... =0) But please have in mind, that if you just "stripe" more filesystems via network, to make them available as one filesystem that you will run into big trouble when something goes wrong. So I hope you won't store important data on this.
What you are trying to do is a so called "storage virtualization" when you want to abstract the storage devices to the clients. There are a lot of software solutions out there for this...
But I think with NBD, you will also have the bottle neck of one server that manages the traffic and on http://nbd.sourceforge.net/ written "...But (also unlike NFS), if someone has mounted NBD read/write, you must assure that no one else will have it mounted...". This means that the NBD will have corrupt data when the mentioned situation occurs.

What about CFS or lustre filesystem? --> http://wiki.lustre.org/index.php?title=Main_Page
I think this will meet your needs, but is a bit tricky to set up. But I wouldn't even store my mp3 collection on a shaky setup with LVM and NFS mounts or NBD.
An easy way would be to set up iscsi targets on those server that should provide the storage. The clients must have installed the iscsi initiator. But you will also have no centralized storage and a lot of iscsi targets, which would mean that the logical configuration would be the same with a lot of NFS or samba shares. But the performance should be a bit better with iscsi.

WooD

wildhorse · Posted: Mon Feb 04, 2008 11:43 pm Post subject:

your_WooDness, I have enough disks (and thus plenty of statistics) to confirm that hard disks sometime do fail. :wink:

NBD supports RAID, and in particular RAID5. NBD and data corruption is no problem since the disks are only being used on my disk servers and I am in charge.

CFS is not an option. Neither is modifying the clients (except for adding an NFS file system entry fstab). Lustre looks interesting. I also looked into other "cluster" or global file systems. But I have to use NFS, at least as front end.

Has anyone ever compared iSCSI vs. NBD?

wildhorse · Posted: Tue Feb 05, 2008 12:41 am Post subject:

Perhaps I should add ATA over Ethernet (AoE) as another option since I only have a dedicated LAN between the disk servers.

So, if there nothing like MNFS, has anyone tested iSCSI vs. NBD vs. AoE?