| View previous topic :: View next topic |
| Author |
Message |
stonevil
n00b
Joined: 13 Apr 2008
Posts: 3
|
 Posted: Sun Apr 13, 2008 4:30 am Post subject: question about mirror network file system |
 |
|
hi.
problem:
2 file servers with decentralise location (one in USA, second in Ukraine).
need online synced file system (with file blocking etc, like gfs).
what best solution for this on gentoo?
pls, not propose rsync and other like software.
tnx. |
|
| Back to top |
|
 |
nativemad
Developer
Joined: 30 Aug 2004
Posts: 918
Location: Switzerland
|
| Posted: Tue Apr 15, 2008 2:45 pm Post subject: |
|
|
Probably drdb with gfs on top!
Or glusterfs with afr-translator!
But i can't tell you, if it works with low-bandwith............  |
|
| Back to top |
|
|
stonevil
n00b
Joined: 13 Apr 2008
Posts: 3
|
| Posted: Tue Apr 15, 2008 2:52 pm Post subject: |
|
|
tnx!
after 2 days of r&t I'm choosing PVFS.
http://www.pvfs.org/ |
|
| Back to top |
|
|
nativemad
Developer
Joined: 30 Aug 2004
Posts: 918
Location: Switzerland
|
| Posted: Tue Apr 15, 2008 3:48 pm Post subject: |
|
|
hmmm... sounds interesting!
Can pvfs do mirroring?? Haven't found useful information so far... |
|
| Back to top |
|
|
stonevil
n00b
Joined: 13 Apr 2008
Posts: 3
|
| Posted: Tue Apr 15, 2008 3:51 pm Post subject: |
|
|
| nativemad wrote: | hmmm... sounds interesting!
Can pvfs do mirroring?? Haven't found useful information so far... |
it's a primary future. |
|
| Back to top |
|
|
nativemad
Developer
Joined: 30 Aug 2004
Posts: 918
Location: Switzerland
|
|
| Back to top |
|
|
richard.scott
Veteran
Joined: 19 May 2003
Posts: 1497
Location: Oxfordshire, UK
|
| Posted: Thu Apr 17, 2008 5:28 pm Post subject: |
|
|
I've been looking into PVFS and I've not been able to find any "mirroring" functions? 
From what I can tell PVFS is a way to stripe a shared filesystem across many nodes.... but if one node goes down then the whole PVFS goes down until you can restore that node and all its data! 
They suggest using Heartbeat and a shared back end disk array to provide High Availability......
However, I've gone for a complicated route of PVFS on OCFS on DRBD to enable me to mirror two servers in real time! 
OCFS is a shared file system that you can mount on more than one node at a time and DRBD does mirroring of partitions between nodes and can be setup as "active/active" so both nodes can be used. OCFS then handles file locking etc on top of that 
I'm not sure how DRBD and OCFS with PVFS would deal with latency and timeouts being used over the internet tho?? |
|
| Back to top |
|
|
nativemad
Developer
Joined: 30 Aug 2004
Posts: 918
Location: Switzerland
|
| Posted: Fri Apr 18, 2008 5:41 am Post subject: |
|
|
| Why do you need pvfs on top of ocfs? |
|
| Back to top |
|
|
richard.scott
Veteran
Joined: 19 May 2003
Posts: 1497
Location: Oxfordshire, UK
|
| Posted: Fri Apr 18, 2008 9:07 am Post subject: |
|
|
| nativemad wrote: | | Why do you need pvfs on top of ocfs? |
You need OCFS to enable both nodes to see and mount the same device at the same time
I guess your not totaly aware of DRBD's function as that is a key reason as to why we need OCFS before PVFS. The DRBD website they describe it as RAID-1 for servers. DRBD mirrors the disk partitions in both servers in real time so its in effect the same device in both nodes at the same time. You could think of it similar to connecting to an external disk in both nodes at the same time.
A normal file system would corrupt if you mounted it on both nodes at the same time so we need OCFS to deal with these issues. PVFS is on top of that so you can cluster together pairs of nodes to create a HA filesystem.
Think of each pair of servers as a virtual PVFS node and these virtual nodes are then linked together within PVFS to make a HA filesystem as in theory if one node goes down then the other is there to take over instantly |
|
| Back to top |
|
|
nativemad
Developer
Joined: 30 Aug 2004
Posts: 918
Location: Switzerland
|
| Posted: Fri Apr 18, 2008 9:36 am Post subject: |
|
|
Ok, then you need pvfs for the failover in an active/active drdb?!?
Then you have at least 4 Servers with 2 active/active drdbs? 2 times the same data, striped across 4 nodes?
-i'm a bit scared of ocfs, as it's not posix compliant... Or isn't it a problem with pvfs on top and drdb on the bottom??!?
How about locking in your setup? is this then automagically done by drdb?
....sorry for my dumb questions.... |
|
| Back to top |
|
|
richard.scott
Veteran
Joined: 19 May 2003
Posts: 1497
Location: Oxfordshire, UK
|
| Posted: Fri Apr 18, 2008 10:33 am Post subject: |
|
|
I'll try and expand, I have the following layers in my setup.
Each virtual node is made up of a pair of servers that have the following:
1. <raw disk partition> - /dev/hda1
2. <drbd> - /dev/drbd0
3. <ocfs2> - /mnt/drbd0
4. <pvfs2> - /mnt/drbd0/pvfs2-storage-space
Why we have each part:
1. The physical disk drive.
2. DRBD is used to create a virtual device that mirrors raw partition data from node to node.
With DRBD any data that is written to the device is written to both drives in both nodes in parallel.
The active/active part is specific to DRBD, and having OCFS2 on top of that enables both nodes to mount the same device.
3. OCFS2 is required so both nodes can mount the same virtual DRBD device (/dev/drbd0).
The old OCFS (v1) I don't think was, but from what I understand OCFS2 is a POSIX compliant file system 
4. PVFS2 joins this node into PVFS and we only ever have 1 node running PVFS-server at any one time.
As far as PVFS2 sees there is only ever 1 half of the pair of nodes running pvfs-server at a time.
Yes, this does mean that only half your available data is in use, but then that's what happens when you mirror data.... its in two places at the same time 
and don't worry..... this stuff is really confusing so no question is a dumb question  |
|
| Back to top |
|
|
nativemad
Developer
Joined: 30 Aug 2004
Posts: 918
Location: Switzerland
|
| Posted: Fri Apr 18, 2008 11:42 am Post subject: |
|
|
Ok, i think i've got it now! 
Thanks for the explanation!
But don't you have a bit latency on the second active node, which is just the pvfs-client? (and quite a lot of networktraffic...?) |
|
| Back to top |
|
|
richard.scott
Veteran
Joined: 19 May 2003
Posts: 1497
Location: Oxfordshire, UK
|
| Posted: Fri Apr 18, 2008 11:47 am Post subject: |
|
|
| nativemad wrote: | Ok, i think i've got it now!
Thanks for the explanation!
But don't you have a bit latency on the second active node, which is just the pvfs-client? (and quite a lot of networktraffic...?) |
There's no pvfs-client running on either of these nodes... one of this pair runs pvfs-server.... the other node keeps pinging the primary node and waits for it to die.... if it dies then the second node takes over and starts pvfs-server and takes the IP of the other node and adds it to its own network interface.
The pvfs-client run's on a 3rd system that can actually use and mount the PVFS2 file system.
I've not found an easy way to add new nodes to a pvfs setup so you can increase available disk space so I'm starting to look at OpenAFS now  |
|
| Back to top |
|
|
nativemad
Developer
Joined: 30 Aug 2004
Posts: 918
Location: Switzerland
|
| Posted: Fri Apr 18, 2008 12:08 pm Post subject: |
|
|
| Quote: | | The pvfs-client run's on a 3rd system that can actually use and mount the PVFS2 file system. |
Ahh, thats the point!
I would like to have just these two 'storage'-nodes, so that i could have a mirrored, loadbalanced imap-backend with failover as a nice addition!
When i got it right, i could just use drdb with ocfs2!?!
---
sorry stonevil, if we just take over your thread! |
|
| Back to top |
|
|
richard.scott
Veteran
Joined: 19 May 2003
Posts: 1497
Location: Oxfordshire, UK
|
| Posted: Fri Apr 18, 2008 12:42 pm Post subject: |
|
|
| nativemad wrote: | | When i got it right, i could just use drdb with ocfs2!?! |
Yep, that'd do the trick.
You could use keepalived, ucarp or heartbeat to monitor the other nodes and then start services on your own node as required.
I prefer keepalived as manages LVS too!
PM me if you want some more advice and we can talk specifics about your situation. |
|
| Back to top |
|
|
nativemad
Developer
Joined: 30 Aug 2004
Posts: 918
Location: Switzerland
|
| Posted: Sat Apr 19, 2008 10:27 am Post subject: |
|
|
Thanks for the offer!!
Maybe i'll come back to you, if i'll get stuck!
First i have to setup some spare servers to prepare my playgroud... |
|
| Back to top |
|
|
MostAwesomeDude
Guru
Joined: 12 Aug 2007
Posts: 373
|
| Posted: Sat Apr 19, 2008 3:51 pm Post subject: |
|
|
Man, all these acronyms scare me. I'm sticking to rsync.
_________________
Don't believe the "n00b" under my name. |
|
| Back to top |
|
|
|
Other Things Gentoo
