| View previous topic :: View next topic |
| Author |
Message |
humbletech99
Veteran
Joined: 26 May 2005
Posts: 1229
Location: London
|
 Posted: Tue Nov 11, 2008 6:10 pm Post subject: High Performance Computing |
 |
|
I am interested in setting up some High Performance Computing clusters and would like to get people's views and experiences on this.
I have 2 requirements:
1. Compute clusters to do fast cpu intensive computations
2. Storage clusters of parallel and extendable filesystems spread across many nodes
Both of these should run across multiple commodity hardware nodes and ideally be Linux/Unix based and open source.
Any feedback welcome.
_________________
The Human Equation:
value(geeks) > value(mundanes) |
|
| Back to top |
|
 |
notHerbert
Advocate
Joined: 11 Mar 2008
Posts: 2228
Location: 45N 73W
|
|
| Back to top |
|
|
humbletech99
Veteran
Joined: 26 May 2005
Posts: 1229
Location: London
|
| Posted: Wed Nov 12, 2008 3:35 pm Post subject: |
|
|
I did, but it doesn't look very comprehensive, very brief, doesn't really explain as much as I would like. Will try to do some of it on spare boxen and see if it works.
I'm really interested in people's experiences, of course I can read a couple of web pages, but I'd like to know if anyone out there is using HPC, what for and how they are doing it.
EDIT: looked more closely at openpbs, looks like some old batch queuing system from the 90s? The only other thing mentioned on that page above is mpich which looks like a glorified remote process starter over rsh/ssh, still figuring this out. Going to go and have a look at beowulf now...
_________________
The Human Equation:
value(geeks) > value(mundanes) |
|
| Back to top |
|
|
didl
Retired Dev
Joined: 09 Sep 2003
Posts: 1106
Location: Pittsburgh, PA
|
| Posted: Wed Nov 12, 2008 7:25 pm Post subject: |
|
|
| humbletech99 wrote: |
I'm really interested in people's experiences, of course I can read a couple of web pages, but I'd like to know if anyone out there is using HPC, what for and how they are doing it.
|
HPC is often used in a scientific context (biomedical, weather prediction, astrophysics, fluid dynamics, etc), in data mining, and weapons research. Scientifc HPC codes
typically have special hardware requirements met in the form of compute clusters or supercomputers in order to do their job for a variety or reasons,
such as sheer computational cost as well as large memory and/or I/O requirements. You will probably have to read much more than a few websites in order to set up one
these clusters or even write a well performing HPC code  |
|
| Back to top |
|
|
coolsnowmen
Veteran
Joined: 30 Jun 2004
Posts: 1479
Location: No.VA
|
| Posted: Wed Nov 12, 2008 8:10 pm Post subject: Re: High Performance Computing |
|
|
| humbletech99 wrote: |
1. Compute clusters to do fast cpu intensive computations
|
"Fast" is all but meaningless in this context. Do you have any actual latency or throughput requirements?
_________________
emerge: there are no ebuilds to satisfy "moo" |
|
| Back to top |
|
|
humbletech99
Veteran
Joined: 26 May 2005
Posts: 1229
Location: London
|
| Posted: Thu Nov 13, 2008 10:56 am Post subject: |
|
|
Throughput, cpu computations done in a given time frame.
I currently have several cpu intensive servers running processes they get from a queue kept in a database.
This means that some could be 100% cpu maxed out while others sit idle. This is just one case of where I would like to see if I can apply HPC to make all the servers work on the same problem at the same time and get the job finished faster.
_________________
The Human Equation:
value(geeks) > value(mundanes) |
|
| Back to top |
|
|
mmoelle1
n00b
Joined: 23 Feb 2008
Posts: 26
Location: Dortmund
|
| Posted: Thu Nov 13, 2008 11:20 am Post subject: |
|
|
Concerning requirement 2: storage clusters.
Personally, I have no real experience with Cluster Computing but our university runs some linux clusters (Configured by professionals, thus, no gentoo  ). From what I have heard from my colleagues, GPFS is not recommendable. The cluster requires rebooting quite frequently which conflicts with long-time simulations (talking about weeks, months, etc.). Our next cluster generation will most likely be built on Lustre http://wiki.lustre.org.
This is just my personal opinion. Even less, it is what colleagues experienced. Do not exclude GPFS from your considerations and install lustre blindly.
Matthias |
|
| Back to top |
|
|
|
Other Things Gentoo
