Building an HPC Cluster...

[alexander]

So I am dedicating this sticky to that, i am currently putting together an HPC cluster to run at home for use, including other things, but mainly for some project computation needs here at hypo (please check out the Strange Numbers thread)

I figured what i wanted to do, i am going to make custom installs of Gentoo on a few machines to start with, i will start a 2 node cluster on machines i have at home, i hope to go to a 4 or 5 node next week, and hopefully to an 8 node in the coming months (actually if i get a decent amount of money from taxes, this may turn to have a large amount of nodes before the summer arrives)

current hardware:

master:
3.4Ghz P4ht 800Mhz fsb
2.5Gb DDR400

node 1:
3.0Ghz P4ht 800MHz fsb
1.5GB ddr400

network: is a dedicated 8 port gigabit switch, intel gigabit cards.

Storage: not decided yet, master node may get a storage facelift, figuring 1-2T should suffice for now this will eliminate the usage of the network to store the data

as i physically get more nodes, i will post them here

first reference: Gentoo Linux Documentation -- High Performance Computing on Gentoo Linux

now i just need to make it happen software-wise

I decided to go with gentoo only because its a totally stripped down distro that i can build up... i didnt want a live distro, it uses too much ram, usually they are inefficient too, i might eventually make a net-boot version, but for now i just want to get it going, and start writing mpi code for it.

I'll post as i go along, on what happens, how it gets resolved, etc

once i have master, new nodes will be allowed to use distcc to build their packages, so each consecutive node will have more processing power to compile with for boot strapping etc

[Turtle]

Originally Posted by alexander

So I am dedicating this sticky to that, i am currently putting together an HPC cluster to run at home for use, including other things, but mainly for some project computation needs here at hypo (please check out the Strange Numbers thread)...

So dude, I prolly wouldn't know a cluster if it bit me in the backside, but I just happened to watch an interview today that Charlie Rose did for 40 minutes with Jen-Hsun Huang, CEO & co-founder of Nvidia. What does this have to do with fast processing you ask? Well, seems Mr. Huang and his gang are applying programmable GPU's to amp up overall raw math processing usually left to the CPU and getting performances as much as 100 times faster.

So, I can't explain it so well maybe, but here is the interview. This is as cutting edge as it gets as far as I can gather from it. Enjoy.

Charlie Rose - A conversation with Jen-Hsun Huang, CEO Nvidia

[dberkholz]

Originally Posted by alexander

first reference: Gentoo Linux Documentation -- High Performance Computing on Gentoo Linux[/url]

Hi there,

Good to see you're using Gentoo! Take that guide with a grain of salt, it's a bit outdated. You probably want Torque/Maui and OpenMPI instead of OpenPBS and MPICH.

[alexander]

i know, i do research well, and well, i've been using gentoo since 2005, you have probably vague understanding of just how many installs of it i had done, lets put it this way, we had weekends at college where my friends and i installed gentoo for any advanced population that came around... that was neat...

yeah i am going to use openmpi and mpitch, distcc... i've also been thinking about Dr.Queue batch job manager, in case someone wanted to do some rendering on it (speaking of multitude of friends i have that do a lot of random things that might want to use the cluster)

[alexander]

db if you are interested i will post details on the installs, they are fairly similar, juuust a little different hardware they are both compiling kernels right now, but that's besides the point... they are stage 3's that will eventually be recompiled... getting them as down to core as i can actually disabling almost everything to conserve power....

i will probably be ordering a couple of Gigabyte boards with Phenom II 920's and 8 gb ram for each, pretty soon, i have to come up with some sort of an enclosure for them, but more then that, with MOHNEH (yes those don't magically appear for anyone) that is currently a bit tight, for now, i am recycling my friend's no longer used equipment, as long as it's like a p4 or a 3000+ or newer series AMD (preferably HT, because i set makeopts to j3, and with symmetric multiprocessing, those procs give interesting stats on compiles), i will use it

base system is almost installed, they are finishing some essentials before first reboot

[alexander]

I was going to work on it from work today, but even though i did not forget to enable ssh on my mac, as i was going to use it to ssh into the boxes that are being worked on, i did forget to disable Little Snitch... which you can NOT disable in command line... that pisses me off, but that's besides the point, you cant unload the kext, you cant shutdown the service... useless...

so i will have wait till later to finish the installs (i have a couple of things left, boot loader, and some packages, but i mostly recompiled all the system libraries and binaries over night (optimized for the processors they are running on CFLAGS="-march=pentium4 -O2 -pipe -fomit-frame-pointer" also i disabled all kinds of USE flags so not to compile sound support, or any X, kde, gnome or gtk libs and a load of other crap that gets loaded on the system)

Oh also note to anyone doing Gentoo installs from stage 2-3 (i use stage3 for the speed and then optimize it)

if you have an error just after unpacking udev do NOT panic. Problem is caused by the differences in the march flags used to compile the binaries you are loading vs the new march flags you set in make.conf

first reemerge gcc

second run gcc-config and set the profile to the i686

third source /etc/profile

forth reemerge libtool

also if ss and com_err are blocking e2fsptogs and e2fsprogs-libs

first fetch everything you might need
# emerge -vuDaf world

second backup ss and com_err
# quickpkg ss com_err

remove ss and com_err and e2fsprogs (if they are blocking themselves)
# emerge -C ss com_err e2fsprogs

emerge the new ones
#emerge -va --oneshot e2fsprogs-libs e2fsprogs

fix brokage
#revdep-rebuild

[alexander]

Update: first 2 machines are finally self-booting, they are almost where i want them to be, just need a couple of more things before they are hpc-ready (boot times under 35 seconds btw)

[alexander]

the master and node1 are now almost there, i have setup distcc to compile using them, this means that i have now the ability to setup new systems a few times faster then normal, i also have gotten 2 more p4 machines, one ht, one not (i think), they will be joining the cluster probably closer to the end of the week (i have to make some space for them near some power sockets)

Plan for tonite is to get openmpi working, as well as torque and maui (gentoo restricts the download of the code, i have to download it, but wget does not download it either) i was having a ton of fun compiling the kernel yesterday, i saw about a 60% increase in speed with just one more machine (granted it was a 3.4 vs 3.0 with nearly twice the memory).

plan for first cluster test next week (probably render something in blender using mpi) (do i hear anyone say MIT box?)

[alexander]

another update, i finally registered downloaded and installed maui, figured out my dyndns update issue on the router and forwarded 22 to the master node (the router runs an ips and a block all by default policy, the box runs iptables that only allows 22 and icmp through from the outside)... i think i covered security fairly well...

Now its only a matter of figuring out the configuration of the batch job manager, openmpi etc, etc, what needs to be installed on the nodes, testing it with the one node and finally throwing more nodes on should be done in the next couple of days...

[alexander]

ok more documentation links:

Torque: torque:torque_wiki [Documentation Wiki]
Maui: Cluster resources :: Docs & Training - Documentation (scroll down to maui section)