Building a Budget-Friendly Linux Cluster: Tips and Tricks from Hak5

[Gunthi]

I have some old computers at home and I want to put them to work by making a cluster, maibe run SETI or other distributed computing related stuff.

I've searched the Internets, found out about Beowulf Clusters, but I can't seem to find any useful tutorial on how to make one.

So I'm calling out for some help.

1.Has anyone here made a cluster?
2.If so, what bibliography/references did you use?
3.What's the best Linux distribution for it?
4.What network hardware do i need?

Thanks in advance!

 

[waht]

I have helped building a cluster, but mostly the hardware part.

Software-wise there was a master node on one server from which all other servers booted redhat over the network. The master node hosted an NTP server which synchronized time with slave servers, and also hosted a bunch of other server stuff like SSH, FTP, DHCP I think.

The software was running MPI implementation in C or C++, and LINPACK to do linear algebra. But how they configured it I have no idea. But it's definitely on my to do list.

Also the network switch that was used to connect up all the servers was expensive with really good response time. Cheap network switches are slow when it comes to switching lots of short and chatty bursts.

 

[Gunthi]

I have helped building a cluster, but mostly the hardware part.

Software-wise there was a master node on one server from which all other servers booted redhat over the network. The master node hosted an NTP server which synchronized time with slave servers, and also hosted a bunch of other server stuff like SSH, FTP, DHCP I think.

The software was running MPI implementation in C or C++, and LINPACK to do linear algebra. But how they configured it I have no idea. But it's definitely on my to do list.

Also the network switch that was used to connect up all the servers was expensive with really good response time. Cheap network switches are slow when it comes to switching lots of short and chatty bursts.

I don't intend to invest a lot of money in this, the pc's are all bellow pentium III so I'm not expecting to build a Deep Blue.

So, regarding the hardware, to connect all the nodes and server, a network switch will do right?

Also, do you know any good books about this?

Thanks ;)

 

[waht]

Here is a mini guide on making a cluster
http://www.mini-itx.com/projects/cluster/

So, regarding the hardware, to connect all the nodes and server, a network switch will do right?

yes, any cheap 10/100 switch will do

 

[Gunthi]

Here is a mini guide on making a cluster
http://www.mini-itx.com/projects/cluster/



yes, any cheap 10/100 switch will do

Thanks for the link ;)

The hardware part is ok but it gets a little bit too advanced for me in the software part...
Do you know tutorials that are a little more accessible?

 

[waht]

Thanks for the link ;)

The hardware part is ok but it gets a little bit too advanced for me in the software part...
Do you know tutorials that are a little more accessible?

I haven't found any more in-depth guide on the net, but there are dozens of books on amazon. But I suppose they assume a more in-depth linux/unix knowledge.

If you are not familiar how to set up a DHCP, SSH, RSH, NFS, NTP, FTP, or Apache on linux it would be a good idea to learn those individually before attempting to actually building the cluster. I would recommend setting up ubuntu server edition on the computers since ubuntu has very good step by step guides.

Secondly, once you have the computers connected and talking to each other, google around on how to setup up MPI on them.

Thirdly, what would you like to run on it? SETI at home doesn't support parallel computing. You could set up SETI on the computers but they would be working on completely independent data sets.

 

[Gunthi]

I haven't found any more in-depth guide on the net, but there are dozens of books on amazon. But I suppose they assume a more in-depth linux/unix knowledge.

If you are not familiar how to set up a DHCP, SSH, RSH, NFS, NTP, FTP, or Apache on linux it would be a good idea to learn those individually before attempting to actually building the cluster. I would recommend setting up ubuntu server edition on the computers since ubuntu has very good step by step guides.

Secondly, once you have the computers connected and talking to each other, google around on how to setup up MPI on them.

Ok, I'll look into those.

Thirdly, what would you like to run on it? SETI at home doesn't support parallel computing. You could set up SETI on the computers but they would be working on completely independent data sets.

I didn't know that. Thanks for the help and patience! ;)

 

[woojay]

I don't know how much it would help to reply to a year-old post, but I recently built a homemade Beowulf cluster using Debian Linux, DRBL, and the Sun Grid Engine (SGE). I've documented the hardware and software configurations http://sites.google.com/site/woojay/helmer" .

 

[Bill Simpson]

Recent example of building a small cluster without spending money on frills

http://revision3.com/hak5

episode 823.

To keep cost down, for example, he didn't buy cases but just used the cardboard boxes that the motherboards came in as the cases.

Sometimes the Hak5 episodes seem like they are just giving hints and perhaps keywords that you will have to go do some digging on your own, or perhaps send them a polite question or two, to figure out how to use that in your own situation.

I'd suggest buying a few Fry's loss leader motherboard/cpu combos that will provide perhaps 100x the performance of the PIII, get a few recycled old cases, power supplies and old low performance pci video cards, leave the sides off, aim cheap house fans at the boards to keep them cool.