** See post on 15/12/2011 for information about Debian 64 bit. It builds fine on Squeeze but not Wheezy. This seems to have to do with the version of mpich2.**
Figuring out how to compile nwchem with mpich support took a little while, but this seems to have worked:
First mpich2 and gfortran need to be installed (since it was not installed on a virgin system there may have been other required packages already installed)
sudo apt-get install mpich2 gfortran
I created a file called myconfig.sh in the nwchem directory, with the following content:
setenv LARGE_FILES TRUE
setenv TCGRSH /usr/local/bin/ssh
setenv NWCHEM_TOP /work/nwchem
setenv NWCHEM_TARGET LINUX64
setenv NWCHEM_MODULES all
setenv USE_MPI y
setenv USE_MPIF y
setenv MPI_LOC /usr
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include/mpich2
setenv LIBMPI "-lfmpich -lmpich"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran >& make.log
do a csh myconfig.sh and you should be good to go.
I then added the following to the end of my ~/.bashrc and sourced it:
PATH=$PATH://work/nwchem/bin/LINUX64
export NWCHEM_EXECUTABLE=/work/nwchem/bin/LINUX64/nwchem
Jobs can then be submitted (assuming that mpd is up) by
mpdrun -n 2 nwchem nameofjob.nw
Lindqvist - a blog about Linux and Science. Mostly.
Pages
▼
28 January 2011
09 January 2011
2. Setting up a cheap-ish ubuntu cluster - the hardware
I've recently become interested in setting up a small computational cluster for simple QM and MD work. Since it doesn't hurt sharing your experience with others, I'll post the details on this blog.
I'm not a computer/linux/computational chemistry expert - the goal of this project is as much to learn how to go about setting up a cluster, as getting a useful computational tool.
First, the hardware - the idea was to get something fairly cheap, but also reasonably useful. I opted for the following hardware (prices in USD):
30.99 V7 NS1132-N6 10/100/1000Mbps Unmanaged Networking Switch 5 x RJ45 2K MAC Address Table
Each node
39.99 BIOSTAR MCP6P3 AM3 NVIDIA GeForce 6150 / nForce 430 Micro ATX AMD Motherboard
77.00 AMD Athlon II X3 445 Rana 3.1GHz 3 x 512KB L2 Cache Socket AM3 95W Triple-Core Desktop Processor
2*21.99 Mushkin Enhanced 2GB 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory Model 20-146-744
7.99 NIC Zonet ZEN3301E 10/ 100/ 1000Mbps PCI 32-Bit Network Adapter 1 x RJ45
24.99 APEX TX-381-C Black Steel Micro ATX Tower Computer Case (needs full size ATX PSU)
13.99 LOGISYS Computer PS480D 480W ATX12V Power Supply
3.99 APEVIA CF12S-BK 120mm Case Fan
I ordered everything from Newegg, and all the parts arrived after three days, as promised.
I already had two SATA harddrives, and two IDE CD ROM drives, in addition to two IDE/PATA cables. The CD ROM drives aren't important, as the mobo bios allows for boot from USB, so you can install your OS that way (at least Ubuntu 10.10).
Assembling the two systems was easy, even though it was the first time I had done anything more complicated than installing a RAM stick or changing a hard drive. I'm not particularly good with my hands, so that really says something.
Here are the steps, in case someone is as useless as I am with hardware:
Preparing the motherboard:
1. The CPU was fitted on the motherboard - being an AM3 socket it was a simple matter of gently dropping the cpu in the socket with the yellow marking on the cpu being aligned with the yellow marking on the mobo.
2. Next the heatsink/fan was fitted. The heatsink arrived with a smooth layer of thermal grease already applied. While it's difficult to see whether the heatsink was in perfect position over the cpu, I did my best and everything went well.
3. The RAM memory was installed. Nothing complicated here.
Preparing the case:
1. The PSU was installed - nothing complicated. Just four screws to attach the PSU to the case.
2. The motherboard panel at the back of the case didn't fit my mobo, so I installed the one which came in the motherboard box. One of the PCI slot covers on the case was removed to make space for the gigabit ethernet card
3. The brass spacers that came with the case were attached to the wall of the case. The spacers are for the mobo to rest on. If you don't install them the mobo won't align with the openings at the back of the case, so there's little risk that you'll forget to do this.
4. A CD drive (IDE) and a harddrive (SATA) were installed, but not connected.
Assembly:
1. The motherboard was attached to the case by resting it on the brass spacers and aligning the holes in the mobo, then attaching the mobo with screws.
2. All the cables connected to the case were clearly labelled, as were the corresponding connections on the motherboard, so it was just a matter of matching one to the other.
3. The PSU cables were fairly simple to connect two - two cables were attached to the corresponding connections on the mobo (just going by shape), the sata power cable was connected to the HDD and a molex connector was connected to the IDE CD ROM drive.
4. The sata cable that came with the case was used to connect the motherboard with the harddrive. I had a couple of IDE cables around, and used it to connect the CD ROM drive to the motherboard. It took two attempts to get the IDE thingy right as I first attached the cable the wrong way round - for an IDE cable with three plugs (one on each end, one slightly offset from the middle of the cable) you want to connect the end with the 'middle' connector closer to it to the CD ROM, and the other end to the MOBO. Makes sense, but getting it right the first time would save you a few minutes. Of course, an easier way of getting around this is probably by setting the jumper to cable-select.
One system was installed by booting from a Ubuntu 10.10 64 CD, and the other by booting from a USB stick prepared using usb-creator-gtk. I could've settled on minimal/server installs instead of full Ubuntu Desktop installs, but as I 1) will later modify grub to boot to terminal instead of starting Gnome, 2) don't have space contraints and 3) want to be able to use the computers as back-up workstations I opted for full installs.
The systems were updated after the install.
I'm not a computer/linux/computational chemistry expert - the goal of this project is as much to learn how to go about setting up a cluster, as getting a useful computational tool.
First, the hardware - the idea was to get something fairly cheap, but also reasonably useful. I opted for the following hardware (prices in USD):
30.99 V7 NS1132-N6 10/100/1000Mbps Unmanaged Networking Switch 5 x RJ45 2K MAC Address Table
Each node
39.99 BIOSTAR MCP6P3 AM3 NVIDIA GeForce 6150 / nForce 430 Micro ATX AMD Motherboard
77.00 AMD Athlon II X3 445 Rana 3.1GHz 3 x 512KB L2 Cache Socket AM3 95W Triple-Core Desktop Processor
2*21.99 Mushkin Enhanced 2GB 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory Model 20-146-744
7.99 NIC Zonet ZEN3301E 10/ 100/ 1000Mbps PCI 32-Bit Network Adapter 1 x RJ45
24.99 APEX TX-381-C Black Steel Micro ATX Tower Computer Case (needs full size ATX PSU)
13.99 LOGISYS Computer PS480D 480W ATX12V Power Supply
3.99 APEVIA CF12S-BK 120mm Case Fan
I ordered everything from Newegg, and all the parts arrived after three days, as promised.
I already had two SATA harddrives, and two IDE CD ROM drives, in addition to two IDE/PATA cables. The CD ROM drives aren't important, as the mobo bios allows for boot from USB, so you can install your OS that way (at least Ubuntu 10.10).
Assembling the two systems was easy, even though it was the first time I had done anything more complicated than installing a RAM stick or changing a hard drive. I'm not particularly good with my hands, so that really says something.
Here are the steps, in case someone is as useless as I am with hardware:
Preparing the motherboard:
1. The CPU was fitted on the motherboard - being an AM3 socket it was a simple matter of gently dropping the cpu in the socket with the yellow marking on the cpu being aligned with the yellow marking on the mobo.
2. Next the heatsink/fan was fitted. The heatsink arrived with a smooth layer of thermal grease already applied. While it's difficult to see whether the heatsink was in perfect position over the cpu, I did my best and everything went well.
3. The RAM memory was installed. Nothing complicated here.
Preparing the case:
1. The PSU was installed - nothing complicated. Just four screws to attach the PSU to the case.
2. The motherboard panel at the back of the case didn't fit my mobo, so I installed the one which came in the motherboard box. One of the PCI slot covers on the case was removed to make space for the gigabit ethernet card
3. The brass spacers that came with the case were attached to the wall of the case. The spacers are for the mobo to rest on. If you don't install them the mobo won't align with the openings at the back of the case, so there's little risk that you'll forget to do this.
4. A CD drive (IDE) and a harddrive (SATA) were installed, but not connected.
Assembly:
1. The motherboard was attached to the case by resting it on the brass spacers and aligning the holes in the mobo, then attaching the mobo with screws.
2. All the cables connected to the case were clearly labelled, as were the corresponding connections on the motherboard, so it was just a matter of matching one to the other.
3. The PSU cables were fairly simple to connect two - two cables were attached to the corresponding connections on the mobo (just going by shape), the sata power cable was connected to the HDD and a molex connector was connected to the IDE CD ROM drive.
4. The sata cable that came with the case was used to connect the motherboard with the harddrive. I had a couple of IDE cables around, and used it to connect the CD ROM drive to the motherboard. It took two attempts to get the IDE thingy right as I first attached the cable the wrong way round - for an IDE cable with three plugs (one on each end, one slightly offset from the middle of the cable) you want to connect the end with the 'middle' connector closer to it to the CD ROM, and the other end to the MOBO. Makes sense, but getting it right the first time would save you a few minutes. Of course, an easier way of getting around this is probably by setting the jumper to cable-select.
One system was installed by booting from a Ubuntu 10.10 64 CD, and the other by booting from a USB stick prepared using usb-creator-gtk. I could've settled on minimal/server installs instead of full Ubuntu Desktop installs, but as I 1) will later modify grub to boot to terminal instead of starting Gnome, 2) don't have space contraints and 3) want to be able to use the computers as back-up workstations I opted for full installs.
The systems were updated after the install.
02 January 2011
1. Setting up a cheap-ish ubuntu cluster - basic set-up
Note: I have two nodes - one master (beryllium) and one slave (lithium). These names will be used in the examples
Once all the hardware was assembled and linux up and running, I installed a host of programs that I'd be using:
Cluster tools:
sudo apt-get install mpich2 vnstat sinfo ganglia-monitor openmpi-*
For compiling:
sudo apt-get install build-essential gfortran gpp cpp
Generally useful software:
sudo apt-get install gnuplot mpqc-openmpi gromacs-mpich gomacs-mpich octave octave-parallel qtoctave gnome-activity-journal maxima
I downloaded the nwchem source from here. We'll deal with the compilation and installation in the next post.
I followed this post to get up and running (scroll down) with nfs. Here's my take on it:
On the master node:
sudo apt-get install nfs-kernel-server nfs-common portmap
We don't want portmap to use 127.0.0.1, so
sudo dpkg-reconfigure portmap
sudo /etc/init.d/portmap restart
sudo mkdir /work
sudo chmod a+wrx /work
sudo nano /etc/exports
Add the following line to the end of the file:
/work *(rw,sync)
sudo /etc/init.d/nfs-kernel-server restart
sudo exportfs -a
One the slave node:
sudo apt-get install portmap nfs-common
sudo mkdir /work
sudo nano /etc/fstab
Add the following line:
beryllium:/work /work nfs rw 0 0
To get it up and running immediately
sudo mount /work
Next, create a file called mpd.hosts - you can use the /work directory or the home directory. List the hosts/nodes and add, without spaces, the number of processors on each node:
lithium:3
beryllium:3
On beryllium
mpd --ncpus=3 &
mpdtrace -l
which returns something akin to beryllium_12345 (192.168.1.3). 12345 is the port number, which will change each time you start mpd
On lithium
mpd --ncpus=3 -h beryllium -p 12345 &
NOTE: I had problems before specifying --ncpus - gave an error about mpich2 consecutive number/IDs. The down-side is that when you submit a job it fills up the master node, then starts jobs on the slave node(s).
Once all the hardware was assembled and linux up and running, I installed a host of programs that I'd be using:
Cluster tools:
sudo apt-get install mpich2 vnstat sinfo ganglia-monitor openmpi-*
For compiling:
sudo apt-get install build-essential gfortran gpp cpp
Generally useful software:
sudo apt-get install gnuplot mpqc-openmpi gromacs-mpich gomacs-mpich octave octave-parallel qtoctave gnome-activity-journal maxima
I downloaded the nwchem source from here. We'll deal with the compilation and installation in the next post.
I followed this post to get up and running (scroll down) with nfs. Here's my take on it:
On the master node:
sudo apt-get install nfs-kernel-server nfs-common portmap
We don't want portmap to use 127.0.0.1, so
sudo dpkg-reconfigure portmap
sudo /etc/init.d/portmap restart
sudo mkdir /work
sudo chmod a+wrx /work
sudo nano /etc/exports
Add the following line to the end of the file:
/work *(rw,sync)
sudo /etc/init.d/nfs-kernel-server restart
sudo exportfs -a
One the slave node:
sudo apt-get install portmap nfs-common
sudo mkdir /work
sudo nano /etc/fstab
Add the following line:
beryllium:/work /work nfs rw 0 0
To get it up and running immediately
sudo mount /work
Next, create a file called mpd.hosts - you can use the /work directory or the home directory. List the hosts/nodes and add, without spaces, the number of processors on each node:
lithium:3
beryllium:3
On beryllium
mpd --ncpus=3 &
mpdtrace -l
which returns something akin to beryllium_12345 (192.168.1.3). 12345 is the port number, which will change each time you start mpd
On lithium
mpd --ncpus=3 -h beryllium -p 12345 &
NOTE: I had problems before specifying --ncpus - gave an error about mpich2 consecutive number/IDs. The down-side is that when you submit a job it fills up the master node, then starts jobs on the slave node(s).