Firstly, I must acknowledge this guide: http://helms-deep.cable.nu/~rwh/blog/?p=159
I FOLLOW THAT POST ALMOST VERBATIM
This post will be more of a "I followed this guide and it actually works on debian testing/wheezy too and here's how" post, since it doesn't add anything significant to the post above, other than detail.
Since I ran into problems over and over again, I'm posting as much as I can here. Hopefully you can ignore most of the post for this reason.
Some reading before you start:
Having toyed with this for a while I've noticed one important factor in getting this to work:
the hostnames you use when you configure SGE
MUST match those returned by hostname. It doesn't matter what you've defined in your /etc/host file. This can obviously cause a little bit of trouble when you've got multiple subnets set up (my computers communicate via a 10/100 net for WAN and a 10/100/1000 net for computations). My front node is called beryllium (i.e. this is what is returned when
hostname is executed) but it's known as corella on the gigabit LAN. Same goes for one of my sub nodes: it's called borax on the giganet and boron on the slow LAN. hostname here returns boron. I should obviously go back and redo this for the gigabit subnet later -- I'm just posting what worked.
While setting it up on the front node takes a little while, the good news is that very little work needs to be done on each node. This would become important when you are working with a large number of nodes -- with the power of xargs and a name list, setting them up on the front node should be a breeze.
My front node is beryllium, and one of my subnodes is boron. I've got key-based, password-less ssh login set up.
Set up your front node before you touch your subnodes. Add all the node name to your front node before even installing gridengine-exec on the subnode.
I've spent a day struggling with this. The order of events listed here is
the first thing that worked. You make modifications at your own peril (and frustration). I tried openjdk with little luck, hence the sun java.
NFS
Finally, I've got nfs set up to share a folder from the front node (~/jobs) to all my subnodes. See here for instructions on how to set it up:
http://verahill.blogspot.com.au/2012/02/debian-testing-wheezy-64-sharing-folder.html
When you use ecce, you can and SHOULD use
local scratch folders i.e. use your nfs shared folder as the runtime folder, but set scratch to e.g. /tmp which isn't an nfs exported folder.
Before you start, stop and purge
if you've tried installing and configuring gridengine in the past, there may be processes and files which will interfere. On
each computer do
ps aux|grep sge
use
sudo kill to kill any sge processes
Then
sudo apt-get purge gridengine-*
First install sun/oracle java on all nodes.
[UPDATE 24 Aug 2013:
openjdk-6-jre or
openjdk-7-jre work fine, so
you can skip this]
There's no sun/oracle java in the debian testing repos anymore, so we'll follow this:
http://verahill.blogspot.com.au/2012/04/installing-sunoracle-java-in-debian.html
sudo apt-get install java-package
Download the jre-6u31-linux-x64.bin from here:
http://java.com/en/download/manual.jsp?locale=en
make-jpkg jre-6u31-linux-x64.bin
sudo dpkg -i oracle-j2re1.6_1.6.0+update31_amd64.deb
Then select your shiny oracle java by doing:
sudo update-alternatives --config java
sudo update-alternatives --config javaws
Do that one every node, front and subnodes. You don't have to do all the steps though: you just built oracle-j2re1.6_1.6.0+update31_amd64.deb so copy that to your nodes, do sudo dpkg -i oracle-j2re1.6_1.6.0+update31_amd64.deb and then do the sudo update-alternatives dance.
Front node:
sudo apt-get install gridengine-client gridengine-qmon gridengine-exec gridengine-master
(at the moment this installs v 6.2u5-7)
I used the following:
Configure automatically: yes
Cell name: rupert
Master hostname: beryllium
=> SGE_ROOT: /var/lib/gridengine
=> SGE_CELL: rupert
=> Spool directory: /var/spool/gridengine/spooldb
=> Initial manager user: sgeadmin
Once it was installed, I added myself as an sgeadmin:
sudo -u sgeadmin qconf -am ${USER}
sgeadmin@beryllium added "verahill" to manager list
and to the user list:
qconf -au ${USER} users
added "verahill" to access list "users"
We add beryllium as a submit host
qconf -as beryllium
beryllium added to submit host list
Create the group allhosts
qconf -ahgrp @allhosts
1 group_name @allhosts
2 hostlist NONE
I made no changes
Add beryllium to the hostlist
qconf -aattr hostgroup hostlist beryllium @allhosts
verahill@beryllium modified "@allhosts" in host group list
qconf -aq main.q
This opens another text file. I made no changes.
verahill@beryllium added "main.q" to cluster queue list
Add the host group to the queue:
qconf -aattr queue hostlist @allhosts main.q
verahill@beryllium modified "main.q" in cluster queue list
1 core on beryllium is added to SGE:
qconf -aattr queue slots "[beryllium=1]" main.q
verahill@beryllium modified "main.q" in cluster queue list
Add execution host
qconf -ae
which opens a text file in vim
I edited hostname (
boron) but nothing else. Saving returns
added host boron to exec host list
Add boron as a submit host
qconf -as boron
boron added to submit host list
Add 3 cores for boron:
qconf -aattr queue slots "[boron=3]" main.q
Add boron to the queue
qconf -aattr hostgroup hostlist boron @allhosts
Here's my history list in case you can't be bother reading everything in detail above.
2015 sudo apt-get install gridengine-client gridengine-qmon gridengine-exec gridengine-master
2016 sudo -u sgeadmin qconf -am ${USER}
2017 qconf -help
2018 qconf user_list
2019 qconf -au ${USER} users
2020 qconf -as beryllium
2021 qconf -ahgrp @allhosts
2022 qconf -aattr hostgroup hostlist beryllium @allhosts
2023 qconf -aq main.q
2024 qconf -aattr queue hostlist @allhosts main.q
2025 qconf -aattr queue slots "[beryllium=1]" main.q
2026 qconf -as boron
2027 qconf -ae
2028 qconf -aattr hostgroup hostlist beryllium @allhosts
2029 qconf -aattr queue slots "[boron=3]" main.q
2030 qconf -aattr hostgroup hostlist boron @allhosts
Next, set up your subnodes:
My example here is a subnode called boron.
On the subnode:
sudo apt-get install gridengine-exec gridengine-client
Configure automatically: yes
Cell name: rupert
Master hostname: beryllium
This node is called boron.
Check whether
sge_execd got start after the install
ps aux|grep sge
sgeadmin 25091 0.0 0.0 31712 1968 ? Sl 13:54 0:00 /usr/lib/gridengine/sge_execd
If not, and only if not, do
/etc/init.d/gridengine-exec start
cat /tmp/execd_messages.*
If there's no message corresponding to the current iteration of sge (i.e. you may have old error messages from earlier attempts) then you're probably in a good place.
Back to the front node:
qhost
HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
-------------------------------------------------------------------------------
global - - - - - - -
beryllium lx26-amd64 6 0.57 7.8G 3.9G 14.9G 597.7M
boron lx26-amd64 3 0.62 3.8G 255.6M 14.9G 0.0
If the exec node isn't recognised (i.e. it's listed but no cpu info or anything else) then you're in a dark place. Probably you'll find a message about "request for user soandso does not match credentials" in your /tmp/execd_messages.* files on the exec node. The only way I got that solved was stopping all sge processes everywhere, purging all gridengine-* packages on all nodes and starting from the beginning -- hence why I posted the history output above.
qstat -f
queuename qtype resv/used/tot. load_avg arch states
---------------------------------------------------------------------------------
main.q@beryllium BIP 0/0/1 0.64 lx26-amd64
---------------------------------------------------------------------------------
main.q@boron BIP 0/0/3 0.72 lx26-amd64
Time to see how far we've got:
Create a file called test.qsub on your front node:
#$ -S /bin/csh
#$ -cwd
tree -L 1 -d
hostname
qsub test.qsub
Your job 2 ("test.qsub") has been submitted
qstat -u ${USER}
job-ID prior name user state submit/start at queue slots ja-task-ID
2 0.00000 test.qsub verahill qw 06/05/2012 14:03:10 1
ls
test.qsub test.qsub.e2 test.qsub.o2
cat test.qsub.[oe]*
.
0 directories
beryllium
Tree could have had more exciting output I s'pose, but I didn't have any subfolders in my run directory.
So far, so good. We still need to set up
parallel environments (e.g. orte, mpi).
Before that,
we'll add another node, which is called
tantalum and has a quadcore cpu.
On the front node:
qconf -as tantalum
qconf -ae
replace template with tantalum
qconf -aattr queue slots "[tantalum=4]" main.q
qconf -aattr hostgroup hostlist tantalum @allhosts
qhost
HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
-------------------------------------------------------------------------------
global - - - - - - -
beryllium lx26-amd64 6 0.67 7.8G 3.7G 14.9G 597.7M
boron lx26-amd64 3 0.14 3.8G 248.0M 14.9G 0.0
tantalum - - - - - - -
On tantalum:
Install java by copying the oracle-j2re1.6_1.6.0+update31_amd64.deb which got created when you set it up the first time.
sudo dpkg -i oracle-j2re1.6_1.6.0+update31_amd64.deb
sudo update-alternatives --config java
sudo update-alternatives --config javaws
Install gridengine:
sudo apt-get install gridengine-exec gridengine-client
On the front node:
qhost
HOSTNAME ARCH NCPU LOAD MEMTOT MEMUSE SWAPTO SWAPUS
-------------------------------------------------------------------------------
global - - - - - - -
beryllium lx26-amd64 6 0.62 7.8G 3.7G 14.9G 601.0M
boron lx26-amd64 3 0.15 3.8G 248.6M 14.9G 0.0
tantalum lx26-amd64 4 4.02 7.7G 977.0M 14.9G 24.1M
qstat -f
queuename qtype resv/used/tot. load_avg arch states
---------------------------------------------------------------------------------
main.q@beryllium BIP 0/0/1 0.71 lx26-amd64
---------------------------------------------------------------------------------
main.q@boron BIP 0/0/3 0.72 lx26-amd64
---------------------------------------------------------------------------------
main.q@tantalum BIP 0/0/4 4.01 lx26-amd64
It's a beautiful thing when everything suddenly works.
Parallel environments:
In order to use all the cores on each node we need to set up parallel environments.
qconf -ap orte
pe_name orte
slots 9999
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule $fill_up
control_slaves FALSE
job_is_first_task FALSE
urgency_slots min
accounting_summary FALSE
To use a parallel environement include #$ -pe orte 3 for 3 slots in your test.qsub:
#$ -S /bin/csh
#$ -cwd
#$ -pe orte 3
tree -L 1 -d
hostname
Submit it:
qsub test.qsub
Your job 14 ("test.qsub") has been submitted
qstat
job-ID prior name user state submit/start at queue slots ja-task-ID
14 0.00000 test.qsub verahill qw 06/05/2012 15:43:25 3
verahill@beryllium:~/mine/qsubtest$ cat test.qsub.*
.
0 directories
boron
It got executed on boron.
We are basically done with a basic setup now. To read more, use google. Some additional info that might be helpful is here:
http://wiki.gridengine.info/wiki/index.php/StephansBlog
We're going to set up a few more parallel environments:
qconf -ap mpi1
pe_name mpi1
slots 9
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule $fill_up
control_slaves FALSE
job_is_first_task TRUE
urgency_slots min
accounting_summary FALSE
qconf -ap mpi2
pe_name mpi2
slots 9
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule 2
control_slaves FALSE
job_is_first_task TRUE
urgency_slots min
accounting_summary FALSE
qconf -ap mpi3
pe_name mpi3
slots 9
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule 3
control_slaves FALSE
job_is_first_task TRUE
urgency_slots min
accounting_summary FALSE
qconf -ap mpi4
pe_name mpi4
slots 9
user_lists NONE
xuser_lists NONE
start_proc_args /bin/true
stop_proc_args /bin/true
allocation_rule 4
control_slaves FALSE
job_is_first_task TRUE
urgency_slots min
accounting_summary FALSE
And we'll call these using the
#$ -pe mpi$totalprocs $totalprocs directive below
We
need to add them (update: you need to add them to a queue. Which one is irrelevant, as long as the environment and queue parameters are consistent) to our main.q file though:
qconf -mq main.q
pe_list make orte mpi1 mpi2 mpi3 mpi4
This obviously isn't the end of my travails -- now I need to get nwchem and gaussian happy.
I've got this in my CONFIG.Dynamic (inside joke) file
NWChem: /opt/nwchem/nwchem-6.1/bin/LINUX64/nwchem
Gaussian-03: /opt/gaussian/g09/g09
perlPath: /usr/bin/perl
qmgrPath: /usr/bin/
SGE {
#$ -S /bin/csh
#$ -cwd
#$ -l h_rt=$wallTime
#$ -l h_vmem=4G
#$ -j y
#$ -pe mpi$totalprocs $totalprocs
}
NWChemCommand {
setenv LD_LIBRARY_PATH "/usr/lib/openmpi/lib:/opt/openblas/lib"
setenv PATH "/bin:/usr/bin:/sbin:/usr/sbin"
mpirun -n $totalprocs /opt/nwchem/nwchem-6.1/bin/LINUX64/nwchem $infile > $outfile
}
Gaussian-03Command{
setenv GAUSS_SCRDIR /scratch
setenv GAUSS_EXEDIR /opt/gaussian/g09/bsd:/opt/gaussian/g09/local:/opt/gaussian/g09/extras:/opt/gaussian/g09
/opt/gaussian/g09/g09 $infile $outfile >g09.log
}
And now everything works!
See below for a few of the annoying errors I encountered during my adventures:
Error -- missing gridengine-client
The gaussian set-up worked fine. The nwchem setup worked on one node but not at all on another -- my problem sounded identical to that described here (two nodes, same binaries, still one works and one doesn't):
http://www.open-mpi.org/community/lists/users/2010/07/13503.php
And it's the same as this one too
http://www.digipedia.pl/usenet/thread/11269/867/
[boron:18333] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file ../../../../../../orte/mca/ess/hnp/ess_hnp_module.c at line 161--------------------------------------------------------------------------It looks like orte_init failed for some reason; your parallel process islikely to abort. There are many reasons that a parallel process canfail during orte_init; some of which are due to configuration orenvironment problems. This failure appears to be an internal failure;here's some additional information (which may only be relevant to anOpen MPI developer):
orte_plm_base_select failed --> Returned value Not found (-13) instead of ORTE_SUCCESS--------------------------------------------------------------------------[boron:18333] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file ../../../orte/runtime/orte_init.c at line 132--------------------------------------------------------------------------It looks like orte_init failed for some reason; your parallel process islikely to abort. There are many reasons that a parallel process canfail during orte_init; some of which are due to configuration orenvironment problems. This failure appears to be an internal failure;here's some additional information (which may only be relevant to anOpen MPI developer):
orte_ess_set_name failed --> Returned value Not found (-13) instead of ORTE_SUCCESS--------------------------------------------------------------------------[boron:18333] [[INVALID],INVALID] ORTE_ERROR_LOG: Not found in file ../../../../../orte/tools/orterun/orterun.c at line 543
It's tooka while to troubleshoot this one. As always, when you're troubleshooting you discover the odd thing or two. On my front node:
/usr/bin/rsh -> /etc/alternatives/rsh
which is normal, but
/etc/alternatives/rsh -> /usr/bin/krb5-rsh
There are some krb packages on tantalum, but nothing on boron
boron:
locate rsh|grep "usr/bin"
/usr/bin/rsh
tantalum:
locate rsh|grep "usr/bin"
/usr/bin/glib-genmarshal
/usr/bin/qrsh
/usr/bin/rsh
sudo apt-get autoremove krb5-clients
Of course, that did not get it working...
The annoying things is that nwchem/mpirun on boron work perfectly together, also when submitting jobs directly via ECCE. It's just with qsub that I am having trouble. Search continues:
On the troublesome node:
aptitude search mpi|grep ^i
i libblacs-mpi-dev - Basic Linear Algebra Comm. Subprograms - D
i A libblacs-mpi1 - Basic Linear Algebra Comm. Subprograms - S
i A libexempi3 - library to parse XMP metadata (Library)
i libopenmpi-dev - high performance message passing library -
i A libopenmpi1.3 - high performance message passing library -
i libscalapack-mpi-dev - Scalable Linear Algebra Package - Dev. fil
i A libscalapack-mpi1 - Scalable Linear Algebra Package - Shared l
i A mpi-default-bin - Standard MPI runtime programs (metapackage
i A mpi-default-dev - Standard MPI development files (metapackag
i openmpi-bin - high performance message passing library -
i A openmpi-checkpoint - high performance message passing library -
i A openmpi-common - high performance message passing library -
Library conflict?
sudo apt-get autoremove mpi-default-*
And then recompile nwchem. Still no change.
Finally I found the real problem:
gridengine-client was missing on the troublesome node. Once I had installed that, everything worked!
Errors:
If your parallel job won't start (sits with qw forever), and qstat -o jobid gives you
scheduling info: cannot run in PE "orte" because it only offers 0 slots
make sure that qstat -f lists all your nodes.
This is good:
qstat -f
queuename qtype resv/used/tot. load_avg arch states
---------------------------------------------------------------------------------
main.q@beryllium BIP 0/0/1 0.71 lx26-amd64
---------------------------------------------------------------------------------
main.q@boron BIP 0/0/3 0.72 lx26-amd64
---------------------------------------------------------------------------------
main.q@tantalum BIP 0/0/4 4.01 lx26-amd64
This is bad:
qstat -f
queuename qtype resv/used/tot. load_avg arch states
---------------------------------------------------------------------------------
main.q@beryllium BIP 0/0/1 0.64 lx26-amd64
To fix it, do
qconf -aattr hostgroup hostlist tantalum @allhosts
on the front node for all your node names (change tantalum to the correct name)
An unhelpful error message:
qstat -u verahill
job-ID prior name user state submit/start at queue slots ja-task-ID
3 0.50000 test.qsub verahill Eqw 06/05/2012 11:45:18 1
cat test.qsub.[eo]*
/builder/src-buildserver/Platform-7.0/src/linux/lwmsg/src/connection-wire.c:325: Should not be here
This came from a faulty qsub directive: I used
#$ -S csh
instead of
#$ -S /bin/csh
i.e. you should use the latter.
I think it's a potentially common enough mistake that I post it here. See
http://helms-deep.cable.nu/~rwh/blog/?p=159 for more errors.
Links to this post:
http://gridengine.org/pipermail/users/2012-November/005207.html
http://web.archiveorange.com/archive/v/JfPLjOHE5fXSiyFH0yzc
