617. SLURM on debian jessie (and compiling Jürgen Rinas' sinfo)

Two issues:
* Sun GridEngine (now Oracle GridEngine) is missing from Debian Jessie. I need a queue manager for my cluster. For now the wheezy package runs fine in debian jessie, but I'd be happier with a supported solution. SLURM is a good alternative here, and I've used it at the TACC.

* SLURM conflicts with Jürgen Rinas' sinfo package, which I use to keep an eye on my cluster. Until this has been resolved, I'll compile and use my own version of sinfo -- basically, I'll rename sinfo and sinfod to sinfo_jr and sinfod_jr. I can't live without sinfo.

---

Compiling sinfo

mkdir ~/tmp/sinfo -p
cd ~/tmp/sinfo

sudo apt-get install build-essential

sudo apt-get autoremove sinfo 

apt-get source sinfo
cd sinfo-0.0.47/
vim debian/rules 

Change

16     dh_auto_configure -- --enable-SIMPLE_USER_CACHE --enable-CPUNO_ADJUST
21     rm $(CURDIR)/debian/sinfo/usr/bin/sshallsinfo
22     rm $(CURDIR)/debian/sinfo/usr/share/man/man1/sshallsinfo.1
25     rm $(CURDIR)/debian/sinfo/usr/lib/*/sinfo/*.la
37     chmod 755 $(CURDIR)/debian/sinfo/usr/share/sinfo/sinfo.pl.cgi

to

16     dh_auto_configure -- --enable-SIMPLE_USER_CACHE --enable-CPUNO_ADJUST --program-suffix=_jr
21     rm $(CURDIR)/debian/sinfojr/usr/bin/sshallsinfo_jr
22     rm $(CURDIR)/debian/sinfojr/usr/share/man/man1/sshallsinfo_jr.1
25     rm $(CURDIR)/debian/sinfojr/usr/lib/*/sinfo/*.la
37     chmod 755 $(CURDIR)/cgi/sinfo.pl.cgi

That's jr for Jürgen Rinas.

Then edit debian/control and change

12 Package: sinfo
15 Conflicts: slurm-client, slurm-llnl (<< 14.03.8-1)

to

12 Package: sinfojr
15 Conflicts: 

Build:

dpkg-buildpackage -us -uc
cd ../
sudo dpkg -i sinfo_0.0.47-3_amd64.deb

I launch sinfodjr at boot by putting the following in /etc/rc.local:

su verahill -c '/usr/sbin/sinfodjr --bcast 192.168.1.255' &

---

SLURM:
I had a look at this post: https://paolobertasi.wordpress.com/2011/05/24/how-to-install-slurm-on-debian/

It looked to easy to be true.

Here's what I ended up doing:

On the MASTER node:

sudo apt-get install slurm-wlm slurmctld slurmd

[..]
Generating a pseudo-random key using /dev/urandom completed.
Please refer to /usr/share/doc/munge/README.Debian for instructions to generate more secure key.
Setting up slurm-client (14.03.9-5) ...
Setting up slurm-wlm-basic-plugins (14.03.9-5) ...
Setting up slurmd (14.03.9-5) ...
Setting up slurmctld (14.03.9-5) ...
Setting up slurm-wlm (14.03.9-5) ...
[..]

open file:///usr/share/doc/slurmctld/slurm-wlm-configurator.easy.html

# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=beryllium
ControlAddr=192.168.1.1
#
#MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/pgid
ReturnToService=2
SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/lib/slurm/slurmd
SlurmUser=slurm
#SlurmdUser=root
StateSaveLocation=/var/lib/slurm/slurmctld
SwitchType=switch/none
TaskPlugin=task/none
#
#
# TIMERS
#KillWait=30
#MinJobAge=300
#SlurmctldTimeout=120
#SlurmdTimeout=300
#
#
# SCHEDULING
FastSchedule=1
SchedulerType=sched/backfill
#SchedulerPort=7321
SelectType=select/linear
#
#
# LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
ClusterName=rupert
#JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
#SlurmctldDebug=3
SlurmctldLogFile=/var/log/slurm/slurmctld.log
#SlurmdDebug=3
SlurmdLogFile=/var/log/slurm/slurmd.log
#
#
# COMPUTE NODES
NodeName=beryllium NodeAddr=192.168.1.1
NodeName=neon NodeAddr=192.168.1.120
PartitionName=All Nodes=beryllium,neon

Copy the above block to /etc/slurm-llnl/slurm.conf

Note the lack of spaces between beryllium and neon in the Nodes= directive.

scontrol show daemons

slurmctld
sudo /usr/sbin/create-munge-key

The munge key /etc/munge/munge.key already exists
Do you want to overwrite it? (y/N) y
Generating a pseudo-random key using /dev/urandom completed.

sudo systemctl enable slurmctld.service
sudo ln -s /var/lib/slurm-llnl /var/lib/slurm
sudo systemctl start slurmctld.service
sudo systemctl status slurmctld.service 

● slurmctld.service - Slurm controller daemon
   Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled)
   Active: active (running) since Tue 2015-07-21 11:16:18 AEST; 40s ago
  Process: 19958 ExecStart=/usr/sbin/slurmctld $SLURMCTLD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 19960 (slurmctld)
   CGroup: /system.slice/slurmctld.service
           └─19960 /usr/sbin/slurmctld

sudo systemctl status munge.service

● munge.service - MUNGE authentication service
   Loaded: loaded (/lib/systemd/system/munge.service; disabled)
   Active: active (running) since Wed 2015-07-08 00:11:18 AEST; 1 weeks 6 days ago
     Docs: man:munged(8)
 Main PID: 25986 (munged)
   CGroup: /system.slice/munge.service
           └─25986 /usr/sbin/munged

Also, add yourself to the group slurm and chmod g+r /var/log/slurm/accounting.

On neon (and later on each node): 
Install slurmd and slurm-client as shown below, then copy the /etc/munge/munge.key from the master node to the execute node. Do the same with /etc/slurm-llnl/slurm.conf. Then enable and restart the services.

sudo apt-get install slurmd slurm-client
sudo ln -s /var/lib/slurm-llnl /var/lib/slurm
sudo systemctl enable slurmd.service
sudo systemctl restart slurmd.service
sudo systemctl enable munge.service
sudo systemctl restart munge.service
sudo systemctl status slurmd.service

On the main host (beryllium) I checked that everything was well:

sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
All          up   infinite      1  idle* beryllium, neon