Pages

28 July 2015

618. Modifying ECCE to work with slurm

UPDATE: ecce stops monitoring the job after 10-20 seconds. The job continues fine though. Working on fixing the monitoring issue. This message will be removed once that's fixed. It was due to $q needing to be lowercase (i.e. 'slurm', not 'Slurm') in eccejobmonitor.



Sun Gridengine has been removed from debian jessie (it's in wheezy and sid). This has given me a good excuse to explore setting up SLURM on my debian cluster. So I did: http://verahill.blogspot.com.au/2015/07/617-slurm-on-debian-jessie-and.html

My setup is very simple, with each node having it's own working directory that they export via NFS to the main node. Also, I never run jobs across several nodes. Because of that, each node has it's own queue. Not how beowulf clusters were supposed to work, but it's the best solution for me (e.g. ROCKS does the opposite -- exports the user dir from the main node, but that makes reading and writing slow where it counts i.e. on the work nodes).

I've currently got this slurm.conf:
ControlMachine=beryllium ControlAddr=192.168.1.1 MpiDefault=none ProctrackType=proctrack/pgid ReturnToService=2 SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid SlurmdSpoolDir=/var/lib/slurm/slurmd SlurmUser=slurm StateSaveLocation=/var/lib/slurm/slurmctld SwitchType=switch/none TaskPlugin=task/none FastSchedule=1 SchedulerType=sched/backfill SelectType=select/linear AccountingStorageType=accounting_storage/filetxt AccountingStorageLoc=/var/log/slurm/accounting ClusterName=rupert JobAcctGatherType=jobacct_gather/none SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd.log NodeName=beryllium NodeAddr=192.168.1.1 NodeName=neon NodeAddr=192.168.1.120 state=unknown NodeName=tantalum NodeAddr=192.168.1.150 state=unknown NodeName=magnesium NodeAddr=192.168.1.200 state=unknown NodeName=carbon NodeAddr=192.168.1.190 state=unknown NodeName=oxygen NodeAddr=192.168.1.180 state=unknown PartitionName=All Nodes=neon,beryllium,tantalum,oxygen,magnesium,carbon default=yes maxtime=infinite state=up PartitionName=mpi4 Nodes=tantalum maxtime=infinite state=up PartitionName=mpi12 Nodes=carbon maxtime=infinite state=up PartitionName=mpi8 Nodes=neon maxtime=infinite state=up PartitionName=mpix8 Nodes=oxygen maxtime=infinite state=up PartitionName=mpix12 Nodes=magnesium maxtime=infinite state=up PartitionName=mpi1 Nodes=beryllium maxtime=infinite state=up
and sinfo returns
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST All* up infinite 6 idle beryllium,carbon,magnesium,neon,oxygen,tantalum mpi4 up infinite 1 idle tantalum mpi12 up infinite 1 idle carbon mpi8 up infinite 1 idle neon mpix8 up infinite 1 idle oxygen mpix12 up infinite 1 idle magnesium mpi1 up infinite 1 idle beryllium

The first step was to figure out what files to edit:

grep -rs "qsub"
apps/siteconfig/QueueManagers:SGE|submitCommand: qsub ##script##
grep -rs "SGE"
apps/scripts/eccejobmonitor: &MsgSendUp("SGE job id '$id' in state '$state'"); [..] apps/siteconfig/Queues:magnesium|queueMgrName: SGE

Here are the files I edited:

apps/siteconfig/QueueManagers:
12 QueueManagers: LoadLeveler \ 13 Maui \ 14 EASY \ 15 PBS \ 16 LSF \ 17 Moab \ 18 SGE \ 19 Shell\ 20 Slurm
185 Shell|jobIdParseExpression: \ [0-9]+ 186 187 ############################################################################### 188 # SLURM 189 # Simple Linux Utility for Resource Management 190 # 191 # 192 Slurm|submitCommand: sbatch ##script## 193 Slurm|cancelCommand: scancel ##id## 194 Slurm|queryJobCommand: squeue 195 Slurm|queryMachineCommand: sinfo -p ##queue## 196 Slurm|queryQueueCommand: squeue -a 197 Slurm|queryDiskUsageCommand: df -k 198 Slurm|jobIdParseExpression: .* 199 Slurm|jobIdParseLeadingText: job
apps/scripts/eccejobmonitor:
2124 LogMsg "Globus status from eccejobstore: $state"; 2125 } 2126 elsif ($q eq 'slurm') 2127 { 2128 $cmd = "squeue 2>&1"; 2129 if (open(QUERY, "$cmd |")) 2130 { 2131 $gotState = 0; 2132 while () 2133 { 2134 LogMsg "JobCheck: Slurm qstat line: $_"; 2135 if (/^\s*$id/) 2136 { 2137 my $state = (split)[5]; 2138 2139 &MsgSendUp("Slurm job id '$id' in state '$state'"); 2140 2141 if (grep {$state eq $_} qw{R 2142 t}) 2143 { 2144 $status = $JOB_STATE_RUNNING; 2145 } 2146 elsif (grep {$state eq $_} qw{PD}) 2147 { 2148 $status = $JOB_STATE_PENDING; 2149 } 2150 $gotState = 1; 2151 last; 2152 } 2153 } 2154 if ($gotState == 0) 2155 { 2156 if ($gJobCheckState != $JOB_STATE_NONE) 2157 { 2158 $status = $JOB_STATE_DONE; 2159 } 2160 } 2161 close QUERY;

Next set up a new machine (or queue) using ecce -admin. Set up a queue -- you won't be able to select Slurm, so select e.g. PBS. Edit the apps/siteconfig/CONFIG.machinename file to e.g.
1 NWChem: /opt/nwchem/Nwchem/bin/LINUX64/nwchem 2 Gaussian-03: /opt/gaussian/g09d/g09/g09 3 perlPath: /usr/bin/ 4 qmgrPath: /usr/bin/ 5 xappsPath: /usr/bin/ 6 7 Slurm { 8 #!/bin/csh 9 #SBATCH -p mpi8 10 #SBATCH --time=$walltime 11 #SBATCH --output=slurm.out 12 #SBATCH --job-name=$submitFile 13 } 14 15 NWChemEnvironment { 16 PYTHONPATH /opt/nwchem/Nwchem/contrib/python 17 } 18 19 NWChemFilesToRemove{ core *.aoints.* *.gridpts.* } 20 21 NWChemCommand { 22 setenv PATH "/bin:/usr/bin:/sbin:/usr/sbin" 23 setenv LD_LIBRARY_PATH "/usr/lib/openmpi/lib:/opt/openblas/lib:/opt/acml/acml5.3.1/gfortran64_fma4_int64/lib:/opt/acml/acml5.3.1/gfortran64_int64/lib:/opt/intel/mkl/lib/intel64" 24 hostname 25 mpirun -n $totalprocs /opt/nwchem/Nwchem/bin/LINUX64/nwchem $infile > $outfile 26 } 27 28 Gaussian-03FilesToRemove{ core *.rwf } 29 30 Gaussian-03Command{ 31 set path = ( /opt/nbo6/bin $path ) 32 setenv GAUSS_SCRDIR /home/me/scratch 33 setenv GAUSS_EXEDIR /opt/gaussian/g09d/g09/bsd:/opt/gaussian/g09d/g09/local:/opt/gaussian/g09d/g09/extras:/opt/gaussian/g09d/g09 34 /opt/gaussian/g09d/g09/g09< $infile > $outfile 35 echo 0 36 } 37 38 Wrapup{ 39 dmesg|tail 40 find ~/scratch/* -name "*" -user me|xargs -I {} rm {} -rf 41 }
Next, edit apps/siteconfig/Queues -- in my case the machine I created is called neon-slurm:
neon-slurm|queueMgrName: Slurm neon-slurm|queueMgrVersion: 2.0~ neon-slurm|prefFile: neon-slurm.Q
And that's all. You should now be able to submit jobs via slurm. There's obviously a lot more than can be done and configured with SLURM, but this was enough to get me up and running, so that I'm now 'future-proofed' in case SGE never comes back into debian stable.

And here's what it looks like when my ecce-submitted jobs are running:

squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 30 mpi8 In_monom me PD 0:00 1 (Resources) 29 mpi8 b_monome me R 16:12 1 neon 31 mpix12 tl_dimer me R 34:28 1 magnesium

617. SLURM on debian jessie (and compiling Jürgen Rinas' sinfo)

Two issues:
* Sun GridEngine (now Oracle GridEngine) is missing from Debian Jessie. I need a queue manager for my cluster. For now the wheezy package runs fine in debian jessie, but I'd be happier with a supported solution. SLURM is a good alternative here, and I've used it at the TACC.

* SLURM conflicts with Jürgen Rinas' sinfo package, which I use to keep an eye on my cluster. Until this has been resolved, I'll compile and use my own version of sinfo -- basically, I'll rename sinfo and sinfod to sinfo_jr and sinfod_jr. I can't live without sinfo.



Compiling sinfo
mkdir ~/tmp/sinfo -p
cd ~/tmp/sinfo
sudo apt-get install build-essential
sudo apt-get autoremove sinfo 
apt-get source sinfo
cd sinfo-0.0.47/
vim debian/rules 

Change
16 dh_auto_configure -- --enable-SIMPLE_USER_CACHE --enable-CPUNO_ADJUST 21 rm $(CURDIR)/debian/sinfo/usr/bin/sshallsinfo 22 rm $(CURDIR)/debian/sinfo/usr/share/man/man1/sshallsinfo.1 25 rm $(CURDIR)/debian/sinfo/usr/lib/*/sinfo/*.la 37 chmod 755 $(CURDIR)/debian/sinfo/usr/share/sinfo/sinfo.pl.cgi
to
16 dh_auto_configure -- --enable-SIMPLE_USER_CACHE --enable-CPUNO_ADJUST --program-suffix=_jr 21 rm $(CURDIR)/debian/sinfojr/usr/bin/sshallsinfo_jr 22 rm $(CURDIR)/debian/sinfojr/usr/share/man/man1/sshallsinfo_jr.1 25 rm $(CURDIR)/debian/sinfojr/usr/lib/*/sinfo/*.la 37 chmod 755 $(CURDIR)/cgi/sinfo.pl.cgi
That's jr for Jürgen Rinas.

Then edit debian/control and change
12 Package: sinfo
15 Conflicts: slurm-client, slurm-llnl (<< 14.03.8-1)
to
12 Package: sinfojr
15 Conflicts: 
Build:
dpkg-buildpackage -us -uc
cd ../
sudo dpkg -i sinfo_0.0.47-3_amd64.deb

I launch sinfodjr at boot by putting the following in /etc/rc.local:
su verahill -c '/usr/sbin/sinfodjr --bcast 192.168.1.255' &

SLURM:
I had a look at this post: https://paolobertasi.wordpress.com/2011/05/24/how-to-install-slurm-on-debian/

It looked to easy to be true.

Here's what I ended up doing:

On the MASTER node:
sudo apt-get install slurm-wlm slurmctld slurmd
[..] Generating a pseudo-random key using /dev/urandom completed. Please refer to /usr/share/doc/munge/README.Debian for instructions to generate more secure key. Setting up slurm-client (14.03.9-5) ... Setting up slurm-wlm-basic-plugins (14.03.9-5) ... Setting up slurmd (14.03.9-5) ... Setting up slurmctld (14.03.9-5) ... Setting up slurm-wlm (14.03.9-5) ... [..]
open file:///usr/share/doc/slurmctld/slurm-wlm-configurator.easy.html
# slurm.conf file generated by configurator easy.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # ControlMachine=beryllium ControlAddr=192.168.1.1 # #MailProg=/bin/mail MpiDefault=none #MpiParams=ports=#-# ProctrackType=proctrack/pgid ReturnToService=2 SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid #SlurmctldPort=6817 SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid #SlurmdPort=6818 SlurmdSpoolDir=/var/lib/slurm/slurmd SlurmUser=slurm #SlurmdUser=root StateSaveLocation=/var/lib/slurm/slurmctld SwitchType=switch/none TaskPlugin=task/none # # # TIMERS #KillWait=30 #MinJobAge=300 #SlurmctldTimeout=120 #SlurmdTimeout=300 # # # SCHEDULING FastSchedule=1 SchedulerType=sched/backfill #SchedulerPort=7321 SelectType=select/linear # # # LOGGING AND ACCOUNTING AccountingStorageType=accounting_storage/none ClusterName=rupert #JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/none #SlurmctldDebug=3 SlurmctldLogFile=/var/log/slurm/slurmctld.log #SlurmdDebug=3 SlurmdLogFile=/var/log/slurm/slurmd.log # # # COMPUTE NODES NodeName=beryllium NodeAddr=192.168.1.1 NodeName=neon NodeAddr=192.168.1.120 PartitionName=All Nodes=beryllium,neon
Copy the above block to /etc/slurm-llnl/slurm.conf

Note the lack of spaces between beryllium and neon in the Nodes= directive.
scontrol show daemons
slurmctld
sudo /usr/sbin/create-munge-key
The munge key /etc/munge/munge.key already exists Do you want to overwrite it? (y/N) y Generating a pseudo-random key using /dev/urandom completed.
sudo systemctl enable slurmctld.service
sudo ln -s /var/lib/slurm-llnl /var/lib/slurm
sudo systemctl start slurmctld.service
sudo systemctl status slurmctld.service 
● slurmctld.service - Slurm controller daemon Loaded: loaded (/lib/systemd/system/slurmctld.service; enabled) Active: active (running) since Tue 2015-07-21 11:16:18 AEST; 40s ago Process: 19958 ExecStart=/usr/sbin/slurmctld $SLURMCTLD_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 19960 (slurmctld) CGroup: /system.slice/slurmctld.service └─19960 /usr/sbin/slurmctld
sudo systemctl status munge.service
● munge.service - MUNGE authentication service Loaded: loaded (/lib/systemd/system/munge.service; disabled) Active: active (running) since Wed 2015-07-08 00:11:18 AEST; 1 weeks 6 days ago Docs: man:munged(8) Main PID: 25986 (munged) CGroup: /system.slice/munge.service └─25986 /usr/sbin/munged
Also, add yourself to the group slurm and chmod g+r /var/log/slurm/accounting.

On neon (and later on each node): 
Install slurmd and slurm-client as shown below, then copy the /etc/munge/munge.key from the master node to the execute node. Do the same with /etc/slurm-llnl/slurm.conf. Then enable and restart the services.

sudo apt-get install slurmd slurm-client
sudo ln -s /var/lib/slurm-llnl /var/lib/slurm
sudo systemctl enable slurmd.service
sudo systemctl restart slurmd.service
sudo systemctl enable munge.service
sudo systemctl restart munge.service
sudo systemctl status slurmd.service

On the main host (beryllium) I checked that everything was well:

sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
All          up   infinite      1  idle* beryllium, neon

26 July 2015

616. SIESTA on debian jessie with debian blacs, scalapack and SIESTA blas

Three posts showing slightly different ways of building SIESTA on debian may seem a bit excessive, but I figured I'd do a post on a simple 'bullet-proof' way of building SIESTA on debian jessie.

For ACML, see http://verahill.blogspot.com.au/2015/07/614-siesta-with-mpi-on-debian-jessie.html
For MKL (not all MKL versions work using that post): http://verahill.blogspot.com.au/2015/07/615-siesta-on-debian-jessie-with-intel.html

See those posts for detailed build instructions. I'll only give you the arch.make here -- for the rest, see either of the above posts.

I've got a node with intel mkl 2013.sp1.3.174 which hasn't got the blacs openmpi libs, so I ended up building SIESTA with the SIESTA BLAS and LAPACK libraries, and using the debian BLACS and SCALAPACK libs.

Install the libs with
sudo apt-get install libblacs-openmpi1 libopenmpi-dev libscalapack-openmpi1 libblacs-mpi-dev libscalapack-mpi-dev


Here's the arch.make:
# # This file is part of the SIESTA package. # # Copyright (c) Fundacion General Universidad Autonoma de Madrid: # E.Artacho, J.Gale, A.Garcia, J.Junquera, P.Ordejon, D.Sanchez-Portal # and J.M.Soler, 1996- . # # Use of this software constitutes agreement with the full conditions # given in the SIESTA license, as signed by all legitimate users. # .SUFFIXES: .SUFFIXES: .f .F .o .a .f90 .F90 SIESTA_ARCH=x86_64-unknown-linux-gnu--unknown FPP= FPP_OUTPUT= FC=mpif90 RANLIB=ranlib SYS=nag SP_KIND=4 DP_KIND=8 KINDS=$(SP_KIND) $(DP_KIND) FFLAGS=-g -O2 FPPFLAGS= -DMPI -DFC_HAVE_FLUSH -DFC_HAVE_ABORT LDFLAGS= ARFLAGS_EXTRA= FCFLAGS_fixed_f= FCFLAGS_free_f90= FPPFLAGS_fixed_F= FPPFLAGS_free_F90= BLAS_LIBS= LAPACK_LIBS= BLACS_LIBS=-L/usr/lib -lblacs-openmpi SCALAPACK_LIBS=-L/usr/lib -lscalapack-openmpi COMP_LIBS=dc_lapack.a liblapack.a libblas.a NETCDF_LIBS= NETCDF_INTERFACE= MPI_LIBS=-L/usr/lib/openmpi/lib -lmpi -lmpi_f90 -lmpi_f77 LIBS=$(SCALAPACK_LIBS) $(BLACS_LIBS) $(LAPACK_LIBS) $(BLAS_LIBS) $(NETCDF_LIBS) $(MPI_LIBS) -lpthread #SIESTA needs an F90 interface to MPI #This will give you SIESTA's own implementation #If your compiler vendor offers an alternative, you may change #to it here. MPI_INTERFACE=libmpi_f90.a MPI_INCLUDE=. #Dependency rules are created by autoconf according to whether #discrete preprocessing is necessary or not. .F.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_fixed_F) $< .F90.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_free_F90) $< .f.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_fixed_f) $< .f90.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_free_f90) $<
At some point in the future, when my nodes are free, I might do a bit of basic performance testing using the different versions.

25 July 2015

615. SIESTA on debian jessie with intel mkl and ifort

It's pretty similar to what I described in http://verahill.blogspot.com.au/2015/07/614-siesta-with-mpi-on-debian-jessie.html, with the main differences being the SCALAPACK, FC, BLACS and BLAS settings in arch.make

I presume that (the sadly no longer free for non-US academics) MKL was set up as shown here: http://verahill.blogspot.com.au/2013/06/465-intel-mkl-math-kernel-library-on.html

I haven't run all the tests on the build yet, but most of the ones that I tried worked, with the exception of the benzene test which came out with "Failure to converge standard eigenproblem", which is described here: http://departments.icmab.es/leem/siesta/Documentation/Manuals/manual-2.0/node47.html and isn't due to the build parameters.

NOTE: this doesn't work on mkl version 2013.sp1.3.174 as the blacs openmpi lib is missing. It does work on 2013.3.163, which is the version I used below. I have no idea why the libraries supplied with mkl are so different.

Follow
sudo apt-get install libopenmpi-dev
sudo mkdir /opt/siesta
sudo chown $USER /opt/siesta
cd /opt/siesta
wget http://departments.icmab.es/leem/siesta/CodeAccess/Code/siesta-3.2-pl-5.tgz
tar xvf siesta-3.2-pl-5.tgz
cd siesta-3.2-pl-5/Obj
sh ../Src/obj_setup.sh
../Src/./configure --enable-mpi

Edit arch.make:
# # This file is part of the SIESTA package. # # Copyright (c) Fundacion General Universidad Autonoma de Madrid: # E.Artacho, J.Gale, A.Garcia, J.Junquera, P.Ordejon, D.Sanchez-Portal # and J.M.Soler, 1996- . # # Use of this software constitutes agreement with the full conditions # given in the SIESTA license, as signed by all legitimate users. # .SUFFIXES: .SUFFIXES: .f .F .o .a .f90 .F90 SIESTA_ARCH=x86_64-unknown-linux-gnu--unknown FPP= FPP_OUTPUT= FC=ifort RANLIB=ranlib SYS=nag SP_KIND=4 DP_KIND=8 KINDS=$(SP_KIND) $(DP_KIND) FFLAGS=-g -O2 FPPFLAGS= -DMPI -DFC_HAVE_FLUSH -DFC_HAVE_ABORT LDFLAGS= ARFLAGS_EXTRA= FCFLAGS_fixed_f= FCFLAGS_free_f90= FPPFLAGS_fixed_F= FPPFLAGS_free_F90= BLAS_LIBS=-L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential LAPACK_LIBS=dc_lapack.a liblapack.a BLACS_LIBS=/opt/intel/mkl/lib/intel64/libmkl_blacs_openmpi_lp64.a SCALAPACK_LIBS=-L/opt/intel/mkl/lib/intel64 -lmkl_scalapack_lp64 COMP_LIBS=dc_lapack.a liblapack.a libblas.a NETCDF_LIBS= NETCDF_INTERFACE= MPI_LIBS= -L/usr/lib/openmpi/lib -lmpi -lmpi_f90 -lmpi_f77 LIBS=$(SCALAPACK_LIBS) $(BLACS_LIBS) $(LAPACK_LIBS) $(BLAS_LIBS) $(NETCDF_LIBS) $(MPI_LIBS) -lpthread #SIESTA needs an F90 interface to MPI #This will give you SIESTA's own implementation #If your compiler vendor offers an alternative, you may change #to it here. MPI_INTERFACE=libmpi_f90.a MPI_INCLUDE=/usr/lib/openmpi/include #Dependency rules are created by autoconf according to whether #discrete preprocessing is necessary or not. .F.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_fixed_F) $< .F90.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_free_F90) $< .f.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_fixed_f) $< .f90.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_free_f90) $<
Then build:
make

You can edit Tests/test.mk to make sure that it's pointing to your siesta executable (or make a symlink to siesta in /opt/siest/siesta-3.2-pl-5/) and that it uses mpirun and the appropriate number of threads.

Then run make in Tests/ to run all the tests.

24 July 2015

614. SIESTA with MPI and acml on debian jessie

One of my students might be using SIESTA for some simulations, and a first step towards that is to set it up on my cluster.

This isn't an optimised build -- right now I'm just looking at having a simple parallell build that runs.

I had a look at http://www.pa.msu.edu/people/tomanek/SIESTA-installation.html and http://pelios.csx.cam.ac.uk/~mc321/siesta.html.
 
NOTE: don't use the int64 acml or openblas BLAS libs, or you'll get SIGSEV due to invalid memory reference when running. NWChem is the complete opposite, and for some reason both the int64 and regulat acml libs have the same names. Not sure how that's supposed to work out on a system with nwchem, which needs the int64 libs.

See here for acml on debian. I've got /opt/acml/acml5.3.1/gfortran64_int64/lib in my /etc/ld.so.conf.d/acml.conf on behalf of nwchem.
 Being lazy, I opted for the debian scalapack and libblacs packages:
 
sudo apt-get install libscalapack-mpi-dev libblacs-mpi-dev libopenmpi-dev

To get the link to the SIESTA code, go to http://departments.icmab.es/leem/siesta/CodeAccess/selector.html

Then, if you're an academic, you can do:
sudo mkdir /opt/siesta
sudo chown $USER /opt/siesta
cd /opt/siesta
wget http://departments.icmab.es/leem/siesta/CodeAccess/Code/siesta-3.2-pl-5.tgz
tar xvf siesta-3.2-pl-5.tgz
cd siesta-3.2-pl-5/Obj
sh ../Src/obj_setup.sh
*** Compilation setup done. *** Remember to copy an arch.make file or run configure as: ../Src/configure [configure_options]
../Src/./configure --help
`configure' configures siesta 2.0 to adapt to many kinds of systems. Usage: ./configure [OPTION]... [VAR=VALUE]... [..] Installation directories: --prefix=PREFIX install architecture-independent files in PREFIX [/usr/local] --exec-prefix=EPREFIX install architecture-dependent files in EPREFIX [PREFIX] By default, `make install' will install all the files in `/usr/local/bin', `/usr/local/lib' etc. You can specify an installation prefix other than `/usr/local' using `--prefix', for instance `--prefix=$HOME'. [..] --enable-mpi Compile the parallel version of SIESTA --enable-debug Compile with debugging support --enable-fast Compile with best known optimization flags Optional Packages: --with-PACKAGE[=ARG] use PACKAGE [ARG=yes] --without-PACKAGE do not use PACKAGE (same as --with-PACKAGE=no) --with-netcdf=<lib> use NetCDF library --with-siesta-blas use BLAS library packaged with SIESTA --with-blas=<lib> use BLAS library --with-siesta-lapack use LAPACK library packaged with SIESTA --with-lapack=<lib> use LAPACK library --with-blacs=<lib> use BLACS library --with-scalapack=<lib> use ScaLAPACK library [..]
../Src/./configure --enable-mpi
checking build system type... x86_64-unknown-linux-gnu checking host system type... x86_64-unknown-linux-gnu [..] checking for mpifc... no checking for mpxlf... no checking for mpif90... mpif90 checking for MPI_Init... no checking for MPI_Init in -lmpi... yes [..] checking for sgemm in /opt/openblas/lib/libopenblas.so... yes checking LAPACK already linked... yes checking LAPACK includes divide-and-conquer routines... yes configure: using DC_LAPACK routines packaged with SIESTA due to bug in library. Linker flag might be needed to avoid duplicate symbols configure: creating ./config.status config.status: creating arch.make
Edit arch.make:
# # This file is part of the SIESTA package. # # Copyright (c) Fundacion General Universidad Autonoma de Madrid: # E.Artacho, J.Gale, A.Garcia, J.Junquera, P.Ordejon, D.Sanchez-Portal # and J.M.Soler, 1996- . # # Use of this software constitutes agreement with the full conditions # given in the SIESTA license, as signed by all legitimate users. # .SUFFIXES: .SUFFIXES: .f .F .o .a .f90 .F90 SIESTA_ARCH=x86_64-unknown-linux-gnu--unknown FPP= FPP_OUTPUT= FC=mpif90 RANLIB=ranlib SYS=nag SP_KIND=4 DP_KIND=8 KINDS=$(SP_KIND) $(DP_KIND) FFLAGS=-g -O2 FPPFLAGS= -DMPI -DFC_HAVE_FLUSH -DFC_HAVE_ABORT LDFLAGS= ARFLAGS_EXTRA= FCFLAGS_fixed_f= FCFLAGS_free_f90= FPPFLAGS_fixed_F= FPPFLAGS_free_F90= BLAS_LIBS=-L/opt/acml/acml5.3.1/gfortran64/lib -lacml LAPACK_LIBS= BLACS_LIBS=-L/usr/lib -lblacs-openmpi -lblacsCinit-openmpi SCALAPACK_LIBS=-L/usr/lib -lscalapack-openmpi COMP_LIBS=dc_lapack.a NETCDF_LIBS= NETCDF_INTERFACE= MPI_LIBS= -L/usr/lib/openmpi/lib -lmpi -lmpi_f90 LIBS=$(SCALAPACK_LIBS) $(BLACS_LIBS) $(LAPACK_LIBS) $(BLAS_LIBS) $(NETCDF_LIBS) $(MPI_LIBS) -lpthread #SIESTA needs an F90 interface to MPI #This will give you SIESTA's own implementation #If your compiler vendor offers an alternative, you may change #to it here. MPI_INTERFACE=libmpi_f90.a MPI_INCLUDE=. #Dependency rules are created by autoconf according to whether #discrete preprocessing is necessary or not. .F.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_fixed_F) $< .F90.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FPPFLAGS) $(FPPFLAGS_free_F90) $< .f.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_fixed_f) $< .f90.o: $(FC) -c $(FFLAGS) $(INCFLAGS) $(FCFLAGS_free_f90) $<
make
cd ../
ln -s Obj/siesta siesta

I added /opt/siesta/siesta-3.2-pl-5 to $PATH.

To test, edit /opt/siesta/siesta-3.2-pl-5/test.mk:
6 #SIESTA=../../../siesta 7 SIESTA=mpirun -n 2 ../../../siesta
Then
cd /opt/siesta/siesta-3.2-pl-5/Tests/h3po4_2
export LD_LIBRARY_CONFIG=/opt/acml/acml5.3.1/gfortran64/lib 
make
>>>> Running h3po4_2 test... ==> Copying pseudopotential file for H... ==> Copying pseudopotential file for O... ==> Copying pseudopotential file for P... ==> Running SIESTA as mpirun -n 2 ../../../siesta ===> SIESTA finished successfully

Also, look at work/h3po4_2.out:
* Running on    2 nodes in parallel
>> Start of run:  24-JUL-2015  21:58:13

                           ***********************       
                           *  WELCOME TO SIESTA  *       
                           ***********************       

reinit: Reading from standard input
[..]
elaps:  optical           1       0.000       0.000     0.00
  
>> End of run:  24-JUL-2015  21:58:20

07 July 2015

613. Debian Jessie: Turn off update pop-ups in gnome, and switching to lxterminal, nemo and ksnapshot

I originally installed the OS on my desktop back 2010 (Lenny) and haven't treated it very nicely (mixed releases, repos and have in general been installing, uninstalling and replacing packages with my own compiled ones -- and have fiddled with a few too many things) so when I was beginning to have issues on my desktop when compiling ECCE -- issues that weren't present on any other systems that were freshly installed -- I decided to start over again and install debian anew. I went straight for Debian Jessie, although I had many reasons to stay with Wheezy, such as systemd and, more importantly, the fact that Sun GridEngine is completely missing in Debian Jessie! See here.

Luckily the debian wheezy package works quite well on jessie -- but that's pure luck. I'm currently on the fence between hoping that SGE continues to work well until the SID version trickles down to backports (IF it does),  or whether to learn how to set up SLURM instead.

Either way, here are a few things that annoyed me in Jessie (more specifically they annoyed me in GNOME) and that had to be fixed:

* Turn off update notifications
I hate being bugged by notifications about updating/upgrading my system. I'm not running windows -- it's unbecoming of a linux desktop to behave like that.

To fix it, go to Settings, Personal/Notifications and untick Package Updater

(simple -- you just need to know it's there)

* Nautilus doesn't do extra pane anymore. Bye bye nautilus.
Instead, install nemo, which is the rigthful heir to nautilus. It pulls in a lot of dependencies, but it's worth it.

To make nemo default do
me@beryllium:$ xdg-mime query default inode/directory
org.gnome.Nautilus.desktop
me@beryllium:$ xdg-mime default nemo.desktop inode/directory application/x-gnome-saved-search
me@beryllium:$ xdg-mime query default inode/directory
nemo.desktop
* Gnome-terminal doesn't do transparency anymore. Bye bye gnome-terminal.
Instead, install lx-terminal. To set it as default in both the OS and gnome, in the terminal do
gsettings set org.cinnamon.desktop.default-applications.terminal exec lxterminal
sudo update-alternatives --config x-terminal-emulator
* Adding firefox and thunderbird to default applications
I also felt compelled to install firefox and add it to the list over available applications in gnome so I could set it as the default browser Edit /var/lib/dpkg/alternatives/x-www-browser so that it reads
 auto
/usr/bin/x-www-browser
x-www-browser.1.gz
/usr/share/man/man1/x-www-browser.1.gz

/usr/bin/chromium
40

/usr/bin/firefox-bin
70

/usr/bin/iceweasel
70
/usr/share/man/man1/iceweasel.1.gz
I also followed this post: http://verahill.blogspot.com.au/2013/11/530-briefly-adding-new-entry-to-default.html
For thunderbird, I only followed the latter post.

* To disable screen saver completely:
gsettings set org.gnome.settings-daemon.plugins.power active false 
* I've got no idea how to elegantly exorcise bijiben-shell-search-provider, which keeps making my CPU usage spike. It's not nice.

* Note that dragging windows no longer works with Alt+left click -- instead use the windows key (Super key). Why this old standard behaviour got nuked I don't understand.

* I installed ksnapshot and set it as the default for prtn scrn instead of the crippled gnome-screenshot. Yes, it pulls in a lot of dependencies, but it's worth it.

Looking at the list above I'm slowly realising that it's probably time to say goodby to gnome for good. It's not going to a place where I want to follow it.

Pity.

There are a few things that I like about gnome 3. Well, there's a single thing that I like that got introduced: quickly searching for programs in the Activities Overview. Turns out that the applications menu wasn't that necessary after all.

The removal of features from nautilus, screenshot, terminal etc annoys me a lot though. Same goes for the removal of the minimize button.

Finally, I only find gnome useable once I've installed the gnome extensions by frippery. Stock gnome is useless to me.