21 May 2012

157. Restarting gaussian (g09) job on an SGE system (qsub)

The Australian University I'm working at has a computational cluster where jobs are submitted using qsub. This post is more like a personal note to myself, but the point about resubmitting jobs may be of use to someone.

This page is useful reading for how these types of scripts with mixed shell and gaussian stuff work: http://www.gaussian.com/g_tech/g_ur/m_running.htm

0. The qsub header (Notes To Myself)

http://cf.ccmr.cornell.edu/cgi-bin/w3mman2html.cgi?qsub(1B)
#$ -S /bin/sh Shell. Can be csh, tcsh etc..
#$ -cwd execute in Current Working Directory
#$ -l h_rt=12:00:00 Maximum allowed run time in hours. -l is a list with resource limits.
#$ -l h_vmem=4G Memory limit. http://www.biostat.jhsph.edu/bit/cluster-usage.html#MemSpec. Should match %mem 4000mb

#$ -j y"-j join. Declares if the standard error stream of the job will be merged with the standard output stream of the job." It creates *.o* and *.p* files with what would've been echoed in the terminal.

#$ -pe g03_smpX XSeems to stand for parallel execution. X would be the number of slots it seems. The g03_smpX seems to be the message passing interface, but not entirely sure. 

1. Setting up simple jobs
First create a standard template, let's call it qsub.header

#!/bin/sh
#$ -S /bin/sh
#$ -cwd
#$ -l h_rt=12:00:00
#$ -l h_vmem=4G
#$ -j y
#$ -pe g03_smp2 2
module load gaussian/g09
time G09 << END > g09_output.log
and save it in your home folder ~.

Create an input file, e.g. water.in and put it in your work directory, e.g. ~/g09


%chk=water
#P ub3lyp/6-31G* opt

water energy

0  1
O
H  1  1.0
H  1  1.0  2  120.0

Put them together:
cat ~/qsub.header > water.qsub
cat ~/g09/water.in >>water.qsub

#!/bin/sh
#$ -S /bin/sh
#$ -cwd
#$ -l h_rt=12:00:00
#$ -l h_vmem=4G
#$ -j y
#$ -pe g03_smp2 2
module load gaussian/g09
time G09 << END > g09_output.log

%chk=water.chk
%nprocshared=6
#P ub3lyp/6-31G* opt

water energy

0  1
O
H  1  1.0
H  1  1.0  2  120.0

2. Restarting jobs

Most likely your home folder is shared across the nodes via nfs.

To find out, submit
#!/bin/sh
#$ -S /bin/sh
#$ -cwd
#$ -l h_rt=12:00:00
#$ -l h_vmem=4G
#$ -j y
#$ -pe g03_smp2 2
pwd
ls -lah
tree -L 1 -d
to get some directory information about the nodes.

Once you have that, just put the absolute path to your .chk file in your restart script, e.g.

#!/bin/sh
#$ -S /bin/sh
#$ -cwd
#$ -l h_rt=12:00:00
#$ -l h_vmem=4G
#$ -j y
#$ -pe g03_smp2 2
module load gaussian/g09
time G09 << END > g09_output.log

%chk=/nfs/home/hpcsci/username/g09/water.chk
%nprocshared=2
#P ub3lyp/6-31G* opt guess=read geom=allcheck



18 May 2012

156. Compiling nwchem 6.1 with internal libs, openmpi, python on debian testing/wheezy

I followed this thread: http://www.nwchem-sw.org/index.php/Special:AWCforum/st/id435/segmentation_fault--running_smal....html

See here for build with external blas: http://verahill.blogspot.com.au/2012/05/building-nwchem-61-on-debian.html

--Start here --
sudo apt-get install libopenmpi-dev python-dev

sudo mkdir /opt/nwchem
sudo chown ${USER} /opt/nwchem
cd /opt/nwchem
wget http://www.nwchem-sw.org/images/Nwchem-6.1-2012-Feb-10.tar.gz
tar xvf Nwchem-6.1-2012-Feb-10.tar.gz
mv nwchem-6.1 nwchem-6.1_internal
cd nwchem-6.1_internal/

Edit src/config/makefile.h and add -lz -lssl to the end of line 1914 (needed by python)

Edit src/tools/GNUmakefile and add two lines as shown below in red (line numbers added by me for illustration only) -- editing needed for compile with internal libs.
355 ifneq ($(BLAS_LIB),)
356     ifeq ($(BLAS_SIZE),4)
357         MAYBE_BLAS = --with-blas4="$(strip $(BLAS_LIB))"
358     else
359         MAYBE_BLAS = --with-blas8="$(strip $(BLAS_LIB))"
360     endif
361 endif
to
355 ifneq ($(BLAS_LIB),)
356     ifeq ($(BLAS_SIZE),4)
357         MAYBE_BLAS = --with-blas4="$(strip $(BLAS_LIB))"
358     else
359         MAYBE_BLAS = --with-blas8="$(strip $(BLAS_LIB))"
360     endif
361 else
362     MAYBE_BLAS = --without-blas
363 endif
Then continue as normal:


export LARGE_FILES=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export PYTHONVERSION=2.7
export PYTHONHOME=/usr
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/lib/openmpi/lib
export MPI_INCLUDE=/usr/lib/openmpi/include
export LIBRARY_PATH=$LIBRARY_PATH:/usr/lib/openmpi/lib
export LIBMPI="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"
cd $NWCHEM_TOP/src
make clean
make nwchem_config
make FC=gfortran

**If you get errors relating to openssl etc. at the end of compile, try building without python support by removing anything in blue. You won't need to do make clean so the rebuild will be quick. 

Building takes ages

Edit your ~/.bashrc and add
export NWCHEM_EXECUTABLE=/opt/nwchem/nwchem-6.1_internal/bin/LINUX64/nwchem
export NWCHEM_BASIS_LIBRARY=/opt/nwchem/nwchem-6.1_internal/src/basis/libraries/
export PATH=$PATH:/opt/nwchem/nwchem-6.1_internal/bin/LINUX64


Links to this post:
http://chemport.ru/forum/viewtopic.php?f=71&t=98589

17 May 2012

155. Gromacs with external fftw3 and blas on debian testing

This is based on http://verahill.blogspot.com.au/2012/03/building-gromacs-with-fftw3-and-openmpi.html

Make sure your build environment is set up:
sudo apt-get install build-essential gfortran libopenmpi-dev


fftw
sudo mkdir /opt/fftw/
sudo chown ${USER} /opt/fftw
mkdir ~/tmp
cd ~/tmp
wget ftp://ftp.fftw.org/pub/fftw/fftw-3.3.2.tar.gz
tar xvf fftw-3.3.2.tar.gz
cd fftw-3.3.2/

./configure --enable-float --enable-mpi --enable-threads --with-pic --prefix=/opt/fftw/fftw-3.3.2/single
make && make install
make clean
./configure --disable-float --enable-mpi --enable-threads --with-pic --prefix=/opt/fftw/fftw-3.3.2/double

make && make install


openblas
sudo mkdir /opt/openblas
sudo chown ${USER} /opt/openblas
cd ~/tmp
wget http://nodeload.github.com/xianyi/OpenBLAS/tarball/v0.1.1
tar xvf v0.1.1
cd xianyi-OpenBLAS-e6e87a2/
wget http://www.netlib.org/lapack/lapack-3.4.1.tgz
make all BINARY=64 CC=/usr/bin/gcc FC=/usr/bin/gfortran USE_THREAD=0 INTERFACE64=1 1> make.log 2>make.err

make PREFIX=/opt/openblas install
cp lib*.*  /opt/openblas/lib

add
export LD_LIBRARY_PATH:$LD_LIBRARY_PATH:/opt/openblas/lib
to your ~/.bashrc

[for later use with nwchem and ecce, add /opt/openblas/lib to /etc/ld.so.conf and do sudo ldconfig]



gromacs


sudo mkdir /opt/gromacs
sudo chown ${USER} /opt/gromacs
cd ~/tmp
wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-4.5.5.tar.gz
tar xvf gromacs-4.5.5.tar.gz
cd gromacs-4.5.5/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/openmpi/lib:/opt/openblas/lib

single
export LDFLAGS="-L/opt/fftw/fftw-3.3.2/single/lib -L/opt/openblas/lib -lopenblas"
export CPPFLAGS="-I/opt/fftw/fftw-3.3.2/single/include -I/opt/openblas/include"

./configure --disable-mpi --enable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_sp --prefix=/opt/gromacs/gromacs-4.5.5
make -j3
make install

double
make distclean
export LDFLAGS="-L/opt/fftw/fftw-3.3.2/double/lib -L/opt/openblas/lib -lopenblas
export CPPFLAGS="-I/opt/fftw/fftw-3.3.2/double/include -I/opt/openblas/include"

./configure --disable-mpi --disable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_dp --prefix=/opt/gromacs/gromacs-4.5.5
make -j3
make install

single + mpi
make distclean
export LDFLAGS="-L/opt/fftw/fftw-3.3.2/single/lib -L/opt/openblas/lib -lopenblas"
export CPPFLAGS="-I/opt/fftw/fftw-3.3.2/single/include -I/opt/openblas/include"

./configure --enable-mpi --enable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_spmpi --prefix=/opt/gromacs/gromacs-4.5.5
make -j3
make install


double + mpi
make distclean
export LDFLAGS="-L/opt/fftw/fftw-3.3.2/double/lib -L/opt/openblas/lib -lopenblas"
export CPPFLAGS="-I/opt/fftw/fftw-3.3.2/double/include -I/opt/openblas/include"

./configure --enable-mpi --disable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_dpmpi --prefix=/opt/gromacs/gromacs-4.5.5
make -j3
make install



Make sure to put this in your ~/.bashrc
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/openmpi/lib:/opt/openblas/lib
export PATH=$PATH:/opt/gromacs/gromacs-4.5.5/bin
 You now have four versions of each binary -- with and without mpi, with single and with double precision.