647. Linda/G16 and ecce + slurm

I had the opportunity to test G16 with Linda on my cluster. Suprisingly, I observed a decent reduction in computational time for geometry optimisation, and a fair reduction for vibrational analysis.

Computational time -- geometry optimisation
1x6 cores: 100%
2x6 cores: 56% (2 nodes)
3x6 cores: 44% (3 nodes)

Computational time -- normal mode computation
1x6 cores: 100%
2x6 cores: 63% (2 nodes)
3x6 cores: 57% (3 nodes)

In no case was the speed up good enough that it actually makes long-term sense to use Linda (need less than 50% for 2 nodes and 33% or less for 3 nodes), but if you're itching for quick results, it might be worth it.

Either way, as part of this I wrote a CONFIG.Linda for ECCE and SLURM:

Gaussian-16: /opt/gaussian/g16a/g16/g16
perlPath: /usr/bin/
qmgrPath: /usr/bin/
xappsPath: /usr/bin/

Slurm {
#!/bin/csh
#SBATCH -p linda
#SBATCH --time=$walltime
#SBATCH --output=slurm.out
#SBATCH --job-name=$submitFile
#SBATCH --nodes=$nodes
#SBATCH --cpus-per-task=$ppn
}

Gaussian-16FilesToRemove{ core }

Gaussian-16Command{
set path = ( /opt/nbo6/bin $path ) 

set nnodes = `scontrol show job $SLURM_JOBID|grep NodeList|sed 's/=/\t/'|gawk '{print $2}'|tail -n 1`
sed -i "2s/^/%LindaWorkers=$nnodes\n/" g16.g16in
sed -i '3s/^/%UseSSH\n/' g16.g16in
sed -i "s/nprocshared.*/nprocshared=$ppn./" g16.g16in

setenv GAUSS_SCRDIR /home/me/scratch
setenv GAUSS_EXEDIR /opt/gaussian/g16a/g16
setenv GAUSS_ARCHDIR /opt/gaussian/g16a/g16/arch
setenv GAUSS_BSDDIR /opt/gaussian/g16a/g16/bsd
setenv GAUSS_LEXEDIR /opt/gaussian/g16a/g16/linda-exe
/opt/gaussian/g16a/g16/g16< $infile > $outfile 
}

The slurm partition linda has all the nodes that has g16.

646. NWChem 6.6 on debian stretch (9)

I've got plenty of posts on how to compile nwchem on this blog. However, as I'm still running debian jessie on my work computers and cluster I have failed to appreciate quite how different things work in debian stretch.

This includes the compilation of ECCE (although owing to updates meant to rectify that at https://github.com/FriendsofECCE/ECCE this isn't necessarily something you'd notice) and NWChem.

Anyway, here's how to build nwchem with openmpi support under debian stretch:

---

1. Download source code and patches

sudo mkdir /opt/nwchem -p
sudo chown $USER:$USER /opt/nwchem
cd /opt/nwchem
mkdir patches6.6
cd patches6.6
for i in Tddft_mxvec20.patch.gz Tools_lib64.patch.gz Config_libs66.patch.gz Cosmo_meminit.patch.gz Sym_abelian.patch.gz Xccvs98.patch.gz Dplot_tolrho.patch.gz Driver_smalleig.patch.gz Ga_argv.patch.gz Raman_displ.patch.gz Ga_defs.patch.gz Zgesvd.patch.gz Cosmo_dftprint.patch.gz Txs_gcc6.patch.gz Gcc6_optfix.patch.gz Util_gnumakefile.patch.gz Util_getppn.patch.gz Gcc6_macs_optfix.patch.gz Xatom_vdw.patch.gz Notdir_fc.patch.gz Hfmke.patch.gz Cdfti2nw66.patch.gz ; do wget http://www.nwchem-sw.org/download.php?f=$i ; done
gunzip *.gz
cat *.patch > consolidated.patch
cd ..
wget http://www.nwchem-sw.org/download.php?f=Nwchem-6.6.revision27746-src.2015-10-20.tar.gz -O Nwchem-6.6.revision27746-src.2015-10-20.tar.gz
tar xvf Nwchem-6.6.revision27746-src.2015-10-20.tar.gz

Apply patches:

cd /opt/nwchem/nwchem-6.6/
cp /opt/nwchem/patches6.6/consolidated.patch .
patch -p0 < consolidated.patch

In addition, you can do the following to print more information so that GabEdit properly shows MOs:

cd nwchem-6.6/
grep -rl "do klo = 0, min(n-1,9), 2"|xargs -I {} sed -i 's/do klo = 0, min(n-1,9), 2/do klo = 0, min(n-1,199), 2/g' {}
grep -rl " 0.15d0,"|grep -v "atomdata"|xargs -I {} sed -i 's/0.15d0/0.01d0/g' {}

---

2. Install dependencies and compile.
 How exactly it is done depends on your compiler and math libs. I've used red for examples with INTEL mkl, blue for AMD acml and orange for the free openblas. ifort is shown in purple and gfortran like this.You should only use one compiler + one math lib.
Hint: also make sure your ld.conf.so.d files contain paths to your math libs.

cd /opt/nwchem/nwchem-6.6
sudo apt-get install build-essential gfortran python2.7-dev libopenmpi-dev openmpi-bin

export NWCHEM_TOP=`pwd`
export LARGE_FILES=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all"
export PYTHONVERSION=2.7
export PYTHONHOME=/usr

export BLASOPT="-L/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64/ -lmkl_core -lmkl_sequential -lmkl_intel_ilp64"
export LIBRARY_PATH="$LIBRARY_PATH:/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64/"

export BLASOPT="-L/opt/acml/acml5.3.1/gfortran64_int64/lib -lacml"
export LIBRARY_PATH="$LIBRARY_PATH:/opt/acml/acml5.3.1/gfortran64_int64/lib"

export BLASOPT="-L/opt/openblas/lib -lopenblas"
export LIBRARY_PATH="$LIBRARY_PATH:/opt/openblas/lib"


export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export ARMCI_NETWORK=SOCKETS

cd $NWCHEM_TOP/src

make clean
make nwchem_config
make FC=ifort 1> make.log 2>make.err
make FC=gfortran 1> make.log 2>make.err

cd $NWCHEM_TOP/contrib
export FC=ifort
export FC=gfortran
./getmem.nwchem

---

3. Run
mpirun -n 8 /opt/nwchem-6.6/bin/LINUX64/nwchem input.nw > output.nw