146. Nwchem with openblas

Openblas seems to be the successor of Gotoblas. http://xianyi.github.com/OpenBLAS/
I willingly admit that I am no expert on this blas library stuff, so I may well be doing something obviously wrong.

1. Compiling and installing openblas
sudo mkdir /opt/openblas
sudo chown ${USER} /opt/openblas
mkdir ~/tmp
cd ~/tmp

download from here http://github.com/xianyi/OpenBLAS/tarball/v0.1.1

tar xvf xianyi-OpenBLAS-v0.1.1-0-g5b7f443.tar.gz

 make all BINARY=64 CC=/usr/bin/gcc FC=/usr/bin/gfortran USE_THREAD=0 INTERFACE64=1 1> make.log 2>make.err

make PREFIX=/opt/openblas install
cp lib*.*  /opt/openblas/lib

cd /opt/openblas/lib
rm libopenblas.so.0
ln -s libopenblas.so libopenblas.so.0

do ls /opt/openblas/lib and note what cpu specific file you have such as libopenblas_barcelona-r0.1.1.so or libopenblas_nehalem-r0.1.1.a

edit /etc/ld.so.conf and add

/opt/openblas/lib

2. Compiling nwchem

sudo apt-get install build-essential gfortran python2.7-dev
mkdir /opt/nwchem/

sudo chown ${USER} /opt/nwchem

cd /opt/nwchem

wget http://www.nwchem-sw.org/images/Nwchem-6.0.tar.gz
tar -xvf Nwchem-6.0.tar.gz
cd nwchem-6.0/

For python support, edit src/config/makefile.h and add -lz -lssl to line 1962 (see here: http://verahill.blogspot.com.au/2012/04/adding-python-support-to-nwchem-under.html)

Next, continue
export LARGE_FILES=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export PYTHONHOME=/usr
export PYTHONVERSION=2.7
export USE_MPI=y
export USE_MPIF=y
export MPI_LOC=/usr/lib/openmpi/lib
export MPI_INCLUDE=/usr/lib/openmpi/include
export BLASOPT="-L/opt/openblas/lib -lopenblas -lopenblas_barcelona-r0.1.1"
export LIBRARY_PATH=$LIBRARY_PATH:/usr/lib/openmpi/lib
export LIBMPI="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"
cd $NWCHEM_TOP/src
make clean
make nwchem_config
make FC=gfortran

Now edit your ~/.bashrc and add

export PATH=$PATH:/opt/nwchem/nwchem-6.0/bin/LINUX64
export LD_LIBRARY_PATH=/opt/openblas/lib:$LD_LIBRARY_PATH 

and you should be done.

You might find that ecce still won't properly execute nwchem jobs (you don't get an error -- it just doesn't work) -- edit your /etc/ld.so.conf and add a line saying

/opt/openblas/lib

Then do sudo ldconfig and it will work

Time to look into benchmarking...

---

So far I've got the following benchmarks (in seconds) for nwchem when optimising benzene with 6-311++g** using mpirun -n 3.

without -lpthread
_____Internal blas_____openblas_____ATLAS_____acml_____
Ta     249.8                    277.7                 335.9           ------
Be     408.2                    421.7                 568.4        
B      546.4                     579.9                 760.2        

Noticing a mention regarding -lpthread online I compiled with that as part of the mpilibs, and lo and behold:

with -lpthread

_____Internal blas_____openblas_____ATLAS_____acml_____
Ta      201.7                    198.9                233.9           ------
Be      371.1                    314.4                390.3        
B        496.6                    416.8                 521.9

* All run using mpirun -n 3. Be=phenom II X6. B= Athlon II X3. Ta= i7-1600 X4. Intel MKL cost money, acml are free.

This doesn't mean that one library is superior to the other -- it just means that under the conditions I employed and in the presence of any mistake I may have made, these are my observations. Your mileage will vary.