424. NWChem 6.3 on Debian Wheezy

Update 23 May 2013: The execution times are pretty much the same as for 6.1.1 with a new patch. I've updated the instructions below to incorporate this new patch (http://www.nwchem-sw.org/images/Iswtch.patch.gz)

Update 21 May 2013: The execution times can be improved considerably by setting

ARMCI_NETWORK=SOCKETS

They are still ca 30% longer than 6.1.1 though due to slower SCF convergence. See http://www.nwchem-sw.org/index.php/Special:AWCforum/st/id834/Nwchem_6.3_running_2-5_times_slo....html

Update 20 May 2013: I did a bit of basic benchmarking. NWChem 6.3 is incredibly slow (ca 190s vs 40s for the 8 core, 3.6 GHz benchmark in http://verahill.blogspot.com.au/2013/05/414-frequency-vs-cores-crude.html). It's parallellising properly from what I can see (i.e. it is not running 8 serial jobs). I've repeated the calc with an unpatched version of nwchem 6.3, and it is just as slow.
 I'll post updates here if I figure this one out.

Original post:
NWChem 6.3 is just out. Here's how to build it for CPU computations.

To build on CentOS 5.6, see http://verahill.blogspot.com.au/2013/05/421-nwchem-63-on-rocks-543centos-56.html

---

Math library:
Use either openblas (for intel or AMD) or ACML (for AMD).

• To set up Openblas, see here:
 http://verahill.blogspot.com.au/2013/05/423-openblas-on-debian-wheezy.html

• To set up ACML, see here:
 http://verahill.blogspot.com.au/2013/05/422-set-up-acml-on-linux.html

---

My GabEdit/Python NWChem patch
This is NOT the patch alluded to in the 23 May update and is optional. It enables python support, and makes the output more verbose so that gabedit can be used as an alternative to ECCE. Hence, it is required if, but only if, you want to enable python and to be able to use GabEdit to open output files.

First create a patch file, e.g. diff.patch.

diff -rupN src.original/config/makefile.h src/config/makefile.h
--- src.original/config/makefile.h 2013-04-15 12:41:45.016853322 +1000
+++ src/config/makefile.h 2013-04-15 12:38:44.933319544 +1000
@@ -2039,7 +2039,7 @@ endif
 
      ifeq ($(BUILDING_PYTHON),python)
 #   EXTRA_LIBS += -ltk -ltcl -L/usr/X11R6/lib -lX11 -ldl
-     EXTRA_LIBS +=    -lnwcutil  -lpthread -lutil -ldl
+     EXTRA_LIBS +=    -lnwcutil  -lpthread -lutil -ldl -lssl -lz
   LDOPTIONS = -Wl,--export-dynamic 
      endif
 ifeq ($(NWCHEM_TARGET),CATAMOUNT)
diff -rupN src.original/ddscf/movecs_pr_anal.F src/ddscf/movecs_pr_anal.F
--- src.original/ddscf/movecs_pr_anal.F 2013-04-15 12:41:45.036852381 +1000
+++ src/ddscf/movecs_pr_anal.F 2013-04-15 12:23:28.100409225 +1000
@@ -195,7 +195,7 @@ c
  22         format(1x,2('  Bfn.  Coefficient  Atom+Function  ',5x))
             write(LuOut,23)
  23         format(1x,2(' ----- ------------  ---------------',5x))
-            do klo = 0, min(n-1,9), 2
+            do klo = 0, min(n-1,199), 2
                khi = min(klo+1,n-1)
                write(LuOut,2) (
      $              int_mb(k_list+k)+1, 
diff -rupN src.original/ddscf/rohf.F src/ddscf/rohf.F
--- src.original/ddscf/rohf.F 2013-04-15 12:41:45.036852381 +1000
+++ src/ddscf/rohf.F 2013-04-15 12:23:28.100409225 +1000
@@ -153,7 +153,7 @@ c
             ilo = 1
             ihi = nmo
          endif
-         call movecs_print_anal(basis, ilo, ihi, 0.15d0, g_movecs, 
+         call movecs_print_anal(basis, ilo, ihi, 0.01d0, g_movecs, 
      $        'ROHF Final Molecular Orbital Analysis', 
      $        .true., dbl_mb(k_eval), oadapt, int_mb(k_irs),
      $        .true., dbl_mb(k_occ))
diff -rupN src.original/ddscf/scf_vec_guess.F src/ddscf/scf_vec_guess.F
--- src.original/ddscf/scf_vec_guess.F 2013-04-15 12:41:45.036852381 +1000
+++ src/ddscf/scf_vec_guess.F 2013-04-15 12:23:28.100409225 +1000
@@ -511,19 +511,19 @@ c
          nprint = min(nclosed+nopen+30,nmo)
          if (scftype.eq.'RHF' .or. scftype.eq.'ROHF') then
             call movecs_print_anal(basis, 1,
-     &           nprint, 0.15d0, g_movecs, 
+     &           nprint, 0.01d0, g_movecs, 
      &           'ROHF Initial Molecular Orbital Analysis', 
      &           .true., dbl_mb(k_eval), oadapt, int_mb(k_irs),
      &           .true., dbl_mb(k_occ))
          else
             nprint = min(nalpha+20,nmo)
             call movecs_print_anal(basis, max(1,nbeta-20),
-     &           nprint, 0.15d0, g_movecs, 
+     &           nprint, 0.01d0, g_movecs, 
      &           'UHF Initial Alpha Molecular Orbital Analysis', 
      &           .true., dbl_mb(k_eval), oadapt, int_mb(k_irs),
      &           .true., dbl_mb(k_occ))
             call movecs_print_anal(basis, max(1,nbeta-20),
-     &           nprint, 0.15d0, g_movecs(2), 
+     &           nprint, 0.01d0, g_movecs(2), 
      &           'UHF Initial Beta Molecular Orbital Analysis', 
      &           .true., dbl_mb(k_eval+nbf), oadapt, int_mb(k_irs+nmo),
      &           .true., dbl_mb(k_occ+nbf))
diff -rupN src.original/ddscf/uhf.F src/ddscf/uhf.F
--- src.original/ddscf/uhf.F 2013-04-15 12:41:45.036852381 +1000
+++ src/ddscf/uhf.F 2013-04-15 12:23:28.096409414 +1000
@@ -144,11 +144,11 @@ C
          enddo
          ihi = max(ihi-1,1)
  9611    continue
-         call movecs_print_anal(basis, ilo, ihi, 0.15d0, g_movecs, 
+         call movecs_print_anal(basis, ilo, ihi, 0.01d0, g_movecs, 
      $        'UHF Final Alpha Molecular Orbital Analysis', 
      $        .true., dbl_mb(k_eval), oadapt, int_mb(k_irs),
      $        .true., dbl_mb(k_occ))
-         call movecs_print_anal(basis, ilo, ihi, 0.15d0, g_movecs(2), 
+         call movecs_print_anal(basis, ilo, ihi, 0.01d0, g_movecs(2), 
      $        'UHF Final Beta Molecular Orbital Analysis', 
      $        .true., dbl_mb(k_eval+nbf), oadapt, int_mb(k_irs+nmo),
      $        .true., dbl_mb(k_occ+nbf))
diff -rupN src.original/mcscf/mcscf.F src/mcscf/mcscf.F
--- src.original/mcscf/mcscf.F 2013-04-15 12:41:45.000854073 +1000
+++ src/mcscf/mcscf.F 2013-04-15 12:23:23.748613695 +1000
@@ -719,7 +719,7 @@ c
       if (util_print('final vectors analysis', print_default))
      $     call movecs_print_anal(basis, 
      $     max(1,nclosed-10), min(nbf,nclosed+nact+10),
-     $     0.15d0, g_movecs, 'Analysis of MCSCF natural orbitals',
+     $     0.01d0, g_movecs, 'Analysis of MCSCF natural orbitals',
      $     .true., dbl_mb(k_evals), .true., int_mb(k_sym), 
      $     .true., dbl_mb(k_occ))
 c     
diff -rupN src.original/nwdft/scf_dft/dft_mxspin_ovlp.F src/nwdft/scf_dft/dft_mxspin_ovlp.F
--- src.original/nwdft/scf_dft/dft_mxspin_ovlp.F 2013-04-15 12:41:45.604825677 +1000
+++ src/nwdft/scf_dft/dft_mxspin_ovlp.F 2013-04-15 12:23:28.228403211 +1000
@@ -184,14 +184,14 @@ c
       call ga_sync()
 c
       call movecs_print_anal(basis,int_mb(k_non),int_mb(k_non)
-     & ,0.15d0,g_alpha,'Alpha Orbitals without Beta Partners',
+     & ,0.01d0,g_alpha,'Alpha Orbitals without Beta Partners',
      &   .false., 0.0 ,.false., 0 , .false., 0 )
 c
       if (nct.GE.2) then
       do i = 2,nct
       ind = int_mb(k_non+i-1)
       call movecs_print_anal(basis,ind,ind
-     & ,0.15d0,g_alpha,' ',
+     & ,0.01d0,g_alpha,' ',
      &   .false., 0.0 ,.false., 0 , .false., 0 )
       enddo
       endif
@@ -350,7 +350,7 @@ c      endif
 c      endif
 c 9990 format(/,18x,'THERE ARE',i3,1x,'UN-PARTNERED ALPHA ORBITALS')
 c
-       call movecs_print_anal(basis, 1, nalp, 0.15d0, g_ualpha,
+       call movecs_print_anal(basis, 1, nalp, 0.01d0, g_ualpha,
      & 'Alpha Orb. w/o Beta Partners (after maxim. alpha/beta overlap)',
      &   .false., 0.0 ,.false., 0 , .false., 0 )
 c
diff -rupN src.original/nwdft/scf_dft/dft_scf.F src/nwdft/scf_dft/dft_scf.F
--- src.original/nwdft/scf_dft/dft_scf.F 2013-04-15 12:41:45.608825490 +1000
+++ src/nwdft/scf_dft/dft_scf.F 2013-04-15 12:23:28.228403211 +1000
@@ -1774,7 +1774,7 @@ c
             else
                blob='DFT Final Beta Molecular Orbital Analysis' 
             endif
-            call movecs_print_anal(ao_bas_han, ilo, ihi, 0.15d0, 
+            call movecs_print_anal(ao_bas_han, ilo, ihi, 0.01d0, 
      &           g_movecs(ispin), 
      &           blob, 
      &           .true., dbl_mb(k_eval(ispin)), oadapt, 
diff -rupN src.original/nwdft/scf_dft_cg/dft_cg_solve.F src/nwdft/scf_dft_cg/dft_cg_solve.F
--- src.original/nwdft/scf_dft_cg/dft_cg_solve.F 2013-04-15 12:41:45.612825303 +1000
+++ src/nwdft/scf_dft_cg/dft_cg_solve.F 2013-04-15 12:23:28.220403588 +1000
@@ -183,7 +183,7 @@ c
             blob = 'DFT Final Beta Molecular Orbital Analysis'
           endif
           call movecs_fix_phase(g_movecs(ispin))
-          call movecs_print_anal(basis, ilo, ihi, 0.15d0,
+          call movecs_print_anal(basis, ilo, ihi, 0.01d0,
      &         g_movecs(ispin),blob,
      &         .true., dbl_mb(k_eval+(ispin-1)*nbf),
      &         oadapt, int_mb(k_irs+(ispin-1)*nbf),

---

Compile NWChem
This examples uses the ACML libs. See e.g. this post for openblas settings.

sudo apt-get install build-essential gfortran python2.7-dev libopenmpi-dev openmpi-bin
sudo mkdir /opt/nwchem
sudo chown $USER:$USER /opt/nwchem
cd /opt/nwchem/
wget http://www.nwchem-sw.org/download.php?f=Nwchem-6.3-src.2013-05-17.tar.gz
mv download.php\?f\=Nwchem-6.3-src.2013-05-17.tar.gz Nwchem-6.3-src.2013-05-17.tar.gz
tar xvf Nwchem-6.3-src.2013-05-17.tar.gz
cd nwchem-6.3-src.2013-05-17/
patch -p0 < diff.patch

patching file src/config/makefile.h
patching file src/ddscf/movecs_pr_anal.F
patching file src/ddscf/rohf.F
patching file src/ddscf/scf_vec_guess.F
patching file src/ddscf/uhf.F
patching file src/mcscf/mcscf.F
patching file src/nwdft/scf_dft/dft_mxspin_ovlp.F
patching file src/nwdft/scf_dft/dft_scf.F
patching file src/nwdft/scf_dft_cg/dft_cg_solve.F

cd src/
wget http://www.nwchem-sw.org/images/Iswtch.patch.gz
gzip -d Iswtch.patch
patch -p0 < Iswtch.patch
cd ../
export LARGE_FILES=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export PYTHONVERSION=2.7
export PYTHONHOME=/usr
export BLASOPT="-L/opt/acml/acml5.3.1/gfortran64_int64/lib -lacml"

export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/lib/openmpi/lib
export MPI_INCLUDE=/usr/lib/openmpi/include
export LIBRARY_PATH="$LIBRARY_PATH:/usr/lib/openmpi/lib:/opt/acml/acml5.3.1/gfortran64_int64/lib"
export LIBMPI="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"

export ARMCI_NETWORK=SOCKETS

cd $NWCHEM_TOP/src
make clean
make nwchem_config
make FC=gfortran 1> make.log 2>make.err
cd $NWCHEM_TOP/contrib
export FC=gfortran
./getmem.nwchem

---

Settings
Create /opt/nwchem/default.nwchemrc

nwchem_basis_library /opt/nwchem/nwchem-6.3-src.2013-05-17/src/basis/libraries/
ffield amber
amber_1 /opt/nwchem/nwchem-6.3-src.2013-05-17/src/data/amber_s/
amber_2 /opt/nwchem/nwchem-6.3-src.2013-05-17/src/data/amber_x/
amber_3 /opt/nwchem/nwchem-6.3-src.2013-05-17/src/data/amber_q/
amber_4 /opt/nwchem/nwchem-6.3-src.2013-05-17/src/data/amber_u/
amber_5 /opt/nwchem/nwchem-6.3-src.2013-05-17/src/data/custom/
spce /opt/nwchem/nwchem-6.3-src.2013-05-17/src/data/solvents/spce.rst
charmm_s /opt/nwchem/nwchem-6.3-src.2013-05-17/src/data/charmm_s/
charmm_x /opt/nwchem/nwchem-6.3-src.2013-05-17/src/data/charmm_x/

Symmlink to this file in each user's home:

ln -s /opt/nwchem/default.nwchemrc ~/.nwchemrc

423. Openblas on Debian Wheezy

Update 23 May 2013: the url for the openblas tarball has been updated and a tab has been removed from the lapack url.

Note that the links to the openblas file tends to die after a while, so you might have to download it manually.

Original post:
I need more modular posts. I've covered how to build openblas several times before, but I need a simple post I can use as a reference. So here it is:

openblas

sudo mkdir /opt/openblas
sudo chown ${USER} /opt/openblas
cd ~/tmp
wget http://github.com/xianyi/OpenBLAS/tarball/v0.2.6

tar xvf v0.2.6
cd xianyi-OpenBLAS-87b4d0c

wget http://www.netlib.org/lapack/lapack-3.4.1.tgz
make all BINARY=64 CC=/usr/bin/gcc FC=/usr/bin/gfortran USE_THREAD=0 INTERFACE64=1 1> make.log 2>make.err
make PREFIX=/opt/openblas install

nm: '/lib/libpthread.so*': No such file
make -j 2 -f Makefile.install install
make[1]: Entering directory `/home/me/tmp/xianyi-OpenBLAS-87b4d0c'
Generating openblas_config.h in /opt/openblas/include
Generating f77blas.h in /opt/openblas/include
Generating cblas.h in /opt/openblas/include
Copying LAPACKE header files to /opt/openblas/lib
Copy the static library to /opt/openblas/lib
cp: cannot stat `libopenblas_opteronp-r0.2.6.a': No such file or directory
make[1]: *** [install] Error 1
make[1]: Leaving directory `/home/me/tmp/xianyi-OpenBLAS-87b4d0c'
make: *** [install] Error 2

Don't worry about the errors -- as long as you have libopenblas_PROCESSOR_rX.X.X.so, you're fine.

cp lib*.* /opt/openblas/lib

add

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openblas/lib

to your ~/.bashrc, and/or add

/opt/openblas/lib

to your /etc/ld.so.conf

422. Set up ACML on linux

These are the same instructions as in post 409B. However, I've decided it's better to do the posts the unix/linux way -- have the do one thing, and do that thing well. It makes life easier for me if I can simply refer back to more modular posts.

Anyway, here's how to set up the ACML libs on debian.

ACML
Download both the 'regular' and the int64 gfortran packages from AMD:
http://developer.amd.com/tools-and-sdks/cpu-development/amd-core-math-library-acml/acml-downloads-resources/#download

tar xvf acml-5-3-1-gfortran-64bit-int64.tgz
tar xvf acml-5-3-1-gfortran-64bit.tgz
sh install-acml-5-3-1-gfortran-64bit-int64.sh

Where do you want to install ACML?  Press return to use
the default location (/opt/acml5.3.1), or enter an alternative path.
The directory will be created if it does not already exist.
> /opt/acml/acml5.3.1

sh install-acml-5-3-1-gfortran-64bit.sh

Where do you want to install ACML?  Press return to use
the default location (/opt/acml5.3.1), or enter an alternative path.
The directory will be created if it does not already exist.
> /opt/acml/acml5.3.1

You'll get something like this:

/opt/acml/acml5.3.1
|-- Doc
|-- gfortran64
|-- gfortran64_fma4
|-- gfortran64_fma4_int64
|-- gfortran64_fma4_mp
|-- gfortran64_fma4_mp_int64
|-- gfortran64_int64
|-- gfortran64_mp
|-- gfortran64_mp_int64
`-- util

where
*  fma4 is for cpus with FMA4 support (use util/cpuid to check)
*  int64 is for double-precision float (integer*8) I think
*  mp is for openmp. For MPI do not use the _mp_ libraries!

Pick your library/ies and add them to the LD_LIBRARY_PATH, e.g.:

echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/acml/acml5.3.1/gfortran64_int64/lib' >> ~/.bashrc
source ~/.bashrc