Showing posts with label centos 5.6. Show all posts
Showing posts with label centos 5.6. Show all posts

21 March 2017

635. Installing R on Rocks 5.4.3

Rocks 5.4.3 is based on CentOS 5.6 which is practically ancient by now (released Jan 2011).

Either way, when dealing with someone else's cluster its better to not fiddle too much with what is already working.

Here's a not at all elegant way of install R on Rocks 5.4.3

sudo yum install R-3.3.2-3.el5.x86_64.rpm libRmath-devel-3.3.2-3.el5.x86_64.rpm libRmath-3.3.2-3.el5.x86_64.rpm R-devel-3.3.2-3.el5.x86_64.rpm R-core-3.3.2-3.el5.x86_64.rpm R-core-devel-3.3.2-3.el5.x86_64.rpm libssh2-0.18-10.el5.x86_64.rpm xdg-utils-1.0.2-4.el5.noarch.rpm texinfo-tex-4.8-14.el5.x86_64.rpm xz-devel-4.999.9-0.3.beta.20091007git.el5.x86_64.rpm texinfo-4.8-14.el5.x86_64.rpm 
[..] Total size: 169 M Downloading Packages: Running rpm_check_debug Running Transaction Test Finished Transaction Test Transaction Test Succeeded Running Transaction Installing : libssh2 1/11 Installing : libRmath 2/11 Installing : texinfo 3/11 Installing : texinfo-tex 4/11 Installing : libRmath-devel 5/11 Installing : xz-devel 6/11 Installing : xdg-utils 7/11 Installing : R-core 8/11 Installing : R-core-devel 9/11 Installing : R-devel 10/11 Installing : R 11/11 Installed: R.x86_64 0:3.3.2-3.el5 R-core.x86_64 0:3.3.2-3.el5 R-core-devel.x86_64 0:3.3.2-3.el5 R-devel.x86_64 0:3.3.2-3.el5 libRmath.x86_64 0:3.3.2-3.el5 libRmath-devel.x86_64 0:3.3.2-3.el5 libssh2.x86_64 0:0.18-10.el5 texinfo.x86_64 0:4.8-14.el5 texinfo-tex.x86_64 0:4.8-14.el5 xdg-utils.noarch 0:1.0.2-4.el5 xz-devel.x86_64 0:4.999.9-0.3.beta.20091007git.el5 Complete!

R version 3.3.2 (2016-10-31) -- "Sincere Pumpkin Patch" Copyright (C) 2016 The R Foundation for Statistical Computing Platform: x86_64-redhat-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details. Natural language support but running in an English locale R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications. Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R. > q() Save workspace image? [y/n/c]: n

13 September 2012

235. CPMD with Netlib's lapack, blas and your own fftw3 on ROCKS 5.4.3/CentOS 5.6

Update 8 Feb 2013:
I somehow had forgot to include some of the instructions for the BLAS part. Fixed now.

This is done pretty much like how it's done on Debian (-march=native didn't work in the BLAS compilation though, nor was -fno-whole-file accepted when compiling cpmd)

1. Compile cmake according to this post:

2. Compile BLAS
sudo mkdir /share/apps/tools/netlib/blas/lib -p
sudo chown $USER /share/apps/tools/netlib -R
mkdir ~/tmp
cd ~/tmp
tar xvf blas.tgz
cd BLAS/

OPTS     = -O3 -shared -m64 -fPIC

make all

gfortran -shared -Wl,-soname, -o *.o -lc
ln -s
cp lib*blas* /share/apps/tools/netlib/blas/lib

3. Compile LAPACK
sudo mkdir /share/apps/tools/netlib/lapack -p
sudo chown $USER /share/apps/tools/netlib -R

tar xvf lapack-3.4.1.tgz
cd lapack-3.4.1/
mkdir build
cd build
ccmake ../

Hit 'c' and edit the values:

 BUILD_COMPLEX                    ON
 BUILD_COMPLEX16                  ON
 BUILD_DOUBLE                     ON
 BUILD_SHARED_LIBS                ON
 BUILD_SINGLE                     ON
 BUILD_STATIC_LIBS                ON
 BUILD_TESTING                    ON
 CMAKE_INSTALL_PREFIX             /share/apps/tools/netlib/lapack
 LAPACKE                          OFF
 LAPACKE_WITH_TMG                 OFF
 USE_OPTIMIZED_BLAS               ON
 USE_XBLAS                        OFF

Hit 'c' again, then hit 'g'.

Edit CMakeCache.txt and add the following lines at the beginning:

# EXTERNAL cache entries

ccmake ../
again, hit 'c', then 'g'.

make install

4. Compile FFTW3
sudo mkdir /share/apps/tools/fftw3
sudo chown $USER /share/apps/tools/fftw3
cd ~/tmp
tar -xvf fftw-3.3.1.tar.gz
cd fftw-3.3.1
make distclean
./configure --enable-float --enable-mpi --enable-threads --with-pic --prefix=/share/apps/tools/fftw3/single
make install
make distclean
./configure --disable-float --enable-mpi --enable-threads --with-pic --prefix=/share/apps/tools/fftw3/double
make install

5. Compile CPMD
I downloaded the cpmd file to a client computer, then uploaded it to the ROCKS front node:
sftp me@rocks:/home/me/tmp

Connected to rocks.
Changing to: /home/me/tmp
sftp> put cpmd-v3_15_3.tar.gz
Uploading cpmd-v3_15_3.tar.gz to /home/me/tmp/cpmd-v3_15_3.tar.gz
cpmd-v3_15_3.tar.gz                100% 2937KB 587.4KB/s   00:05
sftp> exit

I then logged in via ssh as normal.
cd ~/tmp
tar xvf cpmd-v3_15_3.tar.gz
Create a new file LINUX-x86_64-ROCKS

     CFLAGS='-c -O2 -Wall'
     CPP='/lib/cpp -P -C -traditional'
     FFLAGS='-c -O2 -fcray-pointer -fsecond-underscore'
LFLAGS='-L/share/apps/tools/fftw3/double/lib -lfftw3-lfftw3_mpi -lfftw3_threads -I/usr/include -L/share/apps/tools/netlib/blas/lib -lnetblas -L/share/apps/tools/netlib/lapack/lib -llapack -L/opt/openmpi/lib -lpthread -lmpi'
      FC='mpif77 -fbounds-check'
      LD='mpif77 -fbounds-check'

NOTE: I don't think the -I belongs in the LFLAGS statement, but I'm presuming that I put it there for a reason back when I did it the first time.

Go to ~/tmp/CPMD, and edit wfnio.F (basically replace 3 with 2 and remove 'L'):

 15       CHARACTER(len=*) TAG
 63         IF(TAG(1:2).EQ.'NI') THEN
201       IF(TAG(1:2).NE.'NI') THEN
271         IF(TAG(1:2).EQ.'NI') THEN

Finally, edit Makefile and change
  23 LD = f95 -O
  23 LD = mpif77 -fbounds-check

Time to compile

./ LINUX-x86_64-ROCKS > Makefile
sudo mkdir /share/apps/cpmd
sudo chown $USER /share/apps/cpmd
cp cpmd.x /share/apps/cpmd

echo 'export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/tools/netlib/blas/lib:/share/apps/tools/netlib/lapack/lib:/share/apps/tools/fftw3/double/lib' >>~/.bashrc
echo 'export PATH=$PATH:/share/apps/cpmd' >> ~/.bashrc
echo "export PP_LIBRARY_PATH=/share/apps/cpmd/PP_LIBRARY" >>~/.bashrc

You're now done compiling. To test, you need to get some pseudopotential files -- look at e.g. the end of for instructions.

10 September 2012

230. ROCKS 5.4.3, ATLAS and Gromacs on Xeon X3480

After doing another round of 'benchmarks' (there are so many factors that differ between the systems that it's difficult to tell exactly what I'm measuring) I'm back to looking at my BLAS/LAPACK.

So here's compiling ATLAS on a cluster made up of six dual-socket mobos with 2x quadcore XeonX3480 CPUs and 8 Gb RAM. The cluster is running ROCKS 5.4.3, which is a spin based on Centos 5.6. We then compile GROMACS using ATLAS and compare it with Openblas. Please note that I am not an expert on optimisations (or computers or anything) so comparing Openblas vs ATLAS won't tell you which one is 'better'. They are just numbers based on what someone once observed on a particular system under a particular set of circumstances.

Hurdles: I first had to deal with the lapack + bad symbols + recompile with -fPIC problem (solved by using netlib lapack and building shared libraries), then encountered the ' undefined reference to _gfortran_' issue (solved by adding -lgfortran to LDFLAGS).

sudo mkdir /share/apps/ATLAS
sudo chown $USER /share/apps/ATLAS
cd ~/tmp
tar xvf atlas3.9.72.tar.bz2
mkdir build
cd build
.././configure --prefix=/share/apps/ATLAS -Fa alg '-fPIC' --with-netlib-lapack-tarfile=$HOME/tmp/lapack-3.4.1.tgz --shared
OS configured as Linux (1)
Assembly configured as GAS_x8664 (2)
Vector ISA Extension configured as  SSE3 (6,448)
Architecture configured as  Corei1 (25)
Clock rate configured as 3059Mhz
DONE  STAGE 5-1-0 at 15:23
ATLAS install complete.  Examine
ATLAS/bin/<arch>/INSTALL_LOG/SUMMARY.LOG for details.

ls lib/
libatlas.a  libcblas.a  libf77blas.a  libf77refblas.a  liblapack.a  libptcblas.a  libptf77blas.a  libptlapack.a  libtstatlas.a  Makefile
make install

In addition to successful copying you'll also get errors along the lines of

cp: cannot stat `/home/me/tmp/ATLAS/build/lib/libsatlas.dylib': No such file or directory
make[1]: [install_lib] Error 1 (ignored)
cp /home/me/tmp/ATLAS/build/lib/libtatlas.dylib /share/apps/ATLAS/lib/.
cp: cannot stat `/home/me/tmp/ATLAS/build/lib/libtatlas.dylib': No such file or directory
make[1]: [install_lib] Error 1 (ignored)
cp /home/me/tmp/ATLAS/build/lib/libsatlas.dll /share/apps/ATLAS/lib/.
cp: cannot stat `/home/me/tmp/ATLAS/build/lib/libsatlas.dll': No such file or directory
make[1]: [install_lib] Error 1 (ignored)
cp /home/me/tmp/ATLAS/build/lib/libtatlas.dll /share/apps/ATLAS/lib/.
cp: cannot stat `/home/me/tmp/ATLAS/build/lib/libtatlas.dll': No such file or directory
make[1]: [install_lib] Error 1 (ignored)
cp /home/me/tmp/ATLAS/build/lib/ /share/apps/ATLAS/lib/.
cp /home/me/tmp/ATLAS/build/lib/ /share/apps/ATLAS/lib/.
because those files don't exist. 


FFTW3 was first build according to this. The only difference is the install targets (--prefix) -- I put things in /share/apps/gromacs/.fftwsingle and /share/apps/gromacs/.fftwdouble. Gromacs was downloaded and extracted as shown in that post, and /share/apps/gromacs was created.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi/lib:/share/apps/ATLAS/lib
#single precision
export LDFLAGS="-L/share/apps/gromacs/.fftwsingle/lib -L/share/apps/ATLAS/lib -latlas -llapack -lf77blas -lcblas -lgfortran"
export CPPFLAGS="-I/share/apps/gromacs/.fftwsingle/include -I/share/apps/ATLAS/include/atlas"
./configure --disable-mpi --enable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_spa --prefix=/share/apps/gromacs
make -j3
make install
#double precision
make distclean
export LDFLAGS="-L/share/apps/gromacs/.fftwdouble/lib -L/share/apps/ATLAS/lib -latlas -llapack -lf77blas -lcblas -lgfortran"
export CPPFLAGS="-I/share/apps/gromacs/.fftwdouble/include -I/share/apps/ATLAS/include/atlas"
./configure --disable-mpi --disable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_dpa --prefix=/share/apps/gromacs
make -j3
make install
#single + mpi
make distclean
export LDFLAGS="-L/share/apps/gromacs/.fftwsingle/lib -L/share/apps/ATLAS/lib -latlas -llapack -lf77blas -lcblas -lgfortran"
export CPPFLAGS="-I/share/apps/gromacs/.fftwsingle/include -I/share/apps/ATLAS/include/atlas""
./configure --enable-mpi --enable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_spampi --prefix=/share/apps/gromacs
make -j3
make install
#double + mpi
make distclean
export LDFLAGS="-L/share/apps/gromacs/.fftwdouble/lib -L/share/apps/ATLAS/lib -latlas -llapack -lf77blas -lcblas -lgfortran"
export CPPFLAGS="-I/share/apps/gromacs/.fftwdouble/include -I/share/apps/ATLAS/include/atlas"
./configure --enable-mpi --disable-float --with-fft=fftw3 --with-external-blas --with-external-lapack --program-suffix=_dpampi --prefix=/share/apps/gromacs
make -j3
make install
The -lgfortran is IMPORTANT, or you'll end up with 'undefined reference to _gfortran_' type errors.

I ran a 6x6x6 nm box of water for 5 million steps (10 ns) to get a rough idea of the performance.
Make sure to put
in your ~/.bashrc, and to include it in your SGE jobs files (if that's what you use).

I allocated 8 Gb RAM and 8 cores for each run.

Double precision:
Openblas: 10.560 ns/day (11.7 GFLOPS, runtime 8182 seconds)
ATLAS  : 10.544 ns/day (11.6 GFLOPS, runtime 8194 seconds)

Single precision:
Openblas: 17.297 ns/dat (19.1 GFLOPS, runtime 4995 seconds)
ATLAS:   17.351 ns/day (19.2 GFLOPS, runtime 4980 seconds)
That's 15 seconds difference on a 1h 20 min run. I'd say they are identical.

25 July 2012

216. Quantum Espresso on ROCKS 5.4.3 / CentOs 5.6 -- Almost got it.


I've seen just about any error message imaginable when compiling Quantum Espresso. You can most likely get around them by following this guide ad verbatim 
(no parallel environment detected, can't build fortran programs, no linker, parallel vs serial compiler etc.)

This compile is nothing like the easy-breezy one on debian, because of one reason: CentOS 5.6 is old. It's so old that the default gcc version can't compile QE. You'd encounter similar problems with an increasing number of software packages, including ABINIT.

      CentOS 5.6: gcc version 4.1.2 20080704 (Red Hat 4.1.2-50)
      Wheezy:       gcc version 4.7.1 (Debian 4.7.1-2)

The crusty compiler collection means we have to hammer out errors like these during compilation:

In file dfile_star.f90:28
    INTEGER,ALLOCATABLE ::  npert (:), irgq (:)
Error: Attribute at (1) is not allowed in a TYPE definition
 In file dfile_star.f90:37

    REAL(DP),ALLOCATABLE :: gi (:,:), gimq (:), eigen(:)
Error: Attribute at (1) is not allowed in a TYPE definition
 In file dfile_star.f90:42

    COMPLEX(DP),ALLOCATABLE :: u(:,:), t(:,:,:,:), tmq (:,:,:)
Error: Attribute at (1) is not allowed in a TYPE definition

Hence the requirement to first compile GCC 4.7, even if it takes a couple of hours. 

To the students out there swearing (and sweating) over their professors' insistence on using ROCKS/CentOS: compiling a compiler is not a wasted skill and just might be useful in building cross compilation environments for the time when you decide that science can take a hike and there's more money in working on setting up embedded systems. At least it'll buy you a little geek cred. 

Also, it seems that you need to use math and mpi libs compiled with the same compiler as you use to compile quantum espresso. So...lots of steps today as well!

Also, I haven't reported the configure bug for openmpi 1.6. Feel free to reproduce and report it.

I'm not happy that Quantum Espresso ignores the compile flags I've given it and defaults to things it should not default. I've had to jump through way too many hoops to be happy about this. OpenMPI not supporting program-suffix annoys me as well.

This is another massive compile...I'd rate it as my most annoying, troublesome and infuriating one yet.

1. Build GCC (gfortran, g++, gcc) 4.7 according to this post:
Make sure that
export PATH=$PATH:/share/apps/tools/gcc/gcc47/bin

2. Create target directory structure
sudo mkdir /share/apps/libs
sudo chown $USER /share/apps/lib

3. Recompile GNU OpenMP (libgomp)
It's probably easier you start with a freshly extracted gcc-4.7.1
cd ~/tmp/gcc
mkdir newgcc
cp gcc-4.7.1.tar.gz newgcc/
cd newgcc
tar xvf gcc-4.7.1
cd gcc-4.7.1/libgomp
mkdir build
cd build/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/tools/gcc/mpc/lib:/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpfr/lib
.././configure CC=gcc-gcc-4.7 CXX=g++-gcc-4.7 F77=gfortran-gcc-4.7 FC=gfortran-gcc-4.7 --prefix=/share/apps/tools/gcc/gcc47

md5sum /share/apps/tools/gcc/gcc47/lib/
23db68121aacf1e7d896704474e26350  /share/apps/tools/gcc/gcc47/lib/

make install

md5sum /share/apps/tools/gcc/gcc47/lib/
791a1ceb0ea4009e22703b82e8ce1ff4  /share/apps/tools/gcc/gcc47/lib/

4.  Compile OpenMPI using your new GCC 
cd ~/tmp/
tar xvf openmpi-1.6.tar.gz
cd openmpi-1.6
mkdir build
cd build/
export LD_LIBRARY_PATH=/share/apps/tools/gcc/gcc47/lib64:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1:/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/tools/gcc/mpfr/lib

.././configure --prefix=/share/apps/libs/openmpi47 CC=gcc-gcc-4.7 FC=gfortran-gcc-4.7 F77=gfortran-gcc-4.7 F90=gfortran-gcc-4.7 CPP=cpp-gcc-4.7 CXX=g++-gcc-4.7 --with-sge LDFLAGS="-L/share/apps/tools/gcc/gcc47/lib64 -L/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/share/apps/tools/gcc/gmp/lib -L/share/apps/tools/gcc/mpc/lib -L/share/apps/tools/gcc/mpfr/lib -L/share/apps/tools/gcc/gcc47/lib" CPPFLAGS="-I/share/apps/tools/gcc/mpc/include -I/share/apps/tools/gcc/gmp/include -I/share/apps/tools/gcc/mpfr/include -I/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/include"

** There's a bug in the openmpi configure script leading to config failure due to to an extra space being introduced in front of -L when testing the F77 compiler. Exporting the LD_LIBRARY_PATH is done to get around that. **

make install

cd /share/apps/libs/openmpi47bin/
ls mpi* |xargs -I {} ln -s {} {}-gcc-4.7
echo 'export PATH=$PATH:/share/apps/libs/openmpi47/bin' >> ~/.bashrc
source ~/.bashrc

cd /share/apps/libs/openmpi47share/openmpi/
ln -s mpif77-vt-wrapper-data.txt mpif77-gcc-4.7-vt-wrapper-data.txt
ln -s mpif90-vt-wrapper-data.txt mpif90-gcc-4.7-vt-wrapper-data.txt
ln -s mpicc-vt-wrapper-data.txt mpicc-gcc-4.7-vt-wrapper-data.txt
ln -s mpic++-vt-wrapper-data.txt mpic++-gcc-4.7vt--wrapper-data.txt

ln -s mpicxx-vt-wrapper-data.txt mpicxx-gcc-4.7-vt-wrapper-data.txt
ln -s mpif77-wrapper-data.txt mpif77-gcc-4.7-wrapper-data.txt
ln -s mpif90-wrapper-data.txt mpif90-gcc-4.7-wrapper-data.txt
ln -s mpicc-wrapper-data.txt mpicc-gcc-4.7-wrapper-data.txt
ln -s mpic++-wrapper-data.txt mpic++-gcc-4.7-wrapper-data.txt
ln -s mpicxx-wrapper-data.txt mpicxx-gcc-4.7-wrapper-data.txt

5. Compile FFT using your new GCC
cd ~/tmp
tar xvf fftw-3.3.2.tar.gz

cd fftw-3.3.2/
./configure --enable-float --enable-mpi --enable-threads --with-pic --prefix=/share/apps/libs/fftw-3.3.2-gcc47/single CC=gcc-gcc-4.7 FC=gfortran-gcc-4.7 CPP=cpp-gcc-4.7 MPIF90=mpif90-gcc-4.7 F77=gfortran-gcc-4.7 MPICC=mpicc-gcc-4.7 MPIF77=mpif77-gcc-4.7

make install
make check

         FFTW threaded transforms passed basic tests!
Executing "mpirun -np 1 ..
make[3]: *** [check-local] Error 1

That mpirun won't work isn't that surprising since it's still point to /opt/openmpi/bin instead of our mpirun-gcc-4.7

make distclean
./configure --disable-float --enable-mpi --enable-threads --with-pic --prefix=/share/apps/libs/fftw-3.3.2-gcc47/double CC=gcc-gcc-4.7 FC=gfortran-gcc-4.7 CPP=cpp-gcc-4.7 MPIF90=mpif90-gcc-4.7 F77=gfortran-gcc-4.7 MPICC=mpicc-gcc-4.7 MPIF77=mpif77-gcc-4.7

make install
make check
         FFTW threaded transforms passed basic tests!
...followed by the same error as above.
It's fine.

6. Optional: Compile openblas using your new GCC
cd /share/apps/tools/gcc/binutils/bin
ln -s ar-gcc-4.7 gcc-ar

cd ~/tmp
mv v0.1.1 openblas.tar.gz
tar xvf openblas.tar.gz
cd xianyi-OpenBLAS-e6e87a2/

export LD_LIBRARY_PATH=/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/tools/gcc/mpfr/lib

make all BINARY=64 CC=gcc-gcc-4.7 FC=gfortran-gcc-4.7 MPICC=mpicc-gcc-4.7 USE_THREAD=0 INTERFACE64=1 

make PREFIX=/share/apps/libs/openblas47 install

cp libopenblas_* /share/apps/libs/openblas47/lib
cd /share/apps/libs/openblas47/lib
ln -s
ln -s
ln -s
ln -s libopenblas_nehalem-r0.1.1.a libopenblas.a

7. Compile Quantum Espresso using your new GCC
sudo mkdir /share/apps/QE
sudo chown $USER /share/apps/QE
mkdir /share/apps/QE/lapack-3.2

mkdir ~/tmp/QE
cd ~/tmp/QE


Edit environmental_variables
PARA_PREFIX="mpirun-gcc-4.7 -n 8"

Time to configure and compile:
cd ~/tmp/QE
tar xvf espresso-5.0.tar.gz
cd espresso-5.0/
cp lapack-3.2 -R /share/apps/QE/
cp BLAS -R /share/apps/QE/

Create a script called

make clean
export PATH=""
export PATH="/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/ecce/apps/scripts/share/apps/pvm/pvm3/bin/LINUX64:/share/apps/libs/openmpi47/bin:/share/apps/nwchem/nwchem-6.1/bin/LINUX64:/share/apps/sinfo/bin:/share/apps/sinfo/sbin:/share/apps/gromacs/bin:/share/apps/cmake/bin:/share/apps/tools/babel/bin:/share/apps/tools/htop/bin:/share/apps/tools/xmgrace/grace/bin:/share/apps/tools/rasmol/src:/share/apps/tools/strace/bin:/share/apps/tools/gcc/gcc47/bin:/share/apps/tools/gcc/binutils/bin"
which mpif90
read -p "press any key"
export CC=gcc-gcc-4.7
export FC=gfortran-gcc-4.7
export MPIF90=mpif90-gcc-4.7
export MPIF77=mpif77-gcc-4.7
export F77=gfortran-gcc-4.7
export F90=gfortran-gcc-4.7
export CPP=cpp-gcc-4.7
export CXX=g++-gcc-4.7
export CFLAGS=""
export CPPFLAGS="-I/share/apps/tools/gcc/mpc/include -I/share/apps/tools/gcc/gmp/include -I/share/apps/tools/gcc/mpfr/include -I/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/include -I/share/apps/libs/openmpi47/include -I//share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/include"
export LDFLAGS="-L/share/apps/tools/gcc/gcc47/lib -L/share/apps/tools/gcc/gcc47/lib64 -L/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1 -L/share/apps/tools/gcc/gmp/lib -L/share/apps/tools/gcc/mpc/lib -L/share/apps/tools/gcc/mpfr/lib -L/share/apps/libs/openmpi47/lib -L/lib64 -lc"
#export BLAS_LIBS="-L/share/apps/libs/openblas47/lib -lopenblas" 
export BLAS_LIBS="/share/apps/QE/BLAS/blas.a"
export LAPACK_LIBS="/share/apps/QE/lapack-3.2/lapack.a"
export FFT_LIBS="-L/share/apps/libs/fftw-3.3.2-gcc47/double/lib -lfftw3 -lfftw3_mpi -lfftw3_threads"
export MPI_LIBS="-L/share/apps/libs/openmpi47/lib -l:/share/apps/libs/openmpi47/lib/"
export LD_LIBRARY_PATH=/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpfr/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/libs/openmpi47/lib
./configure --prefix=/share/apps/QE/bin|tee conf.log
read -p "Press key to continue"
cd PW
make |tee make.log
cd ../
make all |tee makeall.log

Then run it using


Which gives:

ESPRESSO can take advantage of several optimized numerical libraries
(essl, fftw, mkl...).  This configure script attempts to find them,
but may fail if they have been installed in non-standard locations.
If a required library is not found, the local copy will be compiled.

The following libraries have been found:
  FFT_LIBS=-L/share/apps/libs/fftw-3.3.2-gcc47/double/lib -lfftw3 -lfftw3_mpi -lfftw3_threads
  MPI_LIBS=-L/share/apps/libs/openmpi47/lib -l:/share/apps/libs/openmpi47/lib/
Please check if this is what you expect.

If any libraries are missing, you may specify a list of directories
to search and retry, as follows:
  ./configure LIBDIRS="list of directories, separated by spaces"

Parallel environment detected successfully.\
Configured for compilation of parallel executables.

For more info, read the ESPRESSO User's Guide (Doc/users-guide.tex).
configure: success

and more...Once it's over do

cd ~/tmp/QE/espresso-5.0
cp * -R /share/apps/QE/
echo 'export PATH=$PATH:/share/apps/QE/bin' >> ~/.bashrc
source ~/.bashrc

8. Testing PW -- STUCK
export LD_LIBRARY_PATH=/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpfr/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/libs/openmpi47/lib:/share/apps/libs/openblas47/lib:/share/apps/tools/gcc/gcc47/lib64
cd /share/apps/QE/PW/examples/example02

/share/apps/QE/PW/examples/example02 : starting

This example shows how to use pw.x to compute the equilibrium geometry
of a simple molecule, CO, and of an Al (001) slab.
In the latter case the relaxation is performed in two ways:
1) using the quasi-Newton BFGS algorithm
2) using a damped dynamics algorithm.

  executables directory: /share/apps/QE/bin
  pseudo directory:      /share/apps/QE/pseudo
  temporary directory:   /tmp/QE
  checking that needed directories and files exist...
Downloading O.pz-rrkjus.UPF to /share/apps/QE/pseudo...
Downloading C.pz-rrkjus.UPF to /share/apps/QE/pseudo... done

  running pw.x as: mpirun-gcc-4.7 -n 8  /share/apps/QE/bin/pw.x  

  cleaning /tmp/QE... done
  running the geometry relaxation for CO... from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
 from test_input_xml: Empty input file .. stopping
mpirun-gcc-4.7 noticed that the job aborted, but has no info as to the process
that caused that situation.
Error condition encountered during test: exit status = 2

It's more insidious than that though -- pw.x crashes with "STOP 2" for any type of input.

NOTE: Like on Debian I had segfaults happening when using my own openblas (which works fine with gromacs and nwchem). Hence why I show you how to compile it, yet never use it.

24 July 2012

215. Compiling gcc 4.7.1/gfortran 4.7.1 on Centos 5.6/ROCKS 4.5.3 (and gmp, mpfr, mpc, binutils (ld,as), glibc, libunistring, libtools...)

Update June 2013:
See flakrat's post for a working example:

I haven't updated this post for a long time, and I haven't been using the GCC I compiled this way. See flakrat's post for an up-to-date working version.

Update(s): This is the first time I compile the GCC and because of this I will like be going back, updating this document over the coming days. Make sure to 1) check back in a week and 2) hit refresh.

Updated (given in Melb./Au time zone): 24/7, 25/7

The reason for this post is the outdated version (4.1) of gcc on our ROCKS 5.4.3/CentOS 5.6 cluster.

I looked at installing GCC 4.4.6 using rpm packages I found online, but I'm not used to the Red Hat way of doing things, and e.g. openmpi-devel was requiring a version of libgomp I wanted to update. Ultimately I got scared of messing up a production cluster and decided that compiling, while slower, is a whole lot safer -- especially if you're not comfortable with the local package manager.

So, here's the alternative route of compiling your own.

I'm really not a friend of CentOS or ROCKS. On the other hand, I freely admit that this is probably in large part due to not being used to it. But...during the course of this compilation my feelings have gone from mild annoyance to active, fiery hate. Mostly it has to do with how old everything is, and the difficult in updating anything in ROCKS.

This is probably my most massive compilation, owing to the number of packages I ended up compiling (a lot of these are probably optional). Guile in particular is very, very demanding.

The order in which things are done is not random

1. Look at

2. Download and untar all the sources
NOTE: you should, if possible, select mirrors based on where you are. However, sometimes you're stuck in a shell and just need to get those files downloaded.

mkdir ~/tmp/gcc
cd ~/tmp/gcc

tar xvf gcc-4.7.1.tar.gz
tar xvf gmp-5.0.5.tar.bz2
tar xvf mpfr-3.1.1.tar.gz
tar xvf mpc-1.0.tar.gz

That was the easy bit.

I've also set up a directory called /share/apps/tools/gcc/ with proper permissions already (i.e. whoever is doing the compiling should have write access)

3. Build gmp
cd gmp-5.0.5/
mkdir build
cd build/
.././configure --prefix=/share/apps/tools/gcc/gmp --program-suffix=-gcc47
make install
make check

4. Build mpfr
cd ../../mpfr-3.1.1/
mkdir build
cd build/
.././configure --prefix=/share/apps/tools/gcc/mpfr --program-suffix=-gcc-4.7 --with-gmp=/share/apps/tools/gcc/gmp
make install

Libraries have been installed in:

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the `LD_RUN_PATH' environment variable
     during linking
   - use the `-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to `/etc/'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and manual pages.

make check

It looks like your compile is starting from scratch, but then it ends with:

All 160 tests passed
(1 test was not run)

5. Build mpc
cd ../../mpc-1.0/
mkdir build
cd build/
.././configure --prefix=/share/apps/tools/gcc/mpc --program-suffix=-gcc-4.7 --with-gmp=/share/apps/tools/gcc/gmp --with-mpfr=/share/apps/tools/gcc/mpfr
make install
make check

All 64 tests passed

6. Binutils
'Often' (judging from forum postings...) you can use an older version of the binutils (ld, as) with a newer version of GCC. However, I need crt1.o when compilicing, it's part of glibc, and glibc requires newer versions of ld and as. So, here we go.
tar xvf binutils-2.22.tar.gz
cd binutils-2.22/
mkdir build
cd build/
.././configure --prefix=/share/apps/tools/gcc/binutils --program-suffix=-gcc-4.7
make install
make check

At some point I did:
cd /share/apps/tools/gcc/binutils/bin
ln -s ld-gcc-4.7 ld
ln -s as-gcc-4.7 as
ln -s ar-gcc-4.7 ar
but I don't think it's essential

7. Build gcc
cd ~/tmp/gcc-4.7.1
mkdir build
cd build/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/tools/gcc/gmp/lib:/share/apps/tools/gcc/mpc/lib:/share/apps/tools/gcc/mpfr/lib
.././configure --prefix=/share/apps/tools/gcc/gcc47 --program-suffix=-gcc-4.7 --with-gmp=/share/apps/tools/gcc/gmp --with-mpfr=/share/apps/tools/gcc/mpfr --with-mpc=/share/apps/tools/gcc/mpc LD_FOR_TARGET=ld-gcc-4.7 AS_FOR_TARGET=as-gcc-4.7 --with-ld=/share/apps/tools/gcc/binutils/bin/ld-gcc-4.7 --with-as=/share/apps/tools/gcc/binutils/bin/as-gcc-4.7 --with-ar=/share/apps/tools/gcc/binutils/bin/ar-gcc-4.7 AR_FOR_TARGET=ar-gcc-4.7


This takes A LONG TIME.

make install

To do 'make check' you need to have done the compilations of the optional packages (autogen and dependencies) below.
make check

You now have a shiny new compiler.

Using it is another matter. I'm working on that post at the moment...
You may find the following informative:

gcc-gcc-4.7 --print-search-dirs

install: /share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/

programs: =/share/apps/tools/gcc/gcc47/libexec/gcc/x86_64-unknown-linux-gnu/4.7.1/:/share/apps/tools/gcc/gcc47/libexec/gcc/x86_64-unknown-linux-gnu/4.7.1/:/share/apps/tools/gcc/gcc47/libexec/gcc/x86_64-unknown-linux-gnu/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu/4.7.1/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../x86_64-unknown-linux-gnu/bin/

libraries: =/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../x86_64-unknown-linux-gnu/lib/x86_64-unknown-linux-gnu/4.7.1/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../x86_64-unknown-linux-gnu/lib/../lib64/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../x86_64-unknown-linux-gnu/4.7.1/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../lib64/:/lib/x86_64-unknown-linux-gnu/4.7.1/:/lib/../lib64/:/usr/lib/x86_64-unknown-linux-gnu/4.7.1/:/usr/lib/../lib64/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../../x86_64-unknown-linux-gnu/lib/:/share/apps/tools/gcc/gcc47/lib/gcc/x86_64-unknown-linux-gnu/4.7.1/../../../:/lib/:/usr/lib/

8. glibc
cd ~/tmp/gcc
tar xvf blic-2.14.tar.gz
cd glibc-2.14/
mkdir build
cd build/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/share/apps/tools/gcc/mpc/lib:/share/apps/tools/gcc/mpfr/lib:/share/apps/tools/gcc/gmp/lib
.././configure --prefix=/share/apps/tools/gcc/glibc CC=gcc-gcc-4.7 --with-headers=/usr/src/kernels/2.6.18-238.19.1.el5-x86_64/include
make install

Using the current glibc 2.16 requires kernel>2.6.19 which is why I used 2.14. I don't want to fiddle with installing a new kernel on a production system which is shared between several users (and time zones...):
configure: error: GNU libc requires kernel header files from
Linux 2.6.19 or later to be installed before configuring.
The kernel header files are found usually in /usr/include/asm and
/usr/include/linux; make sure these directories use files from
Linux 2.6.19 or later.  This check uses <linux/version.h>, so
make sure that file was built correctly when installing the kernel header
files.  To use kernel headers not from /usr/include/linux, use the
configure option --with-headers.

Optional -- libtool, libunistring, libffi, bdw-gc, autogen and guile

I'm still stuck on Guile.

If you want to run make check on your gcc build, you need autogen, which needs guile, which needs libtool and libunistring

cd ~/tmp/gcc
tar xvf libtool-2.4.2.tar.gz
cd libtool-2.4.2/
mkdir build
cd build/
.././configure --prefix=/share/apps/tools/gcc/libtool --program-suffix=-2.4.2
make install

make check
(this is very slow)
All 122 tests passed
(2 tests were not run)
## ------------- ##
## Test results. ##
## ------------- ##

104 tests behaved as expected.
22 tests were skipped.

cd ~/tmp/gcc
tar xvf libunistring-0.9.3.tar.gz 
cd libunistring-0.9.3/
mkdir build
cd build/
.././configure --prefix=/share/apps/tools/gcc/libunistring
make install

make check
All 418 tests passed

cd ~/tmp/gcc
tar xvf libffi-3.0.11.tar.gz 
cd libffi-3.0.11/
mkdir build
cd build/
.././configure --prefix=/share/apps/tools/gcc/libffi
make install
cp include -R /share/apps/tools/gcc/libffi/
cp ../src/x86/ffitarget.h /share/apps/tools/gcc/libffi/include/

cd ~/tmp/gcc
tar xvf gc.tar.gz
cd gc-7.2/
mkdir build
cd build/
 .././configure --prefix=/share/apps/tools/gcc/bdw-gc

make check
All 10 tests passed

tar xvf guile-2.0.6.tar.gz
cd guile-2.0.6
mkdir build
cd build/
.././configure --prefix=/share/apps/tools/gcc/guile --with-libltdl-prefix=/share/apps/tools/gcc/libtool --with-libgmp-prefix=/share/apps/tools/gcc/gmp --with-libunistring-prefix=/share/apps/tools/gcc/libunistring LIBFFI_CFLAGS=-I/share/apps/tools/gcc/libffi/include LIBFFI_LIBS=-L/share/apps/tools/gcc/libffi/lib BDW_GC_CFLAGS=-I/share/apps/tools/gcc/bdw-gc/include BDW_GC_LIBS=-L/share/apps/tools/gcc/bdw-gc/lib


../.././libguile/finalizers.c:167: error: static declaration of 'GC_set_finalizer_notifier' follows non-static declaration
/share/apps/tools/gcc/bdw-gc/include/gc/gc.h:177: error: previous declaration of 'GC_set_finalizer_notifier' was here
make[3]: *** [libguile_2.0_la-finalizers.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory `/home/me/tmp/gcc/guile-2.0.6/build/libguile'
make: *** [all] Error 2

tar xvf autogen-5.16.tar.gz
cd autogen-5.16/
mkdir build
cd build/

checking for x86_64-unknown-linux-gnu-gcc... /home/me/tmp/gcc/gcc-4.7.1/build/./gcc/xgcc -B/home/me/tmp/gcc/gcc-4.7.1/build/./gcc/ -B/share/apps/tools/gcc/gcc47/x86_64-unknown-linux-gnu/bin/ -B/share/apps/tools/gcc/gcc47/x86_64-unknown-linux-gnu/lib/ -isystem /share/apps/tools/gcc/gcc47/x86_64-unknown-linux-gnu/include -isystem /share/apps/tools/gcc/gcc47/x86_64-unknown-linux-gnu/sys-include
checking for suffix of object files... configure: error: in `/home/me/tmp/gcc/gcc-4.7.1/build/x86_64-unknown-linux-gnu/libgcc':
configure: error: cannot compute suffix of object files: cannot compile
See `config.log' for more details.
make[2]: *** [configure-stage1-target-libgcc] Error 1
make[2]: Leaving directory `/home/me/tmp/gcc/gcc-4.7.1/build'
make[1]: *** [stage1-bubble] Error 2
make[1]: Leaving directory `/home/me/tmp/gcc/gcc-4.7.1/build'
make: *** [all] Error 2

Solution: export LD_LIBRARY_PATH to include the mpc, mpfr and gmp lib dirs (see above)

Links to this post:

03 June 2012

171. Building ECCE on ROCKS/CentOS

I installed ECCE on a couple of a single workstation with ROCKS, and remotely on a 40 core cluster with ROCKS. The local, workstation install worked fine. I never really bothered much about the cluster install, and only recently looked closer at it. Well, I can launch the 'gateway' but nothing else -- when I click on e.g. the organizer button I get the rocks version of an hourglass that never goes away -- and I don't get any error messages. Turning on logging doesn't yield anything either. 

Ergo, I figured that building it myself  may yield a different result. It didn't on the ROCKS cluster, but everything worked just fine on the single-node ROCKS training box I keep in my office.

CentOS is a bit dated, so you'll need to build your own apr and apr-util. Build apr:
cd /share/apps/utils/
tar xvf apr-1.4.6.tar.gz
cd apr-1.4.6/
./configure --prefix=/share/apps/utils/apr
make install
cd ../
tar xvf apr-util-1.4.1.tar.gz
cd apr-util-1.4.1/
./configure --prefix=/share/apps/utils/apr-util --with-apr=/share/apps/utils/apr/

Time for ecce.
First download 
cd /share/apps/ecce/
tar xvf ecce-v6.3-src.tar.bz2
cd ecce-v6.3/
export ECCE_HOME=/share/apps/ecce/ecce-v6.3
cd build/

Edit build_ecce
889       ./configure --prefix=$ECCE_HOME/${ECCE_SYSDIR}3rdparty/httpd --enable-rewrite --enable-dav --enable-ss-compression
889       ./configure --prefix=$ECCE_HOME/${ECCE_SYSDIR}3rdparty/httpd --enable-rewrite --enable-dav --enable-ss-compression --with-apr=/share/apps/utils/apr/bin/apr-1-config --with-apr-util=/share/apps/utils/apr-util/bin/apu-1-config

Just follow the instructions i.e. hit return, over and over again. Answer no to running tests again. Then run build_ecce again:
Now stuff should be building. Do this another six times. From the README:
"At this stage the script will build one 3rd party package per invocation,
exiting after each package is built.  In order the 3rd party packages that
will be built are:
1. Apache Xerces XML parser
2. Mesa OpenGL
3. wxWidgets C++ GUI toolkit
4. wxPython GUI toolkit
5. Apache HTTP web server"
The httpd build ends with a minor error about "lib" missing. It's fine.

The sixth time ECCE itself is built, and that's the step that takes by far the longest. It finishes with:
 ECCE built and distribution created in /share/apps/ecce/ecce-v6.3
On a single-node desktop I got it to run a seventh time it seemed. The last step finished with the message above though.

Go to your /share/apps/ecce/ecce-v6.3/ dir where you'll find install_ecce.v6.3.csh
Do the install
csh -f install_ecce.v6.3.csh
Follow the instructions.

You may also want to
sudo mv /etc/csh.* ~/
to get rid of the crappy csh config files.

Edit your ~/.bashrc:

alias startecceserver='csh -f /share/apps/ecce/ecce-v6.3/server/ecce-admin/start_ecce_server'
alias stopecceserver='csh -f /share/apps/ecce/ecce-v6.3/server/ecce-admin/stop_ecce_server
export ECCE_HOME=/share/apps/ecce/ecce-v6.3/apps
export PATH=$PATH:${ECCE_HOME}/scripts

and your ~/.cshrc:

setenv ECCE_HOME /share/apps/ecce/ecce-v6.3/apps
set PATH= (/share/apps/nwchem/nwchem-6.1/bin/LINUX64 $PATH)

On my single-node box I had to edit the apps/siteconfig/DataServers and replace with localhost (two instances), as well as the apps/siteconfig/ file (one instance).

In spite of the hassle on the single node box, everything works there -- the builder, organizer etc. all open just fine. The rocks cluster, looks fine, but doesn't work.

The ROCKS Cluster:
Everything seems to work fine -- starting ecce launches the gateway, but clicking on anything sees the centos version of the hourglass churn over and over for all eternity. Nothing happens.

I looked through these two threads, and i also tried the pre-built 32 bit binary. All without luck.

I've also tried editing the site_runtime file:
ECCE_MESA_EXCEPT x86_64:RedHat:Fedora:CentOS
(matches the lsb_release -is output)

02 June 2012

170. tcsh in ROCKS/CentOS with hardcoded csh.cshrc path

WHAT THIS POST DOES: It shows you how to compile your own tcsh which won't be looking at /etc/csh.cshrc. It doesn't show you how to set up the correct .cshrc files. But it certainly allows you to experiment.

Also, keep in mind that since each local node hdd has it's own /bin directory (not exported) you need to make similar changes on each node (i.e. change the /bin/csh symlink -- see below)

(The) csh (startup files) is(are) horribly broken on ROCKS 5.4.3.

For now I've solved it by just moving  /etc/csh.cshrc out of the way, but what we do here is symlink /bin/csh to our own tcsh which been hardcoded to use a non-standard configuration file, so that you can use the standard ROCKS tcsh with /etc/csh.cshrc and your own csh(tcsh) with your own config files.

To be clear: it's not the csh binary which is borked on ROCKS 5.4.3, but the configuration files.

There's a patch for the broken csh -- but when I applied it to a test computer it only got broken-er and prevented the csh from opening and staying open. Good way of getting locked out. So I'm not keen on doing the same thing on someone else's production cluster. Also, I've opted for tcsh since the csh sources come with a bsd style makefile, and I really can't deal with that right now.

What we'll do is hardcode the location of the csh.cshrc file and change it from /etc/csh.cshrc to /share/apps/utils/tcsh

sudo mkdir /share/apps/utils
sudo chown ${USER} /share/apps/utils
cd /share/apps/utils
tar xvf tcsh_6.18.01.orig.tar.gz
 cd tcsh-6.18.01/

Time for find out what to change:
tail -n 9999 *|strings|egrep "/etc/csh.cshrc|<=="

tells us we need to have a look at pathnames.h

124 # define _PATH_DOTCSHRC     "/etc/csh.cshrc"
124 # define _PATH_DOTCSHRC     "/share/apps/utils/custom.tcshrc"

./configure --prefix=/share/apps/utils/tcsh
make install

If all went well:
cat tcsh|strings|grep custom.tcsh
tree /share/apps/utils/tcsh -L 1
|-- bin
`-- share

Obviously, this doesn't really make much of a difference just yet. Now comes the scary part -- and you need root access for this:
 which csh

ls /bin/csh -lah
lrwxrwxrwx 1 root root 4 Feb 23 16:54 /bin/csh -> tcsh
and here's the 'dangerous' stuff:
sudo rm /bin/csh
sudo ln -s /share/apps/utils/bin/tcsh /bin/csh
sudo chown root:root /bin/csh
sudo chmod 777 /bin/csh

Since /bin/csh isn't a binary but a symmlink to tcsh in the /bin directory, we just delete the symlink and create a new one.

We can now make whatever changes we want to our custom.tcshrc while still being able to easily change back to the old setup. I do recognise that we could just have edited /etc/csh.cshrc and /login.cshrc, but I for some reason feel a lot more comfortable using this method.

19 March 2012

111. Ecce (nwchem) on Debian, and ROCKS/Centos

If you're using nwchem chances are that you've considered using ECCE to parse the output:

First of all you'll need to register at -- and you can only do that if you're faculty. Postdocs and PhD students need not apply. Other than that, it's free, but you'll have to wait a couple of days to get your registration approved.

As much as I like nwchem owing to the clear syntax, I feel less warmly about ecce. Don't get me wrong -- it's pretty. It's just feels archaic and cobbled together. Even worse is that it's not open source and that its workings feel a bit opaque at times. Still, there's no better program for visually parsing nwchem output at this point. Anyway...

--start here --
Download the install_ecce.v6.2.rhel5-gcc3.2.3-m32.csh file to ~/tmp/ecce

There's no md5sum supplied but here's what I got:

If you haven't already
sudo apt-get install csh 

OK, moving on...
cd ~/tmp/ecce
chmod +x  install_ecce.v6.2.rhel5-gcc3.2.3-m32.csh

Main ECCE installation menu
0) Help on main menu options
1) Full install
2) Full upgrade
3) Application software install
4) Application software upgrade
5) Server install
6) Server upgrade

Pick 1 if you're installing on your desktop and there's no server that you know of. 

Once the installation is over you get:
!! You MUST perform the following steps in order to use ECCE !!
-- Unless only the user 'me' will be running ECCE,
   start the ECCE server as 'me' with:
-- To register machines to run computational codes, please see
   the installation and compute resource registration manuals
-- To run ECCE each user must source either the runtime_setup
   (csh/tcsh) or (sh/bash/ksh) script in the
   directory /home/me/tmp/ecce/ecce-v6.2/apps/scripts
   from their shell environment setup script.  For example,
   with csh or tcsh, add the following to ~/.cshrc:
     if (-e /home/me/tmp/ecce/ecce-v6.2/apps/scripts/runtime_setup) then
       source /home/me/tmp/ecce/ecce-v6.2/apps/scripts/runtime_setup
Which translates to:
1. sh  /home/me/tmp/ecce/ecce-v6.2/server/ecce-utils/start_ecce_server
2. Sourcing that file makes no sense. Instead, add the following to your ~/.bashrc
export ECCE_HOME=/home/me/tmp/ecce/ecce-v6.2/apps
export PATH=${ECCE_HOME}/scripts:${PATH}

Assuming you've source your ~/.bashrc, start ecce by typing

...which takes an unreasonably long time (ca 1 min) after which you're greeted by
Press Any Key
Type in a password -- any password -- which will be your password from now on.
You're then taken to
Click on Viewer (assuming you've got something to look at)
Pay attention to the fine print
Have a look at the text box in the bottom right corner..and pay attention. In my particular case I have 6 cores and an mpi aware nwchem 6.0 version compiled. I bet that's better than whatever comes bundled with ecce. Also, the

To change you go to the machine browser (see screen shot #2), click on set up remote access and make sure that everything is working by clicking on e.g. processes:

Then click on the Machine menu (top left), select Register Machine while your machine is selected.
You can now change your options.

So, before using ecce you always need to
sh  /home/me/tmp/ecce/ecce-v6.2/server/ecce-utils/start_ecce_server
first. The server will run until you stop it or reboot.
Next, start ecce

Integration with nwchem
Most people would probably set up their nwchem jobs by hand, because it's so simple. All you need to do is to include the statement
ecce_print ecce.out
in the beginning, and you'll get an ecce.out file which you can then IMPORT (not open regularly, but import) into ecce.

Click on Viewer, Import Calculation From Output File, select your ecce out and voilá:
ECCE: homo (benzene)
If you're running debian, you're done now.

ROCKS 5.4.3/Centos 5.6:
This isn't a fix as much as a rant. The problem with ROCKS 5.4.3 is that csh is so broken that it's a struggle just to install ecce. I mean, I do show how to get ecce running in the end, but ROCKS feels like an unfinished piece of work compared to a normal debian install.

--Demonstration only -- don't do --
First back up and ssh-key.csh in /etc/profile.d start by
chmod +x install_ecce.v6.2.rhel5-gcc3.2.3-m32.csh
...and nothing's happening.

You then try just typing in

/etc/profile.d/ line 211: return: can only `return' from a function or sourced script
It appears that you have not set up your ssh key.
This process will make the files:
Generating public/private rsa key pair.
/export/home/me/.ssh/id_rsa already exists.
Overwrite (y/n)? 

Turns out there's a bug in ROCKS 5.4.3.  You can fix that by:
rpm -Uvh

So far so good.
...and nothing. It just exits. Or so you think. But the problem is bigger than that --  try opening a new terminal in e.g. gnome (gnome-terminal or xterm) -- it exits immediately. No error message or anything.

You can get csh to start by moving /etc/csh.cshrc out of the way, but you're still screwed as to opening a new terminal. The only way to get back a working system is to restore and ssh-key.csh.

--- Demonstration over ---

--Start here --
 You could also get around all this by running
csh -f
But then you don't have any env. variables loading and it can lead to problems of its own.

csh -f install_ecce.v6.2.rhel5-gcc3.2.3-m32.csh

The install starts. Just follow the instructions.

After installation, start the server:
csh -f ecce-v6.2/server/ecce-utils/start_ecce_server

Hit enter until you get a workable prompt back...
Edit your ~/.bashrc and add

export ECCE_HOME=/home/me/tmp/ecce/ecce-v6.2/apps
export PATH=${ECCE_HOME}/scripts:${PATH}

Don't bother sourcing your ~/.bashrc. It's easier to just open a new terminal.
and you should be up and running...sort of. Under ROCKS I had problems importing ecce.out files since I had problems actually connecting to the server. Don't know why, but it came down to not being able to open a remote shell on the host.

this worked fine on one box, but not on another one which I was setting up remotely. On that one I had to edit


In particular, I had to change references to

15 March 2012

108. Building local version of sinfo without root/sudo on ROCKS/CentOS

Edit 04/04/2012: there were several errors and omissions. These have been fixed now.

Because I don't want to mess up a cluster which is on a different continent I'm trying to use my superuser powers as little as possible.

Here's how to make a local version of sinfo -- you'll still need to make sinfod runs as a service on all the nodes.

There's no reason the instructions here shouldn't work on most linux distros, including Debian.

cd ~/tmp
tar -xvf boost_1_49_0.tar.gz
cd boost_1_49_0/
./ --prefix=/export/home/me/.libboost

Edit tools/build/user-config.jam and add
using mpi ;
The space between mpi and ; is needed.

Start installation:
./b2 install

cd /export/home/me/.libboost/lib
ln -s
ln -s
ln -s
ln -s
ln -s

cd ~/tmp
wget ""
tar -xvf asio-1.5.3.tar.bz2
cd asio-1.5.3/

./configure --prefix=/export/home/me/.asio --with-boost=/export/home/me/.libboost/include
make install

tar -xvf sinfo-0.0.45.tar.gz
cd sinfo-0.0.45/

export LIBS=-L/export/home/me/.libboost/lib
export CPPFLAGS="-I/export/home/me/.libboost/include -I/export/home/me/.asio/include/"
./configure --prefix=/export/home/me/.sinfo --disable-IPv6

make install 

Getting started:
In order to make something happen at boot you need sudo/root access. However, HPC clusters are rarely rebooted, so even if you launch something as a user it will persist for a long time. If you're lucky the right ports are open -- and they should be open between nodes.

You also need to add this to your ~/.bashrc:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/export/home/me/.libboost/lib

Start sinfod (the daemon) using:
~/.sinfo/sbin/./sinfod --quiet

ps aux |grep sinfod 
will show it it's running

And check that everything is ok using

13 March 2012

105. Nwchem 6.1 with openmpi on ROCKS 5.4.3/CentOS 5.6

EDIT 18 May 2012: 
Compiling nwchem 6.1 with internal libs on debian:
Compiling nwchem 6.1 with openblas on debian:

I can build and use nwchem on ROCKS 5.4.3 -- see instructions below.

EDIT: the gfortran version is GNU Fortran (GCC) 4.1.2 20080704 (Red Hat 4.1.2-50)
On debian, which yields a segfaulting binary, the version is GNU Fortran (Debian 4.6.3-1) 4.6.3

I'm still having no luck building binaries which don't segfault on execution on debian though. The openmpi versions are the same for both ROCKS and debain: 1.4.3.


ROCKS 5.4.3/CentOS
The build is essentially the same as for nwchem-6.0 ( - the single difference is that you need to define USE_MPIF4 or you get errors

To build:

tar -xvf Nwchem-6.1-2012-Feb-10.tar.gz
cd nwchem-6.1/
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/export/home/me/tmp/nwchem-6.1
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/opt/openmpi
export MPI_INCLUDE=/opt/openmpi/include
export LIBRARY_PATH=$LIBRARY_PATH:/opt/openmpi/lib
export LIBMPI="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"
cd $NWCHEM_TOP/src
make clean
make  nwchem_config
make  FC=gfortran

Building takes a little while.

Make sure that you make the reference to your openmpi libs permanent and make life easier by putting the following in your ~/.bashrc or /etc/profile:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi/lib

export NWCHEM_EXECUTABLE=/export/home/me/tmp/nwchem-6.1/bin/LINUX64/nwchem
export NWCHEM_BASIS_LIBRARY=/export/home/me/tmp/nwchem-6.1/src/basis/libraries/

To run on multiple procs do
mpirun -n 3 nwchem input.nw
where 3 is the number of cores

104. Building gromacs with fftw3 and openmpi on ROCKS 5.4.3/CentOS

This guide was heavily modified on 13/03/2012 to remove the need for sudo/root privileges.

Not all flavours of linux are equal. I've always been a Debian man, but have recently become a user of a ROCKS based HPC cluster on a different continent. To make sure that I don't screw things up I'm currently trying to work out how to reliably compile common computational packages under ROCKS 5.4.3, which is CentOS based.

If you installed the bio roll from the beginning you'll have openmpi in /opt/openmpi (rocks_openmpi.x86_64 package), and fftw in /opt/rocks/lib and /opts/rocks/include (fftw.x86_64 package)

If you only installed the basic rolls, you won't have either. Now, you can either download the bio roll and install from there, or you can install the regular openmpi package and compile fftw yourself. In fact, you'll need to do the latter if you want double-precision gromacs anyway.

My goal is to avoid having to use sudo or root at all. I've rewritten this guide a couple of times, so there may be weird annoying errors that I've missed.

Installing openmpi:
If you don't have openmpi in /opt, then you can install it from the base roll
sudo yum install openmpi

You can skip this step IF
1. you have fftw files in /opt/rocks/lib and /opt/rocks/include
2. you only want single precision


tar -xvf fftw-3.3.1.tar.gz
cd fftw-3.3.1

Then use --prefix to tell make where to install the files:

Single precision fftw3 libraries:
make distclean
./configure --enable-float --enable-mpi --enable-threads --with-pic --prefix=/export/home/me/.fftwsingle
make install

Double-precision fftw3 libraries:
make distclean
./configure --disable-float --enable-mpi --enable-threads --with-pic --prefix=/export/home/me/.fftwdouble
make install


First download and extract:

cd ~/tmp

tar -xvf gromacs-4.5.5.tar.gz
cd gromacs-4.5.5/

Before building you need to define where the openmpi libs are i.e.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi/lib
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib64/openmpi/1.4-gcc/lib

We now have three permutations of possible builds:
1. We use the single precision fftw libs in /opt/rocks/lib and /opt/rocks/include
export LDFLAGS=-L/opt/rocks/lib
export CPPFLAGS=-I/opt/rocks/include
./configure --enable-mpi --enable-float --with-fft=fftw3 --program-suffix=_spmpi --prefix=/export/home/me/gromacs
make install

2. We use the single precision fftw libs in /export/home/me/.fftwsingle

export LDFLAGS=-L/export/home/me/.fftwsingle/lib
export CPPFLAGS=-I/export/home/me/.fftwsingle/include
./configure --enable-mpi --enable-float --with-fft=fftw3 --program-suffix=_spmpi --prefix=/export/home/me/gromacs
make install

3. We use the double precision fftw libs in /export/home/me/.fftwdouble

export LDFLAGS=-L/export/home/me/.fftwdouble/lib
export CPPFLAGS=-I/export/home/me/.fftwdouble/include
./configure --enable-mpi --disable-float --with-fft=fftw3 --program-suffix=_ddmpi --prefix=/export/home/me/gromacs
make install


You will now have single and double-precision binaries, e.g.
grompp_spmpi and grompp_ddmpi

Make sure that you define/have defined LD_LIBRARY_PATH in /etc/profile or ~/.bashrc and included the paths to your mpi libs and your fftw libs, e.g.:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi/lib:/export/home/me/.fftwsingle:/export/home/me/.fftwdouble

Actually, it doesn't seem necessary to include the fftw path.

You may also want to include your gromacs bins in your path:
export PATH=$PATH:/export/home/me/gromacs/bin

Dynamic load-balancing seems to be disabled by default, so to use multiple cores run using e.g.
mpirun -n 4 mdrun_spmpi -s inp.tpr -o out.trr etc.



checking size of off_t... configure: error: in `/export/home/me/tmp/gromacs-4.5.5':
configure: error: cannot compute sizeof (off_t)
See `config.log' for more details
./conftest: error while loading shared libraries: cannot open shared object file: No such file or directory
Set LD_LIBRARY_PATH to your openmpi libs e.g.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi/lib

/usr/local/lib/libfftw3f.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
make[3]: *** [] Error 1
Compile fftw3 using the --with-pic switch:
./configure --enable-float --enable-mpi --enable-threads --with-pic 

103. Building nwchem 6.0 on Rocks 5.4.3/CentOS

I've always been a Debian man, but for various reasons I need to be able to compile various scientific packages on a HPC running ROCKS. ROCKS 5.4.3 is based on CentOS 5,6and it turns out that debian is wonderfully easy, accommodating and robust in comparison. Well, since it's not my HPC, CentOS is what I'm stuck with.

Here's how to build nwchem on a rocks 5.4.3 (viper) cluster based on CentOS 5.6 and its ancient kernel.
(Linux  2.6.18-238.19.1.el5 #1 SMP Fri Jul 15 07:31:24 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux )

There are three different approaches:

This method requires no root access.
Check to see whether you've installed the rocks-openmpi package from the bio roll - it should be in /opt/openmpi. Otherwise use yum to install the base-roll openmpi package, which will end up in /usr/lib64/openmpi/1.4-gcc/lib -- you'll need root or sudo to do anything with yum.

For compilation, do
export LIBRARY_PATH=$LIBRARY_PATH:/opt/openmpi/lib
export LIBRARY_PATH=$LIBRARY_PATH:/usr/lib64/openmpi/1.4-gcc/lib/
depending on whether there is an openmpi directory in /opt or not.

You can also put the export line in your below
For execution:
in either you ~/.bashrc (user basis) or /etc/profile (global) put
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi/lib

CASE 2. /opt/openmpi is present; using symlinked libs

mpicc and mpif77 are probably already symlinked, but if not:

sudo ln -s /opt/openmpi/bin/mpicc /usr/bin/mpicc
sudo ln -s /opt/openmpi/bin/mpif77 /usr/bin/mpif77

The following allows for building and running:
sudo ln -s /opt/openmpi/lib/ /usr/lib/
sudo ln -s /opt/openmpi/lib/ /usr/lib/
sudo ln -s /opt/openmpi/lib/ /usr/lib/
sudo ln -s /opt/openmpi/lib/ /usr/lib/
sudo ln -s /opt/openmpi/lib/ /usr/lib64/
sudo ln -s /opt/openmpi/lib/ /usr/lib64/
sudo ln -s /opt/openmpi/lib/ /usr/lib64/
sudo ln -s /opt/openmpi/lib/ /usr/lib64/

the /usr/lib64 symlinks are necessary for execution, or you'll get
./nwchem: error while loading shared libraries: cannot open shared object file: No such file or directory

CASE 3. /opt/openmpi is NOT present; using symlinked libs

yum install openmpi openmpi-devel
And then put in all the symlinks...dunno why this isn't done on install, but there you go.

sudo ln -s /usr/local/lib64/openmpi/1.4-gcc/bin/mpicc  /usr/bin/mpicc
sudo ln -s /usr/local/lib64/openmpi/1.4-gcc/bin/mpif77 /usr/bin/mpif77
sudo ln -s /usr/lib64/openmpi/1.4-gcc/lib/ /usr/lib/
sudo ln -s /usr/lib64/openmpi/1.4-gcc/lib/ /usr/lib/
sudo ln -s /usr/lib64/openmpi/1.4-gcc/lib/ /usr/lib/
sudo ln -s /usr/lib64/openmpi/1.4-gcc/lib/ /usr/lib/

Using the above symlinks compilation will work just fine.
However, in order to actually run nwchem you need
sudo ln -s /usr/lib64/openmpi/1.4-gcc/lib/ /usr/lib64/
sudo ln -s /usr/lib64/openmpi/1.4-gcc/lib/ /usr/lib64/
sudo ln -s /usr/lib64/openmpi/1.4-gcc/lib/ /usr/lib64/
sudo ln -s /usr/lib64/openmpi/1.4-gcc/lib/ /usr/lib64/

or you'll get
./nwchem: error while loading shared libraries: cannot open shared object file: No such file or directory
Finally, make sure we can find our mpirun:
sudo ln -s /usr/lib64/openmpi/1.4-gcc/bin/mpirun /usr/bin/mpirun

Continue here:
We'll be working in /export/home/me/tmp
tar -xvf Nwchem
cd nwchem-6.0

create a file called and stuff it with the following:
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/export/home/me/tmp/nwchem-6.0
export USE_MPI=y
export USE_MPIF=y
export MPI_LOC=/usr/lib64/openmpi/1.4-gcc/lib
export MPI_INCLUDE=/usr/lib64/openmpi/1.4-gcc/include
export LIBMPI="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"
cd $NWCHEM_TOP/src
make clean
make nwchem_config
make FC=gfortran
NOTE: the above works for the case when you installed openmpi yourself (CASE 2 or 3). If it got installed with ROCKS on setup and is present in /opt/openmpi (CASE 1 or 3) change the following:

export MPI_LOC=/opt/openmpi/lib
export MPI_INCLUDE=/opt/openmpi/include
Launch the build


You'll end up with a binary called nwchem in nwchem--6.0/bin/LINUX64 -- you can put a PATH to it in your ~/.bashrc

For execution you will need to make sure nwchem can find the openmpi libs --
will tell you whether the path is included by default.
Otherwise, in either you ~/.bashrc (user basis) or /etc/profile (global) put
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openmpi/lib

If you move nwchem out of the compilation directory (to say /usr/local/nwchem) you may also want to define e.g.

export NWCHEM_TOP=/usr/local/nwchem-6.0

Again, this goes into your .bashrc or /etc/profile, depending on scope.

To use multiple cores, do
mpirun -n 4 nwchem jobname.nw
where the number of cores is 4.

Errors and troubleshooting:
If you get errors about libraries missing or mpicc-related errors you should make sure that you've symlinked everything you need into the /usr/lib folder or set the LIBRARY_PATH (see above). You could probably edit /etc/ld.conf too, but it will get messy with time.

I also tried building using mpich2-1.2 as well as 1.4, but got error messages about undefined references left and right.