11 June 2013

445. GAMESS US 2013 R1 on Debian (Wheezy) -- w/o GPU

Update 27/6/2013:
Please note that Kirill Berezovsky has published a series of posts on GAMESS US, including how to compile it for both CPU and GPU use. See
http://biochemicalmatters.blogspot.com.au/2013/06/gamess-us-frequently-asked-questions_26.html
http://biochemicalmatters.blogspot.ru/2013/06/gamess-us-frequently-asked-questions_1687.html
http://biochemicalmatters.blogspot.ru/2013/06/gamess-us-frequently-asked-questions_1447.html
http://biochemicalmatters.blogspot.com.au/2013/06/gamess-us-frequently-asked-questions.html


Original post:
A new version of GAMESS is out now (2013 R1): https://groups.google.com/forum/?fromgroups#!topic/gamess-announce/8j1esKifzEo

GPU support will be a later post.

0. Install a math library
You can use e.g. acml or atlas. See http://verahill.blogspot.com.au/2013/05/422-set-up-acml-on-linux.html for acml (which I've only had luck with on AMD machines).

To get the debian ATLAS libs do
sudo apt-get install libatlas3-base libatlas-dev

If you want to compile your own ATLAS libs, see e.g. http://verahill.blogspot.com.au/2012/09/rocks-543-atlas-and-gromacs-on-xeon.html or http://verahill.blogspot.com.au/2012/09/compile-atlas-gromacs-nwchem-on-amd-fx.html

You can also link to openblas, as shown in this post: http://verahill.blogspot.com.au/2012/09/compiling-and-testing-gamess-us-on.html


1. Get GAMESS US
Go to http://www.msg.chem.iastate.edu/GAMESS/download/register/
Check the tick boxes next to the architectures you intend to use GAMESS on. Fill out your email address, and hit Submit. You'll receive an email with instructions almost immediately.

The email will contain a URL to an archive with the source code, and a password for downloading it.
I'll presume that you downloaded the file, gamess-current.tar.gz, to ~/Downloads.


2. Untar and prepare
I'm presuming that you don't already have any /opt/gamess or /opt/gamess/gamess directories. In my earlier write-ups (e.g. this) I put everything in /opt/gamess, which isn't a good long-term strategy since you often want to keep earlier versions of computational software alongside newer ones.

sudo apt-get install build-essential gfortran openmpi-bin libopenmpi-dev libboost-all-dev
sudo mkdir /opt/gamess -p
sudo chown $USER:$USER /opt/gamess
cd /opt/gamess
cp ~/Downloads/gamess-current.tar.gz gamess-2013r1.tar.gz
tar xvf gamess-2013r1.tar.gz
mv gamess gamess-2013r1
cd gamess-2013r1/


3. Configure
NOTE: Even if you may have gfortran 4.7 (e.g. Wheezy) you should give the version as 4.6 during configure (see below).


./config
please enter your target machine name: linux64 GAMESS directory? [/opt/gamess/gamess-2013r1] GAMESS build directory? [/opt/gamess/gamess-2013r1] Version? [00] 13 Please enter your choice of FORTRAN: gfortran Please enter only the first decimal place, such as 4.1 or 4.6: 4.6
ACML:
Enter your choice of 'mkl' or 'atlas' or 'acml' or 'none': acml enter this full pathname: /opt/acml/acml5.3.1 Math library 'acml' will be taken from /opt/acml/acml5.3.1/gfortran64_int64/lib
Atlas:
I've compiled ATLAS myself in the past, but the libs have not worked with all programs. Here we use the debian libs instead.
Enter your choice of 'mkl' or 'atlas' or 'acml' or 'none': atlas Please enter the Atlas subdirectory on your system: /usr/lib/atlas-base Math library 'atlas' will be taken from /usr/lib/atlas-base
mpi:
communication library ('sockets' or 'mpi')? mpi Enter MPI library (impi, mvapich2, mpt, sockets): openmpi Please enter your openmpi's location: /usr/lib/openmpi

4. Build
Edit comp and change
1663 # -fno-whole-file suppresses argument's data type checking 1664 set OPT='-O2' 1665 if (".$GMS_DEBUG_FLAGS" != .) set OPT="$GMS_DEBUG_FLAGS"
to
1663 # -fno-whole-file suppresses argument's data type checking 1664 set OPT='-O0' 1665 if (".$GMS_DEBUG_FLAGS" != .) set OPT="$GMS_DEBUG_FLAGS"
or exam44.inp will fail.

cd ddi/
./compddi
cd ../
./compall

If you are building with openmpi, edit lked and change
958 case openmpi: 959 set MPILIBS="-L$GMS_MPI_PATH/lib64" 960 set MPILIBS="$MPILIBS -lmpi" 961 breaksw
to
958 case openmpi: 959 set MPILIBS="-L$GMS_MPI_PATH/lib" 960 set MPILIBS="$MPILIBS -lmpi" 961 breaksw
Make the proper symlinks if you are using ATLAS:
sudo ln -s /usr/lib/atlas-base/libatlas.so.3 /usr/lib/atlas-base/libatlas.so
sudo ln -s /usr/lib/atlas-base/libf77blas.so.3 /usr/lib/atlas-base/libf77blas.so

./lked gamess 13r1 


I use my own script called gmrun:
#!/bin/csh
set TARGET=mpi
set SCR=$HOME/scratch
set USERSCR=/scratch
set GMSPATH=/opt/gamess/gamess-2013r1
set JOB=$1
set VERNO=$2
set NCPUS=$3

if ( $JOB:r.inp == $JOB ) set JOB=$JOB:r
echo "Copying input file $JOB.inp to your run's scratch directory..."
cp $JOB.inp $SCR/$JOB.F05

setenv TRAJECT $USERSCR/$JOB.trj
setenv RESTART $USERSCR/$JOB.rst
setenv INPUT $SCR/$JOB.F05
setenv PUNCH $USERSCR/$JOB.dat
if ( -e $TRAJECT ) rm $TRAJECT
if ( -e  $PUNCH ) rm $PUNCH
if ( -e  $RESTART ) rm $RESTART
source $GMSPATH/gms-files.csh

setenv LD_LIBRARY_PATH /usr/lib/openmpi/lib:$LD_LIBRARY_PATH
mpiexec -n $NCPUS $GMSPATH/gamess.$VERNO.x|tee $JOB.out
cp $PUNCH .

To run, do e.g.
gmrun exam01.inp 13r1 2

8 comments:

  1. Dear lindqvist,

    Today my GAMESS (US) solutions was published at http://biochemicalmatters.blogspot.ru/

    If you interested, please welcome! You can publish my threads, if you want :) Also, I can send you GUI for GAMESS (US) and GUI input file generator, writed on Java.

    ReplyDelete
    Replies
    1. Thank you for the link (and the series of posts!).

      I'll try following the instructions next week to see if I have any more luck with libcchem.

      I added a comment to http://biochemicalmatters.blogspot.com.au/2013/06/gamess-us-frequently-asked-questions_26.htm pointing out potential issues with gfortran 4.7, the optimization level, and test job exam44.inp.

      I'd be interested in seeing the GUI. To date I'm only aware of http://www.chemsoft.ch/qc/gameix.htm

      Although to be fair my 'dream' would be for ECCE to support GAMESS US -- ECCE handles job preparation, submission, monitoring, and everything in between. Unfortunately, it
      1) 'only' handles g03, g98, nwchem, amica and gamess UK
      and
      2) the scripts that prepares the input and reads the output from different software packages are written in Perl, and while I find regex as compact and elegant as the next person, I have a hard time reading and understanding Perl.

      Delete
  2. Hi I was tested the performances of GAMESS-US using various compilers(ifort, gfortran, pgfortran). but the OPEN64 compiler was not tested because no support of GAMESS-US.
    The OPEN64 compiler was best performance about 100% for Gaussian03 at the AMD CPU computers in my experiences.
    Do you have a plan to compile the GAMESS-US with OPEN64 compiler ?
    I am now editing the compile scripts and testing the result having a lot of problems.

    ReplyDelete
  3. Hi I am thinking GAMESS-US compiling with OPEN64 compiler for best performance at AMD CPUs.
    Do you have a plan of GAMESS compiling with OPEN64 compiler ?

    ReplyDelete
    Replies
    1. I don't have any plans on compiling with open64. I used open64 a few years back to try to compile nwchem, and had little luck. If you do sort it out and write it up, feel free to post a link here.

      Delete
  4. Hi,
    Thank you so much for all these topics. I am very-very basic with linux command since I am a pure chemist... but trying to develop calculations.

    Thanks to you, I was able to compile gamess with opempi and atlas. This topic work on ubuntu 13.10. My machine is a HP z600 (2x4 cores, hypertreaded --> 16 cores with 24Gb) and calculations work fine. I have two similr machines, each contains a compiled version according to your topic.

    But now I want to go further and I am trying to built cluster using kestrelHPC (the only one where a full topic help me to buit one) but in this case only ubuntu lucid works or debian 6.0.x.. I install it on a third computer as frontend machine.
    I choose lucid in which the gfortran version is 4.4. Almost everything is ok during compilation. Kestrehpc is correctly installed : ganglia returns me 2 nodes (32processes and 48Gb).

    I have 2 problems for gamess compilation and 1 question about cluster:

    1.1 problem1
    the symlink you put in the topic doesn't work (file not found). Effectively, I looked at the directory and no such file, even not with similar name (libatlas.so.3 and libf77blas.so.3). Do I need to symlink something ?

    1.2. problem 2
    in your gmrun, i seem that the line :
    setenv LD_LIBRARY_PATH /usr/lib/openmpi/lib:$LD_LIBRARY_PATH
    is not correct. message is that the variable LD_LIBRARY_PATH is not set and program cannot find library. Which library ? altas or mpi ? the folder /usr/lib/openmpi/lib exist on my ubuntu lucid.

    2. question : about cluster and compilation
    do I need to specify somewhere the presence of nodes or if I run directly with 32 processes (ie gmrun test.inp 13r1 32 >test.out), the gamess/opempi will automatically send to nodes ?

    Thank you in advance for your help/comments. if you need to contact me, use phaubert2@wanadoo.fr to email me.

    sincerly,
    ph

    ReplyDelete
  5. 1.1 Are you using ATLAS? Have you installed ATLAS? Note also that this was written for debian -- canonical has a way of doing weird things with libraries for no discernable reason e.g. ECCE won't compile on the latest versions of ubuntu but works just fine on debian and fedora derivatives.
    Either way, to figure out where e.g. libatlas.so.3 is do
    locate libatlas|grep so

    1.2 The line is correct, and that's for openmpi. Again, make sure you've install openmpi and that the location is the same under ubuntu as under debian. Do e.g.
    locate libmpi.so

    If LD_LIBRARY_PATH isn't set you can try shortening the line to
    setenv LD_LIBRARY_PATH /usr/lib/openmpi/lib

    2. I'm not sure whether gamess is node-aware. I've basically never done any production work with it. However, you can set up openmpi to split jobs across nodes as it sees fit. See e.g. http://www.open-mpi.org/faq/?category=running#mpirun-scheduling

    Hope that it helps

    ReplyDelete
    Replies
    1. I should add that splitting a job across nodes often incurs a heavy performance penalty, depending on how they are interconnected (e.g. inifinibad vs gigabit ethernet and whether you use e.g. NFS) so it might not be worth the hassle.

      Delete