Showing posts with label gamess us. Show all posts
Showing posts with label gamess us. Show all posts

15 June 2018

652. N/EDA in GAMESS. 4. Running NEDA

Posts 1, 2 and 3.

For this you will need to have linked gamess and nbo.

Here's an example input that works:
andy@carbon:~$ less fast/gamess/mgme_tzvp/Mg_tzvp_opt_neda_2.inp ! File created by the GAMESS Input Deck Generator Plugin for Avogadro $CONTRL SCFTYP=RHF RUNTYP=energy DFTTYP=PBE0 ICHARG=1 MULT=1 NOSYM=1 $END ! $PCM SOLVNT=NEPTANE $END ! $PCMCAV RADII=SUAHF $END ! $SCF DIRSCF=.TRUE. $END $BASIS EXTFIL=.TRUE. GBASIS=DEF2SVP $END $system mwords=2000 memddi=500 $end $INTGRL NOPK=1 $END $NBO MOLUNIT <1> > <2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23> $END $DEL NEDA END $END $DATA Title C1 Mg 12.0 3.14572 1.02487 1.29474 N 7.0 4.95329 0.37304 1.62652 N 7.0 3.11530 2.89665 1.82263 C 6.0 5.44265 -0.98008 1.41792 C 6.0 5.80308 1.30078 2.07039 C 6.0 1.96130 3.78017 1.78886 C 6.0 4.28661 3.37927 2.23846 C 6.0 5.48149 2.64462 2.33178 H 1.0 6.31935 3.22840 2.68338 C 6.0 4.36331 4.81950 2.64988 H 1.0 5.37500 5.09552 2.93515 H 1.0 3.70056 5.01438 3.49762 H 1.0 4.03950 5.47541 1.83769 C 6.0 7.22682 0.90313 2.32491 H 1.0 7.68811 0.50750 1.41630 H 1.0 7.27827 0.11050 3.07627 H 1.0 7.81805 1.74612 2.67254 H 1.0 5.81529 -1.43220 2.34182 H 1.0 4.63475 -1.61377 1.04987 H 1.0 6.24793 -1.02060 0.67837 H 1.0 1.72108 4.18678 2.77556 H 1.0 2.10316 4.62224 1.10460 H 1.0 1.08587 3.22835 1.44552 $END


And here's the output:
Natural Energy Decomposition Analysis (Summary): Component Energy(wfn) Energy(wfn) (kcal/mol) ------------------------------------------------------------------------------ C7H13N2Mg(+) -583.0832392(scf) -582.9738665(loc) CT = -68.632 ES = -394.694 POL = -353.983 XC = -29.974 1. Mg(+2) -199.0566873(def) -199.0582918(cp) DEF(SE) = 1.007( 0.400) 2. C7H13N2(-) -382.6763191(def) -383.3176043(cp) DEF(SE) = 402.412(176.794) --------- E = -443.864 Electrical (ES+POL+SE) : -571.483 Charge Transfer (CT) : -68.632 Core (XC+DEF-SE) : 196.251 ------------ Total Interaction (E) : -443.864 ..... done with NBO analysis .....
A couple of things to note:
* It runs with PCM, but the results seem nonsensical, in addition to it finding a lot more fragments than without PCM (the latter thing can be amended with NAO)

* If I use DEF2TZVP with PBE0 and DIRSCF=.TRUE. It'll get stuck during the NBO run at
437. RY ( 5) H 23 0.00000 0.00000 0.00000 NEXT STEP: Perform one SCF cycle to evaluate the energy of the new density matrix constructed from the deleted NBO Fock matrix. ------------------------------------------------------------------------------ -------------------------- R-PBE0 SCF CALCULATION -------------------------- DENSITY MATRIX CONVERGENCE THRESHOLD= 2.00E-05 COARSE -> FINE DFT GRID SWITCH THRESHOLD= 3.00E-04 (SWITCH IN $DFT) HF -> DFT SWITCH THRESHOLD= 0.00E+00 (SWOFF IN $DFT) DIRECT SCF CALCULATION, SCHWRZ=T FDIFF=T, DIRTHR= 0.00E+00 NITDIR=10 NONZERO BLOCKS ITER EX DEM TOTAL ENERGY E CHANGE DENSITY CHANGE DIIS ERROR INTEGRALS SKIPPED
It's not consuming any CPU at this point (nor is nbo). I'm running w/o DIRSCF now, but it's very slow, and requires ca 30 Gb of scratch space even for a small molecule like this one.

MOLUNIT <1> >
<2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23>
is not necessary for this run. You can simply use NBO $END instead.

651. N/EDA in GAMESS. 3. Running L/CMOEDA

As it turns out, while MOEDA works fine with pure and hybrid functionals, it does not appear to like PCM, giving weird results.

Here's an example of a run that works, with a few key words highlighted:
! File created by the GAMESS Input Deck Generator Plugin for Avogadro $BASIS EXTFIL=.TRUE. GBASIS=DEF2SVP $END ! $PCM SOLVNT=NEPTANE $END ! $PCMCAV RADII=SUAHF $END $CONTRL SCFTYP=RHF RUNTYP=EDA DFTTYP=PBE0 ICHARG=1 MULT=1 $END $CONTRL NOSYM=1 $END $SYSTEM MWORDS=2500 MEMDDI=500 $END $SCF DIRSCF=.TRUE. $END $lmoeda matom(1)=1,22 mcharg(1)=2,-1 mmult(1)=1,1 $end $DATA Test of MOEDA using a Mg coordination compound. C1 Mg 12.0 3.14572 1.02487 1.29474 N 7.0 4.95329 0.37304 1.62652 N 7.0 3.11530 2.89665 1.82263 C 6.0 5.44265 -0.98008 1.41792 C 6.0 5.80308 1.30078 2.07039 C 6.0 1.96130 3.78017 1.78886 C 6.0 4.28661 3.37927 2.23846 C 6.0 5.48149 2.64462 2.33178 H 1.0 6.31935 3.22840 2.68338 C 6.0 4.36331 4.81950 2.64988 H 1.0 5.37500 5.09552 2.93515 H 1.0 3.70056 5.01438 3.49762 H 1.0 4.03950 5.47541 1.83769 C 6.0 7.22682 0.90313 2.32491 H 1.0 7.68811 0.50750 1.41630 H 1.0 7.27827 0.11050 3.07627 H 1.0 7.81805 1.74612 2.67254 H 1.0 5.81529 -1.43220 2.34182 H 1.0 4.63475 -1.61377 1.04987 H 1.0 6.24793 -1.02060 0.67837 H 1.0 1.72108 4.18678 2.77556 H 1.0 2.10316 4.62224 1.10460 H 1.0 1.08587 3.22835 1.44552 $END
The ! comments out the PCM lines. I generated the input from an .xyz file through avogadro, and then edited the file manually to get the charges etc. right.

Here's the output:
DIRECT SCF CALCULATION, SCHWRZ=T FDIFF=F, DIRTHR= 0.00E+00 NITDIR=10 DFT IS SWITCHED OFF, PERFORMING PURE SCF UNTIL SWOFF THRESHOLD IS REACHED. NONZERO BLOCKS ITER EX DEM TOTAL ENERGY E CHANGE DENSITY CHANGE DIIS ERROR INTEGRALS SKIPPED *** ERROR *** IN -DAWRIT- ROUTINE ON NODE 0 DAWRIT HAS REQUESTED A RECORD WITH LENGTH DIFFERENT THAN BEFORE - ABORT FORCED. DAF RECORD 88 NEW LENGTH = 20100 OLD LENGTH = 190 EXECUTION OF GAMESS TERMINATED -ABNORMALLY- AT Thu Jun 14 23:30:05 2018 6712269 WORDS OF DYNAMIC MEMORY USED CPU 0: STEP CPU TIME= 1.73 TOTAL CPU TIME= 2.0 ( 0.0 MIN) TOTAL WALL CLOCK TIME= 2.1 SECONDS, CPU UTILIZATION IS 96.70% DDI Process 0: error code 911 ddikick.x: application process 0 quit unexpectedly. ddikick.x: Fatal error detected. The error is most likely to be in the application, so check for input errors, disk space, memory needs, application bugs, etc. ddikick.x will now clean up all processes, and exit... DDI Process 1: terminated upon request. DDI Process 2: terminated upon request. *** ERROR *** IN -DAWRIT- ROUTINE ON NODE 1 DAWRIT HAS REQUESTED A RECORD WITH LENGTH DIFFERENT THAN BEFORE - ABORT FORCED. DAF RECORD 88 NEW LENGTH = 20100 OLD LENGTH = 190 *** ERROR *** IN -DAWRIT- ROUTINE ON NODE 2 DAWRIT HAS REQUESTED A RECORD WITH LENGTH DIFFERENT THAN BEFORE - ABORT FORCED. DAF RECORD 88 NEW LENGTH = 20100 OLD LENGTH = 190 DDI Process 3: terminated upon request. DDI Process 4: terminated upon request. *** ERROR *** IN -DAWRIT- ROUTINE ON NODE 3 DAWRIT HAS REQUESTED A RECORD WITH LENGTH DIFFERENT THAN BEFORE - ABORT FORCED. DAF RECORD 88 NEW LENGTH = 20100 OLD LENGTH = 190 DDI Process 5: terminated upon request. *** ERROR *** IN -DAWRIT- ROUTINE ON NODE 4 DAWRIT HAS REQUESTED A RECORD WITH LENGTH DIFFERENT THAN BEFORE - ABORT FORCED. DAF RECORD 88 NEW LENGTH = 20100 OLD LENGTH = 190 *** ERROR *** IN -DAWRIT- ROUTINE ON NODE 5 DAWRIT HAS REQUESTED A RECORD WITH LENGTH DIFFERENT THAN BEFORE - ABORT FORCED. DAF RECORD 88 NEW LENGTH = 20100 OLD LENGTH = 190 ddikick.x: Sending kill signal to DDI processes. ddikick.x: Execution terminated due to error(s). unset echo

649. N/EDA in GAMESS. 1. Recompiling GAMESS US with NBO6

I need to do energy decomposition analysis (EDA), but only have licenses for Gaussian and NBO6 (i.e. not ADF, turbomole, QChem etc.). NEDA isn't supported by NBO6 with gaussian (afaik).

NWChem, my usual gaussian alternative, doesn't support NBO6 beyond writing a .47 file.

Enter GAMESS US. I've been trying out gamess every few years, but I've found that it's slow and unreliable (very difficult to get SCF convergence) for the systems I work with (polyanions). Some of this may obviously be down to my lack of familiarity with the code -- there are probably plenty of satisified users of GAMESS US.

Either way, NBO6 suppports NEDA with GAMESS US. Also, GAMESS US does two different types  of   EDA: the common Morokuma-Kitaura (MOROKM) one (although only with HF) and an alternative approach by Su and Li that's referred to by GAMESS as LMOEDA (or CMOEDA).

MOROKM and LMOEDA as supported out of the box by GAMESS, but to get it to do NEDA you need to re-link it against NBO. Luckily it's even easier than the instructions in the NBO gamess file (i.e. no need to edit code).

NOTE: I could only link with gfortran 4.9 (jessie). gfortran 6.3 (stretch) failed to link (messages re -fPIC; recompiling gamess with -fPIC didn't solve it).

To compile gamess, see e.g. http://verahill.blogspot.com/2013/06/4xx-gamess-us-2013-r1-on-debian-wheezy.html

Once you've done the ddi/comp and compall steps, edit lked and search for NBO. Change to

set NBO=true set NBOLIB="/opt/nbo6/bin/gmsnbo.i8.a"
assuming that this location is correct.

Then do lked as in the post above.

11 June 2013

445. GAMESS US 2013 R1 on Debian (Wheezy) -- w/o GPU

Update 27/6/2013:
Please note that Kirill Berezovsky has published a series of posts on GAMESS US, including how to compile it for both CPU and GPU use. See
http://biochemicalmatters.blogspot.com.au/2013/06/gamess-us-frequently-asked-questions_26.html
http://biochemicalmatters.blogspot.ru/2013/06/gamess-us-frequently-asked-questions_1687.html
http://biochemicalmatters.blogspot.ru/2013/06/gamess-us-frequently-asked-questions_1447.html
http://biochemicalmatters.blogspot.com.au/2013/06/gamess-us-frequently-asked-questions.html


Original post:
A new version of GAMESS is out now (2013 R1): https://groups.google.com/forum/?fromgroups#!topic/gamess-announce/8j1esKifzEo

GPU support will be a later post.

0. Install a math library
You can use e.g. acml or atlas. See http://verahill.blogspot.com.au/2013/05/422-set-up-acml-on-linux.html for acml (which I've only had luck with on AMD machines).

To get the debian ATLAS libs do
sudo apt-get install libatlas3-base libatlas-dev

If you want to compile your own ATLAS libs, see e.g. http://verahill.blogspot.com.au/2012/09/rocks-543-atlas-and-gromacs-on-xeon.html or http://verahill.blogspot.com.au/2012/09/compile-atlas-gromacs-nwchem-on-amd-fx.html

You can also link to openblas, as shown in this post: http://verahill.blogspot.com.au/2012/09/compiling-and-testing-gamess-us-on.html


1. Get GAMESS US
Go to http://www.msg.chem.iastate.edu/GAMESS/download/register/
Check the tick boxes next to the architectures you intend to use GAMESS on. Fill out your email address, and hit Submit. You'll receive an email with instructions almost immediately.

The email will contain a URL to an archive with the source code, and a password for downloading it.
I'll presume that you downloaded the file, gamess-current.tar.gz, to ~/Downloads.


2. Untar and prepare
I'm presuming that you don't already have any /opt/gamess or /opt/gamess/gamess directories. In my earlier write-ups (e.g. this) I put everything in /opt/gamess, which isn't a good long-term strategy since you often want to keep earlier versions of computational software alongside newer ones.

sudo apt-get install build-essential gfortran openmpi-bin libopenmpi-dev libboost-all-dev
sudo mkdir /opt/gamess -p
sudo chown $USER:$USER /opt/gamess
cd /opt/gamess
cp ~/Downloads/gamess-current.tar.gz gamess-2013r1.tar.gz
tar xvf gamess-2013r1.tar.gz
mv gamess gamess-2013r1
cd gamess-2013r1/


3. Configure
NOTE: Even if you may have gfortran 4.7 (e.g. Wheezy) you should give the version as 4.6 during configure (see below).


./config
please enter your target machine name: linux64 GAMESS directory? [/opt/gamess/gamess-2013r1] GAMESS build directory? [/opt/gamess/gamess-2013r1] Version? [00] 13 Please enter your choice of FORTRAN: gfortran Please enter only the first decimal place, such as 4.1 or 4.6: 4.6
ACML:
Enter your choice of 'mkl' or 'atlas' or 'acml' or 'none': acml enter this full pathname: /opt/acml/acml5.3.1 Math library 'acml' will be taken from /opt/acml/acml5.3.1/gfortran64_int64/lib
Atlas:
I've compiled ATLAS myself in the past, but the libs have not worked with all programs. Here we use the debian libs instead.
Enter your choice of 'mkl' or 'atlas' or 'acml' or 'none': atlas Please enter the Atlas subdirectory on your system: /usr/lib/atlas-base Math library 'atlas' will be taken from /usr/lib/atlas-base
mpi:
communication library ('sockets' or 'mpi')? mpi Enter MPI library (impi, mvapich2, mpt, sockets): openmpi Please enter your openmpi's location: /usr/lib/openmpi

4. Build
Edit comp and change
1663 # -fno-whole-file suppresses argument's data type checking 1664 set OPT='-O2' 1665 if (".$GMS_DEBUG_FLAGS" != .) set OPT="$GMS_DEBUG_FLAGS"
to
1663 # -fno-whole-file suppresses argument's data type checking 1664 set OPT='-O0' 1665 if (".$GMS_DEBUG_FLAGS" != .) set OPT="$GMS_DEBUG_FLAGS"
or exam44.inp will fail.

cd ddi/
./compddi
cd ../
./compall

If you are building with openmpi, edit lked and change
958 case openmpi: 959 set MPILIBS="-L$GMS_MPI_PATH/lib64" 960 set MPILIBS="$MPILIBS -lmpi" 961 breaksw
to
958 case openmpi: 959 set MPILIBS="-L$GMS_MPI_PATH/lib" 960 set MPILIBS="$MPILIBS -lmpi" 961 breaksw
Make the proper symlinks if you are using ATLAS:
sudo ln -s /usr/lib/atlas-base/libatlas.so.3 /usr/lib/atlas-base/libatlas.so
sudo ln -s /usr/lib/atlas-base/libf77blas.so.3 /usr/lib/atlas-base/libf77blas.so

./lked gamess 13r1 


I use my own script called gmrun:
#!/bin/csh
set TARGET=mpi
set SCR=$HOME/scratch
set USERSCR=/scratch
set GMSPATH=/opt/gamess/gamess-2013r1
set JOB=$1
set VERNO=$2
set NCPUS=$3

if ( $JOB:r.inp == $JOB ) set JOB=$JOB:r
echo "Copying input file $JOB.inp to your run's scratch directory..."
cp $JOB.inp $SCR/$JOB.F05

setenv TRAJECT $USERSCR/$JOB.trj
setenv RESTART $USERSCR/$JOB.rst
setenv INPUT $SCR/$JOB.F05
setenv PUNCH $USERSCR/$JOB.dat
if ( -e $TRAJECT ) rm $TRAJECT
if ( -e  $PUNCH ) rm $PUNCH
if ( -e  $RESTART ) rm $RESTART
source $GMSPATH/gms-files.csh

setenv LD_LIBRARY_PATH /usr/lib/openmpi/lib:$LD_LIBRARY_PATH
mpiexec -n $NCPUS $GMSPATH/gamess.$VERNO.x|tee $JOB.out
cp $PUNCH .

To run, do e.g.
gmrun exam01.inp 13r1 2

09 May 2013

409.A.GAMESS US with GPU support on debian wheezy. This works (probably).


Update 27/6/2013:
Please note that Kirill Berezovsky has published a series of posts on GAMESS US, including how to compile it for both CPU and GPU use. See
http://biochemicalmatters.blogspot.com.au/2013/06/gamess-us-frequently-asked-questions_26.html
http://biochemicalmatters.blogspot.ru/2013/06/gamess-us-frequently-asked-questions_1687.html
http://biochemicalmatters.blogspot.ru/2013/06/gamess-us-frequently-asked-questions_1447.html
http://biochemicalmatters.blogspot.com.au/2013/06/gamess-us-frequently-asked-questions.html


Update 21 May 2013: See the comments below this post. This approach most likely works -- what has been confusing me is the lack of reports of GPU timings in the output, but this doesn't necessarily mean that the GPU isn't being used. The poster below this post, using nvidia-smi, observed GPU usage, although the speed-up was not major.


Update 10/05/2013: fixed libcchem compile.

Everything compiles fine and computations run fine and fast. To date there's only one other detailed step-by-step example of successful compilation of GAMESS with GPU support out there. At least based on google.

For various reasons I'm beginning to suspect that ATLAS isn't working out for me -- I've had issues getting things to converge with ATLAS, but which work fine with ACML (see post B).

I was in part following http://combichem.blogspot.com.au/2011/02/compiling-gamess-with-cuda-gpu-support.html and ./libcchem/aaa.readme.1st

This took a while to hammer out, so the write-up is a bit messy.


Set up
sudo apt-get install libboost-all-dev build-essential g++ gfortran automake nvidia-cuda-toolkit python-cheetah openmpi-bin libopenmpi-dev zlib1g-dev checkinstall
mkdir ~/tmp

Get gamess (see e.g. http://verahill.blogspot.com.au/2012/09/compiling-and-testing-gamess-us-on.html).

Put gamess-current.tar.gz in  ~/tmp

cd ~/tmp
tar xvf gamess-current.tar.gz
sudo mv gamess /opt/gamess_cuda
sudo chown $USER:$USER /opt/gamess_cuda -R


Preparing Boost
Edit /usr/include/boost/mpl/aux_/integral_wrapper.hpp
47 // other compilers (e.g. MSVC) are not particulary happy about it 48 #if BOOST_WORKAROUND(__EDG_VERSION__, <= 238) || defined(__CUDACC__) 49 typedef struct AUX_WRAPPER_NAME type;
Edit /usr/include/boost/mpl/size_t_fwd.hpp
20 21 BOOST_MPL_AUX_ADL_BARRIER_NAMESPACE_OPEN 22 #if defined(__CUDACC__) 23 typedef std::size_t std_size_t; 24 template< std_size_t N > struct size_t; 25 #else 26 template< std::size_t N > struct size_t; 27 #endif 28 29 BOOST_MPL_AUX_ADL_BARRIER_NAMESPACE_CLOSE
Edit /usr/include/boost/mpl/size_t.hpp
19 #if defined(__CUDACC__) 20 #define AUX_WRAPPER_VALUE_TYPE std_size_t 21 #define AUX_WRAPPER_NAME size_t 22 #define AUX_WRAPPER_PARAMS(N) std_size_t N 23 #else 24 #define AUX_WRAPPER_VALUE_TYPE std::size_t 25 #define AUX_WRAPPER_NAME size_t 26 #define AUX_WRAPPER_PARAMS(N) std::size_t N 27 #endif 28

HDF5
You'll have to compile that yourself for now since H5Cpp.h missing in the debian packages.(i.e. cxx support)

mkdir ~/tmp
cd ~/tmp
wget http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.10-patch1.tar.gz
tar xvf hdf5-1.8.10-patch1.tar.gz
cd hdf5-1.8.10-patch1/
export CC=/usr/bin/gcc-4.6 && export CXX=/usr/bin/g++-4.6
./configure --prefix=/opt/gamess_cuda/hdf5 --with-pthread --enable-cxx --enable-threadsafe --enable-unsupported
make
mkdir /opt/gamess_cuda/hdf5/lib -p
mkdir /opt/gamess_cuda/hdf5/include -p
sudo checkinstall
This package will be built according to these values: 0 - Maintainer: [ root@neon ] 1 - Summary: [ hdf5-cxx] 2 - Name: [ hdf5-1.8.10 ] 3 - Version: [ 1.8.10-1 ] 4 - Release: [ 1 ] 5 - License: [ GPL ] 6 - Group: [ checkinstall ] 7 - Architecture: [ amd64 ] 8 - Source location: [ hdf5-1.8.10-patch1 ] 9 - Alternate source location: [ ] 10 - Requires: [ ] 11 - Provides: [ hdf5-1.8.10 ] 12 - Conflicts: [ ] 13 - Replaces: [ ]
Make sure to edit the Version field since Patch-1 leads to an error (must start with digit).
Openmpi 1.6 Can't remember why I ended up compiling it myself instead of using the stock debian version. From here.

sudo apt-get install build-essential gfortran
wget http://www.open-mpi.org/software/ompi/v1.6/downloads/openmpi-1.6.tar.bz2
tar xvf openmpi-1.6.tar.bz2
cd openmpi-1.6/

sudo mkdir /opt/openmpi/
sudo chown ${USER} /opt/openmpi/
./configure --prefix=/opt/openmpi/1.6/ --with-sge

make
make install

compiling libcchem
cd /opt/gamess_cuda/libcchem
edit /opt/gamess_cuda/libcchem/rysq/src/externals/boost/cuda/device_ptr.hpp
  4 #include <cstdlib>
  5 #include <iterator>
  6 #include <stddef.h>
  7 
  8 namespace boost {
Edit /opt/gamess_cuda/libcchem/src/externals/boost/cuda/device_ptr.hpp
  4 #include <cstdlib>
  5 #include <iterator>
  6 #include <stddef.h>
  7 
  8 namespace boost {
  9 namespace cuda {
./configure --with-gamess --with-hdf5=/opt/gamess_cuda/hdf5 CPPFLAGS="-I/opt/gamess_cuda/hdf5/include" --with-cuda=/usr --disable-openmp --prefix=/opt/gamess_cuda/libcchem --with-gpu=fermi --with-integer8 --with-cublas
make
make install

Configure Gamess US Mainly follow this: http://verahill.blogspot.com.au/2012/09/compiling-and-testing-gamess-us-on.html
cd /opt/gamess_cuda
./config
please enter your target machine name: linux64 GAMESS directory? [/opt/gamess_cuda] /opt/gamess_cuda Setting up GAMESS compile and link for GMS_TARGET=linux64 GAMESS software is located at GMS_PATH=/opt/gamess_cuda Please provide the name of the build locaation. This may be the same location as the GAMESS directory. GAMESS build directory? [/home/me/tmp/gamess] Please provide a version number for the GAMESS executable. This will be used as the middle part of the binary's name, for example: gamess.00.x Version? [00] 12r2 Please enter your choice of FORTRAN: gfortran gfortran is very robust, so this is a wise choice. Please type 'gfortran -dumpversion' or else 'gfortran -v' to detect the version number of your gfortran. This reply should be a string with at least two decimal points, such as 4.1.2 or 4.6.1, or maybe even 4.4.2-12. The reply may be labeled as a 'gcc' version, but it is really your gfortran version. Please enter only the first decimal place, such as 4.1 or 4.6: 4.6
Enter your choice of 'mkl' or 'atlas' or 'acml' or 'none': atlas Please enter the Atlas subdirectory on your system: /opt/ATLAS/lib Math library 'atlas' will be taken from /opt/ATLAS If you have an expensive but fast network like Infiniband (IB), and if you have an MPI library correctly installed, choose 'mpi'. communication library ('sockets' or 'mpi')? mpi Enter MPI library (impi, mvapich2, mpt, sockets): openmpi
Please enter your openmpi's location: /opt/openmpi/1.6

Build Gamess US
cd /opt/gamess_cuda/ddi/
./compddi
cd ../

Edit comp
872 # see ~/gamess/libcchem/aaa.readme.1st for more information 873 set GPUCODE=true 874 if ($GPUCODE == true) then
and
1663 # -fno-whole-file suppresses argument's data type checking 1664 set OPT='-O0' 1665 if (".$GMS_DEBUG_FLAGS" != .) set OPT="$GMS_DEBUG_FLAGS"
./compall

Edit lked
69 # 70 set GPUCODE=true 71 # 72 # 5. optional MPQC interface
and
958 case openmpi: 959 set MPILIBS="-L$GMS_MPI_PATH/lib" 960 set MPILIBS="$MPILIBS -lmpi -lpthread" 961 breaksw
and
1214 if ($GPUCODE == true) then 1215 echo " Using 'libcchem' add-in C++ codes for Nvidia/CUDA GPUs." 1216 set GPU_LIBS="-L/opt/gamess_cuda/libcchem/lib -lcchem_gamess -lcchem -lrysq" 1217 set GPU_LIBS="$GPU_LIBS -lcudart -lcublas" 1218 ### GPU_LIBS="$GPU_LIBS -lcudart -lcublas" 1219 set GPU_LIBS="$GPU_LIBS /usr/lib/libboost_thread.a" 1220 set GPU_LIBS="$GPU_LIBS /opt/gamess_cuda/hdf5/lib/libhdf5.a" 1221 set GPU_LIBS="$GPU_LIBS /opt/gamess_cuda/hdf5/lib/libhdf5_cpp.a" 1222 set GPU_LIBS="$GPU_LIBS /opt/gamess_cuda/hdf5/lib/libhdf5_hl.a" 1223 set GPU_LIBS="$GPU_LIBS /opt/gamess_cuda/hdf5/lib/libhdf5.a" 1224 set GPU_LIBS="$GPU_LIBS /opt/ATLAS/lib/libcblas.a" 1225 set GPU_LIBS="$GPU_LIBS -lz" 1226 set GPU_LIBS="$GPU_LIBS -lstdc++" 1227 ### GPU_LIBS="$GPU_LIBS -lgomp" 1228 set GPU_LIBS="$GPU_LIBS -lpthread" 1229 echo " libcchem GPU code's libraries are" 1230 echo "$GPU_LIBS" 1231 else

./lked gamess gpu.12

Create gpurun
#!/bin/csh set TARGET=mpi set SCR=$HOME/scratch set USERSCR=/scratch set GMSPATH=/opt/gamess_cuda set JOB=$1 set VERNO=$2 set NCPUS=$3 @ NUMGPU=1 if ($NUMGPU > 0) then @ NUMCPU = $NCPUS - 1 echo libcchem kernels will use $NUMCPU cores and $NUMGPU GPUs per node... set echo setenv CCHEM_PROFILE 1 setenv NUM_THREADS $NCPUS #--if ($NUMGPU == 0) setenv GPU_DEVICES -1 #--if ($NUMGPU == 2) setenv GPU_DEVICES 0,1 #--if ($NUMGPU == 4) setenv GPU_DEVICES 0,1,2,3 #setenv LD_LIBRARY_PATH /share/apps/cuda/lib64:$LD_LIBRARY_PATH ###### LD_LIBRARY_PATH /usr/local/cuda/lib64:$LD_LIBRARY_PATH unset echo else setenv GPU_DEVICES -1 endif if ( $JOB:r.inp == $JOB ) set JOB=$JOB:r echo "Copying input file $JOB.inp to your run's scratch directory..." cp $JOB.inp $SCR/$JOB.F05 setenv TRAJECT $USERSCR/$JOB.trj setenv RESTART $USERSCR/$JOB.rst setenv INPUT $SCR/$JOB.F05 setenv PUNCH $USERSCR/$JOB.dat if ( -e $TRAJECT ) rm $TRAJECT if ( -e $PUNCH ) rm $PUNCH if ( -e $RESTART ) rm $RESTART source $GMSPATH/gms-files.csh setenv LD_LIBRARY_PATH /opt/openmpi/lib:$LD_LIBRARY_PATH set path= ( /opt/openmpi/bin $path ) mpiexec -n $NCPUS $GMSPATH/gamess.gpu.$VERNO.x|tee $JOB.out cp $PUNCH .

echo 'export PATH=$PATH:/opt/gamess_cuda' >> ~/.bashrc
source ~/.bashrc
chmod +x gpurun
cd test/standard/
 gpurun exam44 12 2


The only evidence of GPU usage in the output is e.g. in exam44.out:
388           -----------------------
389           MP2 CONTROL INFORMATION
390           -----------------------
391           NACORE =        6  NBCORE =        6
392           LMOMP2 =        F  AOINTS = DUP
393           METHOD =        2  NWORD  =               0
394           MP2PRP =        F  OSPT   = NONE
395           CUTOFF = 1.00E-09  CPHFBS = BASISAO
396           CODE   = GPU
397 
398           NUMBER OF CORE -A-  ORBITALS =     6
399           NUMBER OF CORE -B-  ORBITALS =     6

but in the summary only CPU utilisation is mentioned.



I modified rungms:

me@neon:/opt/gamess_cuda/tests/standard$ diff /opt/gamess_cuda/gpurungms /opt/gamess/rungms 
59,62c59,62
< set TARGET=mpi
< set SCR=$HOME/scratch
< set USERSCR=/scratch
< set GMSPATH=/opt/gamess_cuda
---
> set TARGET=sockets
> set SCR=/scr/$USER
> set USERSCR=~$USER/scr
> set GMSPATH=/u1/mike/gamess
67d66
< set NNODES=1
513c512
< set PPN=$3
---
>    set PPN=$4
601c600
<          @ PPN2 = $PPN
---
>          @ PPN2 = $PPN + $PPN
742c741
<    @ NUMGPU=1
---
>    @ NUMGPU=0
752c751
< #      setenv LD_LIBRARY_PATH /share/apps/cuda/lib64:$LD_LIBRARY_PATH
---
>       setenv LD_LIBRARY_PATH /share/apps/cuda/lib64:$LD_LIBRARY_PATH
793c792,793
<       /opt/openmpi/1.6/bin/mpiexec -n $NPROCS $GMSPATH/gamess.$VERNO.x < /dev/null
---
>       mpiexec.hydra -f $PROCFILE -n $NPROCS \
>             /home/mike/gamess/gamess.$VERNO.x < /dev/null