173. ECCE in a virtual machine: step by step using virtualbox

!NOTE!
Pre-built binaries of ECCE now ONLY exist for 64 bit OS -- if you install a 32 bit version of debian you will have to compile ECCE yourself. It's not difficult, but please be aware of it.

!NOTE2! If you provide ECCE with 'localhost' as the hostname during installation, be aware that this will block outside access: http://www.nwchem-sw.org/index.php/Special:AWCforum/st/id858/#post_3178

See this post for ECCE 6.4 of Debian 7 (wheezy): http://verahill.blogspot.com.au/2013/05/434-ecce-64-on-debian-7-in-32-bit.html
!NOTE!

Sadly, not everyone uses linux. Some even refuse to give it a proper consideration. Since the goal here is science rather than OS enlightenment, we'll have to accept that some people will have to run ECCE in a virtual machine rather than using it natively (something like WINE but going the other direction would be nice).

Win/Mac users should not be surprised if graphics heavy applications like ECCE don't run very fast in a virtual machine though, but they will be the source of that frustration themselves.

Here's the whole sequence from downloading an iso to having ECCE ready to submit jobs from inside a virtual machine.

I'm doing all these steps on linux since I simply don't use or have access to windows (other than as a very small virtualbox install), so let me know if there's anything which won't work on windows. I'm particularly interested in XP since that's what most people in academia still use.

You can of course use these instructions on a proper debian installation as well.

The only reason this post is so long is because of the number of screen shots. It is NOT COMPLICATED OR DIFFICULT, so don't be put off by the length.

Index:
1. Install virtualbox
2. Set up the machine
3. Download the debian iso
4. Install debian in virtualbox
5. Customize your debian installation
6. Installing ECCE
7. Setting up ECCE
8. Launching a test calc.
9. Moving files back and forth between virtual debian and windows

---

1. Install Virtualbox
I've tested this step in my virtual XP machine, so it should work fine on a 'real' XP machine.

Download the windows installer: http://download.virtualbox.org/virtualbox/4.1.16/VirtualBox-4.1.16-78094-Win.exe . If the link doesn't work, then go to https://www.virtualbox.org/wiki/Downloads

Start the installer. It appears to be a bit buggy since the 'back' button doesn't work, so make sure that you think about your choices. Otherwise starting over isn't that hard.

 ...and start. You'll get a warning about network interfaces being reset. That's fine.

You get five of those Unsigned driver thingy warnings -- a Microsoft protection racket if you'd ask me (compatibility assurance is a good thing -- forcing developers to pay for warnings to go away isn't). Just click continue.

When you're done, launch:

Download the extensions from here: http://download.virtualbox.org/virtualbox/4.1.16/Oracle_VM_VirtualBox_Extension_Pack-4.1.16-78094.vbox-extpack

Click on Machine, Preferences and select Exentions. Click on the small Blue rhomb with the orange triangle on it on the right side of the window, navigate your way to where you downloaded the extension and install it:

 And this is what it looks like when you're done:

---

2. Set up a new machine
Click on New. Follow the instructions.

A few things to think about:
* Assign a reasonable amount of RAM, whatever that means. I'd suggest 1-1.5 Gb if you have 4 Gb machine. I'm testing 512 Mb in this example to see what the lower requirements are.
* Definitely make your hard drive big enough from the beginning since expanding it isn't that easy.

Here's a series of screen grabs:

 Select Linux, Debian (64 bit if possible)

 Set the amount of RAM. The more the merrier, but leave enough for your OS.

 Use a dynamically expanding HDD.

 10 Gb is a good start, but you'll have to move computational files off of the virtual hard drive periodically. The underlying OS only uses a small amount of storage space, but ECCE -- depending on what you do in it -- can use a lot more.

---

3. Download the iso
Clicking the following URL will hopefully start a download:
http://ftp.nl.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/

If not, then have a look at this page: http://www.debian.org/distrib/netinst and click on AMD64 under Small CDs (180 Mb). This will get you the latest Debian Squeeze, which is a bit old but rock solid. Upgrading to testing is a breeze if you should want to do that later.

---

4. Installing Debian
Select your new machine, and click on settings.

 Click on Storage

 Mount the iso you downloaded

 When all is good it should look like this:

Hit OK.

Two things you'll need to know at this point:

• Right CTRL is your 'host' key. Hit (right) CTRL to release the mouse from the virtual desktop
• Switch between full-screen mode using (right) CTRL+f (host+f)

Now you're ready to start your virtual machine, so click on Start, which brings you to this:

Most of the choices are fairly simple to answer so I won't cover them.
I selected:
Hostname - vecce
Domain name -- (empty)
Pick the mirror country that correspond to the country you're in. As for the actual mirror server, the first choice is normally ok. You can see if your uni is on the list, which will make it even faster.
Don't type anything when it asks for root password (done twice) since it's better (imho) to use sudo.
Our new user is called ecce, and this user will automatically be granted sudo powers.

Once you get to the partitioning, make sure to select Manual, and follow the screenshots below:

 First you need to format your harddrive. Remember, by harddrive we're really talking about a file on your disk, so screwing this up will NOT affect your windows computer in any shape or form.

 Having a Swap partition for a virtual HDD makes absolutely no sense.

Now your base system will install, and you'll eventually get to this screen. De-select the graphical desktop environment since we want to save space and resources -- gnome is wonderful, but demanding. We'll put LXDE on it later.

Install grub to the MBR. Remember, our harddrive is still just a file on your physical harddrive.

That's basically it. Click continue in the screen below, and your virtual system will restart.

 And here's what'll great you on reboot:

You're now ready to log on and do some damage.

5. Customize your debian installation
So log on:

 Make sure you have working internet by e.g. doing
ping -c 3 google.com

Now, type
sudo apt-get install lxde
type your password (you'll get a warning about great reponsibilities etc.), and answer yes to any questions

Once the install has finished, type
sudo shutdown -r now
which will cause a reboot which will bring you to this screen:

I use gnome 3/gnome-shell myself, so I'm not particularly familiar with LXDE, so if anything looks weird in the following steps, that's why.

Log on:

You'll find your terminal (LXTerminal) by clicking on what constitutes the Start Menu down on the left:

First install a web browser (in this case chromium -- essentially a re-branded Chrome):
sudo apt-get install chromium-browser

Start chromium by clicking on the web browser icon in the panel launcher down in the bottom left corner:

6. Installing ECCE
Download ECCE:

Install csh by typing
sudo apt-get install csh 
in the terminal
Then do
cd ~/Downloads
chmod +x install_ecce.v6.3.rhel5-gcc4.1.2-m64
csh install_ecce.v6.3.rhel5-gcc4.1.2-m64

Select Full Install [3]

Change the hostname to localhost
Set the application install path to /home/ecce/ecce-v6.3/apps and the server directory to /home/ecce/ecce-v6.3/server
Then install.

Now, start leafpad:

Open ~/.bashrc (the ~ means it's in your home folder). You do that by clicking on File, Open, then click on file system, home, ecce, hit ctrl+H to show hidden files, and select .bashrc

Add the following to the end of the file:

export ECCE_HOME=/home/ecce/ecce-v6.3/apps
export PATH=$PATH:${ECCE_HOME}/scripts
alias startecceserver='/home/ecce/ecce-v6.3/server/ecce-admin/start_ecce_server'
alias stopecceserver='/home/ecce/ecce-v6.3/server/ecce-admin/stop_ecce_server'

Save, and exit leafpad.

Type
source ~/.bashrc
in your terminal. This will make the terminal read the new settings.
You still can't start since you're missing java:

So, in your terminal, do
sudo apt-get install default-jre

First make sure the ecce server isn't running
stopecceserver
then launch ecce
ecce

You're now being invited to set a new password:

Clicking OK launches what is called the Gateway:

7. Setting up ECCE
You're more than half-way there. The exact set up will depend on what you're trying to achieve and I've documented it elsewhere on this blog.

However, because I'm writing this guide with a couple of specific people in mind, I'll show how to set up a remote ROCKS site to submit to, using node hopping.

In the terminal, type
ecce -admin

 Set up as shown in the screen shot:

Don't forget to hit add/change queue first, then add/change in the bottom left, then close.

Because we're doing a lot of cool, advanced stuff we need to do some manual editing: we need to define the public url, and we need to define our SGE (Sun Grid Engine) settings and finally, we need to set up a couple of environment variables.

In the terminal, do
cd ~/ecce-v6.3/apps/siteconfig/
chmod u+rw CONFIG.rocks
Otherwise the file will be write-protected.

Using leafpad, open the CONFIG.rocks file which is found in /home/ecce/ecce-v6.3/apps/siteconfig/

Add the following (look at the screenshot below):
frontendMachine: rocks.university.edu

SGE {
#$ -S /bin/csh
#$ -cwd
#$ -l h_rt=$walltime
#$ -l h_vmem=$memoryG
#$ -j y
#$ -pe orte $totalprocs    
}

NWChemEnvironment{
            LD_LIBRARY_PATH /usr/lib/openmpi/1.3.2-gcc/lib/
}

NWChemCommand {
        /opt/openmpi/bin/mpirun -n $totalprocs $nwchem $infile > $outfile
}
Gaussian-03Command {
    setenv GAUSS_SCRDIR /tmp
    setenv GAUSS_EXEDIR /share/apps/gaussian/g09/bsd:/share/apps/gaussian/g09/local:/share/apps/gaussian/g09/extras:/share/apps/gaussian/g09
        time /share/apps/gaussian/g09/g09 $infile  $outfile }

Save and exit. (note the $memoryG so you don't put in 4000 but instead use 4 when setting memory during submission in launcher)

In the ECCE gateway, click on Machine Browser

Select rocks and click on Set Up Remote Access, give the user name and password. If all goes well, if you hit e.g. Disk Usage, you should see this:

8. Launching a test calc
Click on Organizer in the Gateway.

If you can't select basis set, theory etc. at this point, close the editor, then open again. Might be because it's the first run ever.

And launch:

 When it's done, it looks like this. Depending on your SGE set-up everything can be very fast, or every step can take a minute. To follow in real-time, click in Viewer. To see the output file, hit Run Mgmt and select tail -f. To look at output fiel from completed run, go to Run Mgmt and select Output.

It's fun, and easy!

9.  Moving files back and forth between your virtual debian and windows
First shut down your virtual machine, then click on settings.

When your virtual machine is running you can access it from e.g. windows using sftp with e.g. Filezilla like this:

---

I hope this guide has been useful. Again, it's long because of the level of detail, not because it's difficult in any way.

Learning how to use ECCE properly is a completely different tutorial. I might put up some examples with increasing complexity in the future.

172. ECCE and a ROCKS cluster: step by step

This is quite similar to a recent post, but here's a step-by-step, detailed account of how to set up ECCE for remote job submission to a ROCKS 5.4.3 cluster (one front node, 4 subnodes)

Coming soon (give it a week): Setting up a virtualbox machine with ecce for (stubborn) windows and ROCKS/CentOS users.

What isn't shown are all the failed attempts and dead-ends I went through and encountered getting to the point where I had a working system. I compiled ECCE. I compiled tcsh. I tried compiling bsd csh, which required me to compile bmake etc. This stuff looks simple, and it is simple -- but not obvious.

NOTE: From the outside we connect to rocks.university.edu. From inside the cluster the submit node is called rocks.local, and the subnodes are called node0, node1, etc. Refer to this naming if you get confused late.

Step 1. Create the site in ecce
From the terminal, do
ecce -admin
and add a new machine

Don't forget to hit Add/Change queue to make the changes to the queue part take effect. Then hit Add/Change. Oh, and pay attention to the Allocation Account tick box - if it's ticked you can't submit anything unless you add an account.  Important: the machine name you add here is the local name or local IP of the submit node -- it's not the 'public' name or url. We'll add that somewhere else later. Don't forget to select the queue manager (I forgot in the screen shot)

Close.

2. Editing your CONFIG file
Since you're already in the terminal, go to ecce-v6.3/apps/siteconfig

Take a quick peek at your Machines file (no editing):

Machines line
rocks rocks.local Dell beo Intel 40:5 ssh :NWChem:Gaussian-03 MN:RD:SD:UN:PW:Q:TL

Take another look at rocks.Q -- there's probably nothing to edit here either:

rocks.Q# Queue details for rocks
Queues:    nwchem
nwchem|minProcessors:       1
nwchem|maxProcessors:       40
nwchem|runLimit:       100000
nwchem|memLimit:       0
nwchem|scratchLimit:       0

Finally, do some editing of your CONFIG.rocks file.

CONFIG.rocks

NWChem: /share/apps/nwchem/nwchem-6.1/bin/LINUX64/nwchem
Gaussian-03: /share/apps/gaussian/g09/g09
perlPath: /usr/bin/
qmgrPath: /opt/gridengine/bin/lx26-amd64
sourcefile: /home/rocksuser/.cshrc
frontendMachine: rocks.university.edu

SGE {
#$ -S /bin/csh
#$ -cwd
#$ -l h_rt=$walltime
#$ -l h_vmem=$memoryG
#$ -j y
#$ -pe orte $totalprocs  
}

NWChemEnvironment{
            LD_LIBRARY_PATH /usr/lib/openmpi/1.3.2-gcc/lib/
}

NWChemCommand {
        /opt/openmpi/bin/mpirun -n $totalprocs $nwchem $infile > $outfile
}
Gaussian-03Command {
    setenv GAUSS_SCRDIR /tmp
    setenv GAUSS_EXEDIR /share/apps/gaussian/g09/bsd:/share/apps/gaussian/g09/local:/share/apps/gaussian/g09/extras:/share/apps/gaussian/g09
        time /share/apps/gaussian/g09/g09 $infile  $outfile }

Obviously, your variables will be different. NOTE that memory is in gigabyte here. You could also do $memoryM for megabyte. Just adjust your launcher requirements accordingly.

Step 3. Making csh modifications on the ROCKS cluster
On the main node just use the root password (or become sudo) and move /etc/csh.cshrc and /etc/csh.login out of the way (backing them up is a good idea). It doesn't seem like you need to make any changes csh-wise to the subnodes.

Step 4. Finalising our set up
Start ecce the normal way (e.g. run ecce from the terminal)
In the Gateway, start the Machine Browser, highlight 'rocks' and click on Setup Remote Access.

Do what you're told.

Step 5. Submit to your heart's content!

NOTE: the option to set the amount of memory is not shown in the launcher window above: my mistake. You can edit you apps/siteconfig/Machines file and add :MM at the end of the line, e.g.

Dynamic beryllium       Unspecified     Unspecified     Unspecified     18:3    ssh     :NWChem:Gaussian-03     MN:RD:SD:UN:PW:Q:TL:MM

171. Building ECCE on ROCKS/CentOS

I installed ECCE on a couple of a single workstation with ROCKS, and remotely on a 40 core cluster with ROCKS. The local, workstation install worked fine. I never really bothered much about the cluster install, and only recently looked closer at it. Well, I can launch the 'gateway' but nothing else -- when I click on e.g. the organizer button I get the rocks version of an hourglass that never goes away -- and I don't get any error messages. Turning on logging doesn't yield anything either. 

Ergo, I figured that building it myself  may yield a different result. It didn't on the ROCKS cluster, but everything worked just fine on the single-node ROCKS training box I keep in my office.

---

CentOS is a bit dated, so you'll need to build your own apr and apr-util. Build apr:

cd /share/apps/utils/

wget http://mirror.mel.bkb.net.au/pub/apache//apr/apr-1.4.6.tar.gz

wget http://mirror.mel.bkb.net.au/pub/apache//apr/apr-util-1.4.1.tar.gz

tar xvf apr-1.4.6.tar.gz

cd apr-1.4.6/

./configure --prefix=/share/apps/utils/apr

make

make install

cd ../

tar xvf apr-util-1.4.1.tar.gz

cd apr-util-1.4.1/

./configure --prefix=/share/apps/utils/apr-util --with-apr=/share/apps/utils/apr/

---

Time for ecce.

First download 

cd /share/apps/ecce/

tar xvf ecce-v6.3-src.tar.bz2

cd ecce-v6.3/

export ECCE_HOME=/share/apps/ecce/ecce-v6.3

cd build/

Edit build_ecce

889       ./configure --prefix=$ECCE_HOME/${ECCE_SYSDIR}3rdparty/httpd --enable-rewrite --enable-dav --enable-ss-compression

to

889       ./configure --prefix=$ECCE_HOME/${ECCE_SYSDIR}3rdparty/httpd --enable-rewrite --enable-dav --enable-ss-compression --with-apr=/share/apps/utils/apr/bin/apr-1-config --with-apr-util=/share/apps/utils/apr-util/bin/apu-1-config

./build_ecce

Just follow the instructions i.e. hit return, over and over again. Answer no to running tests again. Then run build_ecce again:

./build_ecce

Now stuff should be building. Do this another six times. From the README:

"At this stage the script will build one 3rd party package per invocation,
exiting after each package is built.  In order the 3rd party packages that
will be built are:
1. Apache Xerces XML parser
2. Mesa OpenGL
3. wxWidgets C++ GUI toolkit
4. wxPython GUI toolkit
5. Apache HTTP web server"

The httpd build ends with a minor error about "lib" missing. It's fine.

The sixth time ECCE itself is built, and that's the step that takes by far the longest. It finishes with:

 ECCE built and distribution created in /share/apps/ecce/ecce-v6.3

On a single-node desktop I got it to run a seventh time it seemed. The last step finished with the message above though.

Go to your /share/apps/ecce/ecce-v6.3/ dir where you'll find install_ecce.v6.3.csh

Do the install
csh -f install_ecce.v6.3.csh
Follow the instructions.

You may also want to
sudo mv /etc/csh.* ~/
to get rid of the crappy csh config files.

Edit your ~/.bashrc:

alias startecceserver='csh -f /share/apps/ecce/ecce-v6.3/server/ecce-admin/start_ecce_server'
alias stopecceserver='csh -f /share/apps/ecce/ecce-v6.3/server/ecce-admin/stop_ecce_server
export ECCE_HOME=/share/apps/ecce/ecce-v6.3/apps
export PATH=$PATH:${ECCE_HOME}/scripts

and your ~/.cshrc:

setenv ECCE_HOME /share/apps/ecce/ecce-v6.3/apps
set PATH= (/share/apps/nwchem/nwchem-6.1/bin/LINUX64 $PATH)

On my single-node box I had to edit the apps/siteconfig/DataServers and replace eccetera.emsl.pnl.gov with localhost (two instances), as well as the apps/siteconfig/jndi.properties file (one instance).

In spite of the hassle on the single node box, everything works there -- the builder, organizer etc. all open just fine. The rocks cluster, looks fine, but doesn't work.

---

The ROCKS Cluster:
Everything seems to work fine -- starting ecce launches the gateway, but clicking on anything sees the centos version of the hourglass churn over and over for all eternity. Nothing happens.

I looked through these two threads, and i also tried the pre-built 32 bit binary. All without luck.

I've also tried editing the site_runtime file:

ECCE_MESA_OPENGL true
ECCE_MESA_EXCEPT x86_64:RedHat:Fedora:CentOS

(matches the lsb_release -is output)

170. tcsh in ROCKS/CentOS with hardcoded csh.cshrc path

WHAT THIS POST DOES: It shows you how to compile your own tcsh which won't be looking at /etc/csh.cshrc. It doesn't show you how to set up the correct .cshrc files. But it certainly allows you to experiment.

Also, keep in mind that since each local node hdd has it's own /bin directory (not exported) you need to make similar changes on each node (i.e. change the /bin/csh symlink -- see below)

(The) csh (startup files) is(are) horribly broken on ROCKS 5.4.3.

For now I've solved it by just moving  /etc/csh.cshrc out of the way, but what we do here is symlink /bin/csh to our own tcsh which been hardcoded to use a non-standard configuration file, so that you can use the standard ROCKS tcsh with /etc/csh.cshrc and your own csh(tcsh) with your own config files.

To be clear: it's not the csh binary which is borked on ROCKS 5.4.3, but the configuration files.

There's a patch for the broken csh -- but when I applied it to a test computer it only got broken-er and prevented the csh from opening and staying open. Good way of getting locked out. So I'm not keen on doing the same thing on someone else's production cluster. Also, I've opted for tcsh since the csh sources come with a bsd style makefile, and I really can't deal with that right now.

What we'll do is hardcode the location of the csh.cshrc file and change it from /etc/csh.cshrc to /share/apps/utils/tcsh

sudo mkdir /share/apps/utils
sudo chown ${USER} /share/apps/utils
cd /share/apps/utils
 wget http://ftp.de.debian.org/debian/pool/main/t/tcsh/tcsh_6.18.01.orig.tar.gz
tar xvf tcsh_6.18.01.orig.tar.gz
 cd tcsh-6.18.01/

Time for find out what to change:
tail -n 9999 *|strings|egrep "/etc/csh.cshrc|<=="


tells us we need to have a look at pathnames.h
Change

124 # define _PATH_DOTCSHRC     "/etc/csh.cshrc"
to
124 # define _PATH_DOTCSHRC     "/share/apps/utils/custom.tcshrc"

./configure --prefix=/share/apps/utils/tcsh
make
make install

If all went well:
cat tcsh|strings|grep custom.tcsh

/share/apps/utils/custom.tcshrc

and

tree /share/apps/utils/tcsh -L 1
/share/apps/utils/tcsh
|-- bin
`-- share

Obviously, this doesn't really make much of a difference just yet. Now comes the scary part -- and you need root access for this:
 which csh

/bin/csh

ls /bin/csh -lah

lrwxrwxrwx 1 root root 4 Feb 23 16:54 /bin/csh -> tcsh

and here's the 'dangerous' stuff:
sudo rm /bin/csh
sudo ln -s /share/apps/utils/bin/tcsh /bin/csh

sudo chown root:root /bin/csh
sudo chmod 777 /bin/csh

Since /bin/csh isn't a binary but a symmlink to tcsh in the /bin directory, we just delete the symlink and create a new one.

We can now make whatever changes we want to our custom.tcshrc while still being able to easily change back to the old setup. I do recognise that we could just have edited /etc/csh.cshrc and /login.cshrc, but I for some reason feel a lot more comfortable using this method.

169. ECCE, node hopping and cluster nodes without direct access to WAN

NOTE: I'm still working on this. But this sort of works for now. More details coming soon.
Update:  I've posted better solutions more recently for use with SGE. See aspects of e.g. http://verahill.blogspot.com.au/2012/06/ecce-in-virtual-machine-step-by-step.html and http://verahill.blogspot.com.au/2012/06/ecce-and-inaccessible-cluster-nodes.html

It's not easy choosing a good title for this post which describes the purpose of it succinctly yet clearly (sort of like this sentence), so here's what we're dealing with:

The Problem:
You can access the submit node from off-site. You can't access the subnodes directly from off-site. This post shows you how you can submit to each subnode directly. A better technical solution is obviously to use qsub on the main node. Having said that, with very little modification this method can also be adapted to the situation described here: http://verahill.blogspot.com.au/2012/05/port-redirection-with-eccenwchem.html

A low-tech, home-built example is the following:
-eth0 - Main  node - eth1 - (subnodes: node0, node1, node2, node3)/

eth0 is WAN, and eth1 is LAN. You can ssh to the main node from the 'internet'. There's no queue manager like SGE installed -- submission of jobs is done by logging onto each node and executing the commands there.

The example:

Main node:
    my.cluster.edu
Nodes:
   compute-0-0
   compute-0-1
   compute-0-2
   compute-0-3

---

The most obvious solution:
do port redirection. The downside is that it requires some technical skills of the users, and anything with networking and ssh is a PITA if they insist on using windows.

---

The smarter solution:
I was informed of this solution here.

ROCKS-specific stuff
If your cluster is running ROCKS 5.4.3 and you're having issues opening csh shells, just move /etc/csh.cshrc out of the way as a crude fix.
sudo mv /etc/csh.cshrc ~/
Don't forget to do this on all the nodes if they have local /etc folders! And it's not that easy -- the passwords aren't the same on the nodes as on the main node.
So, on the main node:
rocks set host sec_attr compute-0-1 attr=root_pw
sudo su
# cat /etc/hosts
#ssh 10.1.255.253
#mv /etc/csh.cshrc .
#exit

As the user you'll be running as, edit/create your ~/.cshrc

setenv GAUSS_SCRDIR /tmp
setenv ECCE_HOME /share/apps/ecce/apps
set PATH= (/share/apps/nwchem/nwchem-6.1/bin/LINUX64 $PATH)
setenv LD_LIBRARY_PATH /opt/openmpi/lib:/share/apps/openblas/lib

Repeat on all nodes.

SIDE NOTE: What I'm ultimately looking to achieve on the ROCKS cluster is front-node managed SGE submissions. Easy, you say? Well, ECCE submits a

setenv PATH /opt/gridengine/bin/lx26-amd64/:/usr/bin/:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/bin/X11:${PATH}

and for some reason the ROCKS/CentOS (t)csh can't handle (most of the time) adding ${PATH} to itself since it's too long unless "s are added (it seems) i.e. this works consistently:

setenv PATH "/opt/gridengine/bin/lx26-amd64/:/usr/bin/:/bin:/usr/sbin:/sbin:/usr/X11R6/bin:/usr/bin/X11:${PATH}"

On my debian system either works fine (bsd-csh).

ANYWAY -- continue

Ecce:
Start setting it up using
ecce -admin

After that, do some editing in ecce-6.3/apps/siteconfig/

CONFIG.voldemort:

xappsPath: /usr/X11R6/bin
NWChem: /share/apps/nwchem/nwchem-6.1/bin/LINUX64/nwchem
Gaussian-03: /share/apps/gaussian/g09/g09
sourceFile:  /etc/csh.login
frontendMachine: my.cluster.edu

Machines:

compute-0-0 compute-0-0.local Dell HPCS3 Phase 1 Cluster AMD 2.6 GHz Opteron 40:5 ssh :NWChem:Gaussian-03MN:RD:SD:UN:PW

170. Compiling PVM and XPVM on ROCKS 5.4.3

And we're back to ROCKS again.

NOTE: I haven't actually tested the binaries and libs compiled here. I think they should work. But I don't know for sure.

PQS works with openmpi, mpich and PVM. Our vanilla ROCKS install already has openmpi and mpich. There's a package called rocks-pvm, but the size is 50 kb and didn't seem to actually install anything precompiled, so I removed it and decided to go the compilation way instead.

The paths here are specific to the cluster I did this on, so customise as needed.

sudo mkdir /share/apps/pvm
sudo chown ${USER} /share/apps/pvm
cd /share/apps/pvm
wget http://www.netlib.org/pvm3/pvm3.4.6.tgz
tar xvf pvm3.4.6.tgz
cd pvm3/
export PVM_ROOT=`pwd`
make

Time to set up environment variables. Either edit /etc/profile or ~/.bashrc, depending on powers and reach., and add

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:share/apps/pvm/pvm3/lib/LINUX64
export PATH=$PATH:/share/apps/pvm/pvm3/bin/LINUX64
export PVM_ROOT=/share/apps/pvm/pvm3

Changes won't take effect until you source the file, or open a new terminal.


I profess to be ignorant about how to actually use pvm, so no testing just yet.

---

So I also stumbled across xpvm, which sounds (and looks) neat.

wget http://www.netlib.org/pvm3/xpvm/XPVM.src.1.2.5.tgz
cd /share/apps/pvm
tar xvf XPVM.src.1.2.5.tgz
cd xpvm/

Time to do some housekeeping before compiling:
It requires:
1. PVM 3.3.0 or later.
2. TCL 7.3 or later.
3. TK 3.6.1 or later.

I find
/usr/share/tk8.4
/usr/share/tcl8.4
so I might be ok. We just compiled pvm 3.4.6, so it should be alright.

First figure out where stuff is:

locate libtk|grpe so

/usr/lib/libtk.so
/usr/lib/libtk8.4.so
/usr/lib64/libtk.so
/usr/lib64/libtk8.4.so

locate libtcl|grep so

/usr/lib/libtcl.so
/usr/lib/libtcl8.4.so
/usr/lib64/libtcl.so
/usr/lib64/libtcl8.4.so
/usr/lib64/tclx8.4/libtclx8.4.so

These are fairly standard locations, so they should already be searched by ld -- no need to specify them thus.

Include is potentially worse since they'd typically need the development packages.
locate tcl | grep "\.h"

[..]
/usr/include/tcl-private/generic/tcl.h
[..]

locate tk|grep "\.h"

[..]
/usr/include/tk-private/generic/tk.h
[..]

So we /should/ be fine.

We also need the X11 libs and headers:
locate libX11

/usr/lib/libX11.so
/usr/lib/libX11.so.6
/usr/lib/libX11.so.6.2.0
/usr/lib64/libX11.so
/usr/lib64/libX11.so.6
/usr/lib64/libX11.so.6.2.0

locate X11|grep include

[..]
/usr/include/X11
[..]

Finally,

locate libdl

/lib/libdl-2.5.so
/lib/libdl.so.2
/lib64/libdl-2.5.so
/lib64/libdl.so.2
/usr/lib/libdl.a
/usr/lib/libdl.so
/usr/lib64/libdl.a
/usr/lib64/libdl.so

I'll specify the lib locations even though in some of these particular cases it isn't necessary:

Edit xpvm/src/Makefile.aimk and set (line numbers added by me):
19  PVMVERSION = -DUSE_PVM_34
Comment out line 42:
 42 #TCLTKHOME  =  $(HOME)/TCL
and
 44 TCLTKHOME  =   /usr/include
Change

 47 TCLINCL     =   -I$(TCLTKHOME)/tcl-private/generic
 48 TKINCL      =   -I$(TCLTKHOME)/tk-private/generic
and

 57 TCLLIBDIR   =   -L/usr/lib64/tclx8.4
 58 TKLIBDIR    =   -L/usr/lib64
and

 70 TCLLIB      =   -ltcl8.4

 71 TKLIB       =   -ltk8.4

and

83 XINCL       = -L/usr/include/X11

84 XLIBDIR     = -L/usr/lib64

and finally,

 96 SHLIB       = -ldl

Fell asleep? Time to get compiling.

export XPVM_ROOT=/share/apps/pvm/xpvm

export TCL_LIBRARY=

/usr/share/tcl8.4

export TK_LIBRARY=/usr/share/tk8.4

cd ${XPVM_ROOT}
make

[..]
Installing xpvm.tcl
Installing globs.tcl
Installing procs.tcl
Installing util.tcl
make[1]: Leaving directory `/share/apps/pvm/xpvm/src/LINUX64'

The beautiful thing is that the xpvm binary automagically ends up in the pvm3/bin/LINUX64 directory, so no need to fiddle with path.



In theory everything should work now if you log in with ssh -XC. However I get
xpvm

libpvm [pid2607] /tmp/pvmd.502: No such file or directory
libpvm [pid2607]: Can't Start PVM: Can't start pvmd

I'm not actually running -- nor have I ever run -- anything with pvm.

touch /tmp/pvmd.502

xpvm

libpvm [pid4219]: mksocs() read addr file: wrong length read
Connecting to PVMD already running... libpvm [pid4219]: mksocs() read addr file: wrong length read
libpvm [pid4219]: mksocs() read addr file: wrong length read
libpvm [pid4219]: mksocs() read addr file: wrong length read
libpvm [pid4219]: pvm_mytid(): Can't contact local daemon
libpvm [pid4219]: Error Joining PVM: Can't contact local daemon

I mean, it looks like it should work, once pvm is being used.

169. ECCE (PNNL/EMSL) and documentation

The title makes it sound like this is going to be a very dry and boring post, but that's not the intention.

Any current or fledgling user of ECCE will probably agree with me when I say that ECCE is somewhat under-documented (there are practical and understandable reasons for this. Besides, no-one like writing documentation). Most of the documentation which does exist is written for earlier versions of ECCE which used a different work flow. Other features are completely undocumented or actually finding the information is difficult.

As it turns out a lot of the technical documentation is to be found in the ecce-6.3/apps/siteconfig/ directory as many of the settings files are self-documenting. Another thing that has been pointed out to me is that the release notes are paid extra attention to because of the lack of a large, coherent body of documentation.

In particular look at:
site_runtime
Some minor settings you mostly won't have to bother with. You may want to look at the OPENGL settings though.
Also, if you're trouble-shooting you can uncomment the ECCE_RCOM_LOGMODE line

# send remote communications output to the console for diagnosing problems
#ECCE_RCOM_LOGMODE true

This allows ECCE to echo every remote command it executes.

submit.site
A treasure trove of ideas and information. Anything you set here will be the default. Defaults can be overriden in the site-specific CONFIG.<> files. It's a judgement call whether to put a specific setting here or in the CONFIG.<> files. Personally, I always change to

NWChemCommand {
  mpirun -n $totalprocs  $nwchem $infile > $outfile
}

as a default setting here.

remote_shells.site
Editing this can allow you to adapt to restrictions in what kind of shells you can run on the remote site or node. The documentation here is a bit iffier.

CONFIG-Examples/ 
Examples of configurations, as the name would suggest. It could do with an overview of the different sites and situations in the README file though.

168. Port redirection with ECCE/nwchem

Update 1/6/2012: There may be an alternative, better way of doing this: "Another feature that may or may not be useful to you with this special node that is setup with a higher ulimit for submitting your ECCE jobs is that ECCE has a "hop" feature that lets it go from a main login node on a machine to other nodes before actually running commands (e.g. submitting jobs). If you look at the $ECCE_HOME/siteconfig/CONFIG-Examples/CONFIG.mpp2 file, you'll see this "frontendMachine" directive that is what is used to do this. I'm thinking this might allow you to skip the port redirect options with ssh and just "hop" to your special node from regular login node on the compute host. But, I don't think I'd worry about it if what you have now is working fine."
What I describe below works fine, but it does require that users are able to set up the port redirect themselves. The solution hinted at above would be a better technical solution for a larger group of users with varying technical abilities since it should be enough to copy CONFIG.<> and <>.Q files.
Here's how to use main-subnode hopping: http://verahill.blogspot.com.au/2012/06/ecce-and-inaccessible-cluster-nodes.html

Original post:

If you for some reason can't connect to your computational cluster on port 22, here's how to sort it out.

---

Setting it up the first time

Adding a node:

I have access to an idle computer off-campus, which sits behind a trusty old linksys router. To access it directly I need to use port forwarding.

Edit  ecce-6.3/apps/siteconfig/remote_shells.site and add

ssh_p9999: ssh -p 9999|scp -P 9999

The miniscule p for ssh and capital P for scp are important. They do the same thing, but the different programmes expect difference cases. 

In the ECCE gateway (the small floating window with icons that start when you start ECCE) select Machine Browser. Under Machine, select Register Machine:

The localhost bit is the key here -- since you'll be doing port redirect you'll formally connect to a port on localhost. Hit Add/Change.
In the main Machine Browser window, highlight oxygen and click on Setup Remote Access. Your ssh_p9999 thingy should show up there. Don't bother testing anything just yet (i.e. Machine status etc.). Since I'm writing this from memory I don't know whether you need to have the port-redirect active at this point or not. If you do, see below under Running.

It really is that simple. See below for how to actually use it.

Adding a remote site:

I work at an Australian university where I appear to be the only person using ECCE. Thus, while ulimit on the local SGE managed cluster is a meagre 32 procs, this hasn't been a problem up till now. However, ECCE launches five procs per job, so using up my procs allocation I've been locked out of the cluster on a regular basis.

As a solution, I've been offered my very own shiny submit node with a heftier 37k procs allowed. The downside is that it's only accessible from the standard submit node. Luckily, it's not much more difficult than doing port redirects for a remote node.

Edit  ecce-6.3/apps/siteconfig/remote_shells.site and add

ssh_p5454: ssh -p 5454scp -P 5454

The miniscule p for ssh and capital P for scp are important. 

In the terminal, run
ecce -admin

Most of the fields above should fairly self-explanatory. A few things to watch out for:

• ECCE actually looks at the proc to node ratio and will impose strict limitations on the number of cores you can use per node. 50/20 means that if you want to use four cores ECCE forces the use of two nodes. Depending on how you run your jobs (configuration) this may or may not have any real impact. To be safe, pick something like 700 cores and 20 nodes.
• Path means path. I think ECCE defaults to giving the perl path as /usr/bin/perl, but it should be /usr/bin. Same goes for the qsub path.
• You need to create a queue. The queue name isn't used anywhere other than in ECCE, so it can be a smart way of setting up defaults. What I'm saying is: it's not that important to get it 'right' since it bears no relation to anything on your cluster.

Click on close.

In 'regular' ecce (i.e. started without -admin) go to the machine browser window, highlight the added site, hit Set up Remote Access, and pick ssh_p5454 as shown below. Don't bother testing anything just yet (e.g. Machine status etc.). Since I'm writing this from memory I don't know whether you need to have the port-redirect active at this point or not. If you do, see below under Running.

As always, setting up a site takes a bit of customisation. Here's my ecce-6.3/apps/siteconfig/CONFIG.gn54 on my ecce workstation.

NWChem: /opt/sw/nwchem-6.1/bin/nwchem
Gaussian-03: /usr/local/bin/G09
perlPath: /usr/bin
qmgrPath: /opt/n1ge62/bin/lx24-amd64
SGE {
#$ -S /bin/csh
#$ -cwd
#$ -l h_rt=$wallTime
#$ -l h_vmem=4G
#$ -j y
}

NWChemEnvironment{
    LD_LIBRARY_PATH /usr/lib/openmpi/1.3.2-gcc/lib/
    PATH /opt/n1ge62/bin/lx24-amd64/
}
NWChemCommand {
#$ -pe mpi_smp$totalprocs  $totalprocs
module load nwchem/6.1
mpirun -n $totalprocs $nwchem $infile > $outfile
}
Gaussian-03Command {
#$ -pe g03_smp4 4
module load gaussian/g09
time G09< $infile > $outfile }

and here's my ecce-6.3/apps/siteconfig/gn54.Q  on my ecce workstation.

# Queue details for gn54
Queues:    nwchem squ8
nwchem|minProcessors:       1
nwchem|maxProcessors:       8
nwchem|runLimit:       4320
nwchem|memLimit:       4000
nwchem|scratchLimit:       0
squ8|minProcessors:       1
squ8|maxProcessors:       6
squ8|runLimit:       4320
squ8|memLimit:       4000
squ8|scratchLimit:       0 

Finally, you need to make sure that  nwchem can find everything - put a file called .nwchemrc in your home folder on the remote node with the correct paths in it, e.g.

nwchem_basis_library /opt/sw/nwchem-6.1/data/libraries/
nwchem_nwpw_library /opt/sw/nwchem-6.1/data/libraryps/
ffield amber
amber_1 /opt/sw/nwchem-6.1/data/amber_s/
amber_2 /opt/sw/nwchem-6.1/data/amber_q/
amber_3 /opt/sw/nwchem-6.1/data/amber_x/
amber_4 /opt/sw/nwchem-6.1/data/amber_u/
spce /opt/sw/nwchem-6.1/data/solvents/spce.rst
charmm_s /opt/sw/nwchem-6.1/data/charmm_s/
charmm_x /opt/sw/nwchem-6.1/data/charmm_x/

That's it.

---

Running

Starting port redirect:
Before you can pipe anything through to your supersecure remote node/remote site, you need to open a teminal window and do

ssh -C username@remoteserver -L 9999:hiddenserver:22

for the remote node above, or

ssh -C username@remoteserver -L 5454:hiddenserver:22

for the remote site above.

Just to make the syntax clear:
Remote node:
My linksys route is at IP address 110.99.99.99, and the remote node behind it is at 192.168.1.106. My username is verahill

ssh -C verahill@110.99.99.99 -L 9999:192.168.1.106:22

Remote site:
The standard submit node is called msgln4.university.au, and the hidden node is called gn54.university.au. My username is lindqvist

ssh -C lindqvist@msgln4.university.au -L 5454:gn54.university.au:22

You may need to install and use autossh if you keep on being booted off due to inactivity! The syntax is identical.

In ECCE:
Everything in ECCE now works exactly like before - just select the target computer/site/node and go.

---

What doesn't work:
So far I haven't been able to sort out the whole 'open' remote terminal, which means that tail -f output doesn't work either. I'm leaving that for a rainy day with too much time on my hands.

167. ECCE/Nwchem on An Australian University computational cluster using qsub with g09/nwchem

EDIT:
I've just learned the First Rule of Remote Computing:
always start by checking the number of concurrent processes you're allowed on the head node, or you can lock yourself out faster that you can say "IT support'.

do
ulimit -u
If it's anywhere under 1000, then you need to be careful.
Default ulimit on ROCKS: 73728
Default ulimit on Debian/Wheezy:  63431
Ulimit on the Oz uni cluster: 32

ECCE launches FIVE processes per job.
Each pipe you add to a command launches another proc. Logging in launches a proc -- if you've reached your quota, you can't log in until a processes finishes.

cat test.text|sed 's/\,/\t/g'|gawk '{print $2,$3,$4}' 
yields three processes -- ten percent of my entire quota.

NOTE:
Running something on a cluster where you have limited access is very different from a cluster you're managing yourself. Apart from knowing the physical layout, you normally have sudo powers on a local cluster.

On potential issue is excessive disk usage -- both in terms of storage space and in terms of raw I/O (writing to an nfs mounted disk is not efficient anyway)
So in order to cut down on that:
1. Define a scratch directory using e.g. (use the correct path)
scratch_dir /scratch
The point being that /scratch is a local directory on the execution node

2. Make sure that you specify

dft
     direct
     ..
end

or even

dft
    noio
    ...
end

to do as little disk caching as possible.

I accidentally ended up storing 52 GB of aoints files from a single job. It may have been what locked me out of the submit node for three hours...

A good way to check your disk-usage is
ls -d * |xargs du -hs

Now, continue reading:

---

Setting everything up the first time:
First figure out where the mpi libs are:
qsub.tests:

#!/bin/sh
#$ -S /bin/sh
#$ -cwd
#$ -l h_rt=00:14:00
#$ -l h_vmem=4G
#$ -j y
locate libmpi.so

Assuming that the location is /usr/lib/openmpi/1.3.2-gcc/lib/, put 

export LD_LIBRARY_PATH=/usr/lib/openmpi/1.3.2-gcc/lib/

in your ~/.bashrc

Next, look at ls /opt/sw/nwchem-6.1/data -- if there's a default.nwchemrc file, then
ln -s /opt/sw/nwchem-6.1/data/default.nwchemrc ~/.nwchemrc

If not, create ~/.nwchemrc with the locations of the different basis sets, amber files and plane-wave sets listed as follows:

nwchem_basis_library /opt/sw/nwchem-6.1/data/libraries/
nwchem_nwpw_library /opt/sw/nwchem-6.1/data/libraryps/
ffield amber
amber_1 /opt/sw/nwchem-6.1/data/amber_s/
amber_2 /opt/sw/nwchem-6.1/data/amber_q/
amber_3 /opt/sw/nwchem-6.1/data/amber_x/
amber_4 /opt/sw/nwchem-6.1/data/amber_u/
spce /opt/sw/nwchem-6.1/data/solvents/spce.rst
charmm_s /opt/sw/nwchem-6.1/data/charmm_s/
charmm_x /opt/sw/nwchem-6.1/data/charmm_x/

---

Using nwchem:
A simple qsub file would be:

#!/bin/sh
#$ -S /bin/sh
#$ -cwd
#$ -l h_rt=00:14:00
#$ -l h_vmem=4G
#$ -j y
#$ -pe orte 4
module load nwchem/6.1
time mpirun -n 4 nwchem  test.nw > nwchem.out

with test.nw being the actual nwchem input file which is present in your cwd (current working directory).

---

Using nwchem with ecce:
This is the proper way of using nwchem. If you haven't already, look here: http://verahill.blogspot.com.au/2012/05/setting-up-ecce-with-qsub-on-australian.html

Then edit your  ecce-6.3/apps/siteconfig/CONFIG.msgln4  file:

NWChem: /opt/sw/nwchem-6.1/bin/nwchem
Gaussian-03: /usr/local/bin/G09
perlPath: /usr/bin/perl
qmgrPath: /usr/bin/qsub

SGE {
#$ -S /bin/csh
#$ -cwd
#$ -l h_rt=$wallTime
#$ -l h_vmem=4G
#$ -j y
}

NWChemFilesToDelete{ core *.aoints.* }

NWChemEnvironment{
    LD_LIBRARY_PATH /usr/lib/openmpi/1.3.2-gcc/lib/
}

NWChemCommand {
#$ -pe mpi_smp4  4
module load nwchem/6.1

mpirun -n $totalprocs $nwchem $infile > $outfile
}

Gaussian-03Command {
#$ -pe g03_smp4 4
module load gaussian/g09

time G09< $infile > $outfile }

Gaussian-03FilesToDelete{ core *.rwf }

Wrapup{
find /scratch/* -name "*" -user $USER |xargs -I {} rm {} -rf
}

And you should be good to go. IMPORTANT: don't copy the settings blindly -- what works at your uni might be different from what works at my uni. But use the above as an inspiration and validation of your thought process. The most important thing to look out for in terms of performance is probably your -pe switch.

---

Since I'm having problems with the low ulimit, I wrote a small bash script which I've set to run every ten minutes as a cronjob. Of course, if you've used up your 32 procs you can't run the script...also, instead of piping stuff right and left (each pipe creates another fork/proc) I've written it so it dumps stuff to disk. That way you have a list over procs in case you need to kill something manually:

 The script: ~/clean_ps.sh

date 
ps ux>~/.job.list
ps ux|gawk 'END {print NR}'

cat ~/.job.list|grep "\-sh \-i">~/.job2.list
cat ~/.job2.list|gawk '{print$2}'>~/.job3.list
cat ~/.job3.list|xargs -I {} kill -15 {}

cat ~/.job.list|grep "echo">~/.job4.list
cat ~/.job4.list|gawk '{print$2}'>~/.job5.list
cat ~/.job5.list|xargs -I {} kill -15 {}

cat ~/.job.list|grep "notty">~/.job6.list
cat ~/.job6.list|gawk '{print$2}'>~/.job7.list
cat ~/.job7.list|xargs -I {} kill -15 {}

cat ~/.job.list|grep "perl">~/.job8.list
cat ~/.job8.list|gawk '{print$2}'>~/.job9.list
cat ~/.job9.list|xargs -I {} kill -15 {}

qstat -u ${USER} 
ps ux |gawk 'END {print NR}' 
echo "***" 

and the cron job is set up using
crontab -e
 */10 * * * * sh ~/clean_ps.sh>> ~/.cronout

Obviously this kills any job monitoring from the point of view of ecce. However, it keeps you from being locked out. You can manually check the job status using qstat -u ${USER}, then reconnect when a job is ready. Not that convenient, but liveable.

166. Briefly: nvidia API mismatch on debian when running ecce

UPDATE: There's a much better way to do this: "One thing that I did notice is your issues with OpenGL where you suggested moving the shared libraries to another directory. While that's perfectly workable, this would be another instance where consulting the $ECCE_HOME/siteconfig/site_runtime file would be useful. There you would learn about the $ECCE_MESA_OPENGL and $ECCE_MESA_EXCEPT variables that control whether to use the ECCE-supplied GL libraries or native ones (e.g. hardware OpenGL card drivers) on your machine." I'll update this post again when I've had a time to look into it. Lecture slides and grant rejoinders don't write themselves...

Original post:
If you get an error along the lines of this:
 http://www.linuxquestions.org/questions/debian-26/api-mismatch-nvidia-kernel-module-871115/
only when you're running ECCE i.e. there's an API mismatch error with a difference in kernel module version vs the nvidia driver component (in my case 295.49 and 290.10, respectively), thenyou may want to have a look in your apps folder before you launch a major investigation, e.g.

ecce-6.3/apps/rhel5-gcc4.1.2-m64/3rdparty/mesa/liblibGL.so    libGL.so.295.49  libGLU.so.1           libnvidia-glcore.so.295.49
libGL.so.1  libGLU.so        libGLU.so.1.3.071100  libnvidia-tls.so.295.49

You can symmlink the correct drivers, or -- which is even easier -- just move your 3rdparty/mesa library to e.g.3rdparty/bakmesa and see if it solves it

165. Approach to computing reorganisational energies using nwchem

I set out to reproduce Malagoli and Brédas in Chemical Physics Letter, 2000, 327, 13-17 (Link). Essentially it's a paper on calculating reorganisational energies in a few simple organic species, such as biphenyl.  I've already covered this work to some extent (here: http://verahill.blogspot.com.au/2012/05/dft-gridsize-ecce-defaults-to-medium.html ) but here's a step-by-step walkthrough and nwchem:

Approach (using biphenyl):
1. Draw biphenyl -- the dihedral angle between the two rings should be about 38 degrees
2. Optimize structure using b3lyp/6-31+g in gas phase. Gives you E1
3. Change the charge and multiplicity to +1 and 2, respectively.
4. Calculate single-point energy using the previously optimised structure (from step 2, i.e. don't optimise). Gives E2.
5. Now, optimise structure. Gives E3.
6. Change the charge and multiplicity to 0 and 1. Calculate single point energy of the optimised structure in step 5. Gives E4.
7. ΔE=(E2-E3)+(E4-E1). Convert energies from Hartree to eV by multiplying by 27.2107

---

Using nwchem:

title "Calculating E1"
start neutral_ground_state
geometry
 C     3.58691     -1.70661e-06     0.000280946
 C     2.87790     -0.739281     0.946515
 C     1.48493     -0.738173     0.945097
 C     0.747197     0.00000     0.000278762
 C     1.48488     0.738183     -0.944553
 C     2.87785     0.739273     -0.945948
 C     -0.747197     0.00000     0.000278762
 C     -1.48488     0.00000     -1.19873
 C     -2.87785     0.00000     -1.20050
 C     -1.48493     0.00000     1.19927
 C     -3.58691     0.00000     0.000281534
 C     -2.87790     0.00000     1.20107
 H     4.67272     9.42775e-05     0.000158092
 H     3.40921     -1.32260     1.69312
 H     0.975305     -1.32691     1.69865
 H     3.40931     1.32254     -1.69249
 H     -3.40931     0.00000     -2.14788
 H     -0.975305     0.00000     2.15554
 H     -4.67272     0.00000     0.000125630
 H     -3.40921     0.00000     2.14853
 H     -0.975288     0.00000     -2.15502
 H     0.975288     1.32693     -1.69812
end
charge 0
basis "ao basis" cartesian print
  H library "6-31G**"
  C library "6-31G**"
end
dft
    direct
    grid fine
    xc b3lyp
    mult 1
end
task dft optimize

and

title "Calculating E2"
start cation_excited_state
geometry units angstrom
 C     0.00000     -3.56301     0.00000
 C     -1.13927     -2.85928     -0.393841
 C     -1.13879     -1.46545     -0.394153
 C     0.00000     -0.742814     0.00000
 C     1.13879     -1.46545     0.394153
 C     1.13927     -2.85928     0.393841
 C     0.00000     0.742814     0.00000
 C     1.13879     1.46545     -0.394153
 C     1.13927     2.85928     -0.393841
 C     -1.13879     1.46545     0.394153
 C     0.00000     3.56301     0.00000
 C     -1.13927     2.85928     0.393841
 H     0.00000     -4.64896     0.00000
 H     -2.02827     -3.39662     -0.711607
 H     -2.02148     -0.928265     -0.727933
 H     2.02827     -3.39662     0.711607
 H     2.02827     3.39662     -0.711607
 H     -2.02148     0.928265     0.727933
 H     0.00000     4.64896     0.00000
 H     -2.02827     3.39662     0.711607
 H     2.02148     0.928265     -0.727933
 H     2.02148     -0.928265     0.727933
end
charge 1
basis "ao basis" cartesian print
  H library "6-31G**"
  C library "6-31G**"
end
dft
    direct
    grid fine
    xc b3lyp
    mult 2
end
task dft energy

and

title "Calculating E3"
start cation_ground_state
geometry units angstrom
 C     0.00000     -3.56301     0.00000
 C     -1.13927     -2.85928     -0.393841
 C     -1.13879     -1.46545     -0.394153
 C     0.00000     -0.742814     0.00000
 C     1.13879     -1.46545     0.394153
 C     1.13927     -2.85928     0.393841
 C     0.00000     0.742814     0.00000
 C     1.13879     1.46545     -0.394153
 C     1.13927     2.85928     -0.393841
 C     -1.13879     1.46545     0.394153
 C     0.00000     3.56301     0.00000
 C     -1.13927     2.85928     0.393841
 H     0.00000     -4.64896     0.00000
 H     -2.02827     -3.39662     -0.711607
 H     -2.02148     -0.928265     -0.727933
 H     2.02827     -3.39662     0.711607
 H     2.02827     3.39662     -0.711607
 H     -2.02148     0.928265     0.727933
 H     0.00000     4.64896     0.00000
 H     -2.02827     3.39662     0.711607
 H     2.02148     0.928265     -0.727933
 H     2.02148     -0.928265     0.727933
end
charge 1
basis "ao basis" cartesian print
  H library "6-31G**"
  C library "6-31G**"
end
dft
    direct
    grid fine
    xc b3lyp
    mult 2
end
task dft optimize

and

title "Calculating E4"
start neutral_excited_state
geometry
 C     0.00000     -3.54034     0.00000
 C     -1.20296     -2.84049     -0.216000
 C     -1.20944     -1.46171     -0.206253
 C     0.00000     -0.721866     0.00000
 C     1.20944     -1.46171     0.206253
 C     1.20296     -2.84049     0.216000
 C     0.00000     0.721866     0.00000
 C     1.20944     1.46171     -0.206253
 C     1.20296     2.84049     -0.216000
 C     -1.20944     1.46171     0.206253
 C     0.00000     3.54034     0.00000
 C     -1.20296     2.84049     0.216000
 H     0.00000     -4.62590     0.00000
 H     -2.12200     -3.38761     -0.395378
 H     -2.13673     -0.938003     -0.401924
 H     2.12200     -3.38761     0.395378
 H     2.12200     3.38761     -0.395378
 H     -2.13673     0.938003     0.401924
 H     0.00000     4.62590     0.00000
 H     -2.12200     3.38761     0.395378
 H     2.13673     0.938003     -0.401924
 H     2.13673     -0.938003     0.401924
end
charge 0
basis "ao basis" cartesian print
  H library "6-31G**"
  C library "6-31G**"
end
dft
    direct
    grid fine
    xc b3lyp
    mult 1
end
task dft energy

And it all gives:

E1: Total DFT energy =     -463.321927500065
E2: Total DFT energy =     -463.035336642074
E3: Total DFT energy =     -463.042292962541
E:4 Total DFT energy =     -463.315725187090

ΔE=-463.035336642074-(-463.042292962541)+(-463.315725187090-(-463.321927500065))=0.013158633442Hartree= .3580556270002294 eV ≅ 0.36 eV i.e. the same as the paper.