04 June 2012

173. ECCE in a virtual machine: step by step using virtualbox

!NOTE!
Pre-built binaries of ECCE now ONLY exist for 64 bit OS -- if you install a 32 bit version of debian you will have to compile ECCE yourself. It's not difficult, but please be aware of it.

!NOTE2! If you provide ECCE with 'localhost' as the hostname during installation, be aware that this will block outside access: http://www.nwchem-sw.org/index.php/Special:AWCforum/st/id858/#post_3178

See this post for ECCE 6.4 of Debian 7 (wheezy): http://verahill.blogspot.com.au/2013/05/434-ecce-64-on-debian-7-in-32-bit.html
!NOTE!

Sadly, not everyone uses linux. Some even refuse to give it a proper consideration. Since the goal here is science rather than OS enlightenment, we'll have to accept that some people will have to run ECCE in a virtual machine rather than using it natively (something like WINE but going the other direction would be nice).

Win/Mac users should not be surprised if graphics heavy applications like ECCE don't run very fast in a virtual machine though, but they will be the source of that frustration themselves.

Here's the whole sequence from downloading an iso to having ECCE ready to submit jobs from inside a virtual machine.

I'm doing all these steps on linux since I simply don't use or have access to windows (other than as a very small virtualbox install), so let me know if there's anything which won't work on windows. I'm particularly interested in XP since that's what most people in academia still use.

You can of course use these instructions on a proper debian installation as well.

The only reason this post is so long is because of the number of screen shots. It is NOT COMPLICATED OR DIFFICULT, so don't be put off by the length.

Index:
1. Install virtualbox
2. Set up the machine
3. Download the debian iso
4. Install debian in virtualbox
5. Customize your debian installation
6. Installing ECCE
7. Setting up ECCE
8. Launching a test calc.
9. Moving files back and forth between virtual debian and windows


1. Install Virtualbox
I've tested this step in my virtual XP machine, so it should work fine on a 'real' XP machine.

Download the windows installer: http://download.virtualbox.org/virtualbox/4.1.16/VirtualBox-4.1.16-78094-Win.exe . If the link doesn't work, then go to https://www.virtualbox.org/wiki/Downloads

Start the installer. It appears to be a bit buggy since the 'back' button doesn't work, so make sure that you think about your choices. Otherwise starting over isn't that hard.

 ...and start. You'll get a warning about network interfaces being reset. That's fine.

You get five of those Unsigned driver thingy warnings -- a Microsoft protection racket if you'd ask me (compatibility assurance is a good thing -- forcing developers to pay for warnings to go away isn't). Just click continue.


When you're done, launch:



Download the extensions from here: http://download.virtualbox.org/virtualbox/4.1.16/Oracle_VM_VirtualBox_Extension_Pack-4.1.16-78094.vbox-extpack



Click on Machine, Preferences and select Exentions. Click on the small Blue rhomb with the orange triangle on it on the right side of the window, navigate your way to where you downloaded the extension and install it:

 And this is what it looks like when you're done:




2. Set up a new machine
Click on New. Follow the instructions.

A few things to think about:
* Assign a reasonable amount of RAM, whatever that means. I'd suggest 1-1.5 Gb if you have 4 Gb machine. I'm testing 512 Mb in this example to see what the lower requirements are.
* Definitely make your hard drive big enough from the beginning since expanding it isn't that easy.

Here's a series of screen grabs:

 Select Linux, Debian (64 bit if possible)
 Set the amount of RAM. The more the merrier, but leave enough for your OS.


 Use a dynamically expanding HDD.
 10 Gb is a good start, but you'll have to move computational files off of the virtual hard drive periodically. The underlying OS only uses a small amount of storage space, but ECCE -- depending on what you do in it -- can use a lot more.





3. Download the iso
Clicking the following URL will hopefully start a download:
http://ftp.nl.debian.org/debian/dists/squeeze/main/installer-amd64/current/images/

If not, then have a look at this page: http://www.debian.org/distrib/netinst and click on AMD64 under Small CDs (180 Mb). This will get you the latest Debian Squeeze, which is a bit old but rock solid. Upgrading to testing is a breeze if you should want to do that later.


4. Installing Debian
Select your new machine, and click on settings.

 Click on Storage
 Mount the iso you downloaded
 When all is good it should look like this:
Hit OK.

Two things you'll need to know at this point:

  • Right CTRL is your 'host' key. Hit (right) CTRL to release the mouse from the virtual desktop
  • Switch between full-screen mode using (right) CTRL+f (host+f)

Now you're ready to start your virtual machine, so click on Start, which brings you to this:


Most of the choices are fairly simple to answer so I won't cover them.
I selected:
Hostname - vecce
Domain name -- (empty)
Pick the mirror country that correspond to the country you're in. As for the actual mirror server, the first choice is normally ok. You can see if your uni is on the list, which will make it even faster.
Don't type anything when it asks for root password (done twice) since it's better (imho) to use sudo.
Our new user is called ecce, and this user will automatically be granted sudo powers.

Once you get to the partitioning, make sure to select Manual, and follow the screenshots below:
 First you need to format your harddrive. Remember, by harddrive we're really talking about a file on your disk, so screwing this up will NOT affect your windows computer in any shape or form.







 Having a Swap partition for a virtual HDD makes absolutely no sense.


Now your base system will install, and you'll eventually get to this screen. De-select the graphical desktop environment since we want to save space and resources -- gnome is wonderful, but demanding. We'll put LXDE on it later.
Install grub to the MBR. Remember, our harddrive is still just a file on your physical harddrive.

That's basically it. Click continue in the screen below, and your virtual system will restart.
 And here's what'll great you on reboot:

You're now ready to log on and do some damage.

5. Customize your debian installation
So log on:

 Make sure you have working internet by e.g. doing
ping -c 3 google.com

Now, type
sudo apt-get install lxde
type your password (you'll get a warning about great reponsibilities etc.), and answer yes to any questions

Once the install has finished, type
sudo shutdown -r now
which will cause a reboot which will bring you to this screen:

I use gnome 3/gnome-shell myself, so I'm not particularly familiar with LXDE, so if anything looks weird in the following steps, that's why.

Log on:
You'll find your terminal (LXTerminal) by clicking on what constitutes the Start Menu down on the left:
First install a web browser (in this case chromium -- essentially a re-branded Chrome):
sudo apt-get install chromium-browser
Start chromium by clicking on the web browser icon in the panel launcher down in the bottom left corner:
6. Installing ECCE
Download ECCE:



Install csh by typing
sudo apt-get install csh 
in the terminal
Then do
cd ~/Downloads
chmod +x install_ecce.v6.3.rhel5-gcc4.1.2-m64
csh install_ecce.v6.3.rhel5-gcc4.1.2-m64
Select Full Install [3]

Change the hostname to localhost
Set the application install path to /home/ecce/ecce-v6.3/apps and the server directory to /home/ecce/ecce-v6.3/server
Then install.

Now, start leafpad:

Open ~/.bashrc (the ~ means it's in your home folder). You do that by clicking on File, Open, then click on file system, home, ecce, hit ctrl+H to show hidden files, and select .bashrc
Add the following to the end of the file:
export ECCE_HOME=/home/ecce/ecce-v6.3/apps
export PATH=$PATH:${ECCE_HOME}/scripts
alias startecceserver='/home/ecce/ecce-v6.3/server/ecce-admin/start_ecce_server'
alias stopecceserver='/home/ecce/ecce-v6.3/server/ecce-admin/stop_ecce_server'




Save, and exit leafpad.

Type
source ~/.bashrc
in your terminal. This will make the terminal read the new settings.
You still can't start since you're missing java:

So, in your terminal, do
sudo apt-get install default-jre

First make sure the ecce server isn't running
stopecceserver
then launch ecce
ecce
You're now being invited to set a new password:


Clicking OK launches what is called the Gateway:

7. Setting up ECCE
You're more than half-way there. The exact set up will depend on what you're trying to achieve and I've documented it elsewhere on this blog.

However, because I'm writing this guide with a couple of specific people in mind, I'll show how to set up a remote ROCKS site to submit to, using node hopping.

In the terminal, type
ecce -admin
 Set up as shown in the screen shot:

Don't forget to hit add/change queue first, then add/change in the bottom left, then close.

Because we're doing a lot of cool, advanced stuff we need to do some manual editing: we need to define the public url, and we need to define our SGE (Sun Grid Engine) settings and finally, we need to set up a couple of environment variables.

In the terminal, do
cd ~/ecce-v6.3/apps/siteconfig/
chmod u+rw CONFIG.rocks
Otherwise the file will be write-protected.


Using leafpad, open the CONFIG.rocks file which is found in /home/ecce/ecce-v6.3/apps/siteconfig/


Add the following (look at the screenshot below):
frontendMachine: rocks.university.edu

SGE {
#$ -S /bin/csh
#$ -cwd
#$ -l h_rt=$walltime
#$ -l h_vmem=$memoryG
#$ -j y
#$ -pe orte $totalprocs    
}

NWChemEnvironment{
            LD_LIBRARY_PATH /usr/lib/openmpi/1.3.2-gcc/lib/
}

NWChemCommand {
        /opt/openmpi/bin/mpirun -n $totalprocs $nwchem $infile > $outfile
}
Gaussian-03Command {
    setenv GAUSS_SCRDIR /tmp
    setenv GAUSS_EXEDIR /share/apps/gaussian/g09/bsd:/share/apps/gaussian/g09/local:/share/apps/gaussian/g09/extras:/share/apps/gaussian/g09
        time /share/apps/gaussian/g09/g09 $infile  $outfile }




Save and exit. (note the $memoryG so you don't put in 4000 but instead use 4 when setting memory during submission in launcher)

In the ECCE gateway, click on Machine Browser

Select rocks and click on Set Up Remote Access, give the user name and password. If all goes well, if you hit e.g. Disk Usage, you should see this:

8. Launching a test calc
Click on Organizer in the Gateway.






If you can't select basis set, theory etc. at this point, close the editor, then open again. Might be because it's the first run ever.


And launch:
 When it's done, it looks like this. Depending on your SGE set-up everything can be very fast, or every step can take a minute. To follow in real-time, click in Viewer. To see the output file, hit Run Mgmt and select tail -f. To look at output fiel from completed run, go to Run Mgmt and select Output.
It's fun, and easy!

9.  Moving files back and forth between your virtual debian and windows
First shut down your virtual machine, then click on settings.



When your virtual machine is running you can access it from e.g. windows using sftp with e.g. Filezilla like this:






I hope this guide has been useful. Again, it's long because of the level of detail, not because it's difficult in any way.

Learning how to use ECCE properly is a completely different tutorial. I might put up some examples with increasing complexity in the future.

25 comments:

  1. Hello There

    I am trying to install ECCE on windows 7 by following instructions here. I am stuck at slide no 19 where debian screen opens up and donot go further and on bottom bar on RHS the bov with 'V' in it is not activated. When I cosed and start again it shows " FATAL : no bootable medium found. System halted."

    Please help.

    Thank you

    Shaheen

    ReplyDelete
    Replies
    1. The lack of a "V" just means that your cpu doesn't support virtualization, which is fine.

      To help me troubleshoot: what version of windows and what version of debian are you using e.g. 64 bit or 32 bit? And which version of virtualbox?

      I also have to admit that I don't have any experience of using virtualbox on windows other than XP, and that was a couple of years ago.

      Delete
    2. Thanks for the reply.

      My computer has windows 7 and debian I downloaded from the link you have provided here --i.e. 64 bit. And vitual box Version 4.1.16.

      Thank you

      Shaheen

      Delete
    3. Shaheen,
      try the 32 bit version of debian and see if that helps. Remember that you'll have to download the 32 bit version of ecce as well in that case.

      When I think of it I think I saw a similar issue setting up 64 bit debian on win 7 for a student. The 32 bit worked fine.

      Delete
    4. Thank you

      I did it and it worked. But at step/ slide 32 , I forgot to de-select "graphical desktop environment ". Can I change that later?

      Thank you

      Shaheen

      Delete
    5. Hi

      I have got stuck again. I have downloaded 32 bit ecce, when I did csh install.........csh after chmod, then I got a message---bash: csh: command not found

      I have graphical desktop and its different from the slides shown here. I donot know if its ok?

      Thank you

      Shaheen

      Delete
    6. Yes, you can change the graphical environment -- just install lxde. When you start you computer (slide 39), highlight/type your user name and under session select the desktop environment you want to use (lxde in the slides).
      However, gnome2 (which you're using) is absolutely fine. Ecce will run under any desktop environment.

      as for csh -- install it:
      sudo apt-get install csh

      Delete
    7. Thank you so much ,I have installed csh and using genome2. But when I reached to a point to start ECCE I got following error message:

      ecce@debian:~/Downloads$ source ~/.bashrc
      ecce@debian:~/Downloads$ e
      bash: e: command not found
      ecce@debian:~/Downloads$ ecce
      Starting ECCE server ...
      /home/ecce/ecce-v6.4/server/httpd/bin/apachectl start: httpd started
      [1] 2549
      Failed to execute main task. Reason: java.lang.NoClassDefFoundError: org.springframework.core.CollectionFactory

      Starting ECCE ... please wait

      Copying default ~/.nwchemrc file to run NWChem locally
      javax.jms.JMSException: Could not connect to broker URL: tcp://localhost:8095. Reason: java.io.IOException: Connection refused

      ERROR: The ECCE application software cannot
      establish a connection to the ECCE Server
      at this time.

      Please contact your ECCE administrator to check
      that the ECCE Server is running and that the
      ECCE application software configuration file
      '/home/ecce/ecce-v6.4/apps/siteconfig/DataServers'
      contains the correct URL for your ECCE Server.

      Could please you suggest me some solution for this problem.

      Thank you

      Delete
    8. Check whether the ecce server is running. I sometimes have to start the server manually
      /home/ecce/ecce-v6.4/server/ecce-admin/./start_ecce_server

      and then try running
      ecce
      again.

      Delete
    9. Thank you.

      I tried these steps and it gave me again same errors. So what I did? I followed the java installation step you have here. And the manually started the ecce and it worked. Thank you for the help.

      sf

      Delete
    10. Hello Lindqvist,

      I got another question. I am installing all this on my laptop so do I need to do all steps from settingup ROCKS and so on? I have added chinook, frankline and localhost for remote access. Is that ok?

      Thank you

      Shaheen

      Delete
    11. Shaheen,
      how you set up your submit hosts will depends on what they look like. If you don't have cluster, but only want to run jobs on your laptop, you should use a set-up that points ecce towards your compiled nwchem version (the ecce version runs in serial, not parallel).

      e.g. for local jobs (i.e. for testing things) without any queue manager I do:

      CONFIG.beryllium
      NWChem: /opt/nwchem/nwchem-6.1/bin/LINUX64/nwchem
      Gaussian-03: /opt/gaussian/g09/g09
      perlPath: /usr/bin/perl

      And my Machines file has this line in it which sets the number of cores to six and number of nodes to 1 (6:1):
      beryllium beryllium Unspecified Unspecified Unspecified 6:1 ssh :NWChem:Gaussian-03 MN:RD:SD:UN:PW

      Delete
  2. Thank you So Much.

    shaheen

    ReplyDelete
  3. Hello Lindqvist,

    I have used your instructions to install ECCE on Mac OS X (Snow Leopard) with the required changes to install on the mac laptop. I tried to start ECCE by issuing the ecce command on the terminal. It gave the following error:

    ERROR: The ECCE application software cannot
    establish a connection to the ECCE Server
    at this time.

    Following your advice I started the server manually by doing
    /home/ecce/ecce-v6.4/server/ecce-admin/./start_ecce_server

    but no I get the following error:

    /home/ecce/ecce-v6.4/server/httpd/bin/httpd: 1: /home/ecce/ecce-v6.4/server/httpd/bin/httpd: ELF : not found
    /home/ecce/ecce-v6.4/server/httpd/bin/httpd: 2: /home/ecce/ecce-v6.4/server/httpd/bin/httpd: : not found
    /home/ecce/ecce-v6.4/server/httpd/bin/httpd: 3: /home/ecce/ecce-v6.4/server/httpd/bin/httpd: Syntax error: "(" unexpected
    /home/ecce/ecce-v6.4/server/httpd/bin/apachectl start: httpd could not be started
    Unable to execute apachectl start

    Any help will be much appreciated.

    ReplyDelete
    Replies
    1. There should be no space in /./ in your ecce-admin/./start_ecce_server line. An isolated . is translated into the source command, and the source command only works on shell scripts.

      Delete
    2. If that doesn't solve your problem let me know what debian release and ecce version you are using, and anything else that would help me reproduce the error.

      Delete
    3. Thanks for the prompt reply. That was a typo. There wasn't any space when I manually started the server. That being said, I tried the whole process again on my office computer (Mac OS X Snow Leopard) and get the same errors.

      Here are the specifications as requested:
      ECCE - 6.4
      Debian - 7.0.0 (debian-7.0.0-i386-netinst.iso)

      Am I missing some installation because of the minimalist net install of Debian. Let me know your thoughts on this.

      Thank you.

      Delete
    4. I'll try to reproduce your errors within the next 36 hours.

      However, could you just confirm that you are using the 32 bit version of ECCE? (to remove the most obvious source of error)

      I don't know macs well, but I take it that the host system is 64 bit? (don't think this matters)

      Have you made any changes to your .bashrc, e.g. added anything in the ecce directory to PATH? Or put any aliases in there?

      Finally, do
      uname -a
      file /home/ecce/ecce-v6.4/server/httpd/bin/httpd
      which httpd

      That you installed using the net iso shouldn't matter since you're pulling in all the required packages from the net during installation.

      To me your error is most definitely due to the httpd for some reason being executed as a shell script and not as a binary. One of the more common reasons for this is that you accidentally put a space around a . so that it is interpreted as 'source' and not as 'here'.

      Delete
    5. OK, I've figured it out. I set up a virtual machine with debian 7 i386, and got ready to install ECCE 6.4, when I remembered that Gary has stopped supplying a 32 bit version of ECCE.

      So your problem should simply be that you are trying to install a 64 bit version of ECCE on a 32 bit version of Debian.

      Delete
    6. See here for your options: http://verahill.blogspot.com.au/2013/05/434-ecce-64-on-debian-7-in-32-bit.html

      Delete
    7. Thanks Lindqvist!
      I will try following the instructions from your new post.

      Delete
    8. Let me know how it worked out.

      Delete
  4. Dear Lindqvist,

    I followed your instruction on compiling ECCE in CentOS system and installed ECCE with the compiled executable. It works. I am able to run nwchem remotely on another server with ECCE. I did the job on an office workstation willing to make ECCE accessible for all colleagues.

    Here is what I did but failed:

    1. I installed ecce as a user in my own folder on the workstation. ~/ecce-v6.4/

    2. After successful runs of nwchem with this correctly installed ECCE, I tried to modify the system boot sequence as the following:
    a. I added a file /etc/profile.d/ecce.sh

    #!/bin/sh

    if [ -e /home/user/ecce-v6.4/apps/scripts/runtime_setup.sh ]; then
    . /home/user/ecce-v6.4/apps/scripts/runtime_setup.sh
    fi

    export ECCE_HOME=/home/user/ecce-v6.4/apps
    export PATH=${ECCE_HOME}/scripts:${ECCE_HOME}/scripts/parsers:$PATH

    b. I added a line in /etc/rc.d/rc.local

    /home/nckusrc/ecce-v6.4/server/ecce-admin/start_ecce_server

    After restarting the machine, when I run ECCE I got an error message saying the structure library cannot be located and ecce aborted. I did try to recover the setting for my own use but ECCE keep coming up different error messages.

    Can you advise a correct way of setting up ECCE so that everyone can have an individual account and run on the same machine?

    ReplyDelete
    Replies
    1. I don't know the answer off-hand -- and I'll see if I can look into it and write up a post about it. In the meantime a crude alternative (and this is probably how ecce is intended to be used) is to have each user run the client version of ecce, and use that to connect to the server version of ecce.

      I've done something similar in the past, but need to figure out exactly what I did.

      It's a good question though.

      Delete
    2. I've created a post showing how to set up a multi-user ECCE environment:
      http://verahill.blogspot.com.au/2013/06/447-multiuser-ecce.html

      Again, it is based on each user having their own client software installed (although that's not a necessity -- it's just easier) and using that to connect to a common ECCE server.

      The users cannot access other users files, and the files are managed by the ECCE server.

      Delete