28. Compiling/Building nwchem with mpich on debian 64 bit (15/12/2011) -- observations of squeeze, wheezy, sid and experimental

** NOTE: the nwchem build in the debian repos (6.0-1) does not support mpich. It will not run in parallel. It seems like 6.0-2 will, but it's not in the repos yet, and looking at the package status page I get a bit worried about the likelihood of seeing a finished build for amd64 **

** The solution to building on debian testing and above (and presumably ubuntu >10.10) is here: http://verahill.blogspot.com/2011/12/building-nwchem-on-debian-testing-64.html **

** NOTE: according to this post mpd isn't needed in newer (>=1.3) versions. It is needed -- and provided -- by mpich2 1.2**

So, I've had problems building nwchem on debian testing for about a year now. Actually, building nwchem is pretty straightforward, but building it with mpich2 support doesn't seem to work.

I also noted that mpd doesn't refer to an mpich daemon anymore, leading me to suspect that maybe the version of mpich2 (now at 1.4; was at 1.2 when I built on ubuntu early in the year) could be part of the problem.

Long story short: it builds with mpich2 support on debian stable (squeeze), but not debian testing (wheezy).


----------------------------------
SQUEEZE
----------------------------------
Here's what I did: I installed debian stable 64 bit in virtualbox from the business-card iso (standard tools + ssh. No desktop environment). I made sure the system (squeeze) was up to date, downloaded nwchem-6.0 and extracted it in /home/myhome/nwchem/nwchem-6.0  and create a build file, myconfig.sh, with the following content:
export LARGE_FILES=TRUE
export TCGRSH=/usr/local/bin/ssh
export NWCHEM_TOP=/home/myhome/nwchem/nwchem-6.0
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export USE_MPI=y
export USE_MPIF=y
export MPI_LOC=/usr
export MPI_INCLUDE=$MPI_LOC/include/mpich2
export LIBMPI="-lfmpich -lmpich"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran

Only thing remaining before building is:
sudo apt-get install mpich2 gfortran

mpich2: version 1.2

sh myconfig.sh then starts the build process which takes about 10-20 minutes in a virtual machine. Everything good. Works as should -- I only have one core in the virtual machine, so can't do much testing. It builds without a hitch though. In addition mpd, mpdtrace etc are all present.

 Time to take a snapshot. 1.7 Gb.
----------------------------------
WHEEZY
----------------------------------
I next edited /etc/apt/sources.list and replaced all instances of squeeze with wheezy (including commenting out the last two lines (wheezey-updates / squeeze-updates), did sudo apt-get update and sudo apt-get dist-upgrade

mpich2: version 1.4.1.
mpd and mpdtrace are now gone. Only mpi* around.

Tried sh myconfig.sh again (added a make clean before make nwchem_config). It builds for a long time - 10-20 minutes -- ultimately the build fails with
..
..
/usr/lib/libmpich.so: undefined reference to `MPL_trid'
..
..
/usr/lib/libmpich.so: undefined reference to `MPL_trinit'
collect2: ld returned 1 exit status
make: *** [all] Error 1

The undefined references are: MPL_trid, MPL_trvalid, MPL_env2int, MPL_trrealloc, MPL_trspace, MPL_trDebugLevel, MPL_TrSetMaxMem, MPL_trlevel, MPL_trmalloc, MPL_putenv, MPL_env2bool, MPL_env2range, MPL_trcalloc, MPL_trfree, MPL_en2str, MPL_trstrdup, MPL_trdump and MPL_trinit

So no luck with Wheezy, which is the system I run on all my computers. I've actually tried quite a few approaches under Wheezy and have managed to complete the odd build, but without getting proper functionality i.e. using mpiexec -n 6 six instances are created instead of one instance spread across six cores so that the system is solved six times instead of just one.


----------------------------------
SID.
----------------------------------
Time to take another snapshot followed by an upgrade.
The wheezy snapshot is 2.2 Gb -- wonder where the 0.5 Gb come from?

Replacing all instances of wheezy with sid using vim (:%s/wheezy/sid/g). Fails to fetch http://security.debian.org/dists/sid/updates/main/binary-amd64/Packages during sudo apt-get update. Oh well. Remove that line from sources.list. Now update works.
Then sudo apt-get dist-upgrade. 140 Mb of archives to get.


As a general thought -- maybe it'd be worth keeping a copy of sid in a virtual machine just to see what's around the corner?

Upgrade done, sudo shutdown -r now
mpich2: version 1.4.1, same as wheezy. No mpd or mpdtrace.

Try building and...starts at 14:17, fails at 14:45. Same errors as for Wheezy.

Last chance -- snapshot (1.9 Gb...), add a ref to experimental and...no updates. They are the same. Not worth trying thus.
------------------------------------


So there you go -- nwchem builds ok on Squeeze using mpich2 ver 1.2, but not on any of the more up-to-date distros. I also wonder about the lack of mpd/mpdtrace -- in Squeeze mpd is the mpich daemon, while in Wheezy and above it's the Music Player Daemon. Something is odd here.

Coming next: can you build nwchem on Wheezy if you pull mpich2 ver 1.2 from the archives? Of course you can. The answer and how-to is here: http://verahill.blogspot.com/2011/12/building-nwchem-on-debian-testing-64.html

27. Linux: some applications

Here are some linux tools which I've found myself using:
_Terminal/cli_:
email -- mutt
chat -- mcabber
spreadsheet -- sc, sed + gawk, python
calculator -- bc
editing -- vim, nano
browser -- elinks
music -- cplay
plotting -- gnuplot
maths -- octave, maxima
Gnome (some of these I use only in the terminal, so I list them twice):
terminal -- guake
latex -- gvim, texmaker, latexilla
editing -- gvim, gedit
plotting -- gnuplot
maths -- octave, maxima
system stats -- conky
email -- evolution
chat -- gajim
browser -- chrome
video -- vlc
music -- mostly cplay in cli, but otherwise vlc
crypto -- truecrypt
password management -- keepassx
I probably spend 90% of my time using vim since, being a 'simple' editor, it's so versatile. Before taking the time to learn vim, I used gedit (and to a smaller extent nano) for the same purpose. Nano is fine, but vim is more ubiquitous (in the form of vi) and has an excellent tutorial included. I simply don't know what the full capabilities of nano are -- maybe it's only as powerful as, say, notepad, or maybe it can do everything vim can.
I gave emacs a serious try about a year ago, and it didn't suit me. I do appreciate that emacs can be considered as an OS almost in itself, but since I believe in the idea of one program - one task, and find vim more user-friendly (not meaning learner-friendly, although it is too) I felt more motivated to get serious with vim. Two weeks with emacs brought me to the same level as two rather relaxed evenings with vim.

26. Linux The Basics: the evolution of a user

I've used linux for a few years by now, and I find that how I do things have changed over this time. In other words, the way you use linux will change and evolve as you learn more. I never experienced this in windows -- there you click your way around, and you quite easily become stuck with a handful of applications and a way of doing things.

In the beginning, I dual-booted. Windows at work, where I would use originlab's origin, micromath scientist, excel etc. Linux at home where I'd browse the web, check email and chat. That lasted for a month or so, until I become confident that virtualbox could handle a copy of XP.

For about a year I persisted in doing most of my personal work in linux, and using virtualbox to run origin, use word to write articles etc. At least that way I only had to boot one operating system and could make the initially frightening step of removing windows for ever (you're a green linux user without anyone to help you, you have a paid job relying on you being able to use your computer, and you sever the ties with an OS which has been with you since 1993? It's frightening)

Making the transition complete required finding native ways of doing things. Instead of using origin I used gnuplot. I started using latex via texmaker. sed + gawk with a little bit of python has been a great stand-in for excel (there's always gnumeric as well). Latex has replaced powerpoint as well - I don't know why it never occurred to me before to use pdfs for presentation. I mean, it makes a whole lot more sense using something that's designed to be portable (fonts is the one theoretical issue) and which is pre-compiled. Rendering on the fly as powerpoint does is just asking for trouble (and as anyone who's ever been to a conference can confirm, a fair proportion of powerpoint presenters experience problems of some sort).

I've also slowly moved form using gnome with its tools, to using the terminal. Some things simply don't need a fancy gui. While at first I used gedit under gnome and nano in the terminal, I now use vim and gvim. While vim takes an hour or two to learn well enough to use, it is worth it for the convenience of a powerful keyboard-driven and ubiquitous editor. I also use vim for editing latex documents. Mutt is good enough for email when you don't need a gui, mcabber is less intrusive than gajim if you're busy, etc. And you can use them remotely. There's nothing that can be done in matlab (which exists for linux) that I can't do in octave. I don't need mathematica since I use maxima (they are not equivalent -- but I use maxima for symbolic math and octave for numerical stuff. I've dabbled in R as well, and while it's powerful I find the behaviour of it being a bit unpredictable -- R tries to second-guess what you want to do, and often gets it right. But not always).

There's no right way (but plenty of wrong ones -- if all you are doing in linux is installing explorer and msn in wine, then why bother?) of using linux, but what I do find satisfying is that you have the freedom to create your own workflow.

Where I am today: using Linux is no big deal. I don't think about it except when explicitly confronted with another OS. It's as second-nature as using windows once was -- you knew there were people out there using something infuriating and ridiculous-looking called MacOS  going 'eep' at inopportune moments, and that there was something hardcore called UNIX (Jurassic Park -- 'It's a UNIX system. I know this!'. That was 1993 btw.) Linux is just the way I do things, and I no longer evangelise. It might be age too -- you tend to be less religious about things as you get older. If people are willing to expand their horizons and feel that using linux makes sense, then I'm willing to help. If they are happy where they are today, that's fine too. Just don't email me any doc, docx or xls files where simple text files or a pdf would do -- that's just presumptuous in the other direction.

The main problem which hasn't been solved to this day is actually word. I can't stand the fetid piece of excrement (most people have at some point been frustrated by self-moving figures or odd formatting incidents), but if you collaborate with other people in writing articles and those people aren't willing to spend the time necessary to learn LaTex, you're pretty much stuck. Well, articles are better written by a single author anyway - writing by committee never flows. Anyway, turns out Office 2003 installs just fine in Wine. At this point, I find it difficult to understand why people insist on using binary formats like .doc and docx even for unformatted text. LaTeX works for me, and it suits my way of working.

Anyway, each to his or her own. The main challenge for a linux user isn't so much how to configure a certain piece of software as in finding out about the existence of the piece of software in the first place.