28. Compiling/Building nwchem with mpich on debian 64 bit (15/12/2011) -- observations of squeeze, wheezy, sid and experimental

** NOTE: the nwchem build in the debian repos (6.0-1) does not support mpich. It will not run in parallel. It seems like 6.0-2 will, but it's not in the repos yet, and looking at the package status page I get a bit worried about the likelihood of seeing a finished build for amd64 **

** The solution to building on debian testing and above (and presumably ubuntu >10.10) is here: http://verahill.blogspot.com/2011/12/building-nwchem-on-debian-testing-64.html **

** NOTE: according to this post mpd isn't needed in newer (>=1.3) versions. It is needed -- and provided -- by mpich2 1.2**

So, I've had problems building nwchem on debian testing for about a year now. Actually, building nwchem is pretty straightforward, but building it with mpich2 support doesn't seem to work.

I also noted that mpd doesn't refer to an mpich daemon anymore, leading me to suspect that maybe the version of mpich2 (now at 1.4; was at 1.2 when I built on ubuntu early in the year) could be part of the problem.

Long story short: it builds with mpich2 support on debian stable (squeeze), but not debian testing (wheezy).


----------------------------------
SQUEEZE
----------------------------------
Here's what I did: I installed debian stable 64 bit in virtualbox from the business-card iso (standard tools + ssh. No desktop environment). I made sure the system (squeeze) was up to date, downloaded nwchem-6.0 and extracted it in /home/myhome/nwchem/nwchem-6.0  and create a build file, myconfig.sh, with the following content:
export LARGE_FILES=TRUE
export TCGRSH=/usr/local/bin/ssh
export NWCHEM_TOP=/home/myhome/nwchem/nwchem-6.0
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export USE_MPI=y
export USE_MPIF=y
export MPI_LOC=/usr
export MPI_INCLUDE=$MPI_LOC/include/mpich2
export LIBMPI="-lfmpich -lmpich"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran

Only thing remaining before building is:
sudo apt-get install mpich2 gfortran

mpich2: version 1.2

sh myconfig.sh then starts the build process which takes about 10-20 minutes in a virtual machine. Everything good. Works as should -- I only have one core in the virtual machine, so can't do much testing. It builds without a hitch though. In addition mpd, mpdtrace etc are all present.

 Time to take a snapshot. 1.7 Gb.
----------------------------------
WHEEZY
----------------------------------
I next edited /etc/apt/sources.list and replaced all instances of squeeze with wheezy (including commenting out the last two lines (wheezey-updates / squeeze-updates), did sudo apt-get update and sudo apt-get dist-upgrade

mpich2: version 1.4.1.
mpd and mpdtrace are now gone. Only mpi* around.

Tried sh myconfig.sh again (added a make clean before make nwchem_config). It builds for a long time - 10-20 minutes -- ultimately the build fails with
..
..
/usr/lib/libmpich.so: undefined reference to `MPL_trid'
..
..
/usr/lib/libmpich.so: undefined reference to `MPL_trinit'
collect2: ld returned 1 exit status
make: *** [all] Error 1

The undefined references are: MPL_trid, MPL_trvalid, MPL_env2int, MPL_trrealloc, MPL_trspace, MPL_trDebugLevel, MPL_TrSetMaxMem, MPL_trlevel, MPL_trmalloc, MPL_putenv, MPL_env2bool, MPL_env2range, MPL_trcalloc, MPL_trfree, MPL_en2str, MPL_trstrdup, MPL_trdump and MPL_trinit

So no luck with Wheezy, which is the system I run on all my computers. I've actually tried quite a few approaches under Wheezy and have managed to complete the odd build, but without getting proper functionality i.e. using mpiexec -n 6 six instances are created instead of one instance spread across six cores so that the system is solved six times instead of just one.


----------------------------------
SID.
----------------------------------
Time to take another snapshot followed by an upgrade.
The wheezy snapshot is 2.2 Gb -- wonder where the 0.5 Gb come from?

Replacing all instances of wheezy with sid using vim (:%s/wheezy/sid/g). Fails to fetch http://security.debian.org/dists/sid/updates/main/binary-amd64/Packages during sudo apt-get update. Oh well. Remove that line from sources.list. Now update works.
Then sudo apt-get dist-upgrade. 140 Mb of archives to get.


As a general thought -- maybe it'd be worth keeping a copy of sid in a virtual machine just to see what's around the corner?

Upgrade done, sudo shutdown -r now
mpich2: version 1.4.1, same as wheezy. No mpd or mpdtrace.

Try building and...starts at 14:17, fails at 14:45. Same errors as for Wheezy.

Last chance -- snapshot (1.9 Gb...), add a ref to experimental and...no updates. They are the same. Not worth trying thus.
------------------------------------


So there you go -- nwchem builds ok on Squeeze using mpich2 ver 1.2, but not on any of the more up-to-date distros. I also wonder about the lack of mpd/mpdtrace -- in Squeeze mpd is the mpich daemon, while in Wheezy and above it's the Music Player Daemon. Something is odd here.

Coming next: can you build nwchem on Wheezy if you pull mpich2 ver 1.2 from the archives? Of course you can. The answer and how-to is here: http://verahill.blogspot.com/2011/12/building-nwchem-on-debian-testing-64.html

27. Linux: some applications

Here are some linux tools which I've found myself using:
_Terminal/cli_:
email -- mutt
chat -- mcabber
spreadsheet -- sc, sed + gawk, python
calculator -- bc
editing -- vim, nano
browser -- elinks
music -- cplay
plotting -- gnuplot
maths -- octave, maxima
Gnome (some of these I use only in the terminal, so I list them twice):
terminal -- guake
latex -- gvim, texmaker, latexilla
editing -- gvim, gedit
plotting -- gnuplot
maths -- octave, maxima
system stats -- conky
email -- evolution
chat -- gajim
browser -- chrome
video -- vlc
music -- mostly cplay in cli, but otherwise vlc
crypto -- truecrypt
password management -- keepassx
I probably spend 90% of my time using vim since, being a 'simple' editor, it's so versatile. Before taking the time to learn vim, I used gedit (and to a smaller extent nano) for the same purpose. Nano is fine, but vim is more ubiquitous (in the form of vi) and has an excellent tutorial included. I simply don't know what the full capabilities of nano are -- maybe it's only as powerful as, say, notepad, or maybe it can do everything vim can.
I gave emacs a serious try about a year ago, and it didn't suit me. I do appreciate that emacs can be considered as an OS almost in itself, but since I believe in the idea of one program - one task, and find vim more user-friendly (not meaning learner-friendly, although it is too) I felt more motivated to get serious with vim. Two weeks with emacs brought me to the same level as two rather relaxed evenings with vim.

26. Linux The Basics: the evolution of a user

I've used linux for a few years by now, and I find that how I do things have changed over this time. In other words, the way you use linux will change and evolve as you learn more. I never experienced this in windows -- there you click your way around, and you quite easily become stuck with a handful of applications and a way of doing things.

In the beginning, I dual-booted. Windows at work, where I would use originlab's origin, micromath scientist, excel etc. Linux at home where I'd browse the web, check email and chat. That lasted for a month or so, until I become confident that virtualbox could handle a copy of XP.

For about a year I persisted in doing most of my personal work in linux, and using virtualbox to run origin, use word to write articles etc. At least that way I only had to boot one operating system and could make the initially frightening step of removing windows for ever (you're a green linux user without anyone to help you, you have a paid job relying on you being able to use your computer, and you sever the ties with an OS which has been with you since 1993? It's frightening)

Making the transition complete required finding native ways of doing things. Instead of using origin I used gnuplot. I started using latex via texmaker. sed + gawk with a little bit of python has been a great stand-in for excel (there's always gnumeric as well). Latex has replaced powerpoint as well - I don't know why it never occurred to me before to use pdfs for presentation. I mean, it makes a whole lot more sense using something that's designed to be portable (fonts is the one theoretical issue) and which is pre-compiled. Rendering on the fly as powerpoint does is just asking for trouble (and as anyone who's ever been to a conference can confirm, a fair proportion of powerpoint presenters experience problems of some sort).

I've also slowly moved form using gnome with its tools, to using the terminal. Some things simply don't need a fancy gui. While at first I used gedit under gnome and nano in the terminal, I now use vim and gvim. While vim takes an hour or two to learn well enough to use, it is worth it for the convenience of a powerful keyboard-driven and ubiquitous editor. I also use vim for editing latex documents. Mutt is good enough for email when you don't need a gui, mcabber is less intrusive than gajim if you're busy, etc. And you can use them remotely. There's nothing that can be done in matlab (which exists for linux) that I can't do in octave. I don't need mathematica since I use maxima (they are not equivalent -- but I use maxima for symbolic math and octave for numerical stuff. I've dabbled in R as well, and while it's powerful I find the behaviour of it being a bit unpredictable -- R tries to second-guess what you want to do, and often gets it right. But not always).

There's no right way (but plenty of wrong ones -- if all you are doing in linux is installing explorer and msn in wine, then why bother?) of using linux, but what I do find satisfying is that you have the freedom to create your own workflow.

Where I am today: using Linux is no big deal. I don't think about it except when explicitly confronted with another OS. It's as second-nature as using windows once was -- you knew there were people out there using something infuriating and ridiculous-looking called MacOS  going 'eep' at inopportune moments, and that there was something hardcore called UNIX (Jurassic Park -- 'It's a UNIX system. I know this!'. That was 1993 btw.) Linux is just the way I do things, and I no longer evangelise. It might be age too -- you tend to be less religious about things as you get older. If people are willing to expand their horizons and feel that using linux makes sense, then I'm willing to help. If they are happy where they are today, that's fine too. Just don't email me any doc, docx or xls files where simple text files or a pdf would do -- that's just presumptuous in the other direction.

The main problem which hasn't been solved to this day is actually word. I can't stand the fetid piece of excrement (most people have at some point been frustrated by self-moving figures or odd formatting incidents), but if you collaborate with other people in writing articles and those people aren't willing to spend the time necessary to learn LaTex, you're pretty much stuck. Well, articles are better written by a single author anyway - writing by committee never flows. Anyway, turns out Office 2003 installs just fine in Wine. At this point, I find it difficult to understand why people insist on using binary formats like .doc and docx even for unformatted text. LaTeX works for me, and it suits my way of working.

Anyway, each to his or her own. The main challenge for a linux user isn't so much how to configure a certain piece of software as in finding out about the existence of the piece of software in the first place.

25. Linux The Basics: Google talk using Gajim or mcabber

Empathy is the stock gnome chat client, gajim is a REALLY versatile python-based chat client which integrates with gpg/pgp and mcabber is a command line chat client.

While I used empathy for a few years, I much prefer gajim. Partly it's the geek cred that comes with using something a bit less common, but mostly it's because it 1) supports gpg, 2) is highly configurable and 3) is easy to use. It's available in the debian repos. I like mcabber over centerim for the same reason - gpg support. Now, if I only had friends who were as paranoid as I...

[gajim]
Jabber ID: youruser@gmail.com
Resource: Gajim
Priority: adjust to status

Hostname: talk.google.com
Port: 5223

If you use gpg, go to the Personal information tab
You don't need to select 'Use gpg agent' (gpg agent segfaults on one of my boxes)


[mcabber]

The following goes into ~/.mcabber/mcabberrc

set color_background=default #allows transparent background

# Account
set jid = youruser@gmail.com
set password = areallylongandcomplicatedpasswordwhichyoullneverremember
set server = talk.google.com
set port = 5222
set ignore_self_presence = 1 #show self in list?

# GPG/PGP
set pgp = 1
set pgp_private_key = "01100110C1C1C1C1" # get the Key ID from seahorse 
set pgp_passphrase_retries = 3

set nickname = blofeld
set spell_enable = 1
set spell_lang = en_GB
set spell_encoding = UTF-8
set cmdhistory_lines = 250
set roster_display_filter = ofdna
set max_history_blocks = 8
set message_autoaway = Auto-away (idle)
set escdelay = 50

alias me = say /me
alias online   = status online
alias away     = status away
alias dnd      = status dnd
alias notavail = status notavail

bind 17 = roster unread_next
bind 24 = roster alternate
bind 269 = roster toggle_offline
bind 276 = roster toggle
bind 521 = buffer up
bind 514 = buffer down

24. Linux and The Basics: configuring Evolution for gmail or fastmail accounts

While it comes more naturally to post a blog entry when you've just managed to fix a problem which has been nagging you, it helps newcomers to the linux community if you post how-to guides for the more basic stuff.

Since I can see what search terms drive people to the blog, I get a very simplistic overview of what people are looking for but which the blog doesn't yet provide. So, here are the gmail and fastmail (http://www.fastmail.fm/) settings.

[gmail]

If you're using google apps with an address along the lines of user@domain.example, then keep the server names (i.e. smtp.gmail.com) but use the full address (user@domain.example) as the user name.

--Receiving email
server type: imap (or imap+)
server: imap.gmail.com
username youruser@gmail.com
Use secure connection: ssl
Authentication type: password

--Sending email
Server type: smtp
server: smtp.gmail.com:587
use secure: tls
type: login
username youruser@gmail.com

--Defaults
Drafts: youruser@gmail.com/Drafts
Sent: youruser@gmail.com/Sent
etc.
*******************************
[fastmail] -- fastmail has a lot of domains. Like for google apps this means you must include the domain name in the username. Here's how it's done for imap.cc


--Receiving email
server type: imap (or imap+)
server: mail.messagingengine.com
username youruser@imap.cc
Use secure connection: ssl
Authentication type: password

--Sending email
Server type: smtp
server: mail.messagingengine.com
server requires authentication: yes
use secure: tls
type: login
username youruser@imap.cc

--Defaults
Drafts: youruser@imap.cc/INBOX/Drafts
Sent: youruser@imap.cc/INBOX/Sent
etc.

23. Configuring Mutt for two IMAP accounts

Edit: fastmail configuration added -- see beneath original post

Here's my situation: I have a personal gmail account, and a google apps gmail account via my job.

A lot of the configuration stuff (gpg in particular) was copy/pasted from some website a few years ago. Kudos to them. If I only remembered who they were, I'd give credit where credit is due.

Since I'm paranoid about protecting my identitity, let's say that the google apps account is vera@hill.com and my gmail account is verahill@gmail.com (it isn't).

Here's how it can be done:
1. create ~/.mutt
2. You want to create the following files in ~/.mutt :
private.account
work.account
muttrc


Optional:
aliases

signature.private
signature.work

3. Here are the contents of the files (the passwords are from http://xkcd.com/936/):

~/.mutt/private.account:

set from=verahill@gmail.com

set hostname="gmail.com"

unset folder

set folder="imaps://verahill@gmail.com@imap.gmail.com"

set postponed="=Drafts"

set record="=Sent"

set smtp_url="smtps://verahill@gmail.com@smtp.gmail.com"

set spoolfile="imaps://imap.gmail.com/INBOX"

set from='Vera Hill <verahill@gmail.com>'

set signature="~/.mutt/signature.private"

~/.mutt/work.account:

set from=vera@hill.com

set hostname="hill.com"

unset folder

set folder="imaps://vera@hill.com@imap.gmail.com"

set smtp_url="smtps://vera@hill.com@smtp.gmail.com"

set from='Vera Hill <vera@hill.com'

set realname='Vera Hill'

set postponed="=Drafts"

set record="=Sent"

set spoolfile="imaps://imap.gmail.com/INBOX"

set signature="~/.mutt/signature.private"

basic ~/.mutt/muttrc: 

set header_cache=~/.mutt/cache/headers

set message_cachedir=~/.mutt/cache/bodies

set certificate_file=~/.mutt/certificates

set sort='threads'

set sort_aux='last-date-received'

set imap_check_subscribed

auto_view text/html #needs ~/.mailcap and install elinks!

set attribution="* %n <%a> wrote\a ecrit:\n"

set pager_index_lines="15" #split view 

# Make sure the inbox updates and bind a key

set timeout=15

set mail_check=60

bind index "i" imap-fetch-mail

#dual boxes specific

macro index <f3> '<sync-mailbox><enter-command>source ~/.mutt/private.account<enter><change-folder>!<enter>'

macro index <f4> '<sync-mailbox><enter-command>source ~/.mutt/work.account<enter><change-folder>!<enter>'

folder-hook 'verahill@gmail.com' 'source ~/.mutt/private.account'

folder-hook 'vera@hill.com' 'source ~/.mutt/work.account'

account-hook imaps://verahill@gmail.com 'set imap_user=verahill@gmail.com imap_pass=correcthorsebatterystaple smtp_pass=correcthorsebatterystaple'

account-hook imaps://vera@hill.com 'set imap_user=vera@hill.com imap_pass=Tr0ub4dor&3 smtp_pass=Tr0ub4dor&3'

source ~/.mutt/private.account # default account

~/.mailcap:

#allows viewing html, but requires that elinks is installed
text/html;                      elinks %s; nametemplate=%s.html
text/html;                      elinks -dump %s; nametemplate=%s.html; copiousoutput

You can do a lot more -- if you want to use an address book (aliases), create a file called ~/.mutt/aliases with each line having an alias of the format:

alias  Obama "Barak Obama" <potus@whitehouse.gov>

Add the following to the end of your muttrc:

 # Addressbook

set alias_file=~/.mutt/aliases

set sort_alias=alias

set reverse_alias=yes

source $alias_file

If you want to use GPG add this to your muttrc (anywhere) and the ID of your key is C1C2C3C4:

# GPG stuff - autosign

set pgp_decode_command="gpg %?p?--passphrase-fd 0? --no-verbose --batch --output - %f"

set pgp_verify_command="gpg --no-verbose --batch --output - --verify %s %f"

set pgp_decrypt_command="gpg --passphrase-fd 0 --no-verbose --batch --output - %f"

set pgp_sign_command="gpg --no-verbose --batch --output - --passphrase-fd 0 --armor --detach-sign --textmode %?a?-u %a? %f"

set pgp_clearsign_command="gpg --no-verbose --batch --output - --passphrase-fd 0 --armor --textmode --clearsign %?a?-u %a? %f"

set pgp_encrypt_only_command="pgpewrap gpg --batch --quiet --no-verbose --output - --encrypt --textmode --armor --always-trust --encrypt-to 0xC1C2C3C4 -- -r %r -- %f"

set pgp_encrypt_sign_command="pgpewrap gpg --passphrase-fd 0 --batch --quiet --no-verbose --textmode --output - --encrypt --sign %?a?-u %a? --armor --always-trust --encrypt-to 0xC1C2C3C4 -- -r %r -- %f"

set pgp_import_command="gpg --no-verbose --import -v %f"

set pgp_export_command="gpg --no-verbose --export --armor %r"

set pgp_verify_key_command="gpg --no-verbose --batch --fingerprint --check-sigs %r"

set pgp_list_pubring_command="gpg --no-verbose --batch --with-colons --list-keys %r" 

set pgp_list_secring_command="gpg --no-verbose --batch --with-colons --list-secret-keys %r" 

set pgp_autosign=yes

set pgp_sign_as=0xC1C2C3C4

set pgp_replyencrypt=yes

set pgp_timeout=1800

set pgp_good_sign="^gpg: Good signature from"

If you prefer vim to nano:

set editor="vim -c 'set tw=70 et' '+/^$' "

set edit_headers=yes

A colour-scheme which happens to work with my set-up for guake:

color index green black plain

color normal green black

color status white black

color body green black plain

color header cyan black plain

color quoted blue black

color tree green black

Not sure what this does (something to do with headers) but it's there in my file:

ignore "Authentication-Results:"

ignore "DomainKey-Signature:"

ignore "DKIM-Signature"

hdr_order Date From To Cc

EDIT:

If you want to add a fastmail account (in my case me@imap.cc), do this:
There's no free smtp service from fastmail, so I use the gmail smtp service

Create imapcc.account in ~/.mutt and fill it with

set from=me@imap.cc
set hostname="imap.cc"
unset folder
set folder="imaps://me@imap.cc@messagingengine.com"
set postponed="=Drafts"
set record="=Sent"
set smtp_url="smtps://verahill@gmail.com@smtp.gmail.com"
set spoolfile="imaps://messagingengine.com/INBOX"
set from='Me at gmail <verahill@gmail.com>'
set signature="~/.mutt/signature.private"

and edit ~/.mutt/muttrc to add the following lines:

macro index <f5> '<sync-mailbox><enter-command>source ~/.mutt/imapcc.account<enter><change-folder>!<enter>'

folder-hook 'me@imap.cc' 'source ~/.mutt/imapcc.account'
account-hook imaps://me@imap.cc 'set imap_user=me@imap.cc imap_pass=ashortpassword smtp_pass=correhorsebatterystaple'

22. gnome-shell/GNOME3 -- acceptance

So after the turmoil of being thrown into a new desktop metaphor, I've finally reached a stage where I'd feel less at home in Gnome 2.3 than 3.0. Truth be told, the new gnome is both pretty and quite responsive. Key to making Gnome 3 usable is, however, that enough shell extensions are installed that it works similar to the old gnome. The application centric desktop may work for casual users (email/browser), but not for traditional linux/unix applications where different parts of a work flow is handled by different applications (e.g. latex -- scripting/composition in one application, compilation by another, inspection in a third; or gnuplot -- data preparation, gnuplot scripting, visualisation of output file).

But once all the shell extensions are installed, it's back to business as usual.

Even Linus seems to be coming around to Gnome 3 after mumbling about defecting to xfce4: https://plus.google.com/102150693225130002912/posts

So far I've only had one unresolved problem - an 'old' desktop with a radeon 5400 hd card won't run gnome-shell on debian testing. No acceleration. Oh well. At least all my other computers are reasonably happy so far.

21. Current Bug -- gnome applications unresponsive to keyboard input

UPDATE 02/02/2012:
It seems like whatever the problem was, it's gone now. An up-to-date testing system with ibus runs smoothly.

26/11/2011
For some reason, after updating today I'm having a hard time entering input in various gnome applications, whereas non-gnome applications work fine.
E.g. I can't type certain letters, and in general it's difficult to provide any inpurt, in gnome-terminal and gnome-session-properties. lxterm and guake work completely normal though.

No sure what is causing this, but it's happening on my Thinkpad SL410 as well as on my Optiplex 910. The only thing they have in common is the operating system (up-to-date debian testing 64 bit).

Don't know enough about it to file a bug report.

Edit: the problem is present in Evolution as well. Still present as of 28/11/2011. Problem exists on a home-built six-core AMD with 1 GB graphics card.

Edit 2: It sounds a bit like this, which was reported in June (!).
 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=631116
It seems to be related to iBus/SCIM


Edit 3: Evolution is also affected, as is gEdit. Turning off iBus 'solves' the problem. Now, why does it affect the standard gnome applications, but not other packages?

At any rate, this bug has given me the incentive to leave gnome-terminal for guake, and to give alpine and thunderbird a try instead of evolution.

I tried
sudo apt-get install ibus-gtk3
sudo apt-get autoremove ibus-gtk

iBus still works. But unfortunately evolution is still way too slow to be useful.

20. Is there no molecular weight calculator in the Debian repos?

UPDATE: see here for an isotopic calculator written in python -- it calculates mass as well: http://verahill.blogspot.com.au/2012/10/isotopic-pattern-caculator-in-python.html

The best molecular weight calculator which I've encountered is Matthew Monroe's Molecular Weight Calculator, which can be found at http://ncrr.pnnl.gov/software/

It does everything. Molecular weights. Isotopic patterns. And so, so, so much more.

It has one major drawback though - it's written for Windows. Luckily, it sort of works under Wine after you've done a bit of wine-trick-ery.

As great as the calculator is, sometimes you only need to calculate the molecular weight of something, and nothing else. Searching the debian repos I can' t find a single dedicated molecular weight calculator. In particular, a command-line driven calculator would be nice.

Seriously - it's a crying shame that the distribution with the largest repos, i.e. Debian, does not have a single passable molecular weight calculator. It is even more surprising given the number of chemistry-related packages which are present.

So, here's what I did:
a quick google on "python molecular weight calculator" brought me to http://pygments.org

A little bit of editing gave the code below, which was saved as molcalc, copied to /usr/bin, followed by sudo chmod +x /usr/bin/molcalc. It can now be called using
molcalc "(Co(CO)5)2"
and returns
The mass of (Co(CO)5)2 is 257.916890.

Here's the code, which is 99.9% the original and 0.1% my modification. All credit thus due to Lee, Freitas and Tucker.

NOTE: it doesn't handle layered parentheses. (Al(NO3)3)2 gets interpreted as Al2(NO3)3.

#!/usr/bin/python2.6
#########################################################################
# Author: Toni Lee with the help of Guilherme Freitas and Becky Tucker. Minor changes by Lindqvist
# Copyright: This module has been placed in the public domain
#########################################################################

#Import regular expressions
import re
import sys
try:
test=sys.argv[1]
except:
quit()

#Create the dictionary (From Becky with a value of 0 inserted for Uus(mass not measurable))
TableofElements ={ 'H':1.00794,'He':4.002602,'Li':6.941,'Be':9.012182,
                        'B':10.811,'C':12.0107,'N':14.0067,'O':15.9994,'F':18.9984032,'Ne':20.1797,
                        'Na':22.98976928,'Mg':24.3050,'Al':26.9815386,'Si':28.0855,
                        'P':30.973762,'S':32.065,'Cl':35.453,'Ar':39.948,'K':39.0983,'Ca':40.078,
                        'Sc':44.955912,'Ti':47.867,'V':50.9415,'Cr':51.9961,'Mn':54.938045,
                        'Fe':55.845,'Ni':58.6934,'Co':58.933195,'Cu':63.546,'Zn':65.38,'Ga':69.723,
                        'Ge':72.64,'As':74.92160,'Se':78.96,'Br':79.904,'Kr':83.798,'Rb':85.4678,
                        'Sr':87.62,'Y':88.90585,'Zr':91.224,'Nb':92.90638,'Mo':95.96,'Tc':98,
                        'Ru':101.07,'Rh':102.90550,'Pd':106.42,'Ag':107.8682,'Cd':112.411,
                        'In':114.818,'Sn':118.710,'Sb':121.760,'Te':127.60,'I':126.90447,
                        'Xe':131.293,'Cs':132.9054519,'Ba':137.327,'La':138.90547,'Ce':140.116,
                        'Pr':140.90765,'Nd':144.242,'Pm':145,'Sm':150.36,'Eu':151.964,'Gd':157.25,
                        'Tb':158.92535,'Dy':162.500,'Ho':164.93032,'Er':167.259,'Tm':168.93421,
                        'Yb':173.054,'Lu':174.9668,'Hf':178.49,'Ta':180.94788,'W':183.84,
                        'Re':186.207,'Os':190.23,'Ir':192.217,'Pt':195.084,'Au':196.966569,
                        'Hg':200.59,'Tl':204.3833,'Pb':207.2,'Bi':208.98040,'Po':210,'At':210,
                        'Rn':220,'Fr':223,'Ra':226,'Ac':227,'Th':232.03806,'Pa':231.03588,
                        'U':238.02891,'Np':237,'Pu':244,'Am':243,'Cm':247,'Bk':247,'Cf':251,
                        'Es':252,'Fm':257,'Md':258,'No':259,'Lr':262,'Rf':261,'Db':262,'Sg':266,
                        'Bh':264,'Hs':277,'Mt':268,'Ds':271,'Rg':272, 'Uus':0
}


#######################################
#Computes the MW of an atom-number pair
#######################################
def getMass(x):
    atom=re.findall('[A-Z][a-z]*',x)
    number=re.findall('[0-9]+', x)
    if len(number) == 0:
        multiplier = 1
    else:
        multiplier = float(number[0])
    atomic_mass=TableofElements[atom[0]]
    return (atomic_mass*multiplier)

################################################################
#Segments formula into atom-number sections (i.e. 'H3' or 'N10')
################################################################
def parseFormula(fragment):
    segments=re.findall('[A-Z][a-z]*[0-9]*',fragment)
    return (segments)

##################################################################################
#Computes total mass of both parenthetical and nonparenthetical formula components
##################################################################################
def molmass(formula):
    parenMass=0
    nonparenMass=0
    while (len(formula)>0):
        #First computes the molecular weight of all parenthetical formulas from left to right
        while (len(re.findall('\(\w*\)[0-9]+', formula))!=0):
            parenthetical=re.findall('\(\w*\)[0-9]+',formula)
            for i in range(0,len(parenthetical)):
                parenMult1 = re.findall('\)[0-9]+', parenthetical[i])
                parenMult2 = re.findall('[0-9]+', parenMult1[0])
                segments =parseFormula(parenthetical[i])
                for i in range(0, len(segments)):
                    parenMass= parenMass + ((getMass(segments[i]))*(float(parenMult2[0])))
            formula=re.sub('\(\w*\)[0-9]+', '', formula)
        #Sums nonparenthetical molecular weights when all parenthetical molecular weights have been summed
        segments = parseFormula(formula)
        for i in range(0, len(segments)):
            nonparenMass=nonparenMass + getMass(segments[i])
        formula=re.sub(formula, '', formula)

    Mass=parenMass+nonparenMass
    return Mass
     
if __name__ == '__main__':
test=test.split(',')
for element in test:
print ('The mass of %(substance)s is %(Mass)f.' % {'substance': \
element, 'Mass': molmass(element)})
 

19. Gnome3/gnome-shell -- 11 days later

So, I stuck it out. The new gnome-shell is fairly functional for me, finally.
The lack of system indicator applets was solved by using conky to overlay information on the desktop.
Frippery's gnome-shell extensions also make the experience less frustrating.

However, it still feels like everything I've done have been finding /work-arounds/ rather than /customisation/. In the end the whole idea of an application-centric desktop is unsuitable for me -- I would expect this to be true for the majority of established linux users.

Edit (25/11/2011):The screengrab shows what my heavily modified desktop looks like.

18. Gnome 3/Gnome-shell -- first impressions. Rant.

Debian testing has now transitioned to gnome-shell/gnome3. It's...different...from gnome 2.32, so be warned. While I'm as frustrated as anyone else who feels that they are being forced to move to a new desktop metaphor for no good reason, I'm trying to keep an open mind. The lack of a bottom panel is pissing me off and disorienting me enormously though. I also would like all my panel applets back - I used to have a good overview over how my computer hardware was doing, and now I have no clue anymore.

<RANT>
Essentially, at this point it seems like gnome-shell is fine for people who do their work in a specific application - like a browser, word processor etc. It absolutely blows if you're using your COMPUTER. I edit code in gedit, render figures in the terminal, inspect them using evince, include them in documents using LaTeX etc. Suddenly I feel I have no overview what's going on. The lack of a bottom panel showing me which applications are open on a specific virtual desktop is very confusing. Using alt+tab to check before I switch is a great time-waster.

For those who haven't used gnome-shell -- yes, you can still have windows side-by-side. You just can't see if you're hiding a window behind another one.

Seriously. I don't see why I can't customise my desktop anymore. Gnome has always (I do realise that this isn't entirely true - functionality is being removed and re-added all the time in most desktop environments) been a bit more restrictive than KDE in terms of granular control and I do understand that this is on purpose -- it's a design philosophy. I guess I just violently disagree with it.

Finally, even though I can't put my finger on WHY, I feel that I suddenly have a tiny screen. It's 23 inches. It's huge. It's made for having lots of windows open side-by-side.

I admit that I'm as resistant to change as anyone, but since I use my computer as part of my work, I need a damned good reason for changing. Few people can afford a few weeks downtime in productivity while learning the ropes if they feel that the change isn't justified.

I'll stick with gnome for a while longer. You can't bitch if you don't give it a chance. But I'll spend those weeks looking closer at the alternatives - XFCE, LXDE, KDE etc.

The point here -- and which was seen with the defections to OSX caused by Windows Vista -- is that if people are forced to learn a new way of working, they might as well explore ALL the options.

Prediction: I'll either stay with gnome-shell (which will hopefully improve as functionality and control is returned), or move to xmonad (another extreme)

I obviously appreciate the fact that I'm using a free and open source collection of software - in theory no-one is forcing me to keep on using GNOME. Nor was I forced to upgrade. In reality, it's not so easy.

Given that gnome 3/gnome-shell is more than just an iterative update, I think it would've have made sense to allow for the installation of gnome 2.32 and gnome3 side-by-side. After all, there's nothing preventing you from running KDE, gnome, xfce and lxde side-by-side. Sure, uptake would be slower - but 'forcing' people to move from one version to another isn't really a good idea either. The way it is now you have no easy way of reverting back to 'normal' if you accidentally, or misguidedly, upgrade to gnome3.

Oh well, ranting is easy. A better use of my time would probably have been to learn how to write gnome-shell-extensions to provide the functionality which I feel is missing.
</RANT>

<How to deal with it>
So, there are a few things which can be done to make the transitions a bit easier to handle -  do an online search for gnome-shell-extensions and download the ones which you think will help. For me, I've got the following installed and active:
Bottom Panel
windowNavigator
Gajim IM integration
Alternative Status Menu
Shut Down Menu
User Themes
Break Dynamic Workspaces
Panel Favorites
Weather
Applications Menu
Move Clock
Auto Move Windows

Make sure that you get the right version of the extension for your version of gnome-shell (gnome-shell --version; currently it's on 3.0.2) since extensions for 3.2 won't necessarily work with 3.0.2 (e.g. the bottom panel extension).

You will also want to install the gnome-tweak-tool and explore what it does. At least you can choose your preferred icon theme, set nautilus to handle the desktop space, bring back maximize/minimize buttons etc.

You may also want to add keyboard shortcuts to the most commonly used application since it's a PITA having to go back to the Activities every time you open e.g. a terminal, nautilus or gedit  instance. I've mapped terminal to ctral+shift+up, google chrome to ctrl+shift+down, nautilus to ctrl+shift+left and gedit to ctrl+shift+right.


If you find that you can't run gnome-shell but only use the fallback mode, check that you haven't got 'compiz --replace' in your start-up programs (gnome-session-properties)

17. Linphone with iinet iitalk voip service on Debian Testing 64 (Wheezy)

We've got the Naked DSL from iinet, and included with it is a 'free' voip service, iitalk. Since we haven't got either a modem supporting voip, or an ATA adapter + regular terrestrial phone, we're aiming for the next best - using a software voip solution on our desktop. Linphone is FOSS and available in the debian repos.

You will have received an email from iinet with the following details:
Your Netphone number:         0xxxxxxx
Your Netphone password:     yyyyyyy
Your Netphone PIN:                 zzzz
VoIP SIP domain:            iinetphone.iinet.net.au
VoIP SIP server:              sip.state.iinet.net.au

where state is act, nsw, vic, wa etc. giving something along the lines of sip.act.iinet.net.au


first
sudo apt-get install linphone

You'll find the GUI version on your gnome menu under 'internet'. Start it.
Go to Linphone/Settings
Click on the manage SIP accounts tab.
There are three main sections on this tab: default identity, Proxy accounts and Privacy
Don't bother with the Default identity or Privacy; instead click on Add under Proxy accounts


Your SIP identity: sip: 0xxxxxxx@sip.state.iinet.net.au
SIP proxy addres: sip: sip.state.iinet.net.au
Route (optional): sip:
Registration duration (sec): 3600
Register at startup: yes

You can also set up the account directly in your ~/.linphonerc :

[proxy_0]
reg_proxy=sip:sip.state.iinet.net.au
reg_route=sip:
reg_identity=sip:0xxxxxxx@sip.state.iinet.net.au
reg_expires=3600
reg_sendregister=1
publish=0
dial_escape_plus=0

[auth_info_0]
username=0xxxx
userid=0xxxxxx
passwd=yyyyyyyyyyy
realm="iinetphone.iinet.net.au"

The number after proxy_ and auth_info_ depends on whether you have other accounts defined as well. Note that the .linphonerc file will contain a fair number of other things. Here's my full .linphonerc (with some information replaced by x's)


[sound]
playback_dev_id=ALSA: default device
ringer_dev_id=ALSA: default device
capture_dev_id=ALSA: default device
remote_ring=/usr/share/sounds/linphone/ringback.wav

[video]
device=V4L2: /dev/video0
size=cif
enabled=0
display=0
capture=0
show_local=0
self_view=1

[GtkUi]
advanced_ui=1
uri0=0xxxxxxxx

[net]
download_bw=0
upload_bw=0
firewall_policy=0
mtu=0

[sip]
sip_port=5060
guess_hostname=1
contact=<sip:xxx@192.168.1.2>
inc_timeout=15
use_info=0
use_rfc2833=0
use_ipv6=0
register_only_when_network_is_up=1
default_proxy=0

[auth_info_0]
username=0xxxxxxx
userid=0xxxxxxxx
passwd=yyyyyyy
realm="iinetphone.iinet.net.au"

[rtp]
audio_rtp_port=7078
video_rtp_port=9078
audio_jitt_comp=60
video_jitt_comp=0
nortp_timeout=30

[audio_codec_0]
mime=speex
rate=32000
enabled=1

[audio_codec_1]
mime=speex
rate=16000
enabled=1

[audio_codec_2]
mime=speex
rate=8000
enabled=1

[audio_codec_3]
mime=GSM
rate=8000
enabled=1

[audio_codec_4]
mime=PCMU
rate=8000
enabled=1

[audio_codec_5]
mime=PCMA
rate=8000
enabled=1

[video_codec_0]
mime=MP4V-ES
rate=90000
enabled=1
recv_fmtp=profile-level-id=3

[video_codec_1]
mime=theora
rate=90000
enabled=1

[video_codec_2]
mime=H263-1998
rate=90000
enabled=1
recv_fmtp=CIF=1;QCIF=1

[video_codec_3]
mime=H263
rate=90000
enabled=1

[video_codec_4]
mime=x-snow
rate=90000
enabled=1

[proxy_0]
reg_proxy=sip:sip.state.iinet.net.au
reg_route=sip:
reg_identity=sip:0xxxxxxxxx@sip.state.iinet.net.au
reg_expires=3600
reg_sendregister=1
publish=0
dial_escape_plus=0

16. Using onboard video card with acceleration on Optiplex 990 on Debian

NOTE: another issue debian users should be aware of is the failure to reboot on sudo shutdown -r now. The fix is easy though: change GRUB_CMDLINE_LINUX in /etc/default/grub to read GRUB_CMDLINE_LINUX="reboot=pci"

--Start Here ---
The videocard (nvidia quadro something or other) my work Optiplex 990 came with only has a 'displayport', and the supplied adapter for it only has DVI-D (no analogue signal, so a VGA converter won't work). Since my 24" monitor doesn't have DVI, and I hate HDMI, and at the same time appreciate graphics acceleration (can't live without my wobbly windows in compiz), I had to generate an /etc/X11/xorg.conf which would allow me to use the motherboard graphics capabilities. Here's that xorg.conf (ripped out the video card, used smxi to generate an xorg-driver dependent xorg.conf):


Section "ServerLayout"
Identifier     "X.org Configured"
Screen      0  "Screen0" 0 0
Screen      1  "Screen1" RightOf "Screen0"
Screen      2  "Screen2" RightOf "Screen1"
InputDevice    "Mouse0" "CorePointer"
InputDevice    "Keyboard0" "CoreKeyboard"
EndSection

Section "Files"
ModulePath   "/usr/lib/xorg/modules"
FontPath     "/usr/share/fonts/X11/misc"
FontPath     "/usr/share/fonts/X11/cyrillic"
FontPath     "/usr/share/fonts/X11/100dpi/:unscaled"
FontPath     "/usr/share/fonts/X11/75dpi/:unscaled"
FontPath     "/usr/share/fonts/X11/Type1"
FontPath     "/usr/share/fonts/X11/100dpi"
FontPath     "/usr/share/fonts/X11/75dpi"
FontPath     "/var/lib/defoma/x-ttcidfont-conf.d/dirs/TrueType"
FontPath     "built-ins"
EndSection

Section "Module"
Load  "dbe"
Load  "record"
Load "dri"
Load  "dri2"
Load "glx"
Load  "extmod"
EndSection

Section "InputDevice"
Identifier  "Keyboard0"
Driver      "kbd"
EndSection

Section "InputDevice"
Identifier  "Mouse0"
Driver      "mouse"
Option    "Protocol" "auto"
Option    "Device" "/dev/input/mice"
Option    "ZAxisMapping" "4 5 6 7"
EndSection

Section "Monitor"
Identifier   "Monitor0"
VendorName   "Monitor Vendor"
ModelName    "Monitor Model"
EndSection

Section "Monitor"
Identifier   "Monitor1"
VendorName   "Monitor Vendor"
ModelName    "Monitor Model"
EndSection

Section "Monitor"
Identifier   "Monitor2"
VendorName   "Monitor Vendor"
ModelName    "Monitor Model"
EndSection

Section "Device"
Identifier  "Card0"
Driver "intel"
BusID       "PCI:0:2:0"
EndSection

Section "Device"
Identifier  "Card1"
Driver "intel"
BusID       "PCI:0:2:0"
EndSection

Section "Device"
  Identifier  "Card2"
Driver "intel"
BusID       "PCI:0:2:0"
EndSection

Section "Screen"
Identifier "Screen0"
Device     "Card0"
Monitor    "Monitor0"
SubSection "Display"
Viewport   0 0
Depth     1
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     4
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     8
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     15
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     16
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     24
EndSubSection
EndSection

Section "Screen"
Identifier "Screen1"
Device     "Card1"
Monitor    "Monitor1"
SubSection "Display"
Viewport   0 0
Depth     1
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     4
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     8
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     15
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     16
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     24
EndSubSection
EndSection

Section "Screen"
Identifier "Screen2"
Device     "Card2"
Monitor    "Monitor2"
SubSection "Display"
Viewport   0 0
Depth     1
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     4
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     8
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     15
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     16
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     24
EndSubSection
EndSection

Section "Extensions"
# Option      "Composite" "disable"
# Option      "RENDER" "disable"
EndSection

15. ORTEP-3 for Windows (Winortep) under Wine

Note: for those who use Mac OS X and get sent here by google, you can also use wine to run ortep3v2. See here for instructions on how to install wine: http://wiki.winehq.org/MacOSX

The installation file can be downloaded from http://www.chem.gla.ac.uk/~louis/software/ortep3/. Install in the same fashion as you would with any other windows program under wine. I will assume that you have installed it in c:\ortep3v2

Request a license the usual way. Make a file called Ortep3-license in the c:\ortep3v2 directory (/home/you/.wine/drive_c/ortep3v2). Put the license information in it:

########################################################################
Program license for : Ortep3 for Windows
User name  : XXXXXXXXXX
User email : XXXX@XXXXX.XXX
User affiliation : Department of Chemistry, XXXXX, XXXXX
Windows version  : Windows XP
License number   : XXXXXXXXXX
########################################################################

So far everything is the same as for a regular windows installation.
The one difference arrives when it comes to setting the ORTEP3DIR variable:
go to the linux shell, type regedit
Navigate to HKEY_CURRENT_USER/Environment
If you don't have the Environment directory, create a new key with that name, then create a string value which you call ORTEP3DIR. Double-click on the data field, and set it to c:\\ortep3v2

You should now be ready to go.

Winortep can write pov-ray files -- and they are well-written, logical and very easy to read/modify

WinOrtep is a full-featured program which can do just about anything.

14. Evolution email and Google Calendar(s)

The process is already described here:
http://www.ubuntugeek.com/how-to-sync-google-and-evolution-calendars-in-ubuntu.html.

It's not much of a post, but it may be of help to someone who tried the Google option when setting up the calendar sync in Evolution and found it less than satisfactory.

Edit 16/12/2011:
Recently (last two days, version 3.0.3-3) been getting error messages in evolution when trying to connect to my google calendar. While using the method described above for a while I switched to using 'google' instead of 'caldav' as the calendar type at some point. Well, it doesn't work anymore.

Here's how to use google calendar in Evolution:
New -> Calendar
Type -- CalDav


URL: caldav://www.google.com/calendar/dav/myemail%40gmail.com/events/
Use SSL: yes
User name: myemail@gmail.com
User email: myemail@gmail.com

Then clock 'Browse server for calendar and select the
+ /calendar/dav/myemail%40gmail.com/
         - myemail@gmail.com


Whether  you want to copy content offline or use it as default calendar is a personal choice.

Edit: 16/12/2011 -- 2 hours later
Well, it's not working anymore. Same TLS handshake BS as before.

"Error loading calendar
Cannot open calendar: Unexpected HTTP status code 6 returned (Peer failed to perform TLS handshake)"

What I can't understand is why there is no good description of this problem on the web, given that Evolution is the default email and calendar client for gnome. I don't like thunderbird/icedove+sunbird/iceowl much, but things seem to work better in general there.

13. phpSheduleIt - basic installation

phpScheduleIt is a nice web-driven instrument scheduling system, which I first come into contact with as an NMR user at the University of California at Davis, where it is used to manage the scheduling of the instruments in the NMR facility there.

You will need to have a LAMP server set up (see e.g. http://www.howtoforge.com/ubuntu_debian_lamp_server). You will need apache2, php and mysql to play nicely together. Basically follow steps one and two, then sudo /etc/init.d/apache2 restart.

Download phpScheduleIt from http://www.php.brickhost.com/

A basic way to get it up and running goes as follows:

1. create a directory under /var/www. In our example it will be /var/www/nmr
sudo mkdir /var/www/nmr. Unzip the file you downloaded from www.php.brickhost.com and put the files in the /var/www/nmr directory

2. in your terminal, go to /var/www/nmr and chmod o+x all files (including subdirectories)

3. cp /var/www/nmr/config/config.php.new /var/www/nmr/config/config.php

4. Edit it.
change $conf['app']['weburi'] = 'http://localhost/phpScheduleIt';
to

change $conf['app']['weburi'] = 'http://localhost/nmr'; 

edit other relevant items, such as adminEmail, defaultLanguage, timeFormat, emailType, defaultPassword etc,

The admin email is important, since this will become the administrator account.

5. go to http://localhost/nmr/install

If nothing happens, or you get an error message, it's time to start trouble-shooting. Going to the terminal and

cd /var/www/nmr/install

php index.php

might give some useful information. Most likely you have either a) made an error in config/config.php, or b) you haven't chmod:d the subdirectories.

6.  If all goes well, you are asked for your root user name and root mysql password (which you set during the installation). Thus, login as root with the correct password. Once you're in, hit the button saying 'create tables'. If all went well you can now delete the /var/www/nmr/install directory.

7. Go to http://localhost/nmr

You should be greeted with a log in window. Important: select create new user and enter the email address you gave as the administrator email in the config.php file. This will then automatically become an administrator account. It has nothing to do with the order you create accounts in - it's all about what email address you put in the config.php file. 

8.  Once you've set up the account you are able to create instruments, user groups, booking restrictions etc. 

Anyone who wants to use the facilities can create their own new registered users, BUT only the administrator(s) can assign resources to them. 

9. Be careful with your GMT setting...

This page has some information as well: http://www.m-osaka.com/jp/plaza/readme.html

12. Pulseaudio - controlling where the output goes

The non-free flash plugin can be a bit of a headache, since even if you install it and flash videos play nicely, you may not get any sound. On a laptop, you might not notice it, since the default output seems to be the laptop speakers. However, you may have problems piping the sounds through any attached USB speakers. Same goes for desktops, where the default output may be the sounds ports on your motherboard.

The fix is simple (assuming you have pulseaudio installed)
create an /etc/asound.conf file or a ~/.asoundrc file, with the following in it:

pcm.!default.type pulse
ctl.!default.type pulse

Rebooting should take care of things

11. Sorting out problems with NVIDIA or ATI graphics cards on Debian Testing

This bug has been a problem for a while now when running debian testing - if you do a dist-upgrade, chances are your prorietary drivers get borked.

Symptoms include gdm not starting i.e. no Gnome login window. Instead you get dumped into the terminal.

There are two fixes - one quick and dirty one and one which is more long term.

1. Quick and dirty
If you want Gnome up and running without necessarily using graphics acceleration, simply log in in your terminal and rename your xorg.conf.

sudo mv xorg.conf xorg.conf.old

This will allow you to use the graphics card on your motherboard again. It won't help with a PCI card much.

2. A real fix
SMXI. It may seems scary, but is fairly straightforward. Log in (presumably in the terminal)

sudo su
cd /usr/local/bin && wget -Nc smxi.org/smxi.zip && unzip smxi.zip && smxi


The sudo su logs you in as root. You could also sudo each command individually above.

Follow the instructions and let the smxi script install your graphics drivers.

10. Combining MS data into a matrix

The following script takes a series of files with comma-delimited data named 0v.csv, 10v.csv etc., extracts everything in column 4 (in my case it contains relative abundance values from MS) from row 7 to the bottom, and stores those columns in files called 0.dat, 10.dat etc.

The second column (in my case it contains m/z values) is extracted from one of the csv files, and stored in mz.x.

Next, all the .dat files are pasted together, with the columns side by side, and the mz.x data added as the first column.

The data is rotated using the rotate script I've published earlier.

A file with cone-voltage values, cv.xy, has already been prepared by hand (single column), and is pasted together with the m/z data.

procms.sh
#!/usr/bin.bash
for e in 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 220 240 260 280 300
do
tail -n +8 $e\v.csv | sed 's/\,/\t/g'| gawk '{print $4}' > $e.dat
done

#get m/z
tail -n +8  0v.csv | sed 's/\,/\t/g'| gawk '{print $2}' > mz.x

paste  0.dat 10.dat 20.dat 30.dat 40.dat 50.dat 60.dat 70.dat 80.dat 90.dat 100.dat 110.dat 120.dat 130.dat 140.dat 150.dat 160.dat 170.dat 180.dat 190.dat 200.dat 220.dat 240.dat 260.dat 280.dat 300.dat > all.dat
paste mz.x all.dat > $1dat
rotate $1.dat > $1\rot.dat
paste cv.xy $1\rot.dat > $1\tot.dat

9. rotating a matrix

I lifted this off of some website at some point, but can't remember where and so can't give credit.

rotate.sh
gawk '
{
    for (i=1; i<=NF; i++)  {
        a[NR,i] = $i
    }
}
NF>p { p = NF }
END {  
    for(j=1; j<=p; j++) {
        str=a[1,j]
        for(i=2; i<=NR; i++){
            str=str" "a[i,j];
        }
        print str
    }
}' $1

8. Very rudimentary peak-finding

The following scripts looks through a file with one x column and several y columns, and returns values larger than a certain cutoff.

Usage:
findpeaks filename columnnumber cutoffvalue
e.g.
findpeaks test.dat 3 0.01

findpeaks:
#!/usr/bin/python2.6
import sys
infile=sys.argv[1]
colno=int(sys.argv[2])
cutoff=float(sys.argv[3])

minsig=100
maxsig=0


spectra=open(infile,'r')



for spectrum in spectra:
spectrum=spectrum.rstrip('\n')
spectrum=spectrum.split(' ')
# print spectrum

try:
spectrum[colno]=float(spectrum[colno])
if spectrum[colno]>cutoff:
print spectrum[1],' ',spectrum[colno]

if spectrum[colno]<minsig:
minsig=spectrum[colno]
if spectrum[colno]>maxsig:
maxsig=spectrum[colno]


except:
a=0
maxsig=0
print "not working: "

print "max sig: ",maxsig,", minsig: ",minsig
spectra.close

7. Processing 1D Bruker nmr data

Bruker 1D binary NMR files can be processed using a combination of cat, grep, sed, gawk and od, together with python and octave (w/ octave-optim) for some fancy line-fitting.

 brukdig2asc:
 #!/bin/bash
#usage: brukdig2asc
SW=`cat acqus | grep 'SW_h' | sed 's/\=/\t/g' | gawk '{print $2}'| tr -d '\n'`
TD=`cat acqus | grep 'TD=' | sed 's/\=/\t/g' | gawk '{print $2}'| tr -d '\n'`
O=`cat acqus | grep '$O1=' | sed 's/\=/\t/g' | gawk '{print $2}'`
SFO=`cat acqus | grep 'SFO1=' | sed 's/\=/\t/g' | gawk '{print $2}'`
#TIME=16384
#SWEEP=23809.5238095238
#AQ=`echo "1/(23809.5238095238/(16384/2))" | bc -lq`
cp fid fid.bin
ls fid.bin | cpio -o | cpio -i --swap -u
od -An -t dI -v -w8 fid.bin| gawk '{print NR,$1,$2}'| sed '1,64d' >fid.asc1
pynmr $SW $TD $O $SFO
makespec

pynmr:
#!/usr/bin/python2.6
import sys
#print str(sys.argv)
sweepwidth=float(sys.argv[1])
nopts=int(sys.argv[2])
centrefreq=float(sys.argv[3])
basefreq=float(sys.argv[4])

aq=1/(sweepwidth/(nopts/2))
#print str(sweepwidth),str(nopts)
f=open('fid.asc1','r')
g=open('fid.asc','w')
for line in f:
    line=line.rstrip('\n')
    line=line.split(' ')
#    print line
    freq=float(line[0])/(nopts/2)*sweepwidth+(centrefreq-sweepwidth/2)
    line[0]=(float(line[0])/(nopts/2))*aq
    g.write(str(line[0])+'\t'+str(line[1])+'\t'+str(line[2])+'\t'+str(freq)+'\n')
f.close
g.close

makespec:

#!/bin/bash
octave --silent --eval "fid=load('fid.asc');
#make xaxis
[nopts b]=size(fid);
aq=max(fid(:,1));
sw=nopts/aq;
freqx=linspace(0,sw,nopts)';

#apodizing
lb=5/10000;
fid(:,2)=fid(:,2).*exp(-lb.*freqx);
fid(:,3)=fid(:,3).*exp(-lb.*freqx);

#phasing
spec=[fid(:,1) real(fftshift(fft(fid(:,2)+i*fid(:,3)))) imag(fftshift(fft(fid(:,2)+i*fid(:,3))))];
[a b]=size(spec); spec(a/2,2:3)=[0 0];
phc=linspace(0,2*pi,180);
maxsig=0;k=1;
for n=1:180;
        localmax=max( real( (spec(:,2)+i*spec(:,3)).*exp(i*phc(n)) ));
        if (localmax>maxsig)
                maxsig=localmax;
                k=n;
        endif
endfor;
#simple baseline
absd=inline('m+t*0','t','m');
guess=0;
[f m kvg iter corp covp covr stdresid z r2]=leasqr(fid(:,4),real((spec(:,2)+i*spec(:,3)).*exp(i*phc(k))),guess,absd);
#disp(m)
#disp(sqrt(diag(covp)))

#make spectrum
spectrum=[fid(:,4) real((spec(:,2)+i*spec(:,3)).*exp(i*phc(k)))-m imag((spec(:,2)+i*spec(:,3)).*exp(i*(phc(k)+pi/2)))-m];

#fitting
pkg load optim
[a b]=max(spectrum(:,2));
centre=fid(b,4);
guess=[10 max(spec(:,2))]; #centre width height
#disp(guess)
lorentzian=inline('p(2)*(1/pi)*(p(1)/2)./((t-centre).^2+(0.5*p(1))^2)','t','p');
[f p r2]=leasqr(fid((b-150):(b+150),4),spectrum((b-150):(b+150),3),guess,lorentzian);

#filter out artefacts from fitting set
filtered=[0 0];
res=floor((max(fid(:,4))-min(fid(:,4)))/nopts);
for l=(b-ceil(5*p(1)/res)):(b+5*ceil(p(1))/res)
delta=lorentzian(fid(l,4),p)-spectrum(l,2);

if (delta>(lorentzian(fid(l,4),p))/1.2)
# do nothing
else
filtered=[filtered; fid(l,4) spectrum(l,2)];
endif
endfor

filtered=[ filtered(2:size(filtered(:,2)),1) filtered(2:(size(filtered(:,2))),2)  ];
[f p r2]=leasqr(filtered(:,1),filtered(:,2),p,lorentzian);

#disp(p')
#disp(r2)
params=[centre centre/67.8 max(lorentzian(fid(:,4),p)) p(1) 1.000 p(2)];
disp(params)
#save
spex=[fid(:,4) real((spec(:,2)+i*spec(:,3)).*exp(i*phc(k)))-m imag((spec(:,2)+i*spec(:,3)).*exp(i*(phc(k)+pi/2)))-m lorentzian(fid(:,4),p)];
save spectrum.dat spex;"

6. Miscellaneous scripts

The following two scripts look through a set of files with paired xy values but not necessarily the same number of x values in each file, extracts all the x values from all the files (script numero uno), then makes sure that all x values are present in all files (and zero-fills to make up for missing data). In short, it's a way of creating files that can be used to generate a matrix of values where all rows have the same number of fields.

The first script:
homogenise.sh
 #!/bin/bash
for e in {0..200..10} {220..300..20}
    do
        cat $e.dat | gawk '{ printf("%s\t",$1) }'
        echo ""
    done

The second script:

makelist.py
#!/usr/bin/python2.6
import sys
infile=sys.argv[1]

f=open(infile,'r')
arr=[]

for line in f:
    line=line.rstrip('\n')
    line=line.split('\t')
    try:
        for i in line:
            i=float(i)
            if i not in arr:
                arr+=[i]
    except:
        arr=arr       

f.close

arr.sort()
mylist=arr
last = mylist[-1]
for i in range(len(mylist)-2, -1, -1):
    if last == mylist[i]:
        del mylist[i]
    else:
        last = mylist[i]

voltages=[0,10,20,30,40,50,60,70,80,90,100,110,120,130,140,150,160,170,180,190,200,220,240,260,280,300]
myys=[0]*len(mylist)

for n in voltages:
    print "voltage: ',n,'\n'
    f=open(str(n)+'.dat','r')
    g=open(str(n)+'full.dat','w')
    arrx=[]
    arry=[]

    for line in f:
        line=line.rstrip('\n')
        line=line.split(' ')
        try:
            line[0]=float(line[0])
            line[1]=float(line[1])
            arrx+=[line[0]]
            arry+=[line[1]]
          
        except:
            #do nothing
            print "fail"

    for i in range(0,len(arrx)-1):

        try:
            myys[mylist.index(arrx[i])]=arry[i]
        except:
            a=0           

    for i in range(0,len(myys)-1):
        g.write(str(mylist[i])+'\t'+str(myys[i])+'\n')

    f.close
    g.close

5. Debian testing - Error cracking CSS key. Check regionset

I don't normally buy or watch DVDs, since the movie studios and distributors are not treating the linux community fairly. However, I was recently given a movie to watch, which turned out to not be as straight-forward as I would've liked it to be.

Long story short, in spite of following the usual routine of installing libdvdcss2 and other relevant packages, the movie would not play. Starting a movie player from the command line, I could see what error messages where associated with the failure to play, and the key was "Error cracking css key".

After having tried a few different things, I ended up installing regionset. When running regionset (sudo regionset) it did not echo a value for the current region which my dvd player was set to - apparently the device had not been locked to a region yet. Setting the region of my dvd player to the region of the dvd caused playback to work for all movie player clients.

Whether playback did not work because the
1. region was not set at all
or
2. region was not set to the same region as the DVD
remains to be found out.

Typically, you can only change the region of your playback device a limited number of times (3-5), so you will want to change it as few times as possible.

4. Getting mouse and keyboard to work on Lenovo Thinkpad SL410 under Debian Testing/Wheezy

For the past few months (it seems) Debian Testing will install just fine on system. On the first boot, however, neither (internal or external) mouse nor keyboard will be responsive and to a casual observer it will appear that the system has hanged. This is fairly well documented and the solution is simple if you have an EXTERNAL keyboard and mouse available (but I'll show it here anyway):
1. Disconnect and reconnect your mouse and keyboard, or, on a laptop, connect a usb keyboard, so that you can log in.
2. Open the terminal  or a console (e.g. ctrl+alt+f1) and proceed to rename /run e.g. sudo mv /run /xrun

You could also delete the folder instead of renaming it. At this point you simply have to reboot and everything should be working without any reconnecting.

NOTE that everything will be responsive during the inital part of the boot - the mouse and keyboard do not stop working until startx/gdm3 starts.

If you do not have an external keyboard you would be advised to install openssh-server during the installation of Debian. This way you should be able to ssh to your laptop from another computer and fix it remotely.

I've set up four systems - three desktops and one laptop - using this approach, and all has been well. The past week a laptop that was set up this way suddenly booted into gdm3 with an unresponsive mouse and keyboard. Plugging in an external keyboard (which I do anyway to slow down the onset of carpal tunnel from using a keyboard at weird angles - I'm not much of a fan of touchpads either) made my laptop usable, but the problem has been annoying - a system which you use for work should be in, well, working order.

At any rate, the /run folder was back. I tried just removing /run/udev, and it didn't help. I couldn't easily remove /run, since it was locked, but by attaching the hdd to another system I could remove it. On boot I got a number of error messages saying that various folders under /var/run/ could not be accessed, and I never made it to either gdm3 or a console. Booting by selecting the 'rescue/recover' option in grub dropped me into a terminal, but I could little beyond creating a /run folder (sudo mkdir /run).

ls /var/run -lah showed that /var/run was linked to /run. /var/lock had also shown up in various error messages. I renamed the /var/run and the /var/lock files (xrun and xlock), made sure that there was no /run folder, and rebooted. Still not happy and no console - complains about various things missing. Booting using the recover option (or is it rescue? Whatever, you'll know when you see it) created the necessary files, but did not create a /run folder. Rebooting and selecting the normal startup option in grub now went off without a hitch, with gdm3 behaving and the touchpad working. It took a few nervous seconds after the start of gdm before the mouse became responsive, but work it did.

So, if you're having the same problem, and you haven't had much luck deleting /run, have a look at /var and see if there are any links.

Now, the big question is: why was /run recreated in the first place?

---

Links to this post:
http://forums.debian.net/viewtopic.php?f=6&t=86208

3. Compiling nwchem on Ubuntu 10.10 64 bit

** See post on 15/12/2011 for information about Debian 64 bit. It builds fine on Squeeze but not Wheezy. This seems to have to do with the version of mpich2.**

Figuring out how to compile nwchem with mpich support took a little while, but this seems to have worked:

First mpich2 and gfortran need to be installed (since it was not installed on a virgin system there may have been other required packages already installed)
sudo apt-get install mpich2 gfortran

I created a file called myconfig.sh in the nwchem directory, with the following content:

setenv LARGE_FILES TRUE
setenv TCGRSH /usr/local/bin/ssh
setenv NWCHEM_TOP /work/nwchem
setenv NWCHEM_TARGET LINUX64
setenv NWCHEM_MODULES all
setenv USE_MPI y
setenv USE_MPIF y
setenv MPI_LOC /usr
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include/mpich2
setenv LIBMPI "-lfmpich -lmpich"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran >& make.log

do a csh myconfig.sh and you should be good to go.

I then added the following to the end of my ~/.bashrc and sourced it:

PATH=$PATH://work/nwchem/bin/LINUX64
export NWCHEM_EXECUTABLE=/work/nwchem/bin/LINUX64/nwchem

Jobs can then be submitted (assuming that mpd is up) by
mpdrun -n 2 nwchem nameofjob.nw

2. Setting up a cheap-ish ubuntu cluster - the hardware

I've recently  become interested in setting up a small computational cluster for simple QM and MD work. Since it doesn't hurt sharing your experience with others, I'll post the details on this blog.
I'm not a computer/linux/computational chemistry expert - the goal of this project is as much to learn how to go about setting up a cluster, as getting a useful computational tool.

First, the hardware - the idea was to get something fairly cheap, but also reasonably useful. I opted for the following hardware (prices in USD):

30.99    V7 NS1132-N6 10/100/1000Mbps Unmanaged Networking Switch 5 x RJ45 2K MAC Address Table

Each node
39.99    BIOSTAR MCP6P3 AM3 NVIDIA GeForce 6150 / nForce 430 Micro ATX AMD Motherboard
77.00    AMD Athlon II X3 445 Rana 3.1GHz 3 x 512KB L2 Cache Socket AM3 95W Triple-Core Desktop Processor 
2*21.99    Mushkin Enhanced 2GB 240-Pin DDR3 SDRAM DDR3 1333 (PC3 10666) Desktop Memory Model 20-146-744
7.99    NIC Zonet ZEN3301E 10/ 100/ 1000Mbps PCI 32-Bit Network Adapter 1 x RJ45
24.99    APEX TX-381-C Black Steel Micro ATX Tower Computer Case (needs full size ATX PSU)
13.99    LOGISYS Computer PS480D 480W ATX12V Power Supply
3.99    APEVIA CF12S-BK 120mm Case Fan 

I ordered everything from Newegg, and all the parts arrived after three days, as promised.

I already had two SATA harddrives, and two IDE CD ROM drives, in addition to two IDE/PATA cables. The CD ROM drives aren't important, as the mobo bios allows for boot from USB, so you can install your OS that way (at least Ubuntu 10.10).

Assembling the two systems was easy, even though it was the first time I had done anything more complicated than installing a RAM stick or changing a hard drive. I'm not particularly good with my hands, so that really says something.

Here are the steps, in case someone is as useless as I am with hardware:
Preparing the motherboard:
1. The CPU was fitted on the motherboard - being an AM3 socket it was a simple matter of gently dropping the cpu in the socket with the yellow marking on the cpu being aligned with the yellow marking on the mobo.
2. Next the heatsink/fan was fitted. The heatsink arrived with a smooth layer of thermal grease already applied. While it's difficult to see whether the heatsink was in perfect position over the cpu, I did my best and everything went well.
3. The RAM memory was installed. Nothing complicated here.

Preparing the case:
1. The PSU was installed - nothing complicated. Just four screws to attach the PSU to the case.
2. The motherboard panel at the back of the case didn't fit my mobo, so I installed the one which came in the motherboard box. One of the PCI slot covers on the case was removed to make space for the gigabit ethernet card
3. The brass spacers that came with the case were attached to the wall of the case. The spacers are for the mobo to rest on. If you don't install them the mobo won't align with the openings at the back of the case, so there's little risk that you'll forget to do this.
4. A CD drive (IDE) and a harddrive (SATA) were installed, but not connected.

Assembly:
1. The motherboard was attached to the case by resting it on the brass spacers and aligning the holes in the mobo, then attaching the mobo with screws.
2. All the cables connected to the case were clearly labelled, as were the corresponding connections on the motherboard, so it was just a matter of matching one to the other.
3. The PSU cables were fairly simple to connect two - two cables were attached to the corresponding connections on the mobo (just going by shape), the sata power cable was connected to the HDD and a molex connector was connected to the IDE CD ROM drive.
4. The sata cable that came with the case was used to connect the motherboard with the harddrive. I had a couple of IDE cables around, and used it to connect the CD ROM drive to the motherboard.  It took two attempts to get the IDE thingy right as I first attached the cable the wrong way round - for an IDE cable with three plugs (one on each end, one slightly offset from the middle of the cable) you want to connect the end with the 'middle' connector closer to it to the CD ROM, and the other end to the MOBO. Makes sense, but getting it right the first time would save you a few minutes. Of course, an easier way of getting around this is probably by setting the jumper to cable-select.

One system was installed by booting from a Ubuntu 10.10 64 CD, and the other by booting from a USB stick prepared using usb-creator-gtk. I could've settled on minimal/server installs instead of full Ubuntu Desktop installs, but as I 1) will later modify grub to boot to terminal instead of starting Gnome, 2) don't have space contraints and 3) want to be able to use the computers as back-up workstations I opted for full installs.

The systems were updated after the install.

1. Setting up a cheap-ish ubuntu cluster - basic set-up

Note: I have two nodes - one master (beryllium) and one slave (lithium). These names will be used in the examples

Once all the hardware was assembled and linux up and running, I installed a host of programs that I'd be using:

Cluster tools:
sudo apt-get install mpich2 vnstat sinfo ganglia-monitor openmpi-*

For compiling:
sudo apt-get install build-essential gfortran gpp cpp 

Generally useful software:
sudo apt-get install gnuplot mpqc-openmpi gromacs-mpich gomacs-mpich octave octave-parallel qtoctave gnome-activity-journal maxima


I downloaded the nwchem source from here. We'll deal with the compilation and installation in the next post.

I followed this post to get up and running (scroll down) with nfs. Here's my take on it:

On the master node:
sudo apt-get install nfs-kernel-server nfs-common portmap

    We don't want portmap to use 127.0.0.1, so
sudo dpkg-reconfigure portmap

sudo /etc/init.d/portmap restart
sudo mkdir /work
sudo chmod a+wrx /work

sudo nano /etc/exports
  Add the following line to the end of the file:
 /work *(rw,sync)

sudo /etc/init.d/nfs-kernel-server restart
sudo exportfs -a

One the slave node:
sudo apt-get install portmap nfs-common
sudo mkdir /work

sudo nano /etc/fstab
   Add the following line:
beryllium:/work /work nfs   rw   0   0


To get it up and running immediately
sudo mount /work

Next, create a file called mpd.hosts - you can use the /work directory or the home directory. List the hosts/nodes and add, without spaces, the number of processors on each node:

lithium:3
beryllium:3


On beryllium
mpd --ncpus=3 &
mpdtrace -l

which returns something akin to beryllium_12345 (192.168.1.3). 12345 is the port number, which will change each time you start mpd

On lithium
mpd --ncpus=3 -h beryllium -p 12345 &

NOTE: I had problems before specifying --ncpus - gave an error about mpich2 consecutive number/IDs. The down-side is that when you submit a job it fills up the master node, then starts jobs on the slave node(s).