26 February 2013

345. Replacing gEdit with Kate

For some reason I can't get gEdit to auto-spellcheck as I type anymore. There's simply no option to do it. Funnily enough, my favourite editor vim does it without any issues -- but while I like vim and use it for most of my data processing, I prefer to edit e.g. html in gEdit. Habit, I suppose.

Anyway, while gEdit is great for everything else, it's given me an excuse to familiarise myself a bit more with Kate which I take is the KDE counterpart to gEdit. A counterpart that's on steroids like a lot of KDE applications (that's not necessarily a good thing -- see e.g. vim vs emacs -- but each to their own)

Another cool thing with kate is that you can run it in a vim compatibility mode. (Sure, there's also gVim for the real deal, but gVIM looks ugly on my computer and I'm a shallow person.)

1. Installation
sudo apt-get install kate

2. Associations
Edit ~/.local/share/applications/defaults.list
 text/plain=kate.desktop
 text/x-python=kate.desktop
 text/x-sql=kate.desktop
 text/html=kate.desktop
 text/dat=kate.desktop
 text/xml=kate.desktop

You can associate kate with as many formats are you want this way.

Create ~/.local/share/applications/kate.desktop
[Desktop Entry]
Name=kate
Comment=kate text editor
Exec=/usr/bin/kate
Type=Application
Categories=Office;Accessories;
Set dynamic word wrap, enable vim, override

3. Desktop
In gnome, either navigate through your menu to 'Main Menu' (under 'system tools') or, if you're running a stock gnome 3 without a menu, go to the overview mode (what you end up with when you put the mouse in the top left corner) and type in main menu.

Either way, go to accessories and tick the box next to Kate.

4. Set up kate
Start kate, go to settings, configure kate and set your defaults:
Enable vi mode

Set up automatic spell check

Activate Dynamic Word Wrap

And enjoy:



24 February 2013

344. Compile a kernel (3.8) without using kpkg

Note: There is a much better and easier way: http://verahill.blogspot.com.au/2013/02/342-compiling-kernel-38-on-debian.html . What follows is just done for the sake of exploration.

Post begins:
It is incredibly easy to compile your own kernel on Debian using kernel-package. See e.g. http://verahill.blogspot.com.au/2013/02/342-compiling-kernel-38-on-debian.html

In the interest of learning how to compile a kernel in a more generic way which is applicable to non-debian systems (arch, red hat etc.), here's a method which doesn't rely on kpkg (kernel-package).

The downside is that this method does not produce a set of .deb files, and that you can't uninstall it using apt.

I'll be brief since most things are covered e.g. here.  I looked at this post when writing this.

sudo apt-get install build-essential ncurses-bin
mkdir ~/tmp
cd ~/tmp
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.8.tar.bz2
tar xvf linux-3.8.tar.bz2
cd linux-3.8/
cat /boot/config-`uname -r`>.config
make oldconfig

Once you're done answering the questions, do

time make -j3

This takes 29 minutes (AMD Athlon II X3).

time make -j3 modules

This takes 4 minutes. Finish by running

sudo make modules_install
sudo make headers_install INSTALL_HDR_PATH=/usr/src/linux-3.8.0
sudo make install

Note that the default path for the headers is /usr/include, which doesn't play well with most programs that check for the presence of headers before installation, so use INSTALL_HDR_PATH to specify the destination (at least on debian).

make install takes care of initramfs and grub-update as well and generates
/boot/config-3.8.0
/boot/initrd.img-3.8.0
/boot/System.map-3.8.0
/boot/vmlinuz-3.8.0
And that's really it -- compiling a kernel even without kernel-packages is pretty easy. Reboot and everything should be in working order.

23 February 2013

343. 'track changes' in LaTeX: collaborative writing

While I find latex superior to word processors in almost every single aspect, there is one thing I have to admit that MS/Libre/Open Office have over LaTeX: track changes. Since LaTeX is nothing but code you can obviously use something like git or svn to manage versions of .tex code. There are some differences though -- e.g. git would show you the diff output, but not necessarily the context of the changes, and I'm having a difficult enough time convincing people to use LaTeX without having to worry about setting up a git server as well (I can obviously cheat around it by running a local git server, downloading their file and push it to my git server etc, but then I would be the only one who'd be able to track the changes. Also, it's not elegant).

I have a student who is currently preparing a thesis, and while I've managed to give most of my feedback via the pdf annotation function in mendeley, it doesn't feel natural when it should be a simple matter of editing the .tex directly (yet in a way that leaves the student in control over whether to accept or reject the changes).

So here's an exploration of three different ways of passing comments and corrections back and forth when using LaTeX.

I'm also not the first one to ask this question. See e.g. http://tex.stackexchange.com/questions/3653/what-is-the-best-way-to-track-changes-with-non-computer-people

1. Make a pdf and annotate it
 This is a simple and straightforward method for passing comments back and forth. It is not a very convenient method for making any substantial changes though. On top of that, linux does not currently have any particularly useful program for annotating pdf files. See here: http://verahill.blogspot.com.au/2013/02/338-annotating-pdfs-in-linux-revisited.html for more information.

Quite apart from the lack of tool, I really want to emphasize that making small annotations is not a replacement for a tool that allows you to really stir up the text.

2. Using git
You can easily set up your own git server and use that for versioning. http://verahill.blogspot.com.au/2013/01/324-setting-up-private-git-server.html

You can see the version differences using a couple of methods, but gitk might be the easiest one. The downside with this is that you lose some of the context of the changes. Also, it shows you the code that was changed, the changes in the final document (e.g. formatting, figures etc.).

Still it's a pretty intuitive and easy way of tracking changes -- and in terms of versioning it is certainly the best solution. You can fork, roll back and generally have a perfect account of how the document has evolved.
git in particular makes it easy to see added code

In terms of small changes like fixing typos it is a lot more subtle though




3. latexdiff
latexdiff is an easy to use tool that's in the debian repos. Simply do
latexdiff version1.tex version2.tex > diff.tex

and then open diff.tex and compile it to see the changes.

Pleasantly suprised




4. Other solutions
there are plenty of solutions for introducing mark-up directly into the tex document, like trackchanges, changes etc. I've used them in the past but it lack elegance when doing heavier collaborative writing.

Thoughts

I can't help but think that the best solution is to combine git with latexdiff -- or even a special fork of git that uses latexdiff.

Googling shows that apparently I"m not the only one who thinks so: http://gitorious.org/git-latexdiff#more as well as http://tex.stackexchange.com/questions/1325/using-latexdiff-with-git
I haven't actually tested any of those methods yet, and it still more or less requires a common git server.

20 February 2013

342. Compiling Kernel 3.8 on Debian Testing/Wheezy

NOTE: It seems like series 3.8 has issues with intel (i915) graphics -- it occasionally generates kworker threads that causes unresponsiveness as seen by slow mouse and keyboard response when e.g. plugging or unplugging mains power. No issues on e.g. nvidia though.
http://verahill.blogspot.com.au/2013/03/368-slow-mouse-and-keyboard-triggered.html
http://forums.gentoo.org/viewtopic-p-7278760.html
https://bbs.archlinux.org/viewtopic.php?pid=1248190

Post:

Kernel 3.8 is out now. Not much to say -- the compilation works well using the standard method. The compressed kernel is about 81 Mb to download.

The approach below shows how to compile the kernel on Debian. If you're interested in a more generic approach, see this post: http://verahill.blogspot.com.au/2013/02/344-compile-kernel-38-without-using-kpkg.html

NOTE: kernel 3.8 -- in contrast to the 3.7 series -- now compiles fine on AMD FX 8150.

NOTE: kernel 3.8 plays well with nvidia dkms

Here we go:
sudo apt-get install kernel-package fakeroot build-essential ncurses-dev
mkdir ~/tmp
cd ~/tmp
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.8.tar.bz2
tar xvf linux-3.8.tar.bz2
cd linux-3.8/
cat /boot/config-`uname -r`>.config
make oldconfig

You will be asked a lot of questions -- how many depends on what version you upgrade from. If in doubt, pick the default answer (i.e. hit enter). If really in doubt, use google.

Then continue:
make-kpkg clean

Do
make menuconfig

if you want to make any specific changes to the kernel (e.g. add support for certain devices)

Then continue:
time fakeroot make-kpkg -j4 --initrd kernel_image kernel_headers

As usual 4 is the number of threads you wish to launch -- make it equal to the number of cores that you have for optimum performance during compilation (more about that here).

The build takes around 20 minutes on a four-core intel i5-2400 with -j4, and 14 minutes on an fx-8150 with -j8 (96 minutes with -j1).

Install:
sudo dpkg -i ../linux-image-3.8.0_3.8.0-10.00.Custom_amd64.deb ../linux-headers-3.8.0_3.8.0-10.00.Custom_amd64.deb

New stuff/Questions:


Offload RCU callback processing from boot-selected CPUs (RCU_NOCB_CPU) [N/y/?] (NEW) *
Memory placement aware NUMA scheduler (NUMA_BALANCING) [N/y/?] (NEW) *
Enable to assign a node which has only movable memory (MOVABLE_NODE) [N/y/?] (NEW)
Allow for memory hot-add (MEMORY_HOTPLUG) [Y/n] y
Allow for balloon memory compaction/migration (BALLOON_COMPACTION) [Y/n/?] (NEW)
Set default setting of cpu0_hotpluggable (BOOTPARAM_HOTPLUG_CPU0) [N/y/?] (NEW
Debug CPU0 hotplug (DEBUG_HOTPLUG_CPU0) [N/y/?] (NEW)
ACPI tables can be passed via uncompressed cpio in initrd (ACPI_INITRD_TABLE_OVERRIDE) [N/y/?] (NEW)
Support multiple cpuidle drivers (CPU_IDLE_MULTIPLE_DRIVERS) [N/y/?] (NEW)
"NOTRACK" target support (DEPRECATED) (NETFILTER_XT_TARGET_NOTRACK) [N/m] (NEW

Default SCTP cookie HMAC encoding
  > 1. Enable optional MD5 hmac cookie generation (SCTP_DEFAULT_COOKIE_HMAC_MD5) (NEW)
    2. Enable optional SHA1 hmac cookie generation (SCTP_DEFAULT_COOKIE_HMAC_SHA1) (NEW)
    3. Use no hmac alg in SCTP cookie generation (SCTP_DEFAULT_COOKIE_HMAC_NONE) (NEW)
  choice[1-3?]:   Enable optional MD5 hmac cookie generation (SCTP_COOKIE_HMAC_MD5) [Y/?] (NEW) y

Enable optional SHA1 hmac cookie generation (SCTP_COOKIE_HMAC_SHA1) [N/y/?] (NEW) *
Enable optional MD5 hmac cookie generation (SCTP_COOKIE_HMAC_MD5) [Y/?] (NEW) y
Distributed ARP Table (BATMAN_ADV_DAT) [N/y/?] (NEW)
Kvaser CAN/USB interface (CAN_KVASER_USB) [N/m/?] (NEW)
LSI MPT Fusion SAS 3.0 Device Driver (SCSI_MPT3SAS) [N/m/?] (NEW)
Chelsio Communications FCoE support (SCSI_CHELSIO_FCOE) [N/m/?] (NEW) *
Marvell 88E6060 ethernet switch chip support (NET_DSA_MV88E6060) [N/m/y/?] (NEW)
Marvell 88E6085/6095/6095F/6131 ethernet switch chip support (NET_DSA_MV88E6131) [N/m/y/?] (NEW)
Marvell 88E6123/6161/6165 ethernet switch chip support (NET_DSA_MV88E6123_61_65) [N/m/y/?] (NEW) *
Cadence devices (NET_CADENCE) [Y/n/?] (NEW)
AT91RM9200 Ethernet support (ARM_AT91_ETHER) [N/m/y/?] (NEW)
Cadence MACB/GEM support (MACB) [N/m/y/?] (NEW)
Broadcom devices (NET_VENDOR_BROADCOM) [Y/?] y
Marvell MDIO interface support (MVMDIO) [N/m/y/?] (NEW)
CDC MBIM support (USB_NET_CDC_MBIM) [N/m/?] (NEW)
Atheros Wireless Cards (ATH_CARDS) [N/m/?] (NEW)
Atheros AR5523 wireless driver support (AR5523) [N/m/?] (NEW)
Wilocity 60g WiFi card wil6210 support (WIL6210) [N/m/?] (NEW) *
Realtek RTL8723AE PCIe Wireless Network Adapter (RTL8723AE) [N/m/?] (NEW)
ARC UART driver support (SERIAL_ARC) [N/m/y/?] (NEW) *
CBUS I2C driver (I2C_CBUS_GPIO) [N/m/?] (NEW)
TS-5500 DIO blocks and compatibles (GPIO_TS5500) [N/m/y/?] (NEW) 
TI BQ2415x battery charger driver (CHARGER_BQ2415X) [N/m/?] (NEW)
Board level reset or power off (POWER_RESET) [N/y/?] (NEW) *

 Default Thermal governor
  > 1. step_wise (THERMAL_DEFAULT_GOV_STEP_WISE) (NEW)
    2. fair_share (THERMAL_DEFAULT_GOV_FAIR_SHARE) (NEW)
    3. user_space (THERMAL_DEFAULT_GOV_USER_SPACE) (NEW)
  choice[1-3?]:   Fair-share thermal governor (FAIR_SHARE) [N/y/?] (NEW)

Step_wise thermal governor (STEP_WISE) [Y/?] (NEW) y
User_space thermal governor (USER_SPACE) [N/y/?] (NEW)
SSB GPIO driver (SSB_DRIVER_GPIO) [N/y/?] (NEW) *
BCMA GPIO driver (BCMA_DRIVER_GPIO) [N/y/?] (NEW)
Support for Realtek PCI-E card reader (MFD_RTSX_PCI) [N/m/y/?] (NEW)
TI ADC / Touch Screen chip support (MFD_TI_AM335X_TSCADC) [N/m/y/?] (NEW)
Support for Nano River Technologies Viperboard (MFD_VIPERBOARD) [N/m/?] (NEW)
Support for Retu multi-function device (MFD_RETU) [N/m/?] (NEW) *
Maxim MAX8973 voltage regulator  (REGULATOR_MAX8973) [N/m/?] (NEW)
TI TPS51632 Power Regulator (REGULATOR_TPS51632) [N/m/?] (NEW)
Siano SMS1xxx based MDTV receiver (SMS_USB_DRV) [N/m/?] (NEW)
Siano SMS1xxx based MDTV via SDIO interface (SMS_SDIO_DRV) [N/m/?] (NEW)   *
Stanton Control System 1 MIDI (SND_SCS1X) [N/m/?] (NEW) *
ION iCade arcade controller (HID_ICADE) [N/m/?] (NEW)
HID over I2C transport layer (I2C_HID) [N/m/?] (NEW) *
Renesas R-Car USB phy support (USB_RCAR_PHY) [N/m/?] (NEW)   *
SDHCI support for ACPI enumerated SDHCI controllers (MMC_SDHCI_ACPI) [N/m/?] (NEW)
NXP PCF8523 (RTC_DRV_PCF8523) [N/m/?] (NEW)
Philips PCF8563/Epson RTC8564 (RTC_DRV_PCF8563) [M/n/?] m
Userspace platform driver with generic irq and dynamic memory (UIO_DMEM_GENIRQ) [N/m/?] (NEW)
Microsoft Hyper-V Balloon driver (HYPERV_BALLOON) [N/m/?] (NEW) *
SystemBase PCI Multiport UART (SB105X) [N/m/y/?] (NEW)
TTY over Firewire (FIREWIRE_SERIAL) [N/m/?] (NEW) *
F2FS filesystem support (EXPERIMENTAL) (F2FS_FS) [N/m/y/?] (NEW) *
Enable CIFS debugging routines (CIFS_DEBUG) [Y/n/?] (NEW)
Simplified Mandatory Access Control Kernel Support (SECURITY_SMACK) [N/y/?] (NEW)
Camellia cipher algorithm (x86_64/AES-NI/AVX) (CRYPTO_CAMELLIA_AESNI_AVX_X86_64) [N/m/y/?] (NEW)



Links to this post:
http://forum.notebookreview.com/lenovo/575569-linux-x220-41.html
http://www.reddit.com/r/CrunchBang/comments/196zx5/whats_the_best_way_to_install_a_new_kernel_in/
http://www.lubuntu.ru/forum/viewtopic.php?t=1125
http://forums.linuxmint.com/viewtopic.php?f=143&p=711457
https://www.linux.org.ru/forum/linux-install/9128362
http://stackoverflow.com/questions/16573913/compile-new-kernel-on-debian-wheezy
http://crunchbang.org/forums/viewtopic.php?id=26825
http://mygoodluck.org/debian-obnovil-yadro-do-38

19 February 2013

341. Upgrading/installing BankID on 64 bit linux

Note: the post below is aimed at installing BankID on Debian (should be ok for ubuntu/mint too). For Arch Linux, see here (item 16)

There are a few ways to get around the rotten behaviour of bankid. This is one of them:

NOTE: to install nspluginwrapper you need to enable the stable/squeeze repos by e.g. adding
deb http://ftp.au.debian.org/debian/ squeeze main contrib non-free
to your /etc/apt/sources.list.

That's normally reasonably safe since apt by default pulls in the newest package and I haven't had any issues. Just be careful though.

You can also install nspluginwrapper by compiling it as shown here: http://verahill.blogspot.com.au/2013/03/366-nspluginwrapper-on-debian.html


Note that there's a FOSS alternative in Fribid (http://verahill.blogspot.se/2012/02/debian-testing-wheezy-64-fribid-as.html) which seems to be working perfectly -- and if you can use it, use it. The main limitation is that in practice you'll have to collect your certificate/ID with it, since newer versions of BankID saves the ID in an incompatible format. Like many foreigners, I don't have the opportunity to visit Sweden for the sole sake of picking up a new ID, so I'm stuck with BankID. But you may not be.



0. Things to install:
sudo apt-get install iceweasel nspluginwrapper ia32-libs

1. Download BankID and uninstall any previous installations
cd ~/Downloads
mkdir bankid
cd bankid
wget https://install.bankid.com/Download?defaultFileId=Linux -O bankid.tar.gz
tar xvf bankid.tar.gz
cd BISP-4.19.1.11663/
sudo sh install.4.19.1.11663.sh u

2. If you're upgrading, make sure to remove any previous libplugins.so
sudo updatedb && locate libplugins.so
/home/me/Downloads/bankid/BISP-4.19.1.11663/libplugins.so /usr/lib/mozilla/plugins/libplugins.so /usr/lib/mozilla/plugins/npwrapper.libplugins.so /usr/lib/nspluginwrapper/plugins/npwrapper.libplugins.so
sudo nspluginwrapper -r /usr/lib/mozilla/plugins/npwrapper.libplugins.so sudo rm /usr/lib/mozilla/plugins/libplugins.so sudo rm /usr/lib/nspluginwrapper/plugins/npwrapper.libplugins.so

3. Install the new version
sudo sh install.4.19.1.11663.sh i
Installing BankID Security Application ln: failed to create symbolic link `/usr/lib/firefox-addons/plugins': No such file or directory WARNING: Failed installing plugin for Firefox 3. Manually add symlink to libplugins.so in your Firefox 3 plugin directory if this browser is to be used. Installation complete.
sudo nspluginwrapper --install /usr/local/lib/personal/libplugins.so

4. Test your installation
Don't bother with test.bankid.com since the idiots won't let you test anything that identifies itself as 64 bit (more about that later). Instead
5. Testing against test.bankid.com
Everything is in working order but for some idiotic reason bankid.com won't even allow you to test you fancy new 64 bit installation -- and it all boils down the useragent string in iceweasel/firefox identifying itself as running on a 64 bit system (paradoxically, a real 32 bit browser running in a chroot won't work either since the kernel is 64 bit -- in schroot you can use personality=linux32 to get around it, but good luck dealing with the massive memory leaks).

Anyway,
  • open your iceweasel browser
  • type in about:config in the address bar
  • promise that you'll be careful
  • right-click on the page, select New, String
  • In the first box, type general.useragent.override
  • In the second box paste Mozilla/5.0 (X11; Linux i686; rv:10.0.12) Gecko/20100101 Firefox/10.0.12 Iceweasel/10.0.12
You can now go to https://test.bankid.com. I find this a bit humiliating though, and you can use bankid everywhere but bankid.com without having to set the useragent to identify your system as being i686.


If you're having issues, the first thing to check is about:plugins in firefox:


18 February 2013

340. Issues when compiling Nwchem 6.1.1 -- missing *.fh in src/include

The problem:
If you modify the nwchem sources, build, do a make realclean and then pack them up to export your patched sources to other nodes you might find that compiling doesn't work that well, yielding errors such as (I'll list them all to help google):

make[1]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/stdio.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(basis.o)'.  Stop.
make: *** [libraries] Error 1

and

make[1]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/nwc_const.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(basis.o)'.  Stop.
make: *** [libraries] Error 1
nwchem.F:3:0: fatal error: errquit.fh: No such file or directory
compilation terminated.
stubs.F:15:0: fatal error: errquit.fh: No such file or directory

and

make[1]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/errquit.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(basis.o)'.  Stop.
make: *** [libraries] Error 1

and

make[1]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/util.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(bas_input.o)'.  Stop.

and

/opt/nwchem/nwchem-6.1.1-src_mod/src/include/util.fh:1:0: fatal error: printlevels.fh: No such file or directory
compilation terminated.
make[1]: *** [/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(bas_input.o)] Error 1
make[1]: *** Waiting for unfinished jobs....

and

basisP.F: In function 'nbf_from_ucont':
basisP.F:427:0: warning: '__result_nbf_from_ucont' may be used uninitialized in this function [-Wmaybe-uninitialized]
make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[1]: warning: -jN forced in submake: disabling jobserver mode.
make[1]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/msgids.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(geom.o)'.  Stop.

and

make[1]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/bitops.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(pstat_alloc.o)'.  Stop.

and

In file included from pstat_alloc.F:12:0:
/opt/nwchem/nwchem-6.1.1-src_mod/src/include/bitops.fh:11:0: fatal error: bitops_decls.fh: No such file or directory
compilation terminated.
make[1]: *** [/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(pstat_alloc.o)] Error 1

and

In file included from pstat_alloc.F:12:0:
/opt/nwchem/nwchem-6.1.1-src_mod/src/include/bitops.fh:12:0: fatal error: bitops_funcs.fh: No such file or directory
compilation terminated.
make[1]: *** [/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(pstat_alloc.o)] Error 1

and

make[2]: warning: -jN forced in submake: disabling jobserver mode.
make[2]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/itri.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwcutil.a(sym_sh_pair.o)'.  Stop.
make[2]: *** Waiting for unfinished jobs....

and

make[2]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/bgj.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libnwints.a(exactd_mem.o)'.  Stop.

and

make[1]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/numerical_constants.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/librimp2.a(rimp2_v_e2.o)'.  Stop.

and

make[2]: *** No rule to make target `/opt/nwchem/nwchem-6.1.1-src_mod/src/include/util_sgroup.fh', needed by `/opt/nwchem/nwchem-6.1.1-src_mod/lib/LINUX64/libdntmc.a(gibbs.o)'.  Stop.

and

nwchem.F:11:0: fatal error: bgj_common.fh: No such file or directory

Solution:
This has been mentioned before on the nwchem forum, but not in explicit enough detail.
The solution is to copy a series of files from src/util and to remove make realclean from your build instructions (or at least do the copying after the make realclean step).

Copy these files
cp src/util/stdio.fh src/include/
cp src/util/nwc_const.fh src/include/
cp src/util/errquit.fh src/include/
cp src/util/util.fh src/include/
cp src/util/printlevels.fh src/include/
cp src/util/msgids.fh src/include/
cp src/util/bitops.fh src/include/
cp src/util/bitops_decls.fh src/include/
cp src/util/bitops_funcs.fh src/include/
cp src/util/itri.fh src/include/
cp src/util/bgj.fh src/include/
cp src/util/numerical_constants.fh src/include/
cp src/util/util_sgroup.fh src/include/
cp src/util/bgj_common.fh src/include/

Then build, using e.g.
export LARGE_FILES=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export PYTHONVERSION=2.7
export PYTHONHOME=/usr
export BLASOPT="-L/opt/openblas/lib -lopenblas"
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/lib/openmpi/lib
export MPI_INCLUDE=/usr/lib/openmpi/include
export LIBRARY_PATH=$LIBRARY_PATH:/usr/lib/openmpi/lib:/opt/openblas/lib
export LIBMPI="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran 1>make.log 2>make.err
export FC=gfortran
cd ../contrib
./getmem.nwchem

15 February 2013

339. Compiling ncdu on ROCKS 5.4.3/Centos 5.6

du is nice, but ncdu gives a better overview. Nothing odd about building it though:

mkdir ~/tmp
cd ~/tmp
wget http://dev.yorhel.nl/download/ncdu-1.9.tar.gz
tar xvf ncdu-1.9.tar.gz
cd ncdu-1.9/
sudo mkdir /share/apps/tools/ncdu -p
sudo chown $USER /share/apps/tools/ncdu
./configure --prefix=/share/apps/tools/ncdu
make
make install
echo 'export PATH=$PATH:/share/apps/tools/ncdu/bin' >> ~/.bashrc
source ~/.bashrc

Start by running
ncdu

13 February 2013

338. Annotating PDFs in linux -- revisited. Still no obvious solution

Update 28/02/2014: I've had a look at I, Librarian and Master PDF Editor here: http://verahill.blogspot.com.au/2014/02/558-more-options-for-pdf-annotation.html

There are a a few main reasons why even platform agnostic people have trouble moving to linux. One of them is poor compatibility of open solutions with MS Office (incompatibility between MS Equations and libre/openoffice is my main gripe, and some may find the lack of native EndNote an issue), and one quite particular to academics is the lack of proper pdf annotation (traditionally you edit your galley proofs by making annotations on the pdf -- my latest paper from Wiley came back with a doc file though, which was a bit surprising, but promising)

I've gotten by in the past by using 'pdf x-change viewer', which is a windows program, under wine. However, there are a lot of things I don't like about that solution, and the search for a native solution has continued to the point where I'm willing to throw money at it.

Why the exercise is unfair
The problem is that our definition of what 'works' and what doesn't is based on what we expect it to look like -- and that's based on our experience. I have colleagues that consider libreoffice 'crap' because it behaves or looks different from MS Office. While I agree that it's not viable as a replacement to office when collaborating with Office users who refuse to use libreoffice, it works fine if libreoffice is all you use. Anyway, it's unfair. Same goes for pdf annotation -- we expect it to look and function like adobe acrobat, simply because that's what we're used to. Any deviation is akin to a bug.

So keep that in mind when I write off some of the alternatives that actually do work -- just not in the way I expect them to.

I've found a lot of people (online) swearing by e.g.  Xournal, so even if it doesn't work for me, you may decide differently.

Commercial software in academia

I don't normally like buying software which is critical to my work. There are many reasons for it, and cost is only a minor one (although I don't like spending tax payer money on overpriced software).

Instead, if the software is critical to the science that I'm doing, I prefer to write my own algorithms in octave or python, and if it's a bit more peripheral I'm still wary of becoming reliant on a piece of software that may one day disappear -- either because I'm forced to upgrade through planned obsolescence, or because the company goes bankrupt/discontinues the software without releasing it as open source.

I cannot recommend Mendeley. And that's for two reasons:
1. it installs a list file in /etc/apt/sources.list.d/ without asking. That's the way malware (and google...) behaves!
2. There's plenty of mention about how it's 'free', but the free version is very restrictive, and if you create a private group you're stuck with a nagging message saying that you "must" upgrade if you want to add more users/groups. A simple one-time message would suffice.

Maybe I'm overreacting, but I don't appreciate this behaviour at all. Beyond that the annotation function works fine, so you decide for yourself.

Anyway, here's a short list of programs I've considered:
(and FOSS= Free and Open Source Software)



Evince
Evince is FOSS and now (since when?) supports adding annotations. It's not working very well (sometimes doesn't save annotation, slow), and you can't delete annotations, so be careful what you're saving.
Evince


pdfedit
pdfedit is another standard linux package. It does highlighting well, but annotates by adding text on top of the document -- not as a sticky note. FOSS.
pdfedit


flpsed
Does annotation, but as text superimposed on the pdf document, not as a collapsible sticky note. FOSS..

Qoppa PDF Studio
This is java based, and runs on Linux. I must already now say that there's one aspect of it that I really don't like: it's available in a Standard and a Professional version. That's the kind of artificial crippling of software that Microsoft likes to engage in, but I though we were beyond that on Linux...the price, $89, is too steep for something that I'd only use for annotations. Note also that the trial version puts a big nasty watermark over everything -- but you can hardly fault them for that, since it's not free. Closed source. Commercial.
Other than that it works, although it's not as pretty as mendeley.

Qoppa PDF studio


Xournal
Xournal can export annotated PDFs, but it doesn't do annotation in the same way as the other programs i.e. using sticky notes. Instead you can simply add text on top of the pdf, and it doesn't really do it for me. FOSS..
Xournal -- the annotations are not easy to spot


Whyteboard
It draws on top of PDF using imagemagick. FOSS.

Mendeley
While it's meant mainly for reference management, this does proper PDF annotation as well, and is platform agnostic. However, it is closed source, requires you to log in (even if you're using the desktop client) every time you use it, and needs you to explicitly keep documents (at least their titles etc.) out of the shared web catalogue. Other than that the pdf annotation works beautifully. I get really annoyed by the requirement to log in even when working offline though. It's free in the sense of gratis though -- but only up to 100 Mb of shared document space, you can only have one private group, and it can only have two members (+you). If you want more you need to pay (see e.g. here).
In practical terms, it seems to use GMT to time stamp annotationsand I haven't found an obvious way
of changing that (without going online). Also, it installs a file into /etc/apt/sources.list.d/ without asking.
Mendeley


Misc
Other 'solutions' that pop up is Okular which work in a roundabout way -- i.e the annotations aren't stored as part of the pdf. Again, it looks pretty, but the annotations are not exported with the pdf.
UPDATE: Note that this doesn't seem to be an issue anymore -- see comments below and this post: https://groakat.wordpress.com/2013/08/27/annotating-pdf-with-okular/ -- note that the new version is NOT in either wheezy or jessie i.e. they won't work.

okular


I tried FoxIt reader as well which claims to do annotation but doesn't on Linux -- and the windows version is not functional under linux/wine.

PDF X-change viewer/wine pops up so often as a suggested solution that it's beginning to look like spam. It does work though:
PDF Xchange viewer


An online only option is http://www.pdfescape.com -- but the paranoid part of me doesn't like the idea of uploading documents that are meant to be private.

12 February 2013

337. Modifying Nwchem 6.1.1 to work with GabEdit

Karol Strutynski left the following comment on a post about NWChem and Gabedit:

Hello,
I have one important comment:
The vectors coefficients in the nwchem output are incomplete!
The default behaviour of nwchem is to print 10 first coefficients with value bigger than 0.15. For systems with many atoms it is not enough, usually its not even close.

This behaviour is hard-coded in the nwchem source.
To change this you must search each instance of movecs_print_anal in the source code and replace 0.15d0 for smaller value in appropriate calls.
Furthermore you must change one loop in the src/ddscf/movecs_pr_anal.F file and around 200 line there will be loop:
do klo = 0, min(n-1,9), 2
You must increase the range of this loop, for something more reasonable like:
do klo = 0, min(n-1,199), 2

After recompiling the nwchem will print more coefficients and the gabedit will produce more reliable orbitals.

Best regards,
Karol Strutynski

So let's modify NWChem. I'll be modifying the 27th of June release of NWChem 6.1.1, which you'll obtain as Nwchem-6.1.1-src.2012-06-27.tar.gz from http://www.nwchem-sw.org/index.php/Download.


Change the number in red to something smaller (I tried 0.01d0) in the following files:
 /src/ddscf/uhf.F
 146  9611    continue
 147          call movecs_print_anal(basis, ilo, ihi, 0.15d0, g_movecs,
 148      $        'UHF Final Alpha Molecular Orbital Analysis',
 149      $        .true., dbl_mb(k_eval), oadapt, int_mb(k_irs),
 150      $        .true., dbl_mb(k_occ))
 151          call movecs_print_anal(basis, ilo, ihi, 0.15d0, g_movecs(2),
 152      $        'UHF Final Beta Molecular Orbital Analysis',
 153      $        .true., dbl_mb(k_eval+nbf), oadapt, int_mb(k_irs+nmo),
 154      $        .true., dbl_mb(k_occ+nbf)

/src/ddscf/scf_vec_guess.F
506          if (scftype.eq.'RHF' .or. scftype.eq.'ROHF') then
507             call movecs_print_anal(basis, 1,
508      &           nprint, 0.15d0, g_movecs,
509      &           'ROHF Initial Molecular Orbital Analysis',
510      &           .true., dbl_mb(k_eval), oadapt, int_mb(k_irs),
511      &           .true., dbl_mb(k_occ))
512          else
513             nprint = min(nalpha+20,nmo)
514             call movecs_print_anal(basis, max(1,nbeta-20),
515      &           nprint, 0.15d0, g_movecs,
516      &           'UHF Initial Alpha Molecular Orbital Analysis',
517      &           .true., dbl_mb(k_eval), oadapt, int_mb(k_irs),
518      &           .true., dbl_mb(k_occ))
519             call movecs_print_anal(basis, max(1,nbeta-20),
520      &           nprint, 0.15d0, g_movecs(2),
521      &           'UHF Initial Beta Molecular Orbital Analysis',
522      &           .true., dbl_mb(k_eval+nbf), oadapt, int_mb(k_irs+nmo),
523      &           .true., dbl_mb(k_occ+nbf))

/src/ddscf/rohf.F
155          endif
156          call movecs_print_anal(basis, ilo, ihi, 0.15d0, g_movecs,
157      $        'ROHF Final Molecular Orbital Analysis',
158      $        .true., dbl_mb(k_eval), oadapt, int_mb(k_irs),
159      $        .true., dbl_mb(k_occ))

/src/mcscf/mcscf.F
680       if (util_print('final vectors analysis', print_default))
681      $     call movecs_print_anal(basis,
682      $     max(1,nclosed-10), min(nbf,nclosed+nact+10),
683      $     0.15d0, g_movecs, 'Analysis of MCSCF natural orbitals',
684      $     .true., dbl_mb(k_evals), .true., int_mb(k_sym),
685      $     .true., dbl_mb(k_occ))
686 c

/src/nwdft/scf_dft_cg/dft_cg_solve.F
166           call movecs_fix_phase(g_movecs(ispin))
167           call movecs_print_anal(basis, ilo, ihi, 0.15d0,
168      &         g_movecs(ispin),blob,
169      &         .true., dbl_mb(k_eval+(ispin-1)*nbf),
170      &         oadapt, int_mb(k_irs+(ispin-1)*nbf),
171      &         .true., dbl_mb(k_occ+(ispin-1)*nbf))
172         enddo

/src/nwdft/scf_dft/dft_scf.F
1736             call movecs_print_anal(ao_bas_han, ilo, ihi, 0.15d0,
1737      &           g_movecs(ispin),
1738      &           blob,
1739      &           .true., dbl_mb(k_eval(ispin)), oadapt,
1740      &           int_mb(k_ir+(ispin-1)*nbf_ao),
1741      &           .true., dbl_mb(k_occ+(ispin-1)*nbf_ao))

/src/nwdft/scf_dft/dft_mxspin_ovlp.F
186       call movecs_print_anal(basis,int_mb(k_non),int_mb(k_non)
187      & ,0.15d0,g_alpha,'Alpha Orbitals without Beta Partners',
188      &   .false., 0.0 ,.false., 0 , .false., 0 )
189 c
190       if (nct.GE.2) then
191       do i = 2,nct
192       ind = int_mb(k_non+i-1)
193       call movecs_print_anal(basis,ind,ind
194      & ,0.15d0,g_alpha,' ',
195      &   .false., 0.0 ,.false., 0 , .false., 0 )
196       enddo
197       endif

352 c
353        call movecs_print_anal(basis, 1, nalp, 0.15d0, g_ualpha,
354      & 'Alpha Orb. w/o Beta Partners (after maxim. alpha/beta overlap)',
355      &   .false., 0.0 ,.false., 0 , .false., 0 )
356 c
Otherwise once could presumably edit the header in ./src/ddscf/movecs_pr_anal.F directly and substitute thresh. At a minimum you should edit that file according to Karol's instructions: change the number in red below to e.g. 199.

/src/ddscf/movecs_pr_anal.F
198             do klo = 0, min(n-1,9), 2
199                khi = min(klo+1,n-1)
200                write(LuOut,2) (
201      $              int_mb(k_list+k)+1,
202      $              dbl_mb(k_vecs+int_mb(k_list+k)),
203      $              (byte_mb(k_tags+int_mb(k_list+k)*16+m),m=0,15),
204      $              k = klo,khi)
205  2             format(1x,2(i5,2x,f12.6,2x,16a1,4x))
206             enddo

Compilation
At this point you should be able to follow post 242. Briefly: Compiling NWChem 6.1.1 with Python on Debian Testing (Wheezy) and compile nwchem with python etc. Don't forget to edit /src/config/makefile.h for python support as shown in that post. Once you're done with that you can compare the GabEdit plots with and without the modification.

Alternatively, if you're simply making changes to a copy of nwchem that you've compiled before, you can speed thing up by a factor of ca 300 by following this post:
http://verahill.blogspot.com.au/2013/04/380-modifying-nwchem-code-without-full.html



The difference:
I ran a job on benzene as described in post 281. Visualising NWChem output with GabEdit. I chose to run use the ELF (electron localisation function) on output from the unmodified and modified nwchem binaries. It's a pretty big difference:

Original

Modified

08 February 2013

336. Compiling ATLAS, netblas, lapack and openblas on Arch Linux

Here's another Arch post.

I was a bit surprised to find that there's no ATLAS in the standard Arch repositories (it is in AUR though), so here's how to build some of the more common math libraries for yourself:


ATLAS

pacman -S wget base-devel gcc-fortran cpupower
sudo systemctl enable cpupower

To build ATLAS you should set the governor for your CPU to performance to get the best optimization:

cpupower frequency-set -g performance
sudo cp /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor

Basically copy the scaling_governor to all cpus (cpu0, cpu1, cpu2 ...) as shown in the last line above. When you set the governor back to e.g. ondemand, follow the same steps.

sudo mkdir /opt/ATLAS
chown ${USER} /opt/ATLAS
mkdir -p ~/tmp/atlas
cd ~/tmp/atlas
wget http://www.netlib.org/lapack/lapack-3.4.2.tgz
wget http://downloads.sourceforge.net/project/math-atlas/Stable/3.10.1/atlas3.10.1.tar.bz2
tar xvf atlas3.10.1.tar.bz2
mkdir build/
cd build/
../ATLAS/./configure --prefix=/opt/ATLAS -Fa alg '-fPIC' --with-netlib-lapack-tarfile=$HOME/tmp/atlas/lapack-3.4.2.tgz --shared
make
make install

Simple as that. You can now change the governor back
cpupower frequency-set -g ondemand
sudo cp /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
sudo cp /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
...


netlib BLAS and lapack
pacman -S wget base-devel gcc-fortran cmake
sudo mkdir /opt/netlib
sudo chown $USER /opt/netlib
mkdir /opt/netlib/blas/lib -p
mkdir -p ~/tmp/blas
cd ~/tmp/blas
wget http://www.netlib.org/blas/blas.tgz
tar xvf blas.tgz
cd BLAS/

Edit make.inc
OPTS = -O3 -shared -m64 -march=native -fPIC
make all
gfortran -shared -Wl,-soname,libnetblas.so -o libblas.so.1.0.1 *.o -lc
ln -s libblas.so.1.0.1 libnetblas.so
cp lib*blas* /opt/netlib/blas/lib
cd ../
wget http://www.netlib.org/lapack/lapack-3.4.2.tgz
tar xvf lapack-3.4.2.tgz
mkdir /opt/netlib/lapack
mkdir build/
cd build/
ccmake ../lapack-3.4.2/ -DCMAKE_INSTALL_PREFIX=/opt/netlib/lapack -DBUILD_SHARED_LIBS=ON -DUSE_OPTIMIZED_BLAS=ON 

Hit c twice to configure, then g to generate.
Edit CMakeCache.txt and add the following lines at the beginning:
########################
# EXTERNAL cache entries
########################
 BLAS_FOUND:STRING=TRUE
 BLAS_GENERIC_FOUND:BOOL=TRUE
 BLAS_GENERIC_blas_LIBRARY:FILEPATH=/opt/netlib/blas/lib/libnetblas.so
 BLAS_LIBRARIES:PATH=/opt/netlib/blas/lib/libnetblas.so

Do
ccmake ../lapack-3.4.2/

again, then hit c once, then g.
Next,
make
make install

Done.

Openblas
Copied from here: http://verahill.blogspot.com.au/2013/02/334-compiling-nwchem-with-openmpi-and.html

Download from http://github.com/xianyi/OpenBLAS/tarball/v0.1.1

pacman -S wget base-devel gcc-fortran
sudo mkdir /opt/openblas
sudo chown $USER /opt/openblas
tar xvf xianyi-OpenBLAS-v0.1.1-0-g5b7f443.tar.gz
cd xianyi-OpenBLAS-e6e87a2/
make all BINARY=64 CC=/usr/bin/gcc FC=/usr/bin/gfortran USE_THREAD=0 INTERFACE64=1 1> make.log 2>make.err
make PREFIX=/opt/openblas install
cp lib*.*  /opt/openblas/lib


07 February 2013

335. Compiling sinfo in Arch linux

Note: I'm not abandoning Debian. I'm just going on a roadtrip with Arch to see whether we can all happily co-exist in the same machine park.

sinfo is available in AUR, but it's broken and won't build. Luckily it's very easy to build it yourself.

(see here for debian and here for centos)

mkdir ~/tmp
cd ~/tmp
sudo pacman -S wget base-devel boost asio
wget http://www.ant.uni-bremen.de/whomes/rinas/sinfo/download/sinfo-0.0.47.tar.gz
tar xvf sinfo-0.0.47.tar.gz
cd sinfo-0.0.47/
sed -i 's/\-mt//g' sinfo/Makefile.*
sed -i 's/\-mt//g' sinfod/Makefile.*
./configure
make
sudo make install


Not sure about a suitable systemd script (I'll update somewhere down the line). To start sinfod manually do
sudo sinfod --quiet 

if you have multiple ifaces you can set the --bcastaddress.

To start sinfo itself, well, run sinfo
sinfo

Not much fun with only one node...

334. Compiling nwchem with openmpi and python on Arch linux

Here's the reason why I gave my virtual machine 30 Gb in post 333 -- to be able to evaluate whether I can figure out how to build all the software that I need on Arch.

Behold my surprise when I realised that there's no need for separate -dev packages, as is the case on Debian i.e. the headers are generally installed together with the package (so e.g. python is enough -- you don't need python-dev as well).

While debian is probably the best choice for my nodes (I want stability -- not the latest flashiest stuff), nwchem is a good test case since I've been playing with it for years, and it's not available in the pacman or AUR repos.

Your mileage with openblas will vary depending on your hardware. ACML is an alternative on e.g. FX-8150. ATLAS doesn't seem to work with NWChem when I try it, but I'm not sure what I'm doing wrong. See the original post for examples on how to link to other math libs.

I'm mainly looking at this post: http://verahill.blogspot.com.au/2012/09/briefly-compiling-nwchem-611-with.html


Dependencies:
pacman -S wget base-devel gcc-fortran tcsh openmpi

Openblas:
Download from http://github.com/xianyi/OpenBLAS/tarball/v0.1.1

sudo mkdir /opt/openblas
sudo chown $USER /opt/openblas
tar xvf xianyi-OpenBLAS-v0.1.1-0-g5b7f443.tar.gz
cd xianyi-OpenBLAS-e6e87a2/
make all BINARY=64 CC=/usr/bin/gcc FC=/usr/bin/gfortran USE_THREAD=0 INTERFACE64=1 1> make.log 2>make.err
make PREFIX=/opt/openblas install
cp lib*.*  /opt/openblas/lib

Nwchem:
sudo mkdir /opt/nwchem
sudo chown $USER /opt/nwchem
cd /opt/nwchem
wget http://www.nwchem-sw.org/images/Nwchem-6.1.1-src.2012-06-27.tar.gz
tar xvf Nwchem-6.1.1-src.2012-06-27.tar.gz
cd nwchem-6.1.1-src/

Edit nwchem-6.1.1-src/src/config/makefile.h and edit line 1957 as shown in this post.

Then continue:
export LARGE_FILES=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export PYTHONVERSION=2.7
export PYTHONHOME=/usr
export BLASOPT="-L/opt/openblas/lib -lopenblas"
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/lib/openmpi
export MPI_INCLUDE=/usr/include
export LIBRARY_PATH=$LIBRARY_PATH:/usr/lib/openmpi:/opt/openblas/lib
export LIBMPI="-L/usr/lib/openmpi -lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"
cd $NWCHEM_TOP/src
make clean
make nwchem_config
make FC=gfortran 1> make.log 2>make.err
export FC=gfortran
cd ../contrib
./getmem.nwchem

Note that some of the locations are a little bit different from debian.

Edit your ~/.bashrc and add:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/openblas/lib export PATH=$PATH:/opt/nwchem/nwchem-6.1.1-src/bin/LINUX64
You can now test your new binary by running a job, e.g. co.nw:
title "co nmr" geometry c 0 0 0 o 0 0 1.13 end basis * library "6-311+g*" end dft direct grid fine mult 1 xc HFexch 0.05 slater 0.95 becke88 nonlocal 0.72 vwn_5 1 perdew91 0.81 end task dft optimize
Run:

nwchem co.nw

or

mpirun -n 2 nwchem co.nw

06 February 2013

333. Arch linux with Gnome in virtualbox

Not too different from what I've posted before (330, 331), but I'm just so excited that I can't help myself, so here we go:

I'll presume that you've got virtualbox installed.


0. Get the Arch iso:
If you're allowed to use bittorrent:

sudo apt-get install bittorrent squashfs-tools
cd ~/Downloads/
btdownloadcurses https://www.archlinux.org/iso/2013.01.04/archlinux-2013.01.04-dual.iso.torrent

If you're not allowed to use bittorrent:

cd ~/Downloads
wget http://mirror.aarnet.edu.au/pub/archlinux/iso/2013.01.04/archlinux-2013.01.04-dual.iso

Note that the links will change with time.



1. Setting up the machine
Start virtualbox and click on New. Select Type=Linux and Version=Arch Linux.

Give the machine at least 512 Mb, although even that is probably a bit on the low end for Gnome.

Create a virtual harddisk. 8 Gb is enough for testing purposes, but if you're actually planning on using the virtual machine you're better off giving it something like 15-20 Gb. VDI is fine when it comes to format, and 'Dynamically allocated' is a safe choice for a normal desktop.








There's a little bit of tinkering left to do. Click on Settings, go to Storage, and mount your iso:





2. First Boot
Click on Star which will boot off of the CD.


You can pick whatever makes you happy here. For the sake of simplicity will pick the first option -- x86_64.

You're now met with the first indication that this ain't Kansas (or Ubuntu):

Remember -- terminals/consoles should give you warm and fuzzy feeling because there is nothing which you can't accomplish in a terminal. A one-click installation option would be nice for checking things out though, but it's actually pretty easy to get started.


Preparing the harddrive
Our first step is to format the harddrive. Remember, it's an imaginary virtual harddrive -- nothing will be touched on your physical harddrive.

Do

fdisk /dev/sda

Hit n to create a new partition. p for primary, 1 to make it partition number 1, make it start at sector 2048, and make it e.g. 2 Gb by having it stop at +2G -- this will be our swap. Hit n again, p again, and make it partition number 2, and accept the default sectors that it suggests. This will be our  /(root) partition.

Hit a, then 2, to make the partition bootable.

Hit t, then 1, then 82 to make a swap disk.

Hit w to finalise the changes.

Do
mkswap /dev/sda1
mkfs.ext4 /dev/sda2

to set up the file systems.

Do
mount /dev/sda2 /mnt

to mount your new ext4 partition.

Install a basic system:
Do
pacman-key --init
pacman-key --populate archlinux

to get set up with pacman. Edit pacman.conf if you want a specific architecture (i686 or x86_64), or the current architecture will be used.

Time to install packages:
pacstrap /mnt base gnome xorg-xinit xorg-server xorg-utils xorg-server-utils virtualbox-guest-modules virtualbox-guest-utils vim mlocate wicd wicd-gtk openssh grub-bios upower

Generate a proto-fstab:
genfstab -p /mnt > /mnt/etc/fstab

Then enter a chroot:
arch-chroot /mnt

Edit /etc/fstab and add a line for swap:
/dev/sda1    none    swap    sw    0 0

Set up your locale and timezone:
ln -s /usr/share/zoneinfo/Australia/Melbourne /etc/localtime
echo 'LANG="en_US.UTF-8"'>/etc/locale
echo 'KEYMAP=us'> /etc/vconsole.conf
echo 'en_US.UTF-8 UTF-8' >> /etc/locale.gen
locale-gen
echo "spinebill" >/etc/hostname

Set up a user:
pacman -S sudo
useradd -p `openssl passwd -1 mypassword` -m verahill
echo "verahill ALL=(ALL) ALL">> /etc/sudoers

Edit /home/verahill/.xinitrc and uncomment
exec gnome-session

Set up grub and booting:
mkinitcpio -p linux
grub-mkconfig -o /boot/grub/grub.cfg
grub-install /dev/sda

You're now ready to start your Arch system. First you need to shut down your current system though:
exit
umount /mnt
shutdown -h now

Once the machine is off, go to Settings, Storage, and remove disk from virtual drive.



Booting:
Boot by clicking on Start.
You might get the odd error message while booting, but everything will be fine.

Log in as the user you created, then enabled the ssh daemon and the wicd daemon:
sudo systemctl enable sshd
sudo systemctl enable wicd

Start wicd-curses
wicd-curses

and wait 10-20 seconds for wicd to detect the network card and starting it


Quit by hitting 'Q'.

To launch gnome, type
startx
evening gnome?
It'll take a little while to start up. This is a pretty ugly Gnome though, but that's the stock background which changes during the day.
daytime gnome?



332. Gnuplot: bug(?) in current debian testing package

Update 2:
"Note that gnuplot uses both "real" and "integer" arithmetic, like FORTRAN and C. Integers are entered as "1", "-10", etc; reals as "1.0", "-10.0", "1e1", 3.5e-1, etc. The most important difference between the two forms is in division: division of integers truncates: 5/2 = 2; division of reals does not: 5.0/2.0 = 2.5. In mixed expressions, integers are "promoted" to reals before evaluation: 5/2e0 = 2.5. The result of division of a negative integer by a positive one may vary among compilers. Try a test like "print -5/2" to determine if your system chooses -2 or -3 as the answer."

http://www.physics.drexel.edu/courses/Comp_Phys/General/Graphics/gnuplot.html

I guess it's a feature. I guess I just need to remind myself of it every now and again...

Update: version 3.7.3 behaves the same way.


I use gnuplot for fitting data for scientific publications and every time I discover a bug it makes me seriously worried that I've somehow published something which will turn out to be incorrect.

It's not without reason -- Gnuplot version 4.4.0 in Debian Testing garbled small and large numbers: http://verahill.blogspot.com.au/2012/02/debian-testing-wheezy-64-bug-in-debian.html

I'm not sure whether this is a bug, but gnuplot 4.6.0 (current version in debian testing) screws up fractional powers.

gnuplot> print 9**(1/2)
1
gnuplot> print 9**(0.5)
3.0
gnuplot> print sqrt(9)
3.0

Somehow this works:
gnuplot> print 10**(2)
100
gnuplot> print 10**(2.0)
100.0
gnuplot> print 10**(6/3)
100
but not this
gnuplot> print 10**(1)
10
gnuplot> print 10**(1/2)
1
gnuplot> print 10**(1/3)
1
gnuplot> print 10**(0.3333333)
2.15443452467292
gnuplot> print 10**(1/3.0)
2.15443469003188

So the error seems to be due to the usual issue of distinguishing between integer values and real values e.g. the following example in Python 2.7:
>>> print 1/3
0
>>> print 1/3.0
0.333333333333

I still don't think this is intended behaviour in gnuplot, and it's certainly dangerous

Upstreams version 4.6.1
wget http://downloads.sourceforge.net/project/gnuplot/gnuplot/4.6.1/gnuplot-4.6.1.tar.gz
tar xvf gnuplot-4.6.1.tar.gz
cd gnuplot-4.6.1/
./configure --program-suffix=-4.6.1 --prefix=$HOME/.gnuplot-dev
make
make install
cd $HOME/.gnuplot-dev/bin
./gnuplot-4.6.1

gnuplot> print 9**(1/3)
1
gnuplot> print 10**(1/3)
1
gnuplot> print 10**(6/3)
100
gnuplot> print 10**(3/6)
1

03 February 2013

331. A full linux install on a USB stick: Arch and LXDE on a thumbdrive

Update 9 June 2013: I've updated the torrent URL for the current image. Note that you will likely have to do this every month or so. See https://www.archlinux.org/download/ for the latest image.

Original post:
Post 330 (How to install Arch linux) got me thinking: one of my most popular posts is post 70 ("Installing Debian on a USB stick -- live usb vs a true and full installation").

I'm new to Arch and LXDE and Openbox -- I normally run Debian with Gnome 3 -- so I'm probably not doing this the best way. But if you're also new to Arch and struggling with the beginner's wiki you may find some ideas here.

I admit: this post and post 330 aren't that different. Here I basically refine what I did in post 330, and adapt it for desktop use. I also don't have to worry about networking here since we're not talking about a headless box anymore, but an interactive system. There'll most likely be a third post at some point which will be a streamlined version of this post. Anyway.

Whatever I do here will obviously also apply to a desktop installation of Arch -- i.e. if you're struggling with LXDE/openbox on Arch, see here.

 Debian is fine and dandy, but the approach I describe there involves using a virtualbox to install debian on a thumbdrive.

In post 330 I learned how to use a chroot to set things up, which speeds things up significantly (you could of course just boot from a live CD etc., but that's too easy).

 Also, while I'm very happy with debian and it gives pretty small installs if done correctly, Arch might just give even leaner installs.[citation needed]

So here's how to install Arch linux and a graphical DE (LXDE -- I've never warmed to XCE for some reason).

We'll do 32 bit Arch here -- a reason to make a bootable linux drive is so that you can travel with your own safe desktop, and 32 bit linux will work on almost all hardware you're likely to encounter, while 64 bit linux is limited to more modern hardware. Note that if you really want to be secure you will need to encrypt the entire USB stick with e.g. truecrypt -- otherwise in particular your gpg/pgp keys are at risk. Ultimately, security is a matter of vigilance and risk reduction.

You do need a running linux machine  (or at least a live disk) -- I don't know how to do this from Windows.

Approach:
1. Formatting the thumbdrive
2. Chrooting the drive and installing stuff with pacstrap
3. Arch-chroot to install the boot loader, setting up internet, and installing a desktop


Formatting the thumbdrive
Plug in your drive to your running linux machine
df -h
rootfs 28G 21G 5.7G 79% / udev 10M 0 10M 0% /dev tmpfs 397M 852K 396M 1% /run /dev/disk/by-uuid/d7479df8-0d84-4236-a92c-b05327b8fa3d 28G 21G 5.7G 79% / tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 2.3G 728K 2.3G 1% /run/shm /dev/sda5 112G 82G 25G 78% /home /dev/sdb2 937M 18M 872M 2% /media/ext /dev/sdb1 1014M 4.5M 1009M 1% /media/win32
sudo umount /dev/sdb1 /dev/sdb2

Time to start fdisk. First we delete any existing partitions (and data...), then we make one partition, and then we'll make it bootable. We won't bother with a swap disk -- we don't have the space and we want to keep disk i/o to a minimum.

sudo fdisk /dev/sdb
Command (m for help): p Disk /dev/sdb: 2063 MB, 2063597056 bytes 64 heads, 62 sectors/track, 1015 cylinders, total 4030463 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x000cf539 Device Boot Start End Blocks Id System /dev/sdb1 62 2079231 1039585 e W95 FAT16 (LBA) /dev/sdb2 2079232 4027519 974144 83 Linux Command (m for help): d Partition number (1-4): 1 Command (m for help): d Selected partition 2 Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): 1 First sector (2048-4030462, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-4030462, default 4030462): Using default value 4030462 Command (m for help): a Partition number (1-4): 1 Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
Create the filesystem:
sudo mkfs.ext4 /dev/sdb1
mke2fs 1.42.5 (29-Jul-2012) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 125952 inodes, 503551 blocks 25177 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=515899392 16 block groups 32768 blocks per group, 32768 fragments per group 7872 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done
sudo fsck /dev/sdb1
fsck from util-linux 2.20.1 e2fsck 1.42.5 (29-Jul-2012) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/sdb1: 11/125952 files (0.0% non-contiguous), 16846/503551 blocks
Make sure to verify you filesystem.


Get the arch iso and set up your chroot:
Note that you will most likely have to update the torrent URL every month or so. Obviously you'll need to change the mount command as well according to the iso name.
sudo apt-get install bittorrent squashfs-tools
cd ~/Downloads/
btdownloadcurses https://www.archlinux.org/releng/releases/2013.06.01/torrent/
sudo mount -o loop archlinux-2013.06.01-dual.iso /mnt
unsquashfs -d arch /mnt/arch/i686/root-image.fs.sfs

If you want x86_64 do unsquashfs -d arch/ /mnt/arch/x86_64/root-image.fs.sfs instead

Time to get busy:
sudo umount /mnt
sudo mount -o loop arch/root-image.fs /mnt
sudo cp /etc/resolv.conf /mnt/etc/resolv.conf
sudo mount -o bind /proc /mnt/proc
sudo mount -o bind /sys /mnt/sys
sudo mount -o bind /dev /mnt/dev
sudo mount -o bind /dev/pts /mnt/dev/pts
sudo chroot /mnt
[root@lithium /]# 

The first chroot:
mount /dev/sdb1 /mnt
mkdir /run/shm
haveged -w 1024
pacman-key --init
pacman-key --populate archlinux

If you're installing i686 (i.e. 32 bit) linux from an x86_64 (i.e. 64 bit) linux, edit /etc/pacman.conf and set Architecture:
Architecture = i686
Continue:
pacstrap /mnt base vim grub-bios openssh wicd lxde-common lxsession desktop-file-utils openbox upower
(196/196) installing openbox [#############] 100% Place menu.xml, rc.xml and autostart in ~/.config/openbox They can be found in /etc/xdg/openbox Optional dependencies for openbox pyxdg: for the xdg-autostart script
This will download a large number of packages (196 packages; ca 178 M download, ca 750 Mb installed), and will take a while. Make sure that all packages end with the architecture you want (e.g. -i686 or -x86_64) or '-any'. I've omitted base-devel above, since you might not need it.

Generate a proto-fstab:
genfstab -p /mnt >> /mnt/etc/fstab

and edit it (/mnt/etc/fstab) to read something like (the uuid will differ):
UUID=c540eff0-0e10-4e7a-9fb6-190a46f38203 / ext4 rw,relatime,data=ordered 0 1 /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0 /dev/sdb1 /media/usb0 auto rw,user,noauto 0 0
The last two lines will automount CDs and usb sticks. It's not necessarily something you do want to allow -- so you may comment them out.


The second chroot:
arch-chroot /mnt

edit /etc/pacman.conf and change Architecture=auto to i686.

pacman -Syy
ln -s /usr/share/zoneinfo/Australia/Melbourne /etc/localtime
echo "kookaburra" > /etc/hostname

Edit /etc/locale.gen and uncomment the locales you want e.g.
en_GB.UTF-8 UTF-8
locale-gen
echo 'LANG="en_GB.UTF-8"'>/etc/locale
echo 'KEYMAP=us'> /etc/vconsole.conf
mkinitcpio -p linux
grub-mkconfig -o /boot/grub/grub.cfg
grub-install /dev/sdb

Don't screw up that last step! If you put the wrong device the arch install won't boot, and most likely not the system you chrooted it from either. i.e. don't accidentally type /dev/sda (or sdb is you have two disks -- it's not difficult to repair, but you'll need a live CD already prepared)

Continuing to set up LXDE/Openbox and network:

pacman -S ifplugd xorg-xinit xorg-server xorg-utils xorg-server-utils

I've only got nvidia boxes at home, but if you're travelling you're likely to run into intel and ati as well. You can probably skip the last package since we're pulling in nouveau

pacman -S xf86-video-nouveau xf86-video-ati xf86-video-intel xf86-video-nv nvidia


At this point startx would get you a black screen with a mouse cursor. Right-clicking on the background yields a menu, but you don't have anything installed.

There are a couple of services we want to run on boot:
systemctl enable wicd
systemctl enable sshd

Edit /etc/wicd/manager-settings.conf;change wired_interface to whatever you're using in udev.

The pc beep on tab is annoying me:
echo "blacklist pcspkr" > /etc/modprobe.d/nobeeep.conf


Create a user:
pacman -S sudo
useradd verahill -m
echo "verahill ALL=(ALL) ALL">> /etc/sudoers
su verahill
cd ~

mkdir ~/.config/openbox -p
cp /etc/xdg/openbox/*.xml ~/.config/openbox
cp /etc/xdg/openbox/* ~/.config/openbox
echo "exec startlxde" > ~/.xinitrc
echo "export DESKTOP_SESSION=LXDE">> ~/.bashrc
exit


Optional:
At this point we have a setup which supports both wired and wireless internet (depending on drivers) and which has a very rudimentary desktop. We're now using 1.4 Gb, out of which 231 M is the package cache.

The range of programs that you may want to install is probably fairly personal, and will also depend on the size of your USB stick. Here's a basic selection which fits me (install as root, or try sudo, although the latter option works best if you actually boot from your USB stick rather than chroot it):


pacman -S truecrypt firefox chromium thunderbird flashplugin

(when asked, ttf-bitstream-vera is fine)

Miscellaneous packages:
pacman -S conky guake leafpad gparted elinks mcabber mutt gajim screen mlocate xterm lxpanel menumaker obmenu obconf alsa-utils volwheel pulseaudio-alsa

To improve the functionality of screen and vim, see here and here. Setting up Mutt and Mcabber is described here and here. Conky is here (item 4).

To sort out sound you might need to create ~/.asoundrc and fill it with:

pcm.!default.type pulse
ctl.!default.type pulse

(Not everyone is a fan of pulseaudio though)

And see this post for a bit more on encrypting communication and files (I travel in China so these things do matter)): http://verahill.blogspot.com.au/2012/05/encrypting-your-email-chat-and-phone-in.html

Some basic science packages that let's me do basic work on the road:
pacman -S octave gnuplot maxima gummi texlive-science gimp

If you pull in all those packages you'll need a pretty big USB drive though (albeit you actually have a small build environment installed) -- it comes in at about 3.4 Gb. Skip the science stuff to save space.

Mandatory:
And I think that's that. Time to safely umount:
exit
exit
umount /mnt
exit
sudo umount /mnt/proc
sudo umount /mnt/sys
sudo umount /mnt/dev/pts
sudo umount /mnt/dev
sudo umount /mnt

To boot:
Plug in your USB stick and hit whatever key or key-combination during BIOS start up is necessary to interrupt boot and allow you to select your boot device. Sometimes it's F9, F2, F12 or del.

On boot:
Your first task  will be to get your internet up and running. Assuming that you haven't already played with /etc/udev/rules.d you need to find out what the interfaces are called. Do ifconfig -a, note the names, and then start wicd-curses. Select Preferences and enter the name of e.g. the wired interface.

Then start your desktop by running startx. There'll be plenty of menu options, but most programs will be missing, but that's what your ~/.config/openbox/menu.xml is for. And that's how far I've gotten. Time to explore.

Potential issues:
It doesn't always work on first try -- for some reason. I had to start from scratch twice before everything worked without a hitch. Often it's easier starting from scratch than trying to fix a basic installation.

You might not be able to run the systemctl commands until you've actually booted from the usb stick.

If you're on a laptop and the mouse/touchpad isn't working, install xf86-input-synaptics.

Consider installing wicd-gtk if you'll be working a desktop environment a lot.