319. Collection of errors when compiling kernel 3.7.x on AMD FX 8150

Update 20th Feb 2013:  Kernel 3.8 compiles fine on AMD FX 8150. Just ignore kernel 3.7.

Original post:

This is a collection of errors
(although you can obviously compile the kernel this way -- just not on AMD FX 8150...)

It does not hold any solutions

See here for how to compile kernel 3.7.0 to 3.7.1:
http://verahill.blogspot.com.au/2012/12/compiling-kernel-37-on-debian.html
and here for how to patch 3.7.1
http://verahill.blogspot.com.au/2012/12/patching-kernel-371-to-fix.html

See here for how to compile 3.7.2 and 3.7.3:
http://verahill.blogspot.com.au/2013/01/321-compiling-kernel-372-on-debian.html

Post continues:
Normally this wouldn't merit its own post, but because I've been annoyed over the inability to compile
kernels on my AMD FX 8150 I decided to collect as much data on the errors as possible.

Basically, kernel 3.7.x fails to build on AMD FX 8150, while kernels 3.6.3 and 3.5 build just fine.
Here's the lspci -n hardware profile for my system, although I'm fairly certain it's due to AMD FX 8150:

00:00.0 0600: 1002:5a14 (rev 02)
00:04.0 0604: 1002:5a18
00:09.0 0604: 1002:5a1c
00:11.0 0106: 1002:4390 (rev 40)
00:12.0 0c03: 1002:4397
00:12.2 0c03: 1002:4396
00:13.0 0c03: 1002:4397
00:13.2 0c03: 1002:4396
00:14.0 0c05: 1002:4385 (rev 42)
00:14.1 0101: 1002:439c (rev 40)
00:14.2 0403: 1002:4383 (rev 40)
00:14.3 0601: 1002:439d (rev 40)
00:14.4 0604: 1002:4384 (rev 40)
00:14.5 0c03: 1002:4399
00:15.0 0604: 1002:43a0
00:15.1 0604: 1002:43a1
00:15.2 0604: 1002:43a2
00:15.3 0604: 1002:43a3
00:16.0 0c03: 1002:4397
00:16.2 0c03: 1002:4396
00:18.0 0600: 1022:1600
00:18.1 0600: 1022:1601
00:18.2 0600: 1022:1602
00:18.3 0600: 1022:1603
00:18.4 0600: 1022:1604
00:18.5 0600: 1022:1605
01:00.0 0300: 10de:0a65 (rev a2)
01:00.1 0403: 10de:0be3 (rev a1)
02:00.0 0c03: 1b6f:7023 (rev 01)
03:06.0 0200: 10ec:8169 (rev 10)
04:00.0 0200: 10ec:8168 (rev 06)

The lower number of errors for kernel 3.7 below is because I did -j1 (so fewer threads to finish), not because it's any better.

Compilation (-O1 example)

mkdir ~/tmp
cd ~/tmp
wget http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.7.2.tar.bz2
tar xvf linux-3.7.2.tar.bz2
cd linux-3.7.2/
cat /boot/config-`uname -r`>.config
make oldconfig
make-kpkg clean

find -name Makefile|xargs -I {} sed -i 's/\-O2/\-O1/g' {}
time fakeroot make-kpkg -j9 --initrd kernel_image kernel_headers

Errors:

---

-O2 on kernel 3.7.2 with AMD FX 8150:

net/mac80211/mesh_hwmp.c:1224:1: internal compiler error: in dwarf2out_finish, at dwarf2out.c:22603
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[3]: *** [net/mac80211/mesh_hwmp.o] Error 1
make[2]: *** [net/mac80211] Error 2
make[2]: *** Waiting for unfinished jobs....
net/netfilter/xt_esp.c:94:1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[3]: *** [net/netfilter/xt_esp.o] Error 1
make[2]: *** [net/netfilter] Error 2
make[1]: *** [net] Error 2
make[1]: *** Waiting for unfinished jobs....
/home/me/tmp/linux-3.7.2/arch/x86/include/asm/msr-index.h:1:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[6]: *** [drivers/net/wireless/ath/ath9k/ar5008_phy.o] Error 1
make[6]: *** Waiting for unfinished jobs....
make[5]: *** [drivers/net/wireless/ath/ath9k] Error 2
make[4]: *** [drivers/net/wireless/ath] Error 2
make[3]: *** [drivers/net/wireless] Error 2
make[3]: *** Waiting for unfinished jobs....
/home/me/tmp/linux-3.7.2/arch/x86/include/asm/elf.h:78:1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[4]: *** [drivers/tty/ipwireless/main.o] Error 1
make[3]: *** [drivers/tty/ipwireless] Error 2
make[3]: *** Waiting for unfinished jobs....
drivers/tty/serial/jsm/jsm_neo.c: In function 'neo_intr':
drivers/tty/serial/jsm/jsm_neo.c:1244:1: internal compiler error: Aborted
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[5]: *** [drivers/tty/serial/jsm/jsm_neo.o] Error 1
make[4]: *** [drivers/tty/serial/jsm] Error 2
make[3]: *** [drivers/tty/serial] Error 2
make[2]: *** [drivers/tty] Error 2
make[2]: *** Waiting for unfinished jobs....
drivers/staging/silicom/bp_proc.c:74:31: warning: 'bp_procfs_dir' defined but not used [-Wunused-variable]
drivers/staging/silicom/bp_proc.c:76:31: warning: 'proc_getdir' defined but not used [-Wunused-function]
{standard input}: Assembler messages:
{standard input}:37643: Error: can't resolve `.LVL231' {*UND* section} - `.Ltext0' {.text section}
{standard input}:37739: Error: can't resolve `.LVL231' {*UND* section} - `.Ltext0' {.text section}
{standard input}:37743: Error: can't resolve `.LVL231' {*UND* section} - `.Ltext0' {.text section}
{standard input}:37783: Error: can't resolve `.LVL231' {*UND* section} - `.Ltext0' {.text section}
make[4]: *** [drivers/staging/speakup/kobjects.o] Error 1
make[3]: *** [drivers/staging/speakup] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [drivers/staging] Error 2
make[2]: *** [drivers/net] Error 2
In file included from include/linux/genhd.h:65:0,
                 from include/linux/blkdev.h:9,
                 from include/scsi/scsi_cmnd.h:5,
                 from drivers/scsi/sym53c8xx_2/sym_glue.h:58,
                 from drivers/scsi/sym53c8xx_2/sym_hipd.c:44:
include/linux/fs.h:1550:49: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[4]: *** [drivers/scsi/sym53c8xx_2/sym_hipd.o] Error 1
make[3]: *** [drivers/scsi/sym53c8xx_2] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [drivers/scsi] Error 2
make[1]: *** [drivers] Error 2
make: *** [debian/stamp/build/kernel] Error 2

and finally

/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x2b377411eead]
/usr/lib/gcc/x86_64-linux-gnu/4.7/cc1[0x4d2f51]
======= Memory map: ========
00400000-00fd3000 r-xp 00000000 08:01 1312897                            /usr/lib/gcc/x86_64-linux-gnu/4.7/cc1
011d2000-011dd000 rw-p 00bd2000 08:01 1312897                            /usr/lib/gcc/x86_64-linux-gnu/4.7/cc1
011dd000-0133c000 rw-p 00000000 00:00 0
01df7000-020b7000 rw-p 00000000 00:00 0                                  [heap]
2b3772f20000-2b3772f40000 r-xp 00000000 08:01 11800025                   /lib/x86_64-linux-gnu/ld-2.13.so
2b3772f40000-2b3772f42000 rw-p 00000000 00:00 0
2b377313f000-2b3773140000 r--p 0001f000 08:01 11800025                   /lib/x86_64-linux-gnu/ld-2.13.so
2b3773140000-2b3773141000 rw-p 00020000 08:01 11800025                   /lib/x86_64-linux-gnu/ld-2.13.so
2b3773141000-2b3773142000 rw-p 00000000 00:00 0
2b3773148000-2b3773151000 r-xp 00000000 08:01 2233201                    /usr/lib/x86_64-linux-gnu/libfakeroot/libfakeroot-sysv.so
2b3773151000-2b3773351000 ---p 00009000 08:01 2233201                    /usr/lib/x86_64-linux-gnu/libfakeroot/libfakeroot-sysv.so
2b3773351000-2b3773352000 rw-p 00009000 08:01 2233201                    /usr/lib/x86_64-linux-gnu/libfakeroot/libfakeroot-sysv.so
2b377336b000-2b377336c000 rw-p 00000000 00:00 0
2b3773370000-2b3773384000 r-xp 00000000 08:01 1321607                    /usr/lib/x86_64-linux-gnu/libmpc.so.2.0.0
2b3773384000-2b3773584000 ---p 00014000 08:01 1321607                    /usr/lib/x86_64-linux-gnu/libmpc.so.2.0.0
2b3773584000-2b3773585000 rw-p 00014000 08:01 1321607                    /usr/lib/x86_64-linux-gnu/libmpc.so.2.0.0
2b3773588000-2b37735e3000 r-xp 00000000 08:01 1321605                    /usr/lib/x86_64-linux-gnu/libmpfr.so.4.1.0
2b37735e3000-2b37737e3000 ---p 0005b000 08:01 1321605                    /usr/lib/x86_64-linux-gnu/libmpfr.so.4.1.0
2b37737e3000-2b37737e5000 rw-p 0005b000 08:01 1321605                    /usr/lib/x86_64-linux-gnu/libmpfr.so.4.1.0
2b37737e8000-2b377384f000 r-xp 00000000 08:01 1321603                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.0.5
2b377384f000-2b3773a4f000 ---p 00067000 08:01 1321603                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.0.5
2b3773a4f000-2b3773a57000 rw-p 00067000 08:01 1321603                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.0.5
2b3773a57000-2b3773a58000 rw-p 00000000 00:00 0
2b3773a58000-2b3773a5a000 r-xp 00000000 08:01 11800014                   /lib/x86_64-linux-gnu/libdl-2.13.so
2b3773a5a000-2b3773c5a000 ---p 00002000 08:01 11800014                   /lib/x86_64-linux-gnu/libdl-2.13.so
2b3773c5a000-2b3773c5b000 r--p 00002000 08:01 11800014                   /lib/x86_64-linux-gnu/libdl-2.13.so
2b3773c5b000-2b3773c5c000 rw-p 00003000 08:01 11800014                   /lib/x86_64-linux-gnu/libdl-2.13.so
2b3773c60000-2b3773c76000 r-xp 00000000 08:01 11796562                   /lib/x86_64-linux-gnu/libz.so.1.2.7
2b3773c76000-2b3773e75000 ---p 00016000 08:01 11796562                   /lib/x86_64-linux-gnu/libz.so.1.2.7
2b3773e75000-2b3773e76000 r--p 00015000 08:01 11796562                   /lib/x86_64-linux-gnu/libz.so.1.2.7
2b3773e76000-2b3773e77000 rw-p 00016000 08:01 11796562                   /lib/x86_64-linux-gnu/libz.so.1.2.7
2b3773e78000-2b3773ef9000 r-xp 00000000 08:01 11800003                   /lib/x86_64-linux-gnu/libm-2.13.so
2b3773ef9000-2b37740f8000 ---p 00081000 08:01 11800003                   /lib/x86_64-linux-gnu/libm-2.13.so
2b37740f8000-2b37740f9000 r--p 00080000 08:01 11800003                   /lib/x86_64-linux-gnu/libm-2.13.so
2b37740f9000-2b37740fa000 rw-p 00081000 08:01 11800003                   /lib/x86_64-linux-gnu/libm-2.13.so
2b37740fa000-2b37740fb000 rw-p 00000000 00:00 0
2b3774100000-2b3774280000 r-xp 00000000 08:01 11796538                   /lib/x86_64-linux-gnu/libc-2.13.so
2b3774280000-2b3774480000 ---p 00180000 08:01 11796538                   /lib/x86_64-linux-gnu/libc-2.13.so
2b3774480000-2b3774484000 r--p 00180000 08:01 11796538                   /lib/x86_64-linux-gnu/libc-2.13.so
2b3774484000-2b3774485000 rw-p 00184000 08:01 11796538                   /lib/x86_64-linux-gnu/libc-2.13.so
2b3774485000-2b377448d000 rw-p 00000000 00:00 0
2b3774490000-2b3774681000 r--p 00000000 08:01 1310939                    /usr/lib/locale/locale-archive
2b3774681000-2b377469e000 rw-p 00000000 00:00 0
2b37746a3000-2b37750d8000 rw-p 00000000 00:00 0
2b3775119000-2b3776444000 rw-p 00000000 00:00 0
2b3776460000-2b3776475000 r-xp 00000000 08:01 11796737                   /lib/x86_64-linux-gnu/libgcc_s.so.1
2b3776475000-2b3776675000 ---p 00015000 08:01 11796737                   /lib/x86_64-linux-gnu/libgcc_s.so.1
2b3776675000-2b3776676000 rw-p 00015000 08:01 11796737                   /lib/x86_64-linux-gnu/libgcc_s.so.1
2b3778000000-2b3778021000 rw-p 00000000 00:00 0
2b3778021000-2b377c000000 ---p 00000000 00:00 0
7ffffd08f000-7ffffd0b2000 rw-p 00000000 00:00 0                          [stack]
7ffffd200000-7ffffd201000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

---

-O1; kernel 3.7.2; AMD FX 8150

fs/ecryptfs/keystore.c:1168:28: warning: 'payload_len' may be used uninitialized in this function [-Wmaybe-uninitialized]
sound/pci/hda/patch_cmedia.c:777:1: internal compiler error: in size_of_die, at dwarf2out.c:7795
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[4]: *** [sound/pci/hda/patch_cmedia.o] Error 1
make[3]: *** [sound/pci/hda] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [sound/pci] Error 2
make[1]: *** [sound] Error 2
make[1]: *** Waiting for unfinished jobs....
In file included from drivers/mtd/chips/cfi_cmdset_0001.c:35:0:
drivers/mtd/chips/cfi_cmdset_0001.c: In function 'cfi_intelext_write_words':
include/linux/mtd/map.h:331:11: warning: 'r.x[0]' may be used uninitialized in this function [-Wmaybe-uninitialized]
drivers/media/usb/gspca/topro.c:4968:1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[5]: *** [drivers/media/usb/gspca/topro.o] Error 1
make[4]: *** [drivers/media/usb/gspca] Error 2
make[3]: *** [drivers/media/usb] Error 2
make[3]: *** Waiting for unfinished jobs....
drivers/net/ethernet/amd/nmclan_cs.c: In function 'nmclan_config':
drivers/net/ethernet/amd/nmclan_cs.c:625:3: warning: 'pcmcia_request_exclusive_irq' is deprecated (declared at include/pcmcia/ds.h:201) [-Wdeprecated-declarations]
make[2]: *** [drivers/media] Error 2
make[2]: *** Waiting for unfinished jobs....
drivers/net/ethernet/intel/ixgb/ixgb_ee.c:604:1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[6]: *** [drivers/net/ethernet/intel/ixgb/ixgb_ee.o] Error 1
make[6]: *** Waiting for unfinished jobs....
make[5]: *** [drivers/net/ethernet/intel/ixgb] Error 2
make[4]: *** [drivers/net/ethernet/intel] Error 2
make[3]: *** [drivers/net/ethernet] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [drivers/net] Error 2
make[1]: *** [drivers] Error 2
make: *** [debian/stamp/build/kernel] Error 2

-O0, kernel 3.7.2, AMD FX 8150

exec make kpkg_version=12.036+nmu3 -f /usr/share/kernel-package/ruleset/minimal.mk debian INITRD=YES 
exec debian/rules  INITRD=YES  kernel_image kernel_headers 
Setup is 16988 bytes (padded to 17408 bytes).
System is 2884 kB
CRC f8376713
drivers/isdn/hardware/eicon/divasi.c:577:1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[5]: *** [drivers/isdn/hardware/eicon/divasi.o] Error 1
make[4]: *** [drivers/isdn/hardware/eicon] Error 2
make[4]: *** Waiting for unfinished jobs....
make[3]: *** [drivers/isdn/hardware] Error 2
make[3]: *** Waiting for unfinished jobs....
make[2]: *** [drivers/isdn] Error 2
make[2]: *** Waiting for unfinished jobs....
fs/reiserfs/bitmap.c: In function 'scan_bitmap_block.constprop.9':
fs/reiserfs/bitmap.c:236:9: warning: 'next' may be used uninitialized in this function [-Wmaybe-uninitialized]
make[1]: *** [drivers] Error 2
make[1]: *** Waiting for unfinished jobs....
make: *** [debian/stamp/build/kernel] Error 2

---

Error messages when compiling kernel 3.7 on AMD FX 8150 with -O2 or -O1

exec make kpkg_version=12.036+nmu3 -f /usr/share/kernel-package/ruleset/minimal.mk debian INITRD=YES 
exec debian/rules  INITRD=YES  kernel_image kernel_headers 
net/xfrm/xfrm_sysctl.c:75:1: internal compiler error: in pop_scope, at c-decl.c:1244
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[3]: *** [net/xfrm/xfrm_sysctl.o] Error 1
make[2]: *** [net/xfrm] Error 2
make[1]: *** [net] Error 2
make: *** [debian/stamp/build/kernel] Error 2

---

Kernel 3.7 with -O0 on AMD FX 8150

exec make kpkg_version=12.036+nmu3 -f /usr/share/kernel-package/ruleset/minimal.mk debian INITRD=YES 
exec debian/rules  INITRD=YES  kernel_image kernel_headers 
In file included from include/acpi/platform/aclinux.h:59:0,
                 from include/acpi/platform/acenv.h:141,
                 from include/acpi/acpi.h:56,
                 from drivers/acpi/acpica/exmisc.c:45:
include/linux/sched.h:1352:9: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
The bug is not reproducible, so it is likely a hardware or OS problem.
make[4]: *** [drivers/acpi/acpica/exmisc.o] Error 1
make[3]: *** [drivers/acpi/acpica] Error 2
make[2]: *** [drivers/acpi] Error 2
make[1]: *** [drivers] Error 2
make: *** [debian/stamp/build/kernel] Error 2

318. qmol 0.3.2: A molecular weight calculator for Linux

Over a year ago I complained about the lack of a decent molecular weight calculator in linux in general, and in Debian Testing in particular.  I eventually managed to hack together a molecular weight calculator in Python as part of an isotopic pattern calculator in Python.

However, interpreted languages like python tend to be a bit slower than compiled languages (generally not critical for a molecular weight calculator, but could be for an isotopic pattern calculator), and, perhaps more importantly, my scripts don't feature a GUI.

I vaguely remember trying to compile Kmol by Tomislav Gountchev over a year ago, and as far as I can recall it wasn't working out since it depended on packages (kde-3) that were too old.

But things are changing.

Thomas Mitterfellner has revived Kmol in the form of qmol. Since it's very recent (version 0.1 was created in November 2012, and we're now at version 0.3.2) it's not found in the Debian repos, and maybe won't for some time given that Debian Testing/Wheezy is frozen.

There are, however, a pre-built .deb file for debian squeeze/stable (and Suse, Ubuntu, Fedora etc.) -- so if you're on stable you do not have to compile. Instead go here: http://download.opensuse.org/repositories/home:/lineinthesand/

qmol is a fairly complete solution, and importantly is highly configurable while at the same time being straightforward to use. In particular I like the ability to define your own chemical groups AND the ability to run it from the command line. It's basically what I've been waiting for with the exception of the lack of an isotopic pattern calculator -- but that may come by version 1.0.

Also, the documentation -- or qmol handbook -- is quite extensive and is available under help.

---

Enough talking -- time for compiling.

sudo apt-get install bzip2 build-essential cmake libqtcore4 libqtgui4 qt4-qmake libqt4-dev
mkdir ~/tmp 
cd ~/tmp
wget http://downloads.sourceforge.net/project/qmol/qmol-0.3.2/qmol-0.3.2.tar.bz2
tar xvf qmol-0.3.2.tar.bz2
mkdir buildqmol
cd buildqmol/
cmake ../qmol-0.3.2
make
sudo make install

Don't worry if you get

-- Looking for Q_WS_X11
-- Looking for Q_WS_X11 - found
-- Looking for Q_WS_WIN
-- Looking for Q_WS_WIN - not found.
-- Looking for Q_WS_QWS
-- Looking for Q_WS_QWS - not found.
-- Looking for Q_WS_MAC
-- Looking for Q_WS_MAC - not found.

during the cmake stage.

---

Usage: Either run qmol from the command line:

qmol 'N(CH3)4'

N(CH3)4 = C4H12N: 74.146 g/mol
C 64.80
H 16.31
N 18.89

The output format can be configured when qmol is in gui mode (Edit/Options).

or launch it by typing

qmol

A virgin window

Previous formulae aggregate at the bottom of the  window

It doesn't handle 0, but then neither does my calculator (yet)

It's very easy to define your own group -- but  only the first letter can be  upper case

Options menu -- you can format the command line output here

It works!

There are only two small things to watch out for: the inability to handle 0 (but you get an error message -- my calculator just give you an erroneous result which is arguably worse...) and the requirement that only the first letter in an abbreviation can be upper case (for reasons of ambiguity -- c.f. e.g. CHO vs C, H, O)

317. Compiling samba 4.0.0 on Debian Testing/Wheezy

Not much to say about this one -- samba/SMB is only one among several linux solutions for sharing files, and probably the easiest one for sharing directories with windows computers. Samba has also been around for about as long as linux, so it has a long and interesting history.

New major versions are always exciting though. Information about Samba 4 can be found here: http://wiki.samba.org/index.php/Samba4

I won't cover configuration here -- partly because I don't know much about it -- so see the link above. To get started you might want to look at this post though: http://verahill.blogspot.com.au/2012/12/300-briefly-sharing-folder-using-samba.html

I also wrote this a long time ago, but haven't checked it for accuracy:
http://verahill.blogspot.com.au/2012/03/building-and-installing-samba-from.html

These instructions will build Samba 4.0.0 with LDAP and Cluster support.
Compilation:

sudo apt-get install build-essential libacl1-dev libattr1-dev libaio-dev libpam0g-dev kfreebsd-headers-9.0-2 libtalloc-dev python-talloc-dev libtevent-dev valgrind libsasl2-dev python2.7-dev libgnutls-dev xsltproc libctdb-dev libldap2-dev libcups2-dev python-ldb-dev ldb-tools libldb-dev checkinstall
mkdir ~/tmp 
cd ~/tmp
wget http://www.samba.org/samba/ftp/stable/samba-4.0.0.tar.gz
tar xvf samba-4.0.0.tar.gz
cd samba-4.0.0/
./configure --prefix=$HOME/.smb4
time make
sudo checkinstall
dpkg -i samba_4.0.0-1_amd64.deb

And you're done. 

Takes about 14 minutes to compile. When you run checkinstall and you're asked

Some of the files created by the installation are inside the home directory: /home

You probably don't want them to be included in the package.
Do you want me to list them?  [n]: n
Should I exclude them from the package? (Saying yes is a good idea)  [n]: n

Answer no, you DO want to include them. You're getting this warning because we set a --prefix and the files aren't going to the / structure like normal programs.

The samba daemon, smbd, is in ~/.smb4/sbin/ and the configuration files are in /etc/samba/.

---

Here's the structure of ~/.smb4:

~/.smb4/
|-- bin
|   |-- cifsdd
|   |-- dbwrap_tool
|   |-- eventlogadm
|   |-- gentest
|   |-- ldbadd
|   |-- ldbdel
|   |-- ldbedit
|   |-- ldbmodify
|   |-- ldbrename
|   |-- ldbsearch
|   |-- locktest
|   |-- masktest
|   |-- ndrdump
|   |-- net
|   |-- nmblookup
|   |-- nmblookup4
|   |-- ntlm_auth
|   |-- oLschema2ldif
|   |-- pdbedit
|   |-- pidl
|   |-- profiles
|   |-- regdiff
|   |-- regpatch
|   |-- regshell
|   |-- regtree
|   |-- rpcclient
|   |-- samba-tool
|   |-- sharesec
|   |-- smbcacls
|   |-- smbclient
|   |-- smbclient4
|   |-- smbcontrol
|   |-- smbcquotas
|   |-- smbget
|   |-- smbpasswd
|   |-- smbspool
|   |-- smbstatus
|   |-- smbta-util
|   |-- smbtorture
|   |-- smbtree
|   |-- testparm
|   `-- wbinfo
|-- etc
|-- include
|   |-- charset.h
|   |-- core
|   |-- credentials.h
|   |-- dcerpc.h
|   |-- dcerpc_server.h
|   |-- dlinklist.h
|   |-- domain_credentials.h
|   |-- gen_ndr
|   |-- gensec.h
|   |-- ldap-util.h
|   |-- ldap_errors.h
|   |-- ldap_message.h
|   |-- ldap_ndr.h
|   |-- ldb_wrap.h
|   |-- libsmbclient.h
|   |-- lookup_sid.h
|   |-- machine_sid.h
|   |-- ndr
|   |-- ndr.h
|   |-- netapi.h
|   |-- param.h
|   |-- passdb.h
|   |-- policy.h
|   |-- read_smb.h
|   |-- registry.h
|   |-- roles.h
|   |-- rpc_common.h
|   |-- samba
|   |-- samba_util.h
|   |-- share.h
|   |-- smb2.h
|   |-- smb2_constants.h
|   |-- smb2_create_blob.h
|   |-- smb2_signing.h
|   |-- smb_cli.h
|   |-- smb_cliraw.h
|   |-- smb_common.h
|   |-- smb_composite.h
|   |-- smb_constants.h
|   |-- smb_ldap.h
|   |-- smb_raw.h
|   |-- smb_raw_interfaces.h
|   |-- smb_raw_signing.h
|   |-- smb_raw_trans2.h
|   |-- smb_request.h
|   |-- smb_seal.h
|   |-- smb_share_modes.h
|   |-- smb_signing.h
|   |-- smb_unix_ext.h
|   |-- smb_util.h
|   |-- smbconf.h
|   |-- smbldap.h
|   |-- tdr.h
|   |-- torture.h
|   |-- tsocket.h
|   |-- tsocket_internal.h
|   |-- util
|   |-- util_ldb.h
|   `-- wbclient.h
|-- lib
|   |-- auth
|   |-- bind9
|   |-- gensec
|   |-- idmap
|   |-- ldb
|   |-- libdcerpc-atsvc.so -> libdcerpc-atsvc.so.0.0.1
|   |-- libdcerpc-atsvc.so.0 -> libdcerpc-atsvc.so.0.0.1
|   |-- libdcerpc-atsvc.so.0.0.1
|   |-- libdcerpc-binding.so -> libdcerpc-binding.so.0.0.1
|   |-- libdcerpc-binding.so.0 -> libdcerpc-binding.so.0.0.1
|   |-- libdcerpc-binding.so.0.0.1
|   |-- libdcerpc-samr.so -> libdcerpc-samr.so.0.0.1
|   |-- libdcerpc-samr.so.0 -> libdcerpc-samr.so.0.0.1
|   |-- libdcerpc-samr.so.0.0.1
|   |-- libdcerpc-server.so -> libdcerpc-server.so.0.0.1
|   |-- libdcerpc-server.so.0 -> libdcerpc-server.so.0.0.1
|   |-- libdcerpc-server.so.0.0.1
|   |-- libdcerpc.so -> libdcerpc.so.0.0.1
|   |-- libdcerpc.so.0 -> libdcerpc.so.0.0.1
|   |-- libdcerpc.so.0.0.1
|   |-- libgensec.so -> libgensec.so.0.0.1
|   |-- libgensec.so.0 -> libgensec.so.0.0.1
|   |-- libgensec.so.0.0.1
|   |-- libndr-krb5pac.so -> libndr-krb5pac.so.0.0.1
|   |-- libndr-krb5pac.so.0 -> libndr-krb5pac.so.0.0.1
|   |-- libndr-krb5pac.so.0.0.1
|   |-- libndr-nbt.so -> libndr-nbt.so.0.0.1
|   |-- libndr-nbt.so.0 -> libndr-nbt.so.0.0.1
|   |-- libndr-nbt.so.0.0.1
|   |-- libndr-standard.so -> libndr-standard.so.0.0.1
|   |-- libndr-standard.so.0 -> libndr-standard.so.0.0.1
|   |-- libndr-standard.so.0.0.1
|   |-- libndr.so -> libndr.so.0.0.1
|   |-- libndr.so.0 -> libndr.so.0.0.1
|   |-- libndr.so.0.0.1
|   |-- libnetapi.so -> libnetapi.so.0
|   |-- libnetapi.so.0
|   |-- libnss_winbind.so -> libnss_winbind.so.2
|   |-- libnss_winbind.so.2
|   |-- libnss_wins.so -> libnss_wins.so.2
|   |-- libnss_wins.so.2
|   |-- libpdb.so -> libpdb.so.0
|   |-- libpdb.so.0
|   |-- libregistry.so -> libregistry.so.0.0.1
|   |-- libregistry.so.0 -> libregistry.so.0.0.1
|   |-- libregistry.so.0.0.1
|   |-- libsamba-credentials.so -> libsamba-credentials.so.0.0.1
|   |-- libsamba-credentials.so.0 -> libsamba-credentials.so.0.0.1
|   |-- libsamba-credentials.so.0.0.1
|   |-- libsamba-hostconfig.so -> libsamba-hostconfig.so.0.0.1
|   |-- libsamba-hostconfig.so.0 -> libsamba-hostconfig.so.0.0.1
|   |-- libsamba-hostconfig.so.0.0.1
|   |-- libsamba-policy.so -> libsamba-policy.so.0.0.1
|   |-- libsamba-policy.so.0 -> libsamba-policy.so.0.0.1
|   |-- libsamba-policy.so.0.0.1
|   |-- libsamba-util.so -> libsamba-util.so.0.0.1
|   |-- libsamba-util.so.0 -> libsamba-util.so.0.0.1
|   |-- libsamba-util.so.0.0.1
|   |-- libsamdb.so -> libsamdb.so.0.0.1
|   |-- libsamdb.so.0 -> libsamdb.so.0.0.1
|   |-- libsamdb.so.0.0.1
|   |-- libsmbclient-raw.so -> libsmbclient-raw.so.0.0.1
|   |-- libsmbclient-raw.so.0 -> libsmbclient-raw.so.0.0.1
|   |-- libsmbclient-raw.so.0.0.1
|   |-- libsmbclient.so -> libsmbclient.so.0.2.0
|   |-- libsmbclient.so.0 -> libsmbclient.so.0.2.0
|   |-- libsmbclient.so.0.2.0
|   |-- libsmbconf.so -> libsmbconf.so.0
|   |-- libsmbconf.so.0
|   |-- libsmbldap.so -> libsmbldap.so.0
|   |-- libsmbldap.so.0
|   |-- libsmbsharemodes.so -> libsmbsharemodes.so.0
|   |-- libsmbsharemodes.so.0
|   |-- libtevent-util.so -> libtevent-util.so.0.0.1
|   |-- libtevent-util.so.0 -> libtevent-util.so.0.0.1
|   |-- libtevent-util.so.0.0.1
|   |-- libtorture.so -> libtorture.so.0.0.1
|   |-- libtorture.so.0 -> libtorture.so.0.0.1
|   |-- libtorture.so.0.0.1
|   |-- libwbclient.so -> libwbclient.so.0.11
|   |-- libwbclient.so.0 -> libwbclient.so.0.11
|   |-- libwbclient.so.0.11
|   |-- mit_samba.so
|   |-- nss_info
|   |-- pkgconfig
|   |-- private
|   |-- process_model
|   |-- python2.7
|   |-- security
|   |-- service
|   |-- vfs
|   `-- winbind_krb5_locator.so
|-- private
|-- sbin
|   |-- nmbd
|   |-- samba
|   |-- samba_dnsupdate
|   |-- samba_kcc
|   |-- samba_spnupdate
|   |-- samba_upgradedns
|   |-- samba_upgradeprovision
|   |-- smbd
|   |-- swat
|   `-- winbindd
|-- share
|   |-- codepages
|   |-- man
|   |-- perl5
|   |-- setup
|   `-- swat
`-- var
    |-- cache
    |-- lib
    |-- lock
    |-- locks
    `-- run

316. Briefly: Automated chroot/sandbox creation

EDIT:
There are plenty of reasons to use chroots, but security is not one of them.

For a practical how-to see e.g
http://pen-testing.sans.org/blog/2012/06/06/escaping-restricted-linux-shells

For a bit of yelling, see
http://yarchive.net/comp/linux/chroot.html

chroot will improve your security by creating an obstacle which may filter out some would-be crackers, but it will not make it secure by any standard. (in spite of what I may have written elsewhere on this blog).

Original post:
I've been using chroot to compile and test stuff so much lately that I figure it was time to automate the process.

Before creating your chroot you'll need a few packages:

sudo apt-get install debootstrap coreutils x11-xserver-utils

The scripts

makechroot.sh

mkdir $HOME/tmp/jail/$1 -p
sudo debootstrap --arch amd64 testing $HOME/tmp/jail/$1 http://ftp.au.debian.org/debian/
sudo cp setupchroot.sh $HOME/tmp/jail/$1/

setupchroot.sh

rm /etc/apt/sources.list
echo 'deb  http://ftp.au.debian.org/debian/ wheezy main contrib non-free' >> /etc/apt/sources.list
apt-get update
apt-get install locales sudo vim
echo 'export LC_ALL="C"'>>/etc/bash.bashrc
echo 'export LANG="C"'>>/etc/bash.bashrc
echo 'export DISPLAY=:0.0' >> /etc/bash.bashrc
echo '127.0.0.1 beryllium >> /etc/hosts'
source /etc/bash.bashrc
adduser sandbox
usermod -g sudo sandbox
echo 'Defaults !tty_tickets' >> /etc/sudoers

launchchroot.sh

xhost +
sudo mount -o bind /proc $1/proc
sudo cp /etc/resolv.conf $1/etc/resolv.conf
sudo chroot $HOME/tmp/jail/$1

How to use
To set up the chroot:

sh makechroot.sh mynewchroot
sudo chroot mynewchroot
root@beryllium:/# sh setupchroot.sh

To use the chroot:

sh launchchroot.sh mynewchroot

Once you're done with the chroot and logged out, do

sudo umount $HOME/tmp/jail/mynewchroot/proc

to unmount the /proc -- you can now delete, copy etc. the directory structure of you chroot.

315. Briefly: Compile firefox 18 on debian testing/wheezy

For no better reason than pure curiosity, here's how to build firefox 18. My regular build environment has so many packages installed these days that it's useless for working out dependencies for other people, so I tried it in a fresh chroot to hammer out the current dependencies beyond a doubt. In other words: it will work for you.

In retrospect I'm not sure if you really need the dev package for libgtk2.0.

sudo apt-get install bzip2 build-essential python zip unzip pkg-config libgtk2.0-dev gconf2 libdbus-1-dev libdbus-glib-1-dev yasm libasound2-dev libcurl4-gnutls-dev libxt-dev mesa-common-dev
mkdir ~/tmp/firefox -p
cd ~/tmp/firefox
wget ftp://ftp.mozilla.org/pub/firefox/releases/18.0/source/firefox-18.0.source.tar.bz2
tar xvf firefox-18.0.source.tar.bz2
mkdir build/
cd build/
../mozilla-release/./configure --prefix=$HOME/.firefox18
make
make install
echo "alias firefox18='$HOME/.firefox18/bin/firefox'" >> $HOME/.bashrc
source $HOME/.bashrc

Launch by typing

firefox18

314. Briefly: Installing talkd on ROCKS 5.4.3

I was asked to set up talkd on our Rocks 5.4.3 cluster (Centos 5.4) . There's no talkd or talk-server packages in the repos on that server.

Note: The general consensus seems to be that talk is
1. insecure and
2. outdated.

To install:

wget http://mirror.centos.org/centos/5/os/x86_64/CentOS/talk-server-0.17-31.el5.x86_64.rpm
yum localinstall talk-server-0.17-31.el5.x86_64.rpm
sudo iptables -A INPUT -p udp --dport 517 -s 127.0.0.1 -j ACCEPT
sudo iptables -A INPUT -p udp --dport 518 -s 127.0.0.1 -j ACCEPT

Above I've added, but haven't yet tried, -s 127.0.0.1 to limit connections from the local computer (localhost). If it doesn't work with -s 127.0.0.1, then try without -- but then be aware of the security implications. These rules also aren't permanent and will be lost on reboot. To make them permanent, edit /etc/sysconfig/iptables.

I couldn't get talk to work before opening the ports and would get

Error on read from talk daemon : Connection refused

Note that talkd uses Xinet and not init -- it will not run as  memory resident daemon, but instead be launched by xinet only when talkd is needed (traffic is detected to the ports associated with talkd). Xinet.d is a bit like a concierge, waking up whomever seems to be the adressee.

Edit both /etc/xinetd.d/talk and /etc/xinetd.d/ntalk. Change to:

# default: off
# description: The talk server accepts talk requests for chatting with users \
#       on other systems.
service talk
{
        flags                   = IPv4
        disable                 = no
        socket_type             = dgram
        wait                    = yes
        user                    = nobody
        group                   = tty
        server                  = /usr/sbin/in.talkd
}

Finally, restart xinet.d (doing chkconfig talk on wasn't enough):

sudo /etc/init.d/xinet.d restart

---

Note: If you or the other user have several terminals open you should figure out which terminal to use. If you're user2, do

ps

PID TTY          TIME CMD
 5455 pts/23   00:00:00 bash
 9321 pts/23   00:00:00 ps

user1 can then do

[user1@host ~]$ talk user2@localhost pts/23

and user2 will see the following in that terminal:

Message from Talk_Daemon@host at 14:49 ...   
talk: connection requested by user1@localhost.localdomain.  
talk: respond with:  talk user1@localhost.localdomain 

If you are user1 and have no idea on what terminal user2 is logged on, you can try

w|grep user2

user2     pts/8    remote:S.0    04Nov12 59:00   0.04s  0.04s /bin/bash
user2     pts/9    remote:S.1    04Nov12 17:28   0.03s  0.03s /bin/bash
user2     pts/10   remote Mon12  2days  0.03s  0.00s ssh -X -v volde
user2     pts/11   local  Mon12  2days  0.29s  0.26s perl eccejobmon
user2     pts/23   remote 14:30  7.00s  0.00s  0.00s -bash

313. Which Office for linux users? Libreoffice, Openoffice, Lotus Symphony, Google Docs, Microsoft Live/Skydrive, Wine/Office2003 and Office2003 in Virtualbox, WPS Office

UPDATE 6 Apr 2013: I've added WPS Office

UPDATE 10 Feb 2013: I've tried Libreoffice 4.0 as well -- no improvement in terms of rendering vs Libreoffice 3.5.4 (below). The start-up times are greatly improved though.

UPDATE 4 Feb. 2013: Equation Editor in Office 2003 on Wine used to be supported -- however, while fixing the editor in office 2010 support for older office versions broke. This will (hopefully) be fixed in future versions and the codeweaver and wine devs are aware of the issue. Bug reports to that effect:
http://bugs.winehq.org/show_bug.cgi?id=32697 and http://www.codeweavers.com/support/tickets/browse/?ticket_id=931759

Most of my publications are via collaborations, and I'm more or less the only one who uses Linux. When I'm in charge, I use latex -- but you can be a zealot, or you can be a pragmatist. Life tends to be easier for the pragmatic. I draw the line at actually using a native installation of Windows though.

A few things are important: figures should render ok, endnote should render ok, I need to use track changes and I should be able to insert equations. Oh, and the templates from the major publishers (RSC, Wiley, ACS) should work.

So, once and for all (well, I should revise this for each major version I suppose) here's an evaluation of  where each solution on Linux falls short and where it does well.

I mocked up a simple article style doc file using Word 2003, Endnote X and Microsoft Equation Editor 3.0 and the Angewandte Chemie template, and then opened it in various alternative solutions.

I tried:

• LibreOffice
• OpenOffice
• Lotus Symphony
• Abiword
• Google Docs/Drive
• Microsoft Live/Skydrive

Here's what I found:

NOTE: when I say 'good' or 'bad' I mean how well they reproduce the look which is found in Word 2003. It's not a judgement on the quality of the word processor as such. I also recognise that for a developer it isnt much fun spending time working on reproducing someone else's crappy and poorly documented software, when you could be implementing new and wonderful things instead.

Outcome: all non-Word 2003 solutions at a minimum had the figure move slightly relative to the text. All solutions did the Endnote references ok, if not the location of them. Some solutions had issues with the greek letters in the equation. All non-MS solutions rendered the equation as an image which couldn't be edited.

See for yourself...


Office 2003/Virtualbox
This is the canonical version -- this is how it was created and how it should look.

Word 2003 in Virtualbox

Libreoffice 1:3.5.4+dfsg-4 
Looks fine, but
* The numbering for the equation (put it there using a tab stop) is too far left
* part of a paragraph now separates the figure from its caption -- the picture hasn't moved, but the first column on the first page seems to be one line shorter, which pushes the text of the previous paragraph past the figure.
* The last paragraph doesn't end on the first page, but in the first column of the second page..
* The second page also looks different, with the endnote bit moved to the second column.

LibreOffice

Openoffice 3.4.1-1
I grabbed the latest debs from http://www.openoffice.org/download/other.html#tested-full
On the second page the Endnote references are in the right place (c.f. Libreoffice).

* Greek letters in Equation replaced by squares (missing font?). (greek letters in text is ok)
* The numbering for the equation (put it there using a tab stop) is too far left
* part of a paragraph now separates the figure from its caption -- the picture hasn't moved, but the first column on the first page seems to be one line shorter, which pushes the text of the previous paragraph past the figure.
* The last paragraph doesn't end on the first page, but in the first column of the second page..

OpenOffice

Lotus Symphony 3.0.1
I downloaded the Lucid i386 deb and installed it in an Ubuntu Lucid virtual machine. Symphony is based on OpenOffice.

* Greek letters in Equation disappeared completely. (greek letters in text is ok)
* The numbering for the equation (put it there using a tab stop) is too far left
* part of a paragraph now separates the figure from its caption -- the picture hasn't moved, but the first column on the first page seems to be one line shorter, which pushes the text of the previous paragraph past the figure.
* The last paragraph doesn't end on the first page, but in the first column of the second page..
* The second page also looks different, with the endnote bit moved to the second column.

Lotus Symphony

Office 2003/Wine
See here for installation: http://verahill.blogspot.com.au/2013/01/307-installing-office-2003-in-wine-on.html
Everything looks fine with one pretty important exception:
* The equation is missing completely. Note that other people may have better luck installing Equation editor under wine -- I haven't managed to install it at all.

Word 2003 in Wine

Abiword
Plenty of issues:
* The author box has move to the top left corner.
* Can't find the table
* Greek letters missing in equation
* Greek letter in text missing
* Lots of stuff missing or misplaced (e.g. Keyword background picture)

Abiword

Google Docs
This was the big surprise -- I've always thought that google docs was essentially as 'good' as Microsoft Office. They have a financial incentive to be, after all. Turns out that the document was all but unrecognisable when opened in Google Docs. By far the worst solution. Note: the out-of-date warning is triggered by Chrome 18. Most similar to Abiword.

Google Docs

Microsoft Live 

Yes, that's right -- Microsoft Live works on Linux. This is the free version you get via Hotmail/Skydrive -- not their Office 365 or whatever they use now.

There are more surprises -- it doesn't render the document faithfully.

View:

* The numbering for the equation (put it there using a tab stop) is too far left. (like Libreoffice)
* part of a paragraph now separates the figure from its caption -- the picture hasn't moved, but the first column on the first page seems to be one line shorter, which pushes the text of the previous paragraph past the figure. (like Libreoffice)
* The last paragraph doesn't end on the first page, but in the first column of the second page. (like Libreoffice)
* The last paragraph doesn't end on the first page, but in the first column of the second page. (like Openoffice)
In other words -- Libreoffice/Openoffice do just as well as Microsoft Live

The interface for the View mode

full -- and no, you can't zoom in and out...

Editing:

In editing mode all it looks like a mess -- see figures.

In addition to the obvious formatting mess,

* greek letters in text show up as squares (problem with fonts on my computer or problems with live?)

* The equation cannot be edited.

However, once you go back to View mode the formatting is restored -- but it'd leave you flying blind while trying to make changes.

Editing mode

The full document -- yes, there's no Zoom...

WPS (Kingsoft)
I only discovered this by accident. Kingsoft (金山软件) is a legitimate Chinese company (Star Office was big in the 90s) and somehow there's a linux version (i386) of their Office suite: http://linux.wps.cn/. The bad news is that it's in Chinese, but that doesn't stop me from evaluating it -- it's such a close copy to Office 2007 that you can pretty much guess how to use it.

You can download it by clicking on this link: http://wdl.cache.ijinshan.com/wps/download/Linux/unstable/wps-office_8.1.0.3724~b1p2_i386.deb

It seems to be reproducing the file pretty well, with the exception of how it renders the equation and the missing stars in the author affiliation section. However, the location of the figure caption is correct, and the second page looks good.

And here's how it looks:

312. Tau + OpenMPI profiling on Debian Testing/Wheezy

Still searching for a way to easily look at the execution of parallel jobs I came across TAU: http://www.cs.uoregon.edu/Research/tau/home.php

You can download without registering, but please do register as the number of registered users tend to be important for funding and evaluation of software development in academia: http://www.cs.uoregon.edu/Research/tau/downloads.php

I'm not really sure about how to use PDT, and I've used Tau without it before without any problems.

The compilation order below is also important -- pdt won't build without libpdb.a which is generated by tau -- but you can't configure tau with -pdt if it doesn't exist.

---

Compiling

sudo mkdir /opt/tau
sudo chown $USER /opt/tau
cd /opt/tau

wget http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/software_werkzeuge_zur_unterstuetzung_von_programmierung_und_optimierung/otf/dateien/OTF-1.12.2salmon.tar.gz
tar xvf OTF-1.12.2salmon.tar.gz
cd /OTF-1.12.2salmon/
./configure --prefix=/opt/tau/OTF
make
make install
cd ../

wget http://tau.uoregon.edu/tau.tgz
tar xvf tau.tgz
cd tau-2.22-p1/
./configure -mpilib=/usr/lib/openmpi/lib -prefix=/opt/tau -openmp -TRACE -iowrapper -otf=/opt/tau/OTF -pthread
make install
cd ../

wget http://tau.uoregon.edu/pdt.tar.gz
tar xvf pdt.tar.gz
cd pdtoolkit-3.18.1/
./configure -prefix=/opt/tau/pdt
make
make install

cd ../tau-2.22-p1/
./configure -mpilib=/usr/lib/openmpi/lib -prefix=/opt/tau -openmp -TRACE -iowrapper -pthread -otf=/opt/tau/OTF -pdt=/opt/tau/pdt

make install

---

Testing
Time to try it out on something parallel.

First set the path

PATH=$PATH:/opt/tau/x86_64/bin

I used nwchem with this input file, co2.nw:

title "co nmr"
geometry
c   0   0   0
o   0   0   1.13
end
basis 
    * library "6-311+G*"
end
property
    shielding
end
dft
 direct
 grid fine
 mult 1
 xc HFexch 0.05 slater 0.95 becke88 nonlocal 0.72 vwn_5 1 perdew91 0.81
end

task dft property

and ran it using

mpirun -n 3 tau_exec nwchem co2.nw

which ends with

Total times  cpu:        4.8s     wall:        7.6s

It's obviously a bit too short, but will do for illustration purposes.

That generates a set of files, profile.*.0.0 -- one for each thread i.e. profile.1.0.0, profile.2.0.0 and profile.3.0.0 in this particular case. There are a lot of options for tracing, using hardware counters etc. -- see http://www.cs.uoregon.edu/Research/tau/docs/newguide/

pprof -s


Reading Profile files in profile.*

FUNCTION SUMMARY (total):
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0       15,813       25,931           3       14276    8643959 .TAU application
 18.8        4,870        4,870       10272           0        474 MPI_Barrier() 
 12.1        3,138        3,138           3           0    1046279 MPI_Init() 
  8.1        2,090        2,090         818           0       2556 MPI_Recv() 
  0.0            9            9           3           0       3173 MPI_Finalize() 
  0.0            3            3          24           0        128 MPI_Bcast() 
  0.0            2            2           6           0        463 MPI_Comm_dup() 
  0.0            1            1         790           0          2 MPI_Comm_size() 
  0.0        0.872        0.872         818           0          1 MPI_Send() 
  0.0        0.294        0.294         841           0          0 MPI_Comm_rank() 
  0.0         0.17         0.17         674           0          0 MPI_Get_count() 
  0.0        0.111        0.111           3           0         37 MPI_Comm_free() 
  0.0        0.026        0.026           3           0          9 MPI_Errhandler_set() 
  0.0        0.024        0.024           6           0          4 MPI_Group_rank() 
  0.0         0.02         0.02           6           0          3 MPI_Comm_compare() 
  0.0        0.015        0.015           4           0          4 MPI_Comm_group() 
  0.0        0.008        0.008           4           0          2 MPI_Group_size() 
  0.0        0.004        0.004           1           0          4 MPI_Group_translate_ranks() 

FUNCTION SUMMARY (mean):
---------------------------------------------------------------------------------------
%Time    Exclusive    Inclusive       #Call      #Subrs  Inclusive Name
              msec   total msec                          usec/call 
---------------------------------------------------------------------------------------
100.0        5,271        8,643           1     4758.67    8643959 .TAU application
 18.8        1,623        1,623        3424           0        474 MPI_Barrier() 
 12.1        1,046        1,046           1           0    1046279 MPI_Init() 
  8.1          696          696     272.667           0       2556 MPI_Recv() 
  0.0            3            3           1           0       3173 MPI_Finalize() 
  0.0            1            1           8           0        128 MPI_Bcast() 
  0.0        0.926        0.926           2           0        463 MPI_Comm_dup() 
  0.0        0.436        0.436     263.333           0          2 MPI_Comm_size() 
  0.0        0.291        0.291     272.667           0          1 MPI_Send() 
  0.0        0.098        0.098     280.333           0          0 MPI_Comm_rank() 
  0.0       0.0567       0.0567     224.667           0          0 MPI_Get_count() 
  0.0        0.037        0.037           1           0         37 MPI_Comm_free() 
  0.0      0.00867      0.00867           1           0          9 MPI_Errhandler_set() 
  0.0        0.008        0.008           2           0          4 MPI_Group_rank() 
  0.0      0.00667      0.00667           2           0          3 MPI_Comm_compare() 
  0.0        0.005        0.005     1.33333           0          4 MPI_Comm_group() 
  0.0      0.00267      0.00267     1.33333           0          2 MPI_Group_size() 
  0.0      0.00133      0.00133    0.333333           0          4 MPI_Group_translate_ranks() 

...which I can't pretend to understand. Reasonably, the first line would be the cpu time and the wall time (4.8 and 7.6 s vs 5,271 and 8,643 ms).

A visual representation can be had by launching paraprof:

paraprof

Now it's time to explore...

The one thing that doesn't seem to work is visualisation of the communication matrix...

---

---

Failed attempt to build with vampirtrace

sudo mkdir /opt/tau
sudo chown $USER /opt/tau
cd /opt/tau

wget http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/software_werkzeuge_zur_unterstuetzung_von_programmierung_und_optimierung/otf/dateien/OTF-1.12.2salmon.tar.gz
tar xvf OTF-1.12.2salmon.tar.gz
cd /OTF-1.12.2salmon/
./configure --prefix=/opt/tau/OTF
make
make install
cd ../

wget http://tu-dresden.de/die_tu_dresden/zentrale_einrichtungen/zih/forschung/software_werkzeuge_zur_unterstuetzung_von_programmierung_und_optimierung/vampirtrace/dateien/VampirTrace-5.14.1.tar.gz
tar xvf VampirTrace-5.14.1.tar.gz
cd VampirTrace-5.14.1/
./configure --prefix=/opt/tau/vampirtrace --with-mpi-dir=/usr/lib/openmpi/lib --with-extern-otf-dir=/opt/tau/OTF
make
make install

wget http://tau.uoregon.edu/tau.tgz
tar xvf tau.tgz
cd tau-2.22-p1/
./configure -mpilib=/usr/lib/openmpi/lib -prefix=/opt/tau -openmp -TRACE -iowrapper -otf=/opt/tau/OTF -vampirtrace=/opt/tau/vampirtrace
make install

It builds fine, but during execution of mpirun -n 2 tau_exec... I get

Error: No matching binding for 'mpi' in directory /opt/tau/x86_64/lib
Available bindings (/opt/tau/x86_64/lib):
Error: No matching binding for 'mpi' in directory /opt/tau/x86_64/lib
Available bindings (/opt/tau/x86_64/lib):
  /opt/tau/x86_64/lib/shared-disable
  /opt/tau/x86_64/lib/shared-disable

311. Compiling MPE for MPI profiling

I've been wanting to get an overview of how my parallel (nwchem, gamess, dalton, etc.) programs are running, and whether there are any obvious bottlenecks other than the network and slow harddrives that I can sort out.

The Australia high performance computer facility in Canberrra uses http://ipm-hpc.sourceforge.net/, but I'm not having much luck compiling it, and the lack of recent updates makes me somewhat less willing to invest too much effort into it.

So I stumbled across MPE instead: http://www.mcs.anl.gov/research/projects/perfvis/download/index.htm#MPE

The problem is that almost all of the links of that page are broken, including those pointing towards the documentation, so I don't actually know how to use it properly. The presence of mpecc in /opt/mpe/bin suggests that it's used as a stand-in for mpicc when compiling, which I'll test some day.


Installing/compiling

cd ~/tmp
wget ftp://ftp.mcs.anl.gov/pub/mpi/mpe/mpe2.tar.gz
tar xvf mpe2.tar.gz
cd mpe2-1.3.0/
./configure MPI_CC=mpicc MPI_F77=mpif77 --prefix=/opt/mpe
make
sudo make install

I found what I looked for in Tau instead: 
http://verahill.blogspot.com.au/2013/01/312-tau-mpi-profiling-on-debian.html

310. Remote mounting using sshfs

I've run out of USB ports on my work desktop, so I occasionally cheat and attach USB drives to one of my compute nodes and transfer the files across the network to my desktop. Since I've got a gigabit switch set up, the speeds are quite acceptable.

NFS isn't really a solution here. Instead, sshfs is the tool to use.

The local and remote computer will be referring to Desktop and Node, respectively. The specific example I'm using here is that of a USB drive manually mounted on the Node, which contains pictures that I want to transfer to my Desktop.

On the Node
The plugged in USB device is found as /dev/sdb, and holds only one partition, /dev/sdb1.

sudo mkdir /media/usbdrive
sudo mount /dev/sdb1 /media/usbdrive

On the Desktop

sudo apt-get install sshfs
sudo mkdir /media/remote
sudo sshfs $USER@Node:/media/usbdrive /media/remote -o allow_other

That's about it. To unmount do

sudo umount /media/remote

and

sudo umount /media/usbdrive

respectively.

309. Installing Office 2003 in Wine on Debian Testing

UPDATE 4 Feb 2013: Equation Editor under Office 2003 used to be support on Wine/Crossover. The devs are aware of it and looking into fixing the regression.


Bug reports to that effect:
http://bugs.winehq.org/show_bug.cgi?id=32697 and http://www.codeweavers.com/support/tickets/browse/?ticket_id=931759


UPDATE: I've since tried this using CrossOver 12 as well. The installation goes much more smoothly (the .exe file works fine), and you basically don't need to do anything except click 'Next' a couple of time. The Equation editor still doesn't work though. There's a free 15 day trial version of CrossOver available at http://www.codeweavers.com/products/. If running Office 2003 is your goal there seems to be little difference between CrossOver and Wine. However, I ended up buying a copy of crossover anyway since at this stage of my life and career it's not that much money and by supporting crossovre you indirectly support wine.

Since I would guess at the lack of Office for Linux as the single most problematic issue facing those wishing to switch from Windows or Mac to Linux, and since it would be something attempted very early on in someone's experience of Linux, it doesn't hurt to show the steps in detail -- especially since there are a few stumbling stones.

The linux community like to suggest new adopters to run Word under Wine, or better yet, to use libre/openoffice. It's an open secret that there are issues though. Most casual users won't notice most of the time,  but problems do exist.

In the particular case of Office 2003 under Wine, my main issue is that there's no way of installing Microsoft Equation Editor from what I can see, and it is a feature I do use when publishing together with other people (sweet, sweet TeX...). Another issue is that figures tend not to show up well or at all (complaints about memory -- most of our collaborative files are >5 Mb) when I use Word under Wine. These aren't issues when running in a virtual machine, which is the way I usually suffer through my bouts with Office.

An added head ache is this: http://wine.1045685.n5.nabble.com/Bug-26358-New-Office-2003-installer-crashes-immediately-td3414859.html and  http://bugs.winehq.org/attachment.cgi?id=35449. Using any of the versions of Wine above I just can't install using the setuppro.exe. This bug report ( http://bugs.winehq.org/show_bug.cgi?id=31942) showed me the simple solution though: using pro11.msi instead.

Anyway, just be aware of the limitations -- you can't install office, then do wine control, "add/remove programs", and add e.g. Microsoft Equation Editor. Likewise, if you try to be smart and do a custom install, and choose advanced customization, the installation will fail and complain about missing CAB files (in the case of equation editor: 01561403.cab). I've also tried installing MathType 4, which shows up under Objects, but is reported as missing when selected.

Otherwise this works fine.

---

Wine
To install the official wheezy version of wine:

sudp dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install wine wine-bin:i386 libwine-bin:i386 libwine:i386

You can also see the following posts on this blog for how to install/compile wine:

1.5.21 (using multiarch)

1.5.19 (using the dev.carbon-project.org rules)

1.5.5 (using the dev.carbon-project.org rules)

1.5.5 (downloading precompiled debs)

For this installation I've used Wine 1.5.19.

---

Installation
First set up cabextract and winetricks and install corefonts etc.

sudo apt-get install cabextract ttf-mscorefonts-installer
wget http://winetricks.org/winetricks
chmod +x winetricks
./winetricks corefonts fontfix

If you have Office 2003 as an OFFICE11.iso file (likely if you're at e.g. a university) then loop mount that file
In the directory where your iso file is:

sudo mkdir /media/OFFICE11
sudo mount -o loop OFFICE11 /media/OFFICE11

mount: warning: /media/OFFICE11/ seems to be mounted read-only.

wine msiexec /i /media/OFFICE11/pro11.msi

If you get any questions about Gecko or Mono, go ahead and let them install. Anyway, this launches the installation -- just do what you'd do under Windows, with the following caveats:

Important: You can't go back and install missing programs, so install all that you want from the beginning. Also, if you try to do advanced customisation and install Equation Editor (maybe other programs too) the installation will fail and complain about a missing CAB file. It hasn't always been so  apparently.

---

Running
Launch it as any other gnome or kde program, or from the terminal do

wine ~/.wine/drive_c/Program\ Files/Microsoft\ Office/OFFICE11/WINWORD.EXE

308. Compiling Wine 1.5.21 with multiarch on Debian Testing -- clean multiarch build

UPDATE 16 May 2013: See here for Wine 1.5.30: http://verahill.blogspot.com.au/2013/05/416-wine-1530-in-chroot.html

Another update (28/1/2013): It seems like your mileage in terms of how well Wine will work for you will vary using this method. It works fine for me and I suspect it's because I've built earlier Wine versions of my systems and have ia32-libs-dev installed. This package is no longer available in Testing. ia32-libs installs the libs you need, but does not provide symlinks between .so and .so.X files -- you can probably do that by hand if necessary though. It's not for beginners.

An additional thing to remember is that the 1.5 series of Wine is a development series -- 1.4 is the stable series which is meant for end users. Progress made in 1.5 will be included in 1.6.


Update2 (26 Jan 2013): there are a couple of issues, and they will depend on how your system is set up.
* libosmesa6-dev:i386 will require mesa-common-dev:i386 which will remove libosmesa6-dev:amd64 and more, which is bad.
* libglu1-mesa-dev:i386 depends on libgl1-mesa-dev:i386 which will remove libgl1-mesa-dev:amd64 and more, which is bad.
*  libgstreamer-plugins-base0.10-dev:i386 depends on libgstreamer0.10-dev:i386 which depends on gir1.2-gstreamer-0.10:i386 which depends on gir1.2-freedesktop:i386 which depends on gir1.2-glib-2.0:i386 which depends on libgirepository-1.0-1:i386 which will remove a whole lot of packages (132 on one of my systems, including gnome, gdm3 etc.)

I'm working on figuring out what's triggering this on some systems but not others. Be aware that that doesn't necessarily mean that I will be successful in doing so -- any help is appreciated.

In the mean time see http://verahill.blogspot.com.au/2013/01/308-compiling-wine-1521-on-debian.html to see how to build wine in a chroot, which is safe. You can then install that .deb package on your normal system and HOPEFULLY there won't be any broken dependencies. You won't need the -dev:i386 packages on the install target.

ldd `which wine` 

linux-gate.so.1 =>  (0x55573000)
 libwine.so.1 => /usr/local/lib/libwine.so.1 (0x55576000)
 libpthread.so.0 => /lib32/libpthread.so.0 (0x556ec000)
 libc.so.6 => /lib32/libc.so.6 (0x55705000)
 libdl.so.2 => /lib32/libdl.so.2 (0x55867000)

so you really don't seem to need much in the way of shared libs installed.

---

Old Update: This build is safe now, and there are no remaining errors (i.e. opencl has been taken care of) (see update2 above for more recent information)

---

I've been building wine (e.g. http://verahill.blogspot.com.au/2012/12/building-wine-1519-on-debian.html) on a system that was set up a few years ago.

Ergo, since ia32libs are on their way out and being replaced by multiarch libs it's time to have a look at what a build from scratch looks like, and what dependencies are needed.

This time we'll build wine the old-fashioned way using configure/make/make install. The biggest challenge was to overcome the annoying

configure: error: Cannot build a 32-bit program, you need to install 32-bit development libraries.

error. The solution to that was simply to install gcc-multilib. 

To work everything out from scratch I've set up a an amd64 chroot of a clean debian testing/wheezy. , It's increasingly appearing as if this is the easiest and safest thing to do.  Hopefully the release of Jessie will get the develop/improvement of multiarch back on the road.

To follow what's new in wine, look here: http://linux.softpedia.com/progChangelog/Wine-Changelog-148.html

...and the obligatory screenshot

---

The chroot environment
Basically as I've shown before:

sudo apt-get install debootstrap coreutils
mkdir -p $HOME/tmp/architectures/wine
cd $HOME/tmp/architectures
sudo debootstrap --arch amd64 testing $HOME/tmp/architectures/wine/ http://ftp.au.debian.org/debian/
sudo chroot wine/
apt-get install locales sudo vim
echo 'export LC_ALL="C"'>>/etc/bash.bashrc
echo 'export LANG="C"'>>/etc/bash.bashrc
source /etc/bash.bashrc
adduser winebuild

usermod -g sudo winebuild
echo 'Defaults !tty_tickets' >> /etc/sudoers

Edit /etc/hosts and add your hostname (here beryllium) to the end of the line e.g.

127.0.0.1 localhost beryllium

Exit, and the log in again

exit
sudo chroot wine

Then change user:

su winebuild
cd ~

Change at the very least the country code for the repo above (n red) so that you're using the closest/fastest one.

---

Compiling Wine
Set up multiarch (we're using amd64 but also want to support i386). I'm setting this up from scratch.

sudo dpkg --add-architecture i386
sudo apt-get update

Get the wine dependencies:

sudo apt-get install libx11-dev:i386 libfreetype6-dev:i386 libxcursor-dev:i386 libxi-dev:i386 libxxf86vm-dev:i386 libxrandr-dev:i386 libxinerama-dev:i386 libxcomposite-dev:i386 libglu-dev:i386 libosmesa-dev:i386 libglu-dev:i386 libosmesa-dev:i386 libdbus-1-dev:i386 libgnutls-dev:i386 libncurses-dev:i386 libsane-dev:i386 libv4l-dev:i386 libgphoto2-2-dev:i386 liblcms-dev:i386 libgstreamer-plugins-base0.10-dev:i386 libcapi20-dev:i386 libcups2-dev:i386 libfontconfig-dev:i386 libgsm1-dev:i386 libtiff-dev:i386 libpng-dev:i386 libjpeg-dev:i386 libmpg123-dev:i386 libopenal-dev:i386 libldap-dev:i386 libxrender-dev:i386 libxml2-dev:i386 libxslt-dev:i386 libhal-dev:i386 libcurl4-openssl-dev:i386

sudo apt-get install gettext prelink bzip2 bison flex oss4-dev checkinstall ocl-icd-libopencl1:i386 opencl-headers

sudo apt-get install gcc-multilib

mkdir ~/tmp/
wget http://mirrors.ibiblio.org/wine/source/1.5/wine-1.5.21.tar.bz2 -O wine-unstable_1.5.21.orig.tar.bz2
tar xvf wine-unstable_1.5.21.orig.tar.bz2
cd wine-1.5.21/
./configure

config.status: executing include/wine commands
config.status: executing Makefile commands
configure: Finished.  Do 'make' to compile Wine.

time make -j4

where -j4 is the number threads to launch, normally number of cores+1 (see here). Took around 15 minutes on a 3 core AMD.

Next

sudo checkinstall

to build a .deb package and install it (preferred solution) or

sudo make install

to just install.
Done!

---

Note that while this build works without issue, the transition to multiarch in debian is not complete and you will likely encounter this problem:

p11-kit: couldn't load module: /usr/lib/i386-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: /usr/lib/i386-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: cannot open shared object file: No such file or directory

If you try to fix it you get

sudo apt-get install gnome-keyring:i386 libcap-ng0:i386 libgck-1-0:i386 libgcr-3-1:i386 gcr:i386 libcap2-bin:i386

The following packages have unmet dependencies:
 libgcr-3-1:i386 : Depends: libgcr-3-common:i386 but it is not installable

This is known. The question is what the consequences of this are -- in this thread it's been suggested that it prevents any program needing to use SSL under wine from working. More at wine HQ here and here. More at Debian here. If I'm reading this right -- and I might not be -- we'll at the least have to wait for gnome-keyring 3.6 to make it out of experimental to sid, then from sid to testing (which is frozen).

 For no particular reason I think this may be what's causing issues with the installation of Office 2003 (and a few other programs, including Agilent HP Chemstation).

---

Links to this page:
http://linux.org.ru/forum/desktop/8571617/page1?lastmod=1358076186524

307. Burning audio CDs from the command line in debian testing/wheezy

I've got a CD burner on a headless box, so burning audio CDs from the command line is a necessity.

I also normally end up burning flash videos that I've converted to mp3s, so I'll show that too. This post is essentially covered (although not very well) already here: http://verahill.blogspot.com.au/2012/01/debian-testing-64-wheezy-small-fixes.html

First install the necessary programmes

sudo apt-get install ffmpeg wodim mpg123

---

Converting flv to mp3
To batch-convert flv files to mp3, do

ls *.flv|xargs -I {}  ffmpeg -i {} -ar 44100 -ab 160k -ac 2 {}.mp3

---

Preparing the files
Rename your files to 01.mp3, 02.mp3 etc. to make the songs burn in that order (since you're using *wav below).

Convert the mp3s to wav files (you could've gone straight from flv to wav above)

for i in *.mp3; do mpg123 --rate 44100 --stereo --buffer 3072 --resync -w "`basename "$i" .mp3`".wav "$i"; done

---

Burn
Burn with

wodim -v -pad speed=1 dev=/dev/cdrw1 -dao -swab *.wav

assuming that cdrw1 is the correct device.

Eject your cd when done.

eject cdrom1

Done.
[There's also no shortage of terminal music players, such as cplay.]

PS. You can burn anything you want from the command line using burn, e.g. an .iso file

sudo apt-get install burn
sudo burn -I -n myiso.iso

If the device you want to burn on is /dev/cdrom1 instead of /dev/cdrom, you can edit that in /etc/burn.conf

306. Insync with Google Drive and Google Docs on Debian Testing/Wheezy

The problem:
1. It would be nice to be able to use Google Docs as a replacement for Microsoft Word until Libre/OpenOffice catch up (post about that later) or the world switches to LaTeX and
2. for that to happen there needs to be an easier way to sync documents between google docs and your harddrive than using email.

The closest thing to that is using Google Drive to keep documents synced, and opening them in Google Docs using your browser.

It's been more than half a year since promising that Google Drive would be available for linux, and Google has yet to actually release anything (here) and it almost looks like vaporware (here).

In lieu of an official solution, there are a few options. One is Grive -- which seems to work with Google Drive but not Google Docs -- and another one is Insync, which isn't open source as it is owned by a start-up. It's the most promising and full-featured solution though, so we'll go with that.


There used to be gdocsfs, but it doesn't seem to be maintained.

The usual caveats about installing things from outside the repos apply, and even more so in this case since the source code is not available.

---

Setting up Insync

sudo apt-get install xdotool python-nautilus libxdo2 gir1.2-nautilus-3.0
mkdir -p ~/tmp/insync
cd ~/tmp/insync
wget http://s.insynchq.com/builds/insync-beta-gnome-cinnamon-common_0.9.34_amd64.deb
wget http://s.insynchq.com/builds/insync-beta-gnome_0.9.34_all.deb
sudo dpkg -i *.deb

If all went well you'll find InSync installed (move mouse to top-right corner, type insync and it should be there). Clicking on it opens a browser tab, in which you're asked to select the gmail account you wish to use.

You're next asked to allow InSync to do various things:

Confirm (you'll then get an email) and associate your machine with the account.

You should now have a new set of folders in $HOME:

/home/me/Insync/
`-- me@gmail.com

If you create a directory either in ~/Insync/me@gmail or in google drive using your browser, the directory should show up in both places (i.e. it's synced) -- assuming that you've got insync running:

insync --headless > /dev/null &

will keep it running in the background. Any doc file copied to the insync folder will now be editable in Google docs by pointing your browser to https://drive.google.com/#my-drive

---

Simple as that.