419. Talking to Myself in Public: DKMS, your own kernel and Virtualbox

Brief version:

check your /lib/modules/KERNEL/build symmlinks if you compile your own kernel.

Original post:

A while back I installed the Oracle-packaged version Virtualbox 4.2 on my system since I had issues with the debian version when going above kernel 3.6 (http://verahill.blogspot.com.au/search?q=kernel+virtualbox)

It's been a bit annoying since I've had to run

sudo service vboxdrv setup

manually every time I upgrade the kernel, but it's not that much hassle. However, I've been seeing error messages such as

Error! Could not locate dkms.conf file.
File:  does not exist.

In addition, I've just upgraded the kernel from 3.8.8-ck1 to 3.9.2-ck1, but not rebooted, so there's no vboxdrv module for the running kernel at the moment. Both kernels were compiled on the machine they were running on.
Anyway, typically I should be able to solve that by doing

sudo dkms autoinstall -k 3.8.8-ck1

but again I get the above error about no dksm.conf file.

---

Troubleshooting:

The debian version
virtualbox-dkms belongs to the debian version and isn't needed for the oracle version -- installing it removes the Oracle version. Anyway, to remind myself of why I wasn't using the debian version (4.1.18-dfsg-2+deb7u1)) in Wheezy I did

sudo apt-get install virtualbox-dkms

which remove the oracle version and installed the debian one.

Building initial module for 3.9.2-ck1
Error! Bad return status for module build on kernel: 3.9.2-ck1 (x86_64)
Consult /var/lib/dkms/virtualbox/4.1.18/build/make.log for more information.
dpkg: error processing virtualbox-dkms (--configure):
 subprocess installed post-installation script returned error exit status 10

dmesg shows

[704511.419271] vboxdrv: disagrees about version of symbol module_layout

OK. False starts.

I then purged virtualbox and reinstalled the debian version:

sudo apt-get purge virtualbox
sudo apt-get install virtualbox-dkms

Same problem.
Looking at https://bbs.archlinux.org/viewtopic.php?id=151965

for i in /var/lib/dkms/*/[^k]*/source; do [ -e "$i" ] || echo "$i";done

/var/lib/dkms/vboxhost/4.1.10/source
/var/lib/dkms/vboxhost/4.2.8/source

sudo rm /var/lib/dkms/vboxhost/4.1.10/source
sudo rm /var/lib/dkms/vboxhost/4.2.8/source

sudo apt-get install virtualbox-dkms

Error! Bad return status for module build on kernel: 3.9.2-ck1 (x86_64)
Consult /var/lib/dkms/virtualbox/4.1.18/build/make.log for more information.
dpkg: error processing virtualbox-dkms (--configure):

sudo less /var/lib/dkms/virtualbox/4.1.18/build/make.log

CC [M]  /var/lib/dkms/virtualbox/4.1.18/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.o
/var/lib/dkms/virtualbox/4.1.18/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c: In function ‘rtR0MemObjNativeMapUser’:
/var/lib/dkms/virtualbox/4.1.18/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1451:38: error: ‘VM_RESERVED’ undeclared (first use in this function)
/var/lib/dkms/virtualbox/4.1.18/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1451:38: note: each undeclared identifier is reported only once for each function it appears in
make[2]: *** [/var/lib/dkms/virtualbox/4.1.18/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.o] Error 1
make[1]: *** [/var/lib/dkms/virtualbox/4.1.18/build/vboxdrv] Error 2
make: *** [_module_/var/lib/dkms/virtualbox/4.1.18/build] Error 2
make: Leaving directory `/home/me/tmp/linux-3.9.2'

The issues is known
http://siduction.org/index.php?name=PNphpBB2&file=viewtopic&t=3097
http://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1086030.html
http://forums.linuxmint.com/viewtopic.php?f=47&t=52496

The oracle version
I gave up and downloaded the newest oracle version again:

wget http://download.virtualbox.org/virtualbox/4.2.12/virtualbox-4.2_4.2.12-84980~Debian~wheezy_amd64.deb
sudo dpkg -i virtualbox-4.2_4.2.12-84980~Debian~wheezy_amd64.deb

Trying to register the VirtualBox kernel modules using DKMS:Error! Your kernel headers for kernel 3.8.8-ck1 cannot be found.
Please install the linux-headers-3.8.8-ck1 package,
or use the --kernelsourcedir option to tell DKMS where it's located

Failed, trying without DKMS ... failed!
Recompiling VirtualBox kernel modules:.
Starting VirtualBox kernel modules:
modprobe vboxdrv failed. Please use 'dmesg' to find out why ... failed!

That last 'failed' is actually ok -- I think it built, but only for 3.9.2-ck1 while the running kernel is 3.8.8-ck1.

So, good enough to use, but not good enough for a blog post, since we're back to square one.
Hmm...

ls /lib/modules/3.8.8-ck1/build -lah
/lib/modules/3.8.8-ck1/build -> /home/me/tmp/ck-kernel/linux-3.8.8

Sure, that where I built it, but it should be pointing at /usr/src/linux-headers-3.8.8-ck1

sudo rm /lib/modules/3.8.8-ck1/build
sudo ln -s /usr/src/linux-headers-3.8.8-ck1/ /lib/modules/3.8.8-ck1/build

I then did the whole purge/install dance again:

sudo apt-get purge virtualbox-4.2
sudo dpkg -i virtualbox-4.2_4.2.12-84980~Debian~wheezy_amd64.deb

Trying to register the VirtualBox kernel modules using DKMS:.
Starting VirtualBox kernel modules:.

dkms status

nvidia, 304.88, 3.8.0, x86_64: installed
nvidia, 304.88, 3.8.8-ck1, x86_64: installed
nvidia, 304.88, 3.8.8, x86_64: installed
nvidia, 304.88, 3.9.2-ck1, x86_64: installed
vboxhost, 4.2.12, 3.8.8-ck1, x86_64: installed

sudo rm /lib/modules/3.9.2-ck1/build
sudo ln -s /usr/src/linux-headers-3.9.2-ck1/ /lib/modules/3.9.2-ck1/build
sudo dkms autoinstall -k 3.9.2-ck1
dkms status

nvidia, 304.88, 3.8.0, x86_64: installed
nvidia, 304.88, 3.8.8-ck1, x86_64: installed
nvidia, 304.88, 3.8.8, x86_64: installed
nvidia, 304.88, 3.9.2-ck1, x86_64: installed
vboxhost, 4.2.12, 3.8.8-ck1, x86_64: installed
vboxhost, 4.2.12, 3.9.2-ck1, x86_64: installed

Happy again. And no dkms.conf errors anymore!

418. POVRay 3.7 on four-eight cores is several orders of magnitudes slower than serial POVRay 3.6

For some reason I get very poor performance when using POVRay 3.7, which can run rendering in parallel. In contrast, POVRay 3.6 -- which is serial -- renders much, much faster.

I had a particular job with partially transparent objects (similar type of job as set up here: http://verahill.blogspot.com.au/2013/05/415-briefly-making-polyhedral-figure-in.html) took

• 2h 25 min (8694 seconds) using POVRay 3.6 running in serial

Smallest Alloc:                   9 bytes
Largest  Alloc:              131080 bytes
Peak memory used:          20730898 bytes
Total Scene Processing Times
  Parse Time:    0 hours  0 minutes  0 seconds (0 seconds)
  Photon Time:   0 hours  0 minutes  0 seconds (0 seconds)
  Render Time:   2 hours 24 minutes 54 seconds (8694 seconds)
  Total Time:    2 hours 24 minutes 54 seconds (8694 seconds)

•  14 hours 5 minutes in POVRay 3.7 rc 7 running in parallel (openmp)

Render Time:
  Photon Time:      No photons
  Radiosity Time:   No radiosity
  Trace Time:      14 hours  5 minutes 42 seconds (50742.481 seconds)
              using 4 thread(s) with 201478.244 CPU-seconds total
POV-Ray finished

on a four-core i5-2400 with 16 Gb RAM.

POVRay 3.7 took 16 hours 45 minutes on an eight core AMD FX 8150 with 32 Gb RAM.

Render Time:
  Photon Time:      No photons
  Radiosity Time:   No radiosity
  Trace Time:      16 hours 45 minutes 34 seconds (60334.108 seconds)
              using 8 thread(s) with 472419.334 CPU-seconds total
POV-Ray finished

I rendered with

povray_3.7 +H1000 +W1000 +A0.01 scene.pov

The general trend applies to povray 3.7-rc6 and whatever povray version I was using in Arch a month ago. It also applies to all linux boxes I've tried it on.

I built povray 3.6 and 3.7 as shown here: http://verahill.blogspot.com.au/2013/05/413-povray-37-rc7-on-debian-wheezy.html

Googling I really only found this: http://news.povray.org/povray.beta-test/thread/%3C455a0770@news.povray.org%3E/?ttop=349052&toff=450
which is from 2006.

First problem!
Version 3.7.0.beta16 was MUCH slower. Version 3.6.1c took 4m46s, while Version3.7.0.beta16 took 7m43s... that's 60% more! Is there a better build or should I be using some different options?

It was never addressed.

I don't know if the performance of 3.7 is worse then 3.6, or if there's some difference in settings that's slowing things down i.e. whether there's something which I'm doing wrong.

A general list over changes between 3.7 and 3,6 is found here: http://wiki.povray.org/content/Documentation:Tutorial_Section_1#Changes_and_New_Features_Summary
but there's nothing that stands out to me.

417. Briefly: Patching kernel 3.9 with the CK patchset: 3.9-ck-1

Nothing strange here -- basically the same as http://verahill.blogspot.com.au/2013/04/395-ck-kernel-on-debian-and-patching.html but with updated links.

I haven't found a good and succinct description of what the -ck patch set does and that I could link to here, but here's what it says on the Arch -ck page:

"..many Archers elect to use this package for the BFS' excellent desktop interactivity and responsiveness under any load situation. Additionally, the bfs imparts performance gains beyond interactivity"

I don't know if there are objective benchmarks that one can use to demonstrate an improvement in 'responsiveness and interactivity'. Subjectively, however, I feel that there's a slight improvement. You decide for yourself.

Begin here

sudo apt-get install kernel-package fakeroot build-essential ncurses-dev
mkdir ~/tmp
cd ~/tmp
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.9.2.tar.xz
tar xvf linux-3.9.2.tar.xz
cd linux-3.9.2/
wget http://ck.kolivas.org/patches/3.0/3.9/3.9-ck1/patch-3.9-ck1.bz2
bunzip2 patch-3.9-ck1.bz2
patch -p1 < patch-3.9-ck1

patching file arch/powerpc/platforms/cell/spufs/sched.c
patching file Documentation/scheduler/sched-BFS.txt
patching file Documentation/sysctl/kernel.txt
patching file fs/proc/base.c
patching file include/linux/init_task.h
patching file include/linux/ioprio.h
patching file include/linux/sched.h
Hunk #6 succeeded at 2738 (offset -10 lines).
patching file init/Kconfig
patching file init/main.c
patching file kernel/delayacct.c
patching file kernel/exit.c
patching file kernel/posix-cpu-timers.c
patching file kernel/sysctl.c
patching file lib/Kconfig.debug
patching file include/linux/jiffies.h
patching file drivers/cpufreq/cpufreq.c
patching file drivers/cpufreq/cpufreq_ondemand.c
patching file drivers/cpufreq/cpufreq_conservative.c
patching file kernel/sched/bfs.c
patching file kernel/sched/Makefile
patching file include/uapi/linux/sched.h
patching file include/linux/sched/rt.h
patching file kernel/stop_machine.c
patching file include/linux/swap.h
patching file mm/memory.c
patching file mm/swapfile.c
patching file mm/vmscan.c
patching file arch/x86/Kconfig
patching file kernel/Kconfig.hz
patching file kernel/Kconfig.preempt
patching file Makefile

make-kpkg clean
cat /boot/config-`uname -r`>.config
make oldconfig

You might now be asked a long series of questions about how the kernel should be configured (or you might not be -- depending on what kernel version you're currently running). In MOST cases you can select the default option (i.e. hit enter) but you should still read each question and consider it. Making a mistake won't break your computer, so don't be scared.

Next, start the compilation (will take a while):

time fakeroot make-kpkg -j4 --initrd kernel_image kernel_headers
sudo dpkg -i ../linux*3.9-ck*.deb

where 4 is the number of cores on your machine (note: it only has to do with compiling -- you can use the compiled binaries on any number of cores).

Anyway, that's all -- you've now patched, compiled and installed a new kernel. And it didn't even hurt.