07 March 2012

96. Optimising, trimming and making your solid state drive last longer on linux

There's an excellent and informative guide here:
http://apcmag.com/how-to-maximise-ssd-performance-with-linux.htm

Please read it. I've tried it on my SSD and it didn't crash my system. In fact, it even seems snappier, but that might be subjective.

While the guide above talks about making Solid State Drives (SSD) last longer and work faster, it equally applies to other solid state media such as USB sticks.

Here we'll apply that guide to a USB stick with debian we installed as shown in this guide: http://verahill.blogspot.com.au/2012/02/installing-debian-on-usb-stick-live-usb.html

--START HERE --
Mount your USB stick as you would any other -- the wonderful things about *NIX is that all configuration options are set in files which you can easily edit.

/etc/fstab
Before editing, it looks like this

proc            /proc           proc    defaults        0       0
UUID=8c0c9c98-e73f-4f5a-b7d7-f91f95c9d8ca /               ext4    errors=remount-ro 0       1
UUID=e3305fbe-8e4e-4d0e-9975-0db692f87d32 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
We'll edit to

  • enable TRIM by adding discard.
    From the manual: "discard/nodiscard - Controls  whether  ext4  should issue discard/TRIM commands to the underlying block device when blocks are freed.  This is useful for SSD devices and sparse/thinly-provisioned LUNs, but it is off by default until sufficient testing has been done."
  • noatime.
    From the man: "Do not update inode access times on this filesystem (e.g., for faster access on the news spool to speed up news servers)."
  • nodiratime
    Same as noatime, for directories
  • move some temporary /var files to the RAM instead of disk
  • the last line puts all log files (/var/log/) in RAM as well. If you do want logs to persist between boots, comment it out
  • 1777 = chmod +trwx, 0755= u+rwx g+rw o+rw
  • I've kept the swap since I might want to use my stick on underpowered machines, but you really probably shouldn't
  • A good source of information on mount options is here: http://www.tuxfiles.org/linuxhelp/fstab.html


proc            /proc           proc    defaults        0       0
UUID=8c0c9c98-e73f-4f5a-b7d7-f91f95c9d8ca /               ext4 noatime,nodiratime,discard,errors=remount-ro 0       1
UUID=e3305fbe-8e4e-4d0e-9975-0db692f87d32 none            swap    sw              0       0
/dev/scd0       /media/cdrom0   udf,iso9660 user,noauto     0       0
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
tmpfs /var/spool tmpfs defaults,noatme,mode=1777 0 0
tmpfs /var/tmp tmpfs defaults,noatime,mode=1777 0 0
tmpfs /var/log tmpfs defaults,noatime,mode=0755 0 0

/etc/default/grub 
To set a system-wide scheduler policy for disk I/O we can edit grub and define elevator as noop, deadline or anticipatory.

Change e.g.
GRUB_CMDLINE_LINUX_DEFAULT="text splash"
to
GRUB_CMDLINE_LINUX_DEFAULT="text splash elevator=deadline"
If you want to know more about schedulers, you can look here: http://www.cyberciti.biz/faq/linux-change-io-scheduler-for-harddisk/


We need to do update grub, and that's a bit trickier, but still not too difficult.
My USB stick is mount at /media/8c0c9c98-e73f-4f5a-b7d7-f91f95c9d8ca/


sudo mount -o bind /dev 8c0c9c98-e73f-4f5a-b7d7-f91f95c9d8ca/dev/
sudo mount -o bind /sys 8c0c9c98-e73f-4f5a-b7d7-f91f95c9d8ca/sys/
sudo mount -o bind /proc 8c0c9c98-e73f-4f5a-b7d7-f91f95c9d8ca/proc/
sudo chroot 8c0c9c98-e73f-4f5a-b7d7-f91f95c9d8ca/
root@beryllium:/# update-grub
Generating grub.cfg ...
Found linux image: /boot/vmlinuz-3.2.0-1-686-pae
Found initrd image: /boot/initrd.img-3.2.0-1-686-pae
Found linux image: /boot/vmlinuz-2.6.32-5-686
Found initrd image: /boot/initrd.img-2.6.32-5-686
grep: input file `/boot/grub/grub.cfg.new' is also the output
done

And exit the chroot jail (type exit)


/etc/rc.local
Add a line to the end
echo 0>/proc/sys/vm/swappiness
to prevent use of swap until the RAM is full.

You're now done.


For an SSD drive you can just boot into your system and make the changes as above, but without fiddling with mounting /dev, /sys, /proc and doing chroot.

95. Cross-compiling 32 bit binaries on 64 bit system

There are many ways of setting up toolchain for cross compiling. This is the method I've had more success with, and feel more comfortable with. It introduces more overhead than some other approaches.

My motivation for wanting to cross compile is because I have a slow 32 bit headless server, and a couple of fast multiu-core 64 bit systems. Compiling kernel 3.2.9 on my 32 bit server takes up to 3 hours, while it takes less than half an hour on a more modern box.


This method allows you to compile without it affecting the host system. Everything will happen in a jail made up chroot. You will only be able to use architectures for which your hardware is compatible i.e. i386 and amd64. Cross-compiling for other architectures is a future project.

-- Start Here --


Debootstrap
create a directory where you can store you architectures e.g.
mkdir -p /home/me/tmp/architectures/i386
cd /home/me/tmp/architectures

sudo apt-get install debootstrap

You will use scripts to point your jailed systems apt to the right place. Here's how to see what's available:

ls /usr/share/debootstrap/scripts/
breezy  edgy  etch-m68k  gutsy  hoary         intrepid  karmic  lucid     natty      oneiric  precise  sarge.buildd      sid      stable   unstable  warty.buildd  woody
dapper  etch  feisty     hardy  hoary.buildd  jaunty    lenny   maverick  oldstable  potato   sarge    sarge.fakechroot  squeeze  testing  warty     wheezy        woody.buildd


Time to get busy:
sudo debootstrap --arch i386 testing /home/me/tmp/architectures/i386/ http://ftp.au.debian.org/debian/

I: Retrieving Release
I: Retrieving Release.gpg
I: Checking Release signature
I: Valid Release signature (key id 9FED2BCBDCD29CDF762678CBAED4B06F473041FA)
I: Retrieving Packages
[..]
I: Configuring perl...
I: Configuring tasksel...
I: Base system installed successfully


You can look at e.g. ftp://ftp.au.debian.org/debian/dists/wheezy/ to see what other architectures are available. In this particular case, it's amd64, armel, i386, ia64, powerpc etc. You won't be able to chroot into an architecture your hardware doesn't support, so you're effectively limited to amd64 and i386. Cross-compiling for other architectures will be a future project.


Go to Jail

make sure you're in /home/me/tmp/architectures
sudo chroot i386
root@tantalum:/#


You're in. To convince yourself it's real -- and that you can't escape from the /home/me/tmp/architectures/i386/ folder -- you can navigate around your system.

You still need to install all your build tools
apt-get install build-essential

Do you want to continue [Y/n]?
[..]
Get:1 http://ftp.au.debian.org/debian/ testing/main libgmp10 i386 2:5.0.4+dfsg-1 [264 kB]
Get:2 http://ftp.au.debian.org/debian/ testing/main libgomp1 i386 4.6.2-12 [27.3 kB]
[..]


Note what architecture we're installing packages for...it's really that simple. Also, no need for sudo since you're root. With only build-essential installed the entire system is ca 450 Mb.

Install kernel-package as well, so you can build 32 bit kernels. This bring it up to ca 550 Mb.

You'll want checkinstall to make .deb packages.

Locales
Finally, set all locale parameters to C by running this in the terminal

export LC_ALL=C
export LC_MESSAGES=C
export LC_COLLATE=C
export LANGUAGE=C
export LANG=C

Save yourself that headache in the future by
apt-get install locale
Exit your chroot jail, and start it again:
exit
sudo chroot i386

Reconfigure:
dpkg-reconfigure locales
and pick the same as your host system, in my case en_AU.UTF-8



Build something


--kernel--
We'll build a 32 bit kernel, but you could really build anything you want.

From outside the chroot jail (i.e. another terminal)
sudo cp ~/tmp/linux3.2.9.tar.bz2 ~/tmp/architectures/i386/usr/src/
Inside the chroot jail
root@tantalum:/# ls /usr/src/
linux-3.2.9.tar.bz2

cd /usr/src/
tar -xvf linux-3.2.9.tar.bz2
cd linux-3.2.9/

either copy a .config from an existing 32 bit kernel version (see http://verahill.blogspot.com.au/2012/03/debian-testing-kernel-329.html for more information), or
make defconfig

fakeroot make-kpkg -jN --initrd --revision=3.2.9 kernel_image kernel_headers
Choose the default for most of the questions. N is the number of cores+1 on the host system, e.g. 5 for a four-core CPU.

With -j5 I made a full build in 1m 21 s. That's a fairly minimal kernel build though.
Using the .config generated from an existing 32 bit debian system and following this I got a full build in about 25 minutes.

--Wine--
Following method 3 on this: http://verahill.blogspot.com.au/2012/01/debian-testingwheezy-64-bit-installing.html
cd /usr/src
wget http://prdownloads.sourceforge.net/wine/wine-1.3.35.tar.bz2
tar -xvf wine-1.3.35.tar.bz2
cd wine-1.3.35

apt-get install bison flex gcc libc6-dev libfontconfig-dev libfreetype6-dev libglu-dev libgsm1-dev libice-dev libjpeg-dev libldap-dev libmpg123-dev libncurses5-dev libopenal-dev libpng-dev libsm-dev libssl-dev libusb-dev libx11-dev libxcomposite-dev libxcursor-dev libxext-dev libxi-dev libxinerama-dev libxml2-dev libxrandr-dev libxrender-dev libxslt-dev libxt-dev libxxf86vm-dev make libcapi20-dev liblcms-dev libsane-dev libhal-dev libdbus-1-dev valgrind prelink libcups2-dev libv4l-dev libncurses5-dev libasound2-dev libz-dev

./configure
make -j5
checkinstall --install=no



In the future
Next time you want to cross compile, just
cd /home/me/tmp/architecture
sudo chroot i386
and you're ready to go.




06 March 2012

94. Virtualisation from the command line -- creating a debian image in virtualbox

I came across these posts and decided to put my own spin on it.
EDIT: fixed the port/device numbers for the dvd drive


Virtual box on the CLI

Here's my take on it. Most of it is self-explanatory. As with everything else Linux, names are case-sensitive.

VBoxManage createvm --name debi_dd --ostype Debian_64 --basefolder /home/me/tmp/virtual

Virtual machine 'debi_dd' is created.
UUID: 3cab43c4-d071-4d71-9e01-22a6f4e80645
Register the vm:
VBoxManage registervm `pwd`/debi_dd.vbox

or you will get
VBoxManage: error: Could not find a registered machine named 'debi_dd'

Create a harddrive:
VBoxManage createhd --filename debi_hd --size 5000 --format VDI --variant Standard


0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: b11e6465-3ac6-441d-9e14-6417135b7cd1
The size is given in mb, VDI is the standard virtualbox format, but you can use  VMDK or VHD too. There are several variants -- Fixed, Standard, Split2G, Stream and ESX. Fixed is fixed size, standard expands as data is added, Split2G splits the HDD into 2GB chunks for filesystems which can't handle large files, Stream is 'optimized for streaming downloads and can be compressed' and I suspect that ESX is referring to VMWare's hypervisor product.

Create a SATA controller (for a win xp guest , use IDE instead)
VBoxManage storagectl debi_dd --name "ATA controller" --add sata --controller IntelAhci

Attach your harddrive
VBoxManage storageattach debi_dd --storagectl "ATA controller" --port 0 --device 0 --type hdd --medium debi_hd.vdi

and a DVD player and point it towards your debian iso
VBoxManage storageattach debi_dd --storagectl "ATA controller" --port 1 --device 0 --type dvddrive --medium ~/jigdo/debian-testing-amd64-CD-1.iso

Configure your VM hardware
VBoxManage modifyvm debi_dd --memory 256 --acpi on --boot1 dvd --clipboard bidirectional --nic1 nat --cableconnected1 on --bridgeadapter1 eth0 --usb on --usbehci on

You will need to have an X server running, either on the computer hosting the virtualbox, or you need to ssh in using ssh -X -C user@server. But it works. Once you have a working linux installation you can run headless.
You can also use remmina to connect via rdp at localhost:3389.

VBoxManage startvm debi_dd

Waiting for VM "debi_dd" to power on...
VM "debi_dd" has been successfully started.
You'll now be doing the usual graphical installation...


** I had some problems with network dhcp, but manually setting the ip, gw, netmask and dns server went fine. It may be particular to my networking set up (3 cards, 2 using static ip)

Make sure to install ssh server so you can easily connect to your running VM.

So you're done installing...

Eject the DVD

VBoxManage storageattach debi_dd --storagectl "ATA controller" --port 1 --device 0 --type dvddrive --medium emptydrive


Associate e.g. localhost:2222 with virtualbox:22
VBoxManage modifyvm "debi_dd" --natpf1 "guestssh,tcp,,2222,,22"

Time to go headless:
VBoxHeadless -s debi_dd
Oracle VM VirtualBox Headless Interface 4.1.8
(C) 2008-2011 Oracle Corporation
All rights reserved.
VRDE server is listening on port 3389.

and 
ssh verahill@localhost -p 2222
to log in

You can also use remmina
Connect to localhost:3389 and you'll be greeted with:
remmina in action


Use
 VBoxManage controlvm debi_dd poweroff
to power off

There's a bit of housekeeping to do with any freshly installed linux machine:
sudo apt-get install bzip2 vim 
sudo updatedb
and edit /etc/apt/sources.list to remove the cd.
You may want to edit /etc/default/grub too and set
GRUB_TIMEOUT=1
GRUB_CMDLINE_LINUX_DEFAULT="text"

Don't forget to sudo update-grub

If you don't want headless mode, just
VBoxManage startvm debi_dd

Removal

VBoxManage storageattach debi_dd --storagectl "ATA controller" --port 1 --device 0 --type dvddrive --medium none
VBoxManage storageattach debi_dd --storagectl "ATA controller" --port 0 --device 0 --type dvddrive --medium none 
VBoxManage unregistervm debi_dd --delete 
VBoxManage closemedium disk debi_hd.vdi --delete 
rm debi_dd/ -rf