18 March 2013

362. Sun Grid Engine: Limit number of running jobs for a user

I don't like copying a post verbatim, but I also want to make sure that I keep track of what I do on my shared cluster, and that I don't forget how to set user based job limits.

Andi Broka made an excellent post at https://lists.sdsc.edu/pipermail/npaci-rocks-discussion/2010-November/049905.html in which two methods were shown (see below).

Resource quota management in general is covered here these days: http://docs.oracle.com/cd/E19957-01/820-0698/gehwk/index.html


1. Number of running jobs: limit for ALL users
Modify the scheduler configuration:
qconf -msconf
2 schedule_interval 0:0:15 3 maxujobs 0 4 queue_sort_method load
Change maxujobs from 0 (unlimited) to a suitable value.

The m in -msconf stands for modify, conf stands for configuration, and the s stands for scheduler.

2. Number of running jobs: limit for a particular user
Modify the resource quota set(s):
qconf -mrqs
1 { 2 name verahill 3 description "specific limitations for user verahill" 4 enabled TRUE 5 limit users verahill to slots=40 6 }

Again, the -m is for modify, and rqs is for resource quota set(s).

Note that slots are the number of processes and not nodes -- 40 slots allows me to use five 8-core nodes.

17 March 2013

361. Installing Debian on a USB stick (from a running Debian system)

Post 70 (Installing Debian on a USB stick -- live usb vs a true and full installation) is receiving a lot of traffic. While the method of installing debian used there -- using virtualbox -- works, it is slow and unnecessarily complex.

A better, faster and easier way is to use a chroot. Sure, it's a bit more command line oriented, but that doesn't necessarily make it harder.
You'll need an empty USB stick. Some sticks are faster than others, but they are all slower than spinning disks, and a lot slower than SSDs (never tried a USB3 stick though).

1. Prepare the USB stick
Attach the USB stick to a computer with Debian. Find out what device it is (do ls /dev/sd* before and after attaching the stick, and if it automounts, you can also check the output of df -h). In this case we'll assume that it's /dev/sdb.


sudo apt-get install util-linux e2fsprogs
sudo umount /dev/sdb*
sudo fdisk /dev/sdb
Command (m for help): o
Building a new DOS disklabel with disk identifier 0x209d6329.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/sdb: 4051 MB, 4051697664 bytes
125 heads, 62 sectors/track, 1021 cylinders, total 7913472 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x209d6329

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): 
Using default response p
Partition number (1-4, default 1): 
Using default value 1
First sector (2048-7913471, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-7913471, default 7913471): 
Using default value 7913471

Command (m for help): a                                                                               
Partition number (1-4): 1

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

It really is that simple to set up the partition table. o wipes the previous table, n creates a new partition, and a makes it bootable. w writes the changes. Now prepare the file system:

sudo mkfs.ext4 /dev/sdb1
mke2fs 1.42.5 (29-Jul-2012) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 247504 inodes, 988928 blocks 49446 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=1015021568 31 block groups 32768 blocks per group, 32768 fragments per group 7984 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done
2. Mount and bootstrap the USB stick

Note that you can change between i386 and amd64, testing and stable and different mirrors, by editing the debootstrap line below. I'd recommend i386 since it will run on 64 bit machines as well, which is useful if you're traveling.

sudo apt-get install debootstrap coreutils
mkdir ~/tmp/usbstick -p
sudo mount /dev/sdb1 ~/tmp/usbstick
sudo debootstrap --arch i386 testing $HOME/tmp/usbstick http://ftp.au.debian.org/debian/
I: Retrieving InRelease I: Checking Release signature I: Valid Release signature (key id 9FED2BCBDCD29CDF762678CBAED4B06F473041FA) I: Retrieving Packages [..] I: Configuring tasksel... I: Configuring tasksel-data... I: Base system installed successfully.

sudo mount -t proc none $HOME/tmp/usbstick/proc
sudo mount --bind /sys $HOME/tmp/usbstick/sys
sudo mount --bind /dev $HOME/tmp/usbstick/dev
sudo cp /etc/resolv.conf $HOME/tmp/usbstick/resolv.conf
sudo chroot $HOME/tmp/usbstick/

3. Basic setup

Edit the sources.list line as needed. The key is to enable non-free so you can install all the non-free firmware so that youre prepared for most types of hardware.

rm /etc/apt/sources.list
echo 'deb  http://ftp.au.debian.org/debian/ testing main contrib non-free' >> /etc/apt/sources.list
apt-get update
apt-get install locales sudo vim
echo 'export LC_ALL="C"'>>/etc/bash.bashrc
echo 'export LANG="C"'>>/etc/bash.bashrc
echo '127.0.0.1 usbstick >> /etc/hosts'
source /etc/bash.bashrc
useradd -m traveller
passwd traveller
usermod -G sudo -a traveller

If you're going to travel with this, you may want to install AIDE to make sure that you're executables haven't been changed (you can also use tripwire). Of course, if anyone has the ability to regenerate the database, then this won't help you.
apt-get install aide

You will have to wait with generating the database until you're properly booted from the USB stick.

4. Make the usbstick bootable
Make sure to install grub to /dev/sdb so that you don't mess up your desktop.

grep -v rootfs /proc/mounts > /etc/mtab
sudo apt-get install grub-pc linux-base linux-image-3.2.0-4-686-pae


You need to set up your /etc/fstab:
blkid /dev/sdb1 -o export|head -n 1 > /etc/fstab

Edit /etc/fstab (your UUID will vary):
UUID=24056f3d-d0c5-4297-adc4-40b2d2007412       /       ext4    errors=remount-ro,user_xattr    0 1

Run
update-grub
Generating grub.cfg ... Found linux image: /boot/vmlinuz-3.2.0-4-686-pae Found initrd image: /boot/initrd.img-3.2.0-4-686-pae done

At this point you've used ca 559 Mb.

5. Installing command line tools
The following installs a curses-based network connection manager, a mail program, a chat program and a browser. All of them work in the terminal:
sudo apt-get install wicd-curses mutt mcabber elinks rcconf

You've now used ca 670 Mb.

To set up Mutt and Mcabber, see here: http://verahill.blogspot.com.au/2011/12/configuring-mutt-for-two-imap-accounts.html
http://verahill.blogspot.com.au/2011/12/linux-basics-google-talk-using-gajim-or.html

6. Installing a desktop
This is luckily very easy. The bad thing is that USB sticks are very slow, so starting a heavy desktop from a USB stick is going to lead to long boot times.

To install gnome (2340 Mb) do
sudo apt-get install gnome rcconf

To install lxde (523 Mb) do
sudo apt-get install lxde rcconf

and to install xfce (457 Mb) do
sudo apt-get install xfce4 rcconf

We installed rcconf to prevent network-manager to start if it has been installed:
Run
rcconf
And unstar whatever you don't want to launch at boot.
To reduce boot times, edit /etc/default/grub and change to
GRUB_CMDLINE_LINUX_DEFAULT="quiet text"
Run
update-grub

You will now have to start whatever desktop you installed manually by running
startx

from the command line i.e. you only start the desktop if you really need it.

Clean your cache to free up some space:

apt-get clean

With xfce4 and cleaning, you've now used a total of 951 Mb.

7. Pull in some extra firmware
In case you'll be using this usb stick on random computers you might want to pull in as many different firmware as you can stomach:
apt-get install firmware-atheros firmware-iwlwifi firmware-ralink firmware-realtek

And so on.
You might also want to install programs such as clamav , but that's your business.

8. Unmounting your usbstick
Once you've installed everything you need:
exit
sudo umount $HOME/tmp/usbstick/sys
sudo umount $HOME/tmp/usbstick/dev
sudo umount $HOME/tmp/usbstick/proc
sudo umount $HOME/tmp/usbstick/

Now you should be able to boot from the USB stick on any computer that supports boot from USB -- and that would be most boxes made in the past five years or so.

Done.


Note that you can generate initramfs manually in debian using
sudo mkinitramfs -o /boot/initrd.img-VER -v VER

where VER e.g. 3.8.0 or 3.2.0-4-amd64 -- the name should be consistent with the vmlinuz, config and system.map suffixes.

Links to this post:
http://atomowki.net/dokuwiki/doku.php
http://andrewgudgel.com/cl-debian-usb.htm

16 March 2013

360. Screenshots from a desktop based Scientific Linux installation

This is a pretty basic post, but linux users span a wide range of skill levels.

I was originally planning to do post 359 via a chroot in Scientific Linux, but decided that doing it via Debian was..well..more convenient (the approach can easily be translated to any distro which has yum, and that includes e.g. Arch, in addition to the obvious suspects)

Anyway, as part of all that I decided to grab some screenshots of the installation process, and since I have them I might as well share them. I did this in virtualbox since I was originally planning on testing out the whole thing there before going to physical hardware.

So, here's Scientific Linux. The easiest approach is just to get whole dvd:
wget http://ftp1.scientificlinux.org/linux/scientific/6.3/x86_64/iso/SL-63-x86_64-2012-08-02-Install-DVD.iso

Launch the installation:






















 



Once the installation finishes, restart/










 
Click on the network icon in the top right corner, and then click on the desired network interface.
Open a terminal and do yum upgrade.