27 June 2013

465. The Intel MKL (Math Kernel Library) on Linux (Debian) -- for free

I've been living under the impression that the Intel MKL weren't free.

In all fairness, since I'm using AMD almost exclusively and the ACML is free -- and OpenBlas has worked fine on my i5-2400-based node -- I haven't had enough motivation to really dig into this. However, as part of a post on compiling GAMESS US, Kirill Berezovsky mentions in passing that you can get both the Intel MKL and ifortran for free for non-commercial use.

To be fair, that's free in the Windows sense, not in the Linux sense. You've still got enough restrictions to make Stallman weep, but it's free enough that we have a fighting chance at evaluation the software -- and use it if it's good enough.

See here for ACML: http://verahill.blogspot.com.au/2013/05/422-set-up-acml-on-linux.html
See here for OpenBlas: http://verahill.blogspot.com.au/2013/05/423-openblas-on-debian-wheezy.html

Anyway, this is about MKL:


0. Register a request
Go to http://software.intel.com/en-us/non-commercial-software-development
Click on what you want to download -- in this case the Intel MKL. Fill out the form and hit submit. Note that you'll need to enable cookies for this to work.


[while I was at it I got the intel parallel studio xe as well -- that's 2 Gb though, so don't get it unless you want it and have a lot of bandwidth to spare]


1. Download
You'll get an email with a link. Click it:


The MKL file is 609 MB, which is a bit bigger than the ACML files which tend to be around 67-70 Mb (each).

Note: the email you get has a serial number. From what I understand it's valid for one year if you want to download updates and new releases. However, nowhere does it say that any installed software will expire, so I presume that you can continue using the MKL libraries indefinitely.


2. Install
In my case the file is called l_mkl_11.0.4.183.tgz and it was downloaded to ~/Downloads
mkdir ~/tmp
cd ~/tmp
cp ~/Downloads/l_mkl_11.0.4.183.tgz .
tar xvf l_mkl_11.0.4.183.tgz
cd l_mkl_11.0.4.183
sudo sh install.sh
Step no: 1 of 7 | Welcome -------------------------------------------------------------------------------- Welcome to the Intel(R) Math Kernel Library 11.0 Update 4 for Linux* installation program. The Flagship of HPC Math Libraries. This library contains highly optimized, extensively threaded, mathematical functions for engineering, scientific, and financial applications that require maximum performance. -------------------------------------------------------------------------------- You will complete the steps below during this installation: Step 1 : Welcome Step 2 : License Step 3 : Activation Step 4 : Intel(R) Software Improvement Program Step 5 : Options Step 6 : Installation Step 7 : Complete -------------------------------------------------------------------------------- Step no: 1 of 7 | Options > Missing Optional Pre-requisite(s) -------------------------------------------------------------------------------- There are one or more optional unresolved issues. It is highly recommended to resolve them all before you continue the installation. You can fix them without exiting from the installation and re-check. Or you can quit from the installation, fix them and run the installation again. -------------------------------------------------------------------------------- Missing optional pre-requisites -- unsupported OS -------------------------------------------------------------------------------- 1. Skip missing optional pre-requisites [default] 2. Show the detailed info about issue(s) 3. Re-check the pre-requisites h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: 1 [..] 13. THIRD PARTY PROGRAMS. The Materials may include third party programs or materials that are governed by the third party's license terms, including without limitation, open source software. The license terms associated with such third party programs or materials govern your use of same, and Intel is not liable for such third party programs or materials. * Other names and brands may be claimed as the property of others -------------------------------------------------------------------------------- Do you agree to be bound by the terms and conditions of this license agreement? Type "accept" to continue or "decline" to back to the previous menu: accept Step no: 3 of 7 | Activation -------------------------------------------------------------------------------- If you have purchased this product and have the serial number and a connection to the internet you can choose to activate the product at this time. Activation is a secure and anonymous one-time process that verifies your software licensing rights to use the product. Alternatively, you can choose to evaluate the product or defer activation by choosing the evaluate option. Evaluation software will time out in about one month. Also you can use license file, license manager, or remote activation if the system you are installing on does not have internet access activation options. -------------------------------------------------------------------------------- 1. I want to activate my product using a serial number [default] 2. I want to evaluate my product or activate later 3. I want to activate either remotely, or by using a license file, or by using a license manager h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: 1 Note: Press "Enter" key to back to the previous menu. Please type your serial number (the format is XXXX-XXXXXXXX): -------------------------------------------------------------------------------- Activation completed successfully. -------------------------------------------------------------------------------- Press "Enter" key to continue: Step no: 4 of 7 | Intel(R) Software Improvement Program -------------------------------------------------------------------------------- Help improve your experience with Intel(R) software Participate in the design of future Intel software. Select 'Yes' to give us permission to learn about how you use your Intel software and we will do the rest. - No Personal contact information is collected - There are no surveys or additional follow-up emails by opting in - You can stop participating at any time Learn more about Intel(R) Software Improvement Program http://software.intel.com/en-us/articles/software-improvement-program With your permission, Intel may automatically receive anonymous information about how you use your current and future Intel software. -------------------------------------------------------------------------------- 1. Yes, I am willing to participate and improve Intel software. (Recommended) 2. No, I don't want to participate in the Intel(R) Software Improvement Program at this time. b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection: 2 Step no: 5 of 7 | Options -------------------------------------------------------------------------------- You are now ready to begin installation. You can use all default installation settings by simply choosing the "Start installation Now" option or you can customize these settings by selecting any of the change options given below first. You can view a summary of the settings by selecting "Show pre-install summary". -------------------------------------------------------------------------------- 1. Start installation Now 2. Change install directory [ /opt/intel/composer_xe_2013.4.183 ] 3. Change components to install [ All ] 4. Show pre-install summary h. Help b. Back to the previous menu q. Quit -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [1]: 1 Step no: 6 of 7 | Installation -------------------------------------------------------------------------------- Each component will be installed individually. If you cancel the installation, components that have been completely installed will remain on your system. This installation may take several minutes, depending on your system and the options you selected. -------------------------------------------------------------------------------- Installing Intel Math Kernel Library 11.0 Update 4 on IA-32 component... done -------------------------------------------------------------------------------- Installing Intel Math Kernel Library 11.0 Update 4 on Intel(R) 64 component... done -------------------------------------------------------------------------------- Finalizing installation... done -------------------------------------------------------------------------------- Press "Enter" key to continue Step no: 7 of 7 | Complete -------------------------------------------------------------------------------- Thank you for installing and for using the Intel(R) Math Kernel Library 11.0 Update 4 for Linux*. Support services start from the time you install or activate your product, so please create your support account now in order to take full advantage of your product purchase. Your Subscription Service support account provides access to free product updates interactive issue management, technical support, sample code, and documentation. To create your support account, please visit the Subscription Services web site https://registrationcenter.intel.com/RegCenter/registerexpress.aspx?clientsn=NBJ N-87S2RM3P To get started using Intel(R) Math Kernel Library 11.0 Update 4 located in /opt/intel/composer_xe_2013.4.183 visit: install-dir/Documentation/en_US/mkl/get_started.html. -------------------------------------------------------------------------------- q. Quit [default] -------------------------------------------------------------------------------- Please type a selection or press "Enter" to accept default choice [q]:

3. Usage
I'll demonstrate using nwchem. I've repeated the build instructions over and over on this blog, but here it goes again:

sudo apt-get install build-essential gfortran python2.7-dev libopenmpi-dev openmpi-bin
sudo mkdir /opt/nwchem -p
sudo chown $USER:$USER /opt/nwchem
cd /opt/nwchem
wget http://www.nwchem-sw.org/download.php?f=Nwchem-6.3.revision1-src.2013-05-28.tar.gz -O Nwchem-6.3.revision1-src.2013-05-28.tar.gz
tar xvf Nwchem-6.3.revision1-src.2013-05-28.tar.gz
cd nwchem-6.3-src.2013-05-28/
export NWCHEM_TOP=`pwd`
export LARGE_FILES=TRUE
export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=`pwd`
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES="all python"
export PYTHONVERSION=2.7
export PYTHONHOME=/usr

export BLASOPT="-L/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64/ -lmkl_sequential -lmkl_core -lmkl_intel_ilp64"
export LIBRARY_PATH="$LIBRARY_PATH:/usr/lib/openmpi/lib:/opt/intel/composer_xe_2013.4.183/mkl/lib/intel64/"

export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/lib/openmpi/lib
export MPI_INCLUDE=/usr/lib/openmpi/include
export LIBMPI="-lmpi -lopen-rte -lopen-pal -ldl -lmpi_f77 -lpthread"
export ARMCI_NETWORK=SOCKETS

cd $NWCHEM_TOP/src

make clean
make nwchem_config
make FC=gfortran 1> make.log 2>make.err

cd $NWCHEM_TOP/contrib
export FC=gfortran
./getmem.nwchem


I had a look at http://www.ichec.ie/support/tutorials/mkl.pdf to better understand what the different libraries do. Still not quite clear though, so I might not be getting optimal performance.


4. Performance
I basically repeated the test job shown here: http://verahill.blogspot.com.au/2013/05/430-briefly-crude-comparison-of.html

The machine has 16 gb ram and an intel i5-2400 cpu (4 cores)

ACML: Total times cpu: 1644.3s wall: 1655.5s
MKL:    Total times  cpu:     1550.0s     wall:     1563.8s

It certainly looks good. While 100 seconds in less than two minutes, and 1550 is 94 %  of 1644, it's a difference of about one day per fortnight, which doesn't make much of a difference on a small single-user cluster, it would make a bit of cumulative difference on our 1400 core multi-user cluster.

26 June 2013

464. bytes2words -- python script

This script does something I could easily do myself in e.g. bc, and so is a bit of a waste of time. However, because I enjoy writing short python scripts, I did it anyway.

Usage:
/bytes2words -i 8gib -o MW
Input: 8gib Output: mw Assuming 64 bit word size Result: 8 gib is 1073 mw
bytes2words:
#!/usr/bin/python
# Converts bytes to words
# The impetus comes from the use of MW as the default memory unit in many computational pieces of software.

import sys

def split_text(s): # from http://stackoverflow.com/questions/12409894
    from itertools import groupby
    for k,g in groupby(s, str.isalpha):
        yield ''.join(list(g))

def getargs(arguments):
 switches={}

 version='0.1'
 
 try:
  if "-i" in arguments:
   switches['i']=arguments[arguments.index('-i')+1]
   print 'Input: %s'% switches['i']

   inputted=list(split_text(switches['i']))
   
   if len(inputted)>1:
    switches['i']=inputted[0]
    switches['unit']=inputted[len(inputted)-1].lower()
    
    if switches['unit'] == 'kib':
     switches['if']=1024
    elif switches['unit'] == 'kb':
     switches['if']=1000
    elif switches['unit'] == 'mb':
     switches['if']=1000*1000
    elif switches['unit'] == 'mib':
     switches['if']=1024*1024
    elif switches['unit'] == 'gb':
     switches['if']=1000*1000*1000
    elif switches['unit'] == 'gib':
     switches['if']=1024*1024*1024
   else:
    switches['if']=1
   
   if len(inputted)>2:
    print 'Illegal input: %s'% inputted
    arguments="--help" 
  else:
   arguments="--help";
 except:
  arguments="--help";
  
 try:
  if "-o" in arguments:
   switches['o']=arguments[arguments.index('-o')+1].lower()
   print 'Output: %s'% switches['o']
   if switches['o'] == 'w':
    switches['of']=1
   elif switches['o'] == 'kiw':
    switches['of']=1024
   elif switches['o'] == 'kw':
    switches['of']=1000
   elif switches['o'] == 'mw':
    switches['of']=1000*1000
   elif switches['o'] == 'miw':
    switches['of']=1024*1024
   elif switches['o'] == 'gw':
    switches['of']=1000*1000*1000
   elif switches['o'] == 'giw':
    switches['of']=1024*1024*1024
   else:
    print 'illegal output argument' % switches['o']
  else:
   arguments="--help";
 except:
  arguments="--help";
 version='0.1'

 try:
  if "-b" in arguments:
   switches['b']=int(arguments[arguments.index('-b')+1])
   print 'Word size: %i bits'% switches['b']
  else:
   print 'Assuming 64 bit word size'
   switches['b']=64;
 except:
  switches['b']=64;

 doexit=0
 try:
  if ("-h" in arguments) or ("--help" in arguments):
   print '\t\t bytes2words version %s' % version
   print ' \t-i\t input in words with units, e.g. 200kb'
   print ' \t-o\t output unit (w,kw,mw,gw)'
   print ' \t-b\t word size (32 or 64 (bit))'
   print 'Exiting'
   doexit=1
 except:
  a=0 # do nothing
 if doexit==1:
  sys.exit(0)
 return switches
 
if __name__=="__main__":
 bitsperbyte=8
 arguments=sys.argv[1:len(sys.argv)]
 switches=getargs(arguments)
 
 print 'Result: %s %s  is %s %s' % ( (switches['i']),switches['unit'],int(switches['if']*(float(switches['i'])/float(switches['b']/bitsperbyte))/float(switches['of'])),switches['o'])
 


463. Very Briefly: Installing ia32-libs

Since the skype and bankid posts on this blog rely on ia32-libs, and the question has popped up a few times:

To install ia32-libs on debian wheezy and up you need to enable multiarch:

sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install ia32-libs