I just tried to build ECCE on Ubuntu LTS 16.04.2 according to http://verahill.blogspot.com/2015/06/611-building-ecce-on-debian-jessie.html and it worked without a hitch. The only things that needed changing were
* from openjdk-7-jdk in debian to openjdk-8-jdk in ubuntu
* src/inv/flclient/flfreetype.c: <ftoutln.h> needed changing to <freetype/ftoutln.h>
Lindqvist - a blog about Linux and Science. Mostly.
Showing posts with label ubuntu. Show all posts
Showing posts with label ubuntu. Show all posts
12 June 2017
18 January 2016
626. Briefly: Gaussian and cloud computing -- Gaussian G09D with Slurm on aws/ec2
Note: you may want to install awscli and euca2ools. I didn't, so I don't actually know whether they are useful.
My instructions are quite rudimentary since I don't have much time to write these blog posts anymore. Hopefully there's enough information to get you through.
AWS
Either way, sign up for AWS. If you already have an amazon ID I think you can use that. Go to https://aws.amazon.com/
Select Launch an Instance and pick the ubuntu AIM and do Launch and Review. I launched it as a t2.micro instance type, as it is free and it's sufficient for set up but not to run jobs.
Hit launch, and create a new key pair. I called mine myfirstkeypair and saved the pem file in my ~/Downloads folder
In my Downloads folder:
I added my id_rsa.pub to ~/.ssh/authorized_keys on the ubuntu AWS image to make logging in via ssh easier -- that way I won't need the pem file.
Set up Gaussian
I then connected with SCP and uploaded my gaussian files -- I went straight for EM64T G09D. It went quite fast at +5 MB/s
Once that was done, on the ubuntu AWS instance I did:
NOTE that you can't run any gaussian jobs under a t2.micro instance. You will have to stop and relaunch as at least a t2.small instance or the jobs will be 'Killed' (that's what is echoed in the terminal when you try to run)
Note that if you terminate an image it will be deleted.
Stop the image and then create a snapshot or an image from it to keep everything you've installed.
Set up Slurm
You'll want a queue manager so that you can launch several jobs in serial. Also, you can set up your script so that it shuts down the image when your job is done to save money.
Benchmark:
c4.2xlarge 2h 11 min [1h 20 min] 8 vcpu/16 Gb
c4.4xlarge 1h 15 min [ 44 min] 16 vcpu/32 Gb
c4.8xlarge 41 min [ 25 min] 36 vcpu/60 Gb
It scales surprisingly well, although not perfectly linearly. It's clear that it's cheaper to use a smaller instance, so if time isn't critical or the larger memory isn't needed, c4.8xlarge is not the first choice.
Dropbox:
You might want to use dropbox to transfer files back and forth, especially finished job files (useful if you shut down the machine using a slurm script as shown below)
Open that link in a browser, then go back to the terminal.
Now, since you don't want to use up space unnecessarily (you're paying for it after all), exclude as many directories as possible. To exclude all existing dropbox dirs, do
Note that it can't handle directories with spaces in the name, so you'll need to polish the list by hand. Next create a directory where you want to run and store your jobs,e .g.
When you run a gaussian job, make sure to specify where the .chk files should end up, e.g.
Stop after execution:
Use a batch script along these lines:
My instructions are quite rudimentary since I don't have much time to write these blog posts anymore. Hopefully there's enough information to get you through.
AWS
Either way, sign up for AWS. If you already have an amazon ID I think you can use that. Go to https://aws.amazon.com/
Select Launch an Instance and pick the ubuntu AIM and do Launch and Review. I launched it as a t2.micro instance type, as it is free and it's sufficient for set up but not to run jobs.
Hit launch, and create a new key pair. I called mine myfirstkeypair and saved the pem file in my ~/Downloads folder
In my Downloads folder:
ssh -i "myfirstkeypair.pem" ubuntu@ec2-11-222-33-444.us-west-2.compute.amazonaws.comI then set a password in the ubuntu AWS image:
sudo passwd ubuntu
I added my id_rsa.pub to ~/.ssh/authorized_keys on the ubuntu AWS image to make logging in via ssh easier -- that way I won't need the pem file.
Set up Gaussian
I then connected with SCP and uploaded my gaussian files -- I went straight for EM64T G09D. It went quite fast at +5 MB/s
scp E6L-103X.tgz ubuntu@ec2-00-111-22-333.us-west-2.compute.amazonaws.com:/home/ubuntu/E6L-103X.tgz
Once that was done, on the ubuntu AWS instance I did:
sudo apt-get install csh sudo mkdir /opt/gaussian cd /opt sudo chown ubuntu gaussian -R cd /opt/gaussian cp ~/E6L-103X.tgz . tar xvf E6L-103X.tgz cd g09 csh bsd/install
echo 'export GAUSS_EXEDIR=/opt/gaussian/g09/bsd:/opt/gaussian/g09/local:/opt/gaussian/g09/extras:/opt/gaussian/g09' >> ~/.bashrc echo 'export GAUSS_SCRDIR=/home/ubuntu/scratch' >> ~/.bashrc echo 'export PATH=$PATH:/opt/gaussian/g09' >> ~/.bashrc source ~/.bashrc mkdir ~/scratch ~/jobs
NOTE that you can't run any gaussian jobs under a t2.micro instance. You will have to stop and relaunch as at least a t2.small instance or the jobs will be 'Killed' (that's what is echoed in the terminal when you try to run)
Note that if you terminate an image it will be deleted.
Stop the image and then create a snapshot or an image from it to keep everything you've installed.
Set up Slurm
You'll want a queue manager so that you can launch several jobs in serial. Also, you can set up your script so that it shuts down the image when your job is done to save money.
sudo apt-get update sudo apt-get install slurm-llnl
ControlMachine=localhost ControlAddr=127.0.0.1 MpiDefault=none ProctrackType=proctrack/pgid ReturnToService=2 SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd SlurmUser=slurm StateSaveLocation=/var/lib/slurm-llnl/slurmctld SwitchType=switch/none TaskPlugin=task/none FastSchedule=1 SchedulerType=sched/backfill SelectType=select/linear AccountingStorageType=accounting_storage/none ClusterName=rupert JobAcctGatherType=jobacct_gather/none SlurmctldLogFile=/var/log/slurm-llnl/slurmctld.log SlurmdLogFile=/var/log/slurm-llnl/slurmd.log NodeName=localhost NodeAddr=127.0.0.1 PartitionName=All Nodes=localhost
sudo /usr/sbin/create-munge-keyEdit /etc/default/munge:
Then runOPTIONS=--force
sudo service slurm-llnl restart
sudo service munge restartTest using slurm.batch
and submit with#!/bin/bash # #SBATCH -p All #SBATCH --job-name=test #SBATCH --output=res.txt # #SBATCH --ntasks=1 #SBATCH --time=10:00 srun hostname srun sleep 60
sbatch slurm.batch
squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
2 All test ubuntu R 0:08 1 localhost
Benchmark:
Using the same opt/freq benchmark as in post 621.#!/bin/csh #SBATCH -p All #SBATCH --time=9999999 #SBATCH --output=slurm.out #SBATCH --job-name=benchmark setenv GAUSS_SCRDIR /home/ubuntu/scratch setenv GAUSS_EXEDIR /opt/gaussian/g09/bsd:/opt/gaussian/g09/local:/opt/gaussian/g09/extras:/opt/gaussian/g09 /opt/gaussian/g09/g09< benchmark.in > benchmark.out
c4.2xlarge 2h 11 min [1h 20 min] 8 vcpu/16 Gb
c4.4xlarge 1h 15 min [ 44 min] 16 vcpu/32 Gb
c4.8xlarge 41 min [ 25 min] 36 vcpu/60 Gb
It scales surprisingly well, although not perfectly linearly. It's clear that it's cheaper to use a smaller instance, so if time isn't critical or the larger memory isn't needed, c4.8xlarge is not the first choice.
Dropbox:
You might want to use dropbox to transfer files back and forth, especially finished job files (useful if you shut down the machine using a slurm script as shown below)
cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf - ~/.dropbox-dist/dropboxdThis computer isn't linked to any Dropbox account... Please visit https://www.dropbox.com/cli_link_nonce?nonce=0011223344556677889900aabbccddeef to link this device. This computer isn't linked to any Dropbox account...
Open that link in a browser, then go back to the terminal.
wget -O - https://www.dropbox.com/download?dl=packages/dropbox.py > dropbox.py sudo mv dropbox.py /usr/local/bin sudo chmod +x d/usr/local/bin/dropbox.py dropbox.py autostart y
Now, since you don't want to use up space unnecessarily (you're paying for it after all), exclude as many directories as possible. To exclude all existing dropbox dirs, do
cd ~/Dropbox dropbox.py exclude add `ld -d */` dropbox.py exclude add `ld *.*` dropbox.py exclude list
Note that it can't handle directories with spaces in the name, so you'll need to polish the list by hand. Next create a directory where you want to run and store your jobs,e .g.
mkdir ~/Dropbox/aws_jobs
When you run a gaussian job, make sure to specify where the .chk files should end up, e.g.
so that you don't use up space/bandwidth for your chk files (unless of course you want to).%chk=/home/ubuntu/scratch/benchmark.chk
Stop after execution:
Use a batch script along these lines:
#!/bin/csh #SBATCH -p All #SBATCH --time=9999999 #SBATCH --output=slurm.out #SBATCH --job-name=benchmark setenv GAUSS_SCRDIR /home/ubuntu/scratch setenv GAUSS_EXEDIR /opt/gaussian/g09/bsd:/opt/gaussian/g09/local:/opt/gaussian/g09/extras:/opt/gaussian/g09 /opt/gaussian/g09/g09< benchmark.in > benchmark.out rm /home/ubuntu/scratch/*.* sudo shutdown -h now
20 November 2014
603. Mobile bankid; works fine in a VM too
Turns out I got it wrong in my earlier post -- mobile bankid has nothing to do with the telecom network.
How it works:
1. Phone: Start BankID on your phone. The program will say that it's waiting for a connection.
2. Computer: You go to your bank/government service web page, select Mobile BankID, then type in your 'person nummer' (like SSN).
3. Phone: BankID on your phone will then prompt you for your PIN.
4. Computer: You're logged in.
It's all happening over the internet. Sure, it might not work if you change SIM, for example, but at least it's not telephony based.
Best of all, I could get an activation code for mobile bankid using the win 7 bankid instance I set up in http://verahill.blogspot.com.au/2014/11/602-surviving-bankid-rant-moving-nge.html and now I never have to use it again.
[if you're on linux and in Scandinavia, just install mobile bankid on your phone or on android in a vm]
So, if you're on linux and you've been using the official bankid application and you happen to be a couple of continents removed from Scandinavia follow this post which basically does the following:
1. Install ubuntu 10.04 LTS in a VM
2. Install bankid 4.19.XXXX or earlier in the ubuntu VM
3. Copy your ~/.personal from your regular linux computer to the ubuntu VM
4. export your key using persadm export to a USB stick
5. Install windows 7 in a VM using the free, legal isos and the free, legal installation key. The copy will expire after 30 days (but will still continue working).
6. Install bankid in the Win 7 VM
7. Under File/Preferences in bankid add the directory on your USB stick with the exported bankid key
8. Log in to your BankID issuer (probably your bank) using bankid 'on file', and request a mobile bankid. You'll get an activation code
9. Install the bankid app on your phone (e.g. bankid from the play store on android)
10. Start the bankid app on your phone, and type in your personnummer and the activation code. Create a pin, which is your password
11. You're done.
I mean, sure, it's a very awkward way of going about it, but it works and is conceptually simple.
Android in a VM
And there's absolutely no reason you can't use mobile bankid it in a virtualbox VM if you want to -- I used an android 4.4 x86 iso and created a virtual machine, and then set up bankid which works fine.
How it works:
1. Phone: Start BankID on your phone. The program will say that it's waiting for a connection.
2. Computer: You go to your bank/government service web page, select Mobile BankID, then type in your 'person nummer' (like SSN).
3. Phone: BankID on your phone will then prompt you for your PIN.
4. Computer: You're logged in.
It's all happening over the internet. Sure, it might not work if you change SIM, for example, but at least it's not telephony based.
Best of all, I could get an activation code for mobile bankid using the win 7 bankid instance I set up in http://verahill.blogspot.com.au/2014/11/602-surviving-bankid-rant-moving-nge.html and now I never have to use it again.
[if you're on linux and in Scandinavia, just install mobile bankid on your phone or on android in a vm]
So, if you're on linux and you've been using the official bankid application and you happen to be a couple of continents removed from Scandinavia follow this post which basically does the following:
1. Install ubuntu 10.04 LTS in a VM
2. Install bankid 4.19.XXXX or earlier in the ubuntu VM
3. Copy your ~/.personal from your regular linux computer to the ubuntu VM
4. export your key using persadm export to a USB stick
5. Install windows 7 in a VM using the free, legal isos and the free, legal installation key. The copy will expire after 30 days (but will still continue working).
6. Install bankid in the Win 7 VM
7. Under File/Preferences in bankid add the directory on your USB stick with the exported bankid key
8. Log in to your BankID issuer (probably your bank) using bankid 'on file', and request a mobile bankid. You'll get an activation code
9. Install the bankid app on your phone (e.g. bankid from the play store on android)
10. Start the bankid app on your phone, and type in your personnummer and the activation code. Create a pin, which is your password
11. You're done.
I mean, sure, it's a very awkward way of going about it, but it works and is conceptually simple.
Android in a VM
And there's absolutely no reason you can't use mobile bankid it in a virtualbox VM if you want to -- I used an android 4.4 x86 iso and created a virtual machine, and then set up bankid which works fine.
19 November 2014
602. Surviving bankid + rant. Exporting bankid files from linux to windows
This post probably isn't interesting unless you live/have lived in Sweden (Scandinavia)/deal with Swedish(Scandinavian) banks and their unbelievably crappy electronic ID solution.
This post will not show you how to use BankID on Linux -- it will show you how to move bankid files from debian linux to windows 7. Nor is the method elegant as it involves using two VMs.
I should make a long rant about how the company, Finansiell ID teknik, behind BankID should be sued into oblivion and the Swedish politicians allowing it to be used to access government service without requiring open specifications be jailed indefinitely. But I'm tired.
[looks like I got into a rant after all]
rant begin:
Long story short: BankID has always been a real pain in the backside to use on linux, and now support has ended altogether. Support for Windows XP has been dropped as well, which will become relevant later.
No more linux announcement: https://support.bankid.com/syskrav
Either way, there's no linux-friendly solution out there, as in moving from v4 to v5 of bankid the format has changed completely. To my understanding, that means that fribid also will not work with providers requiring bankid versions newer than
There's a linux-friendly solution: mobile bankid, which runs on android. However, it seems to require a Swedish (or northern European) SIM card. Actually, I have no idea how it works since there's hardly any bloody information out there -- the bankid.com website is incredibly bad. The most detailed info I've found is this, which says that it's SIM card and service provider dependent: https://www.bankid.no/Dette-er-BankID/BankID-pa-mobil/. On the other hand, this page says it isn't: http://www.stockholm.se/mobiltbankid
[NOPE -- that's not how it works -- it's actually a lot better. See here instead: linktocome ]
In the end it doesn't matter since you'll have to walk into a Swedish bank in person to order a new bankid set-up code. Not something you can do if you're abroad.
There's one other solution available -- ID via the Swedish tax office. Unfortunately it is only available for people residing in Sweden. Anyone who has business with Sweden but lives abroad is SOL.
rant end.
Step 1. Windows 7 in a VM -- overview
So, I had to find a pragmatic solution, and quickly since my current ID key is expiring. I have an old Windows XP disk that I could install in a VM, but since XP isn't supported anymore, that wouldn't work anyway.
I then decided to see how much a copy of Windows 7 costs. I'm not keen on throwing money at MS, but I was still at the point of simply investigating solutions. According to http://www.cheapaussiesoftware.com/microsoft-windows/microsoft-windows-7 the cheapest copy is $124. Not keen.
I am surrounded by computers with Windows 7 Pro OA stickers (that are running debian), so I figured if I could only get my hands on a DVD I could use the license that has already been paid for. I also had a vague memory of the Windows 7 isos being freely and legally available from Microsoft via download.
Using google a bit I eventually found it: http://www.heidoc.net/joomla/technology-science/microsoft/14-windows-7-direct-download-links
which provides Windows 7 isos freely and legally via Digital River. Note that you'll still need to find your own activation key.
Activation key -- you can either:
* get that legally by paying for it. Not interested in paying that much money for something I'm not going to use much.
* use google for a few minutes, find tons of activation keys which appear to work during installation but really don't allow you to activate your copy (go to Computer, Properties and click on Activate Now). This is obviously not legal.
* Put in an installation key (D4F6K-QK3RD-TMVMJ-BBMRX-3MBMV... read more) which allows you to finish the installation but won't activate your copy. Actually, reading the linked post you might not have to enter anything.
Either way, after 30 days your copy will expire. You can still use it -- the only things that change is the background (goes black), you can't install updates anymore, and Aero turns off. See here. This is fine by me.
As OA licenses are limited and I really don't care about using Windows for anything but bankid, I went with the last option.
Installing Windows 7 Ultimate 64 bit in VirtualBox went without a hitch, and I awarded it 30 Gb HDD and 2 gb RAM (I have 4 Tb HDD and 8 Gb RAM on my system). So far so good..
From within Windows 7 I then downloaded bankid v 6.1 from here: https://install.bankid.com/ (ignore the whole linux/ubuntu link -- that's for v 4 which isn't supported by most banks anymore...)
Step 2. Ubuntu (yup) in a VM *yup).
At this point I figured I was doing pretty well. I connected a USB stick to my computer (i.e. debian) and ran persadm.
In desperation I then copied the ~/.personal structure containing the bankid keys from my debian box to my USB stick, fired up an old Ubuntu virtual machine with bankid installed (you may want to download bankid for ubuntu from here for THAT purpose: https://install.bankid.com/)
Running persadmin in the Ubuntu 10.04 LTS 32 bit VM worked perfectly, and I now had the exported .nge file in a folder on my USB stick.
Step 3. Importing the BankID key files
I then fired up the Windows 7 VM, started BankID and went to File/Preferences. Under locations I added the folder on the USB stick that held the exported BankID file, clicked Add, then Save, and everything was good.
Well, apart from the fact that I had to use two VMs, and am now stuck with a Windows 7 VM.
My Win 7 .vdi file is about 9 Gb, which as a tar.gz file is compressed to 3.8 Gb. I've backed it up in three different locations, so hopefully there won't be any issues with losing the bankid due to MS update idiocies.
This post will not show you how to use BankID on Linux -- it will show you how to move bankid files from debian linux to windows 7. Nor is the method elegant as it involves using two VMs.
I should make a long rant about how the company, Finansiell ID teknik, behind BankID should be sued into oblivion and the Swedish politicians allowing it to be used to access government service without requiring open specifications be jailed indefinitely. But I'm tired.
[looks like I got into a rant after all]
rant begin:
Long story short: BankID has always been a real pain in the backside to use on linux, and now support has ended altogether. Support for Windows XP has been dropped as well, which will become relevant later.
No more linux announcement: https://support.bankid.com/syskrav
Either way, there's no linux-friendly solution out there, as in moving from v4 to v5 of bankid the format has changed completely. To my understanding, that means that fribid also will not work with providers requiring bankid versions newer than
There's a linux-friendly solution: mobile bankid, which runs on android. However, it seems to require a Swedish (or northern European) SIM card. Actually, I have no idea how it works since there's hardly any bloody information out there -- the bankid.com website is incredibly bad. The most detailed info I've found is this, which says that it's SIM card and service provider dependent: https://www.bankid.no/Dette-er-BankID/BankID-pa-mobil/. On the other hand, this page says it isn't: http://www.stockholm.se/mobiltbankid
[NOPE -- that's not how it works -- it's actually a lot better. See here instead: linktocome ]
In the end it doesn't matter since you'll have to walk into a Swedish bank in person to order a new bankid set-up code. Not something you can do if you're abroad.
There's one other solution available -- ID via the Swedish tax office. Unfortunately it is only available for people residing in Sweden. Anyone who has business with Sweden but lives abroad is SOL.
rant end.
Step 1. Windows 7 in a VM -- overview
So, I had to find a pragmatic solution, and quickly since my current ID key is expiring. I have an old Windows XP disk that I could install in a VM, but since XP isn't supported anymore, that wouldn't work anyway.
I then decided to see how much a copy of Windows 7 costs. I'm not keen on throwing money at MS, but I was still at the point of simply investigating solutions. According to http://www.cheapaussiesoftware.com/microsoft-windows/microsoft-windows-7 the cheapest copy is $124. Not keen.
I am surrounded by computers with Windows 7 Pro OA stickers (that are running debian), so I figured if I could only get my hands on a DVD I could use the license that has already been paid for. I also had a vague memory of the Windows 7 isos being freely and legally available from Microsoft via download.
Using google a bit I eventually found it: http://www.heidoc.net/joomla/technology-science/microsoft/14-windows-7-direct-download-links
which provides Windows 7 isos freely and legally via Digital River. Note that you'll still need to find your own activation key.
Activation key -- you can either:
* get that legally by paying for it. Not interested in paying that much money for something I'm not going to use much.
* use google for a few minutes, find tons of activation keys which appear to work during installation but really don't allow you to activate your copy (go to Computer, Properties and click on Activate Now). This is obviously not legal.
* Put in an installation key (D4F6K-QK3RD-TMVMJ-BBMRX-3MBMV... read more) which allows you to finish the installation but won't activate your copy. Actually, reading the linked post you might not have to enter anything.
Either way, after 30 days your copy will expire. You can still use it -- the only things that change is the background (goes black), you can't install updates anymore, and Aero turns off. See here. This is fine by me.
As OA licenses are limited and I really don't care about using Windows for anything but bankid, I went with the last option.
Installing Windows 7 Ultimate 64 bit in VirtualBox went without a hitch, and I awarded it 30 Gb HDD and 2 gb RAM (I have 4 Tb HDD and 8 Gb RAM on my system). So far so good..
From within Windows 7 I then downloaded bankid v 6.1 from here: https://install.bankid.com/ (ignore the whole linux/ubuntu link -- that's for v 4 which isn't supported by most banks anymore...)
Step 2. Ubuntu (yup) in a VM *yup).
At this point I figured I was doing pretty well. I connected a USB stick to my computer (i.e. debian) and ran persadm.
No matter what I tried in terms of permissions and destination file systems made any difference.me@beryllium:/media/highio$ persadm export BankID Security Application 4.19.1 Available tokens: 0: (140110 yy.xx) Me - BankID on file 1: (130304 yy.xx) Me - BankID on file Choose token: 0 Enter pin: Enter removable media export directory (must exist): /media/fat32/bid Failed to export token. The reason might be that the export directory is not on a removable media, or that you don't have permission to write to it.
In desperation I then copied the ~/.personal structure containing the bankid keys from my debian box to my USB stick, fired up an old Ubuntu virtual machine with bankid installed (you may want to download bankid for ubuntu from here for THAT purpose: https://install.bankid.com/)
Running persadmin in the Ubuntu 10.04 LTS 32 bit VM worked perfectly, and I now had the exported .nge file in a folder on my USB stick.
In retrospect you might be able to do this in a chrooted ubuntu instance under debian. Who knows?me@me-desktop:~$ persadm export BankID Security Application 4.19.1 Available tokens: 0: (140110 yy.xx) Me - BankID on file 1: (130304 yy.xx) Me - BankID on file Choose token: 0 Enter pin: Enter removable media export directory (must exist): /media/fat32/bid Successfully exported token.
Step 3. Importing the BankID key files
I then fired up the Windows 7 VM, started BankID and went to File/Preferences. Under locations I added the folder on the USB stick that held the exported BankID file, clicked Add, then Save, and everything was good.
Well, apart from the fact that I had to use two VMs, and am now stuck with a Windows 7 VM.
My Win 7 .vdi file is about 9 Gb, which as a tar.gz file is compressed to 3.8 Gb. I've backed it up in three different locations, so hopefully there won't be any issues with losing the bankid due to MS update idiocies.
20 April 2013
393. Solved: XpressConnect on Ubuntu vs Debian. Extra focus on Swinburne university (AU/Melbourne)
Update 4. It's fixed now. There are two fixes:
1. XpressConnect wants ubuntu and checks for it via /etc/lsb-release, a file that doesn't exist on debian. Set it up to pretend that you're using ubuntu, and XpressConnect works.
2. Just configure the network connection manually. This is by far the easiest, as long as you know the settings.
Either way you'll need network-manager, lshw and wireless-tools in order to use XpressConnect. See towards the end of the post for more details.
Presumably this should work on any distro (e.g. arch, centos, rhel, sles), not just debian.
Update 3: Someone at Swinburne managed to connect to the Wifi without using XpressConnect by simply manually configuring the network: http://forums.debian.net/viewtopic.php?f=10&t=103387&p=494412#p494145
Original post:
Until a few days ago I'd never heard of XpressConnect before, but apparently it's a program that sets up your wireless connection automatically, and is used by a number of universities ( http://lmgtfy.com/?q=xpressconnect+site%3A*.edu)
In short, it works on Ubuntu (32 bit 10.04 lts) but not Debian (64 bit wheezy, 32 or 64 bit squeeze). Note that Ubuntu 10.04 has Gnome 2, as does Squeeze. Wheezy has Gnome 3.
Description
You don't need to be anywhere near a uni network to reproduce the issue -- anyone, with or without a wireless card, can test it.
Note: I generated the logs with a copy of XpressConnect downloaded from Swinburne, not the generic version -- hence why it says swinwifi etc.
On debaian, first make sure to install lshw and wireless-tools, and to add /sbin to path in the terminal you're using.
The 'installation' is simple
XpressConnect-DoubleClickToRun is a simple shell script that determines whether system is 32 or 64 bit, downloads a tar file to /tmp, untars it and launches a pre-compiled binary, XpressConnect-x64.
You can also launch XpressConnect-x64 directly by going to /tmp and running it.
Anyway, XpressConnect-x64 then makes sure that you have network-manager running, and tries to determine what your network card is. Based on this, /tmp/netdata.txt is generated. Once this is done, XpressConnect allows you to enter your credentials and sets up your network connection.
Well, that's what happens if you use ubuntu. On Debian (or Arch) you get an error message about 'network configuration' missing.
Delving a bit deeper into it, the binary fails to generate the netdata.txt on debian (and arch), and it seems to have to do with network-manager (or dbus, possibly).
The logs:
XpressConnect-x64 generates a log file (/tmp/xpressconnect.log).
On ubuntu it looks like this:
XpressConnect then generates a file called /tmp/netdata.txt:
I also did a strace on both ubuntu and debian, but strace generates a lot of information. A major difference I observed was the presence of CLOCK_MONOTONIC in the ubuntu log, but this is to synchronize for the CA certificate (or something -- I don't know what I'm talking about). Anyway, it probably just indicates what we already know -- the binary doesn't detect the card on debian but it does on ubuntu.
ldd XpressConnect-x64 gives the following on ubuntu (32 bit):
and this on debian (64 bit):
Evidence of evil and a fix
The evidence
Here's a snippet of the strace output from ubuntu
The evil is that it's highly misleading -- it's not checking for valid networks, it's checking whether you've got ubuntu. And it didn't fail to find a valid network -- it failed to find /etc/lsb-release.
Here's /etc/lsb-release from ubuntu
The fix
Debian doesn't have an lsb-release file, so I simply copied the /etc/lsb-release file from ubuntu to debian. Guess what? XpressConnect now works on debian!
And no, having an empty /etc/lsb-release file doesn't work. Also, editing it (replacing 'ubuntu' with 'debian') also causes XpressConnect to stop working.
Another 'fix' -- manual Configuration:
XpressConnect does not do more than set up your wireless connection i.e. sets the required settings and installs the CA certificate for your particular network. In other words, you can just manually configure your network and you'll probably be fine.
Unfortunately, I haven't found a URL for the CA cert for Swinburne, which is the example that I'm working with (based on a forum post)
I managed to extract most of that by running it in an ubuntu VM. Note that I am nowhere near Swinburne, not staff/a student there and actually can't test this to completion.
The settings are:
You'll need swinwifi.der. Because it is in binary I can't easily post it. However, you can generate it from the .pem file which is in ascii armour:
Put it in the folder ~/.certificates.
The md5sum should be
Finally, do NOT as a rule download certificates from third parties! How can you trust that my intentions are good? At the moment you're choice is between trusting me and Shuttleworth though...
Why bother?
Because an increasing number of university networks have adopted this -- frankly broken -- piece of software, I think it's of some interest to users of other distros to investigate whether there are any simple solutions. This in particular since not all uni networks have instructions for manual configuration anymore. Besides, a challenge is a challenge.
1. XpressConnect wants ubuntu and checks for it via /etc/lsb-release, a file that doesn't exist on debian. Set it up to pretend that you're using ubuntu, and XpressConnect works.
2. Just configure the network connection manually. This is by far the easiest, as long as you know the settings.
Either way you'll need network-manager, lshw and wireless-tools in order to use XpressConnect. See towards the end of the post for more details.
Presumably this should work on any distro (e.g. arch, centos, rhel, sles), not just debian.
Update 3: Someone at Swinburne managed to connect to the Wifi without using XpressConnect by simply manually configuring the network: http://forums.debian.net/viewtopic.php?f=10&t=103387&p=494412#p494145
Original post:
Until a few days ago I'd never heard of XpressConnect before, but apparently it's a program that sets up your wireless connection automatically, and is used by a number of universities ( http://lmgtfy.com/?q=xpressconnect+site%3A*.edu)
In short, it works on Ubuntu (32 bit 10.04 lts) but not Debian (64 bit wheezy, 32 or 64 bit squeeze). Note that Ubuntu 10.04 has Gnome 2, as does Squeeze. Wheezy has Gnome 3.
Description
You don't need to be anywhere near a uni network to reproduce the issue -- anyone, with or without a wireless card, can test it.
Note: I generated the logs with a copy of XpressConnect downloaded from Swinburne, not the generic version -- hence why it says swinwifi etc.
On debaian, first make sure to install lshw and wireless-tools, and to add /sbin to path in the terminal you're using.
sudo apt-get install lshw wireless-tools export PATH=$PATH:/sbin
The 'installation' is simple
cd ~/Downloads wget http://hosted.cloudpath.net/Xavier/Production/tools/XpressConnect-Linux.tar tar xvf XpressConnect-Linux.tar ./XpressConnect-DoubleClickToRunWill download x64 version... --2013-04-18 21:11:16-- http://hosted.cloudpath.net/Xavier/Production//tools/XpressConnect-x64.tar.bz2 Resolving hosted.cloudpath.net (hosted.cloudpath.net)... 72.18.151.75 Connecting to hosted.cloudpath.net (hosted.cloudpath.net)|72.18.151.75|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 6266475 (6.0M) [application/x-bzip2] Saving to: `/tmp/XpressConnect-x64.tar.bz2' 100%[==============================================================================================================================>] 6,266,475 780K/s in 10s 2013-04-18 21:11:27 (600 KB/s) - `/tmp/XpressConnect-x64.tar.bz2' saved [6266475/6266475] files files/logo.jpg network_config.xml resources.properties XpressConnect-x64
XpressConnect-DoubleClickToRun is a simple shell script that determines whether system is 32 or 64 bit, downloads a tar file to /tmp, untars it and launches a pre-compiled binary, XpressConnect-x64.
You can also launch XpressConnect-x64 directly by going to /tmp and running it.
Anyway, XpressConnect-x64 then makes sure that you have network-manager running, and tries to determine what your network card is. Based on this, /tmp/netdata.txt is generated. Once this is done, XpressConnect allows you to enter your credentials and sets up your network connection.
Well, that's what happens if you use ubuntu. On Debian (or Arch) you get an error message about 'network configuration' missing.
Delving a bit deeper into it, the binary fails to generate the netdata.txt on debian (and arch), and it seems to have to do with network-manager (or dbus, possibly).
The logs:
XpressConnect-x64 generates a log file (/tmp/xpressconnect.log).
On ubuntu it looks like this:
And on Debian:Looking for id : 266 Looking for id : 259 Setting start page. Moving to widget 0. Loading configuration page. Starting configuration parse thread. Acquire processing lock. Building NIC data. Starting configuration parser. Validating checksum. Checking license. License validation checked out properly. Today is : 04/19/2013 License expires : The date validation checked out properly. Checking for valid networks for this platform. Only one network configuration found. Auto-selecting it. Auto-selected network 'swinwifi'. Interface : /org/freedesktop/NetworkManager/Devices/0 is in state 8 Interface : /org/freedesktop/NetworkManager/Devices/1 is in state 3 There were no interfaces found that could be autodetected for use. Checking that there are interfaces of the correct type available. Found 1 possible interface(s). Selected interface with object path of /org/freedesktop/NetworkManager/Devices/1 Int type : 2 Set profile index : 11 Going to page 3 Config parser terminated. Set 'static' window labels. ---- Gathering Data ---- Parsing OS information... Checking OS bit depth. OS bits info : i686 OS is 32 bits. Checking hardware information. Looking for extra NIC data. Found driver name for 'wlan0' from DBus. - Driver : rtl8187 Couldn't get capabilities from DBus, trying the hard way. Attempting to release lock. Release processing lock. Set welcome banners. Using DBus UUID for client ID. Using DBus UUID for client ID. Client ID : f3af3f95c2229c756e98556050ed0cc4 Session ID : 6980532961366355275 Upload value (1) : 0 Moving to widget 3.
The platform bit bothers me -- is it checking for ubuntu vs other distros?Looking for id : 266 Looking for id : 259 Setting start page. Moving to widget 0. Loading configuration page. Starting configuration parse thread. Acquire processing lock. Building NIC data. Starting configuration parser. Validating checksum. Checking license. License validation checked out properly. Today is : 04/22/2013 License expires : The date validation checked out properly. Checking for valid networks for this platform. No valid networks found in the configuration. Config parser terminated. Failed to parse configuration. Set 'static' window labels. ---- Gathering Data ---- Parsing OS information... Checking OS bit depth. OS bits info : x86_64 OS is 64 bits. Checking hardware information. Looking for extra NIC data. Found driver name for 'wlan0' from DBus. - Driver : rtl8187 Couldn't get capabilities from DBus, trying the hard way. Attempting to release lock. Release processing lock. Set welcome banners. Using DBus UUID for client ID. Using DBus UUID for client ID. Client ID : fe88ef3cf6716df910e7fa570000000b Session ID : 13542037211366584788 Moving to widget 9. User clicked 'Retry' on the bad config widget. Moving to widget 1. Selected network 'swinwifi'. Going to page 9 Adding page 1 Moving to widget 9.
XpressConnect then generates a file called /tmp/netdata.txt:
*-network description: Ethernet interface product: 82540EM Gigabit Ethernet Controller vendor: Intel Corporation physical id: 3 bus info: pci@0000:00:03.0 logical name: eth0 version: 02 serial: 08:00:27:53:23:4c width: 32 bits clock: 66MHz capabilities: bus_master cap_list ethernet physical configuration: broadcast=yes driver=e1000 driverversion=7.3.21-k5-NAPI firmware=N/A ip=10.0.2.15 latency=64 mingnt=255 multicast=yes resources: irq:10 memory:f0000000-f001ffff ioport:d010(size=8) *-network description: Wireless interface physical id: 1 logical name: wlan0 serial: 00:11:a3:08:12:1d capabilities: ethernet physical wireless configuration: broadcast=yes multicast=yes wireless=IEEE 802.11bg
I also did a strace on both ubuntu and debian, but strace generates a lot of information. A major difference I observed was the presence of CLOCK_MONOTONIC in the ubuntu log, but this is to synchronize for the CA certificate (or something -- I don't know what I'm talking about). Anyway, it probably just indicates what we already know -- the binary doesn't detect the card on debian but it does on ubuntu.
ldd XpressConnect-x64 gives the following on ubuntu (32 bit):
linux-gate.so.1 => (0x006a3000) libdbus-1.so.3 => /lib/libdbus-1.so.3 (0x007b3000) libXrender.so.1 => /usr/lib/libXrender.so.1 (0x00c8b000) libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x008b8000) libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00da4000) libXext.so.6 => /usr/lib/libXext.so.6 (0x00225000) libX11.so.6 => /usr/lib/libX11.so.6 (0x00480000) libz.so.1 => /lib/libz.so.1 (0x00110000) libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0x00125000) librt.so.1 => /lib/tls/i686/cmov/librt.so.1 (0x00665000) libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0x006f2000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00ac3000) libm.so.6 => /lib/tls/i686/cmov/libm.so.6 (0x002d8000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00129000) libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x002fe000) libexpat.so.1 => /lib/libexpat.so.1 (0x00148000) libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00a12000) /lib/ld-linux.so.2 (0x00637000) libXau.so.6 => /usr/lib/libXau.so.6 (0x00c7a000) libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x0096f000)
and this on debian (64 bit):
linux-vdso.so.1 => (0x00007fff50dfb000) libdbus-1.so.3 => /lib/libdbus-1.so.3 (0x00007f2bbb807000) libXrender.so.1 => /usr/lib/libXrender.so.1 (0x00007f2bbb5fd000) libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x00007f2bbb3c7000) libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007f2bbb13f000) libXext.so.6 => /usr/lib/libXext.so.6 (0x00007f2bbaf2d000) libX11.so.6 => /usr/lib/libX11.so.6 (0x00007f2bbabf1000) libz.so.1 => /usr/lib/libz.so.1 (0x00007f2bba9da000) libdl.so.2 => /lib/libdl.so.2 (0x00007f2bba7d6000) librt.so.1 => /lib/librt.so.1 (0x00007f2bba5cd000) libpthread.so.0 => /lib/libpthread.so.0 (0x00007f2bba3b1000) libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f2bba09d000) libm.so.6 => /lib/libm.so.6 (0x00007f2bb9e1a000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f2bb9c04000) libc.so.6 => /lib/libc.so.6 (0x00007f2bb98a2000) libexpat.so.1 => /usr/lib/libexpat.so.1 (0x00007f2bb9679000) libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00007f2bb945d000) /lib64/ld-linux-x86-64.so.2 (0x00007f2bbba58000) libXau.so.6 => /usr/lib/libXau.so.6 (0x00007f2bb9259000) libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x00007f2bb9054000)
Evidence of evil and a fix
The evidence
Here's a snippet of the strace output from ubuntu
and here's Debian:2200 write(8, "The date validation checked out "..., 42) = 42 2200 write(8, "Checking for valid networks for "..., 47) = 47 2200 open("/etc/lsb-release", O_RDONLY|O_LARGEFILE) = 15 2200 read(15, "DISTRIB_ID=Ubuntu\nDISTRIB_RELEAS"..., 8191) = 104 2200 read(15, "", 8191) = 0 2200 close(15)
3079 write(8, "The date validation checked out "..., 42) = 42 3079 write(8, "Checking for valid networks for "..., 47) = 47 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 open("/etc/lsb-release", O_RDONLY) = -1 ENOENT (No such file or directory) 3079 write(8, "No valid networks found in the c"..., 46) = 46 3079 close(11)
The evil is that it's highly misleading -- it's not checking for valid networks, it's checking whether you've got ubuntu. And it didn't fail to find a valid network -- it failed to find /etc/lsb-release.
Here's /etc/lsb-release from ubuntu
DISTRIB_ID=Ubuntu DISTRIB_RELEASE=10.04 DISTRIB_CODENAME=lucid DISTRIB_DESCRIPTION="Ubuntu 10.04.4 LTS"
The fix
Debian doesn't have an lsb-release file, so I simply copied the /etc/lsb-release file from ubuntu to debian. Guess what? XpressConnect now works on debian!
And no, having an empty /etc/lsb-release file doesn't work. Also, editing it (replacing 'ubuntu' with 'debian') also causes XpressConnect to stop working.
Another 'fix' -- manual Configuration:
XpressConnect does not do more than set up your wireless connection i.e. sets the required settings and installs the CA certificate for your particular network. In other words, you can just manually configure your network and you'll probably be fine.
Unfortunately, I haven't found a URL for the CA cert for Swinburne, which is the example that I'm working with (based on a forum post)
I managed to extract most of that by running it in an ubuntu VM. Note that I am nowhere near Swinburne, not staff/a student there and actually can't test this to completion.
 |
| ubuntu -- apparently it's no longer brown... |
Security: WPA & WPA2 Enterprise Authentication: Tunneled TLS Anonymous identity: anonymous CA certificate: swinwifi.der Inner authentication: PAP Username: ***@swinburne.edu Pasword: ******
You'll need swinwifi.der. Because it is in binary I can't easily post it. However, you can generate it from the .pem file which is in ascii armour:
-----BEGIN CERTIFICATE----- MIID9zCCA2CgAwIBAgIBADANBgkqhkiG9w0BAQQFADCBtDELMAkGA1UEBhMCQVUx ETAPBgNVBAgTCFZpY3RvcmlhMRIwEAYDVQQHEwlNZWxib3VybmUxKzApBgNVBAoT IlN3aW5idXJuZSBVbml2ZXJzaXR5IG9mIFRlY2hub2xvZ3kxDDAKBgNVBAsTA0lU UzEeMBwGA1UEAxMVcmFkaXVzLmNjLnN3aW4uZWR1LmF1MSMwIQYJKoZIhvcNAQkB FhRuZXR3b3Jrc0Bzd2luLmVkdS5hdTAeFw0wNzAxMTcwMzU3MDVaFw0xNzAxMTQw MzU3MDVaMIG0MQswCQYDVQQGEwJBVTERMA8GA1UECBMIVmljdG9yaWExEjAQBgNV BAcTCU1lbGJvdXJuZTErMCkGA1UEChMiU3dpbmJ1cm5lIFVuaXZlcnNpdHkgb2Yg VGVjaG5vbG9neTEMMAoGA1UECxMDSVRTMR4wHAYDVQQDExVyYWRpdXMuY2Muc3dp bi5lZHUuYXUxIzAhBgkqhkiG9w0BCQEWFG5ldHdvcmtzQHN3aW4uZWR1LmF1MIGf MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDGwvfVz2DnKxFMYTG1k0QklSHC7vk5 kjIiDkdU3sRTdQ07cQUOI/8wFN4zZXowEz0DwlO1o/YWaZqw27EP85cp9XBndwRK ZZpIv57zrxo8nxJV/mKBpOM7MHpkclju20XoEtrQ7FwTHPbWaaKmSuaMVitWcFVg C3CIYkvQn9pozQIDAQABo4IBFTCCAREwHQYDVR0OBBYEFEo9WNtNMY7jUvvASrE9 Z/OdaeU9MIHhBgNVHSMEgdkwgdaAFEo9WNtNMY7jUvvASrE9Z/OdaeU9oYG6pIG3 MIG0MQswCQYDVQQGEwJBVTERMA8GA1UECBMIVmljdG9yaWExEjAQBgNVBAcTCU1l bGJvdXJuZTErMCkGA1UEChMiU3dpbmJ1cm5lIFVuaXZlcnNpdHkgb2YgVGVjaG5v bG9neTEMMAoGA1UECxMDSVRTMR4wHAYDVQQDExVyYWRpdXMuY2Muc3dpbi5lZHUu YXUxIzAhBgkqhkiG9w0BCQEWFG5ldHdvcmtzQHN3aW4uZWR1LmF1ggEAMAwGA1Ud EwQFMAMBAf8wDQYJKoZIhvcNAQEEBQADgYEASFZ/5oRvTpgs0yQmdXxczUvJhUam 5KTP8MAb8owh1v65VkBFXLdJ27THEDt0SX3ZzwlYIvKcVMoiJcXjE6uLgZPI4AZv 7ogHIU5TOvQsYXWNeCqQMqOTnZtVQPvZmOcHcM1aqiBCAVX3YhIU1S04SccYdKBA PaggsSjsagqoZvA= -----END CERTIFICATE-----
openssl x509 -in swinwifi.pem -out swinwifi.der -outform DER
Put it in the folder ~/.certificates.
The md5sum should be
I've also tried that java version of xpress connect.11843b7d38bc0b024e8356f11d4d7c42 swinwifi.der
Finally, do NOT as a rule download certificates from third parties! How can you trust that my intentions are good? At the moment you're choice is between trusting me and Shuttleworth though...
Why bother?
Because an increasing number of university networks have adopted this -- frankly broken -- piece of software, I think it's of some interest to users of other distros to investigate whether there are any simple solutions. This in particular since not all uni networks have instructions for manual configuration anymore. Besides, a challenge is a challenge.
13 September 2012
234. CPMD with netlib lapack, blas and your own fftw on debian testing
This is a minor update to my previous post on CPMD. Back in the days I had issue linking to my Openblas libs (got a binary which would not run properly) but I've since had success with the netlib lapack and blas libs.
1. Compile the netlib lapack and blas libraries according to this post: http://verahill.blogspot.com.au/2012/09/compiling-netlibs-lapack-and-blas-on.html
2. Compile the fftw libraries according to this post (ignore the sections on Openblas and Gromacs):
http://verahill.blogspot.com.au/2012/05/gromacs-with-external-fftw3-and-blas-on.html
3. Compile CPMD. We'll be following this post in large parts.
Register with cpmd.org. Once you're approved download the cpmd source to ~/tmp.
sudo apt-get install libopenmpi-dev openmpi-bin
Next edit ~/tmp/CPMD/wfnio.F and change the following lines:
Now, in ~/tmp/CPMD, run
And follow everything below 'Done! Almost.' in this post: http://verahill.blogspot.com.au/2012/07/not-solved-compiling-cpmd-on-debian.html
1. Compile the netlib lapack and blas libraries according to this post: http://verahill.blogspot.com.au/2012/09/compiling-netlibs-lapack-and-blas-on.html
2. Compile the fftw libraries according to this post (ignore the sections on Openblas and Gromacs):
http://verahill.blogspot.com.au/2012/05/gromacs-with-external-fftw3-and-blas-on.html
3. Compile CPMD. We'll be following this post in large parts.
Register with cpmd.org. Once you're approved download the cpmd source to ~/tmp.
sudo apt-get install libopenmpi-dev openmpi-bin
cd ~/tmp tar -xvf cpmd-v3_15_3.tar.gz cd CPMD/CONFIGURECreate the file LINUX-x86_64-DEBIAN:
IRAT=2
CFLAGS='-c -O2 -Wall'
CPP='/lib/cpp -P -C -traditional'
CPPFLAGS='-D__Linux -D__PGI -D__GNU -DFFT_FFTW3 -DPARALLEL -DPOINTER8'
FFLAGS='-c -O2 -fcray-pointer -fno-whole-file -fsecond-underscore'
LFLAGS='-l:/opt/fftw/fftw-3.3.2/double/lib/libfftw3.a -l:/opt/fftw/fftw-3.3.2/double/lib/libfftw3_mpi.a -l:/opt/fftw/fftw-3.3.2/double/lib/libfftw3_threads.a -I/usr/include -l:/opt/netlib/blas/lib/libnetblas.so -l:/opt/netlib/lapack/lib/liblapack.so -lpthread -lmpi'
FFLAGS_GROMOS=' $(FFLAGS)'
FC='mpif77 -fbounds-check'
CC='mpicc'
LD='mpif77 -fbounds-check'
Next edit ~/tmp/CPMD/wfnio.F and change the following lines:
15 CHARACTER(len=*) TAG 63 IF(TAG(1:2).EQ.'NI') THEN 201 IF(TAG(1:2).NE.'NI') THEN 271 IF(TAG(1:2).EQ.'NI') THEN
Now, in ~/tmp/CPMD, run
./mkconfig.sh LINUX-x86_64-DEBIAN > Makefile make sudo mkdir /opt/cpmd sudo chown $USER /opt/cpmd cp cpmd.x /opt/cpmd
And follow everything below 'Done! Almost.' in this post: http://verahill.blogspot.com.au/2012/07/not-solved-compiling-cpmd-on-debian.html
19 July 2012
210. Compiling CPMD on Debian Wheezy
[To those coming here from cpmd.org -- I wasn't meant to post in the bug tracking section, but unfortunately I also can't delete posts]
Here's how to compile cpmd on Wheezy. CPMD is the standard implementation of ab initio Carr-Parrinello method for molecular dynamic and would thus be useful to have at hand.
There's no reason to think that Ubuntu and Linux mint won't work just as well (i.e. you may want to look here if you're trying to compile for Ubuntu or Mint)
Errors I encountered on the road are found at the end of the post together with solutions.
COMPILING THE BINARY
First you need to request a username and password from http://cpmd.org/. That'll take a few days.
You may have to install libopenmpi-dev libfftw3-dev libblas-dev first
Once your application is approved and you've downloaded your file (in my case cpmd-v3_15_3.tar.gz) copy it to ~/tmp, then
tar -xvf cpmd-v3_15_3.tar.gz
cd CPMD/CONFIGURE
create CONFIGURE/LINUX-x86_64-DEBIAN
You should now have a working binary. This was a real PITA to sort out.
sudo mkdir /opt/cpmd
sudo chown $USER /opt/cpmd
cp cpmd.x /opt/cpmd
Done! Almost...
Time to get some pseudopotential files from http://cpmd.org/download
Download the following files:
cd /opt/cpmd
ls *.gz|xargs -I {} tar xvf {}
echo "export PP_LIBRARY_PATH=/opt/cpmd/PP_LIBRARY" >>~/.bashrc
echo "export PATH=$PATH:/opt/cpmd" >>~/.bashrc.
source ~/.bashrc
Note:
For some reason cpmd refuses to read my pseudopotential files unless I remove the .psp ending. Doesn't matter how eagerly I'm trying to point cpmd in the right direction. You can quickly mv all the psp files like this:
cp /opt/cpmd/pseudo_extlib/*.psp /opt/cpmd/PP_LIBRARY/
cd /opt/cpmd/PP_LIBRARY/
ls *.psp|sed 's/\./\t/g'|gawk '{print $1}'|xargs -I {} mv {}.psp {}
mv Cu_MT_PBE_SEMI_2.08.psp Cu_MT_PBE_SEMI_2.08
mv Ga.1_GO_LDA.psp Ga.1_GO_LDA
Not that this will guarantee that it'll work. I often find myself putting the full relative path in the .in file:
Test your binary
cd /opt/cpmd/CPMD-test/vdw
mpirun -n 4 cpmd.x ch4-ch4.inp
Almost completely and utterly done.
There are a few more things to do for a full install:
1. cpmd2cube
Download cpmd2xyz-scripts.xyz and cpmd2cube.tar.gz and put them in ~/tmp
cd ~/tmp
tar xvf cpmd2cube.tar.gz
cd cpmd2cube/
./Configure Linux-PC-GFORTRAN-FFTW >Makefile
Edit the Makefile
11 #--------------- Configuration
cp *.x /opt/cpmd/
2. cpdm2xyz-scripts
tar xvf cpmd2xyz-scripts.tar.gz
mv *xyz.pl /opt/cpmd/
ERRORS I encountered on the way to success:
Why I patched wfnio.F
I tried compiling with my own openblas libraries:
Used the following config file base:
IRAT=2
CFLAGS='-c -O1 -Wall'
CPP='/lib/cpp -P -C -traditional'
CPPFLAGS='-D__Linux -D__PGI -D__GNU -DFFT_FFTW -DPOINTER8 -DPARALLEL'
FFLAGS='-c -O2 -fcray-pointer -fno-whole-file -fsecond-underscore'
LFLAGS='-L/opt/fftw/fftw-2.1.5/double/lib -l:libfftw.a -I/opt/fftw/fftw-2.1.5/double/include -I/usr/include -L/opt/openblas/lib -lopenblas'
FFLAGS_GROMOS=' $(FFLAGS)'
FC='mpif90'
CC='mpicc'
LD='mpif90'
I compiled openblas according to http://verahill.blogspot.com.au/2012/05/nwchem-with-openblas.html
If you're having problems with the output below it is because fftw 2 and fftw 3 are not compatible and CPDM uses the fftw 2 iface by default.
mltfft.o: In function `mltfft_fftw_': mltfft.f:(.text+0x958): undefined reference to `fftw_f77_create_plan_' mltfft.f:(.text+0x9af): undefined reference to `fftw_f77_' mltfft.f:(.text+0x9c2): undefined reference to `fftw_f77_destroy_plan_' mltfft.f:(.text+0xaee): undefined reference to `fftw_f77_create_plan_' mltfft.f:(.text+0xb4d): undefined reference to `fftw_f77_' mltfft.f:(.text+0xd03): undefined reference to `fftw_f77_' mltfft.f:(.text+0xdc4): undefined reference to `fftw_f77_' mltfft.f:(.text+0xe6e): undefined reference to `fftw_f77_create_plan_' mltfft.f:(.text+0xe8f): undefined reference to `fftw_f77_create_plan_' collect2: ld returned 1 exit status make: *** [cpmd.x] Error 1
You can either
1. Install fftw2 (sudo apt-get install fftw-dev, fftw2), and put -l:libfftw.so.2 -l:librfftw.so.2 in your LFLAGS.
or
2. Set -DFFT_FFTW3 instead of -DFFT_FFTW.
Here's how to compile cpmd on Wheezy. CPMD is the standard implementation of ab initio Carr-Parrinello method for molecular dynamic and would thus be useful to have at hand.
It took a fair while to hammer out the method, so this post might not read very well. Right now I'm having crashes after the binary has been running for a while -- this point towards system resources more than anything else.
WARNING:
This involves patching a file -- you never know IF that causes problems with the output. To the best of my knowledge, it should be safe.
There's no reason to think that Ubuntu and Linux mint won't work just as well (i.e. you may want to look here if you're trying to compile for Ubuntu or Mint)
Errors I encountered on the road are found at the end of the post together with solutions.
COMPILING THE BINARY
First you need to request a username and password from http://cpmd.org/. That'll take a few days.
You may have to install libopenmpi-dev libfftw3-dev libblas-dev first
Once your application is approved and you've downloaded your file (in my case cpmd-v3_15_3.tar.gz) copy it to ~/tmp, then
tar -xvf cpmd-v3_15_3.tar.gz
cd CPMD/CONFIGURE
create CONFIGURE/LINUX-x86_64-DEBIAN
IRAT=2
CFLAGS='-c -O2 -Wall'
CPP='/lib/cpp -P -C -traditional'
CPPFLAGS='-D__Linux -D__PGI -D__GNU -DFFT_FFTW3 -DPARALLEL -DPOINTER8'
FFLAGS='-c -O2 -fcray-pointer -fno-whole-file -fsecond-underscore'
LFLAGS='-lfftw3 -lfftw3f -I/usr/include -lblas -llapack -lpthread -lmpi'
FFLAGS_GROMOS=' $(FFLAGS)'
FC='mpif77 -fbounds-check'
CC='mpicc'
LD='mpif77 -fbounds-check'
cd ../
Edit the following lines in wfnio.F and change them to:
./mkconfig.sh LINUX-x86_64-DEBIAN > MakefileEdit the following lines in wfnio.F and change them to:
15 CHARACTER(len=*) TAG
63 IF(TAG(1:2).EQ.'NI') THEN
201 IF(TAG(1:2).NE.'NI') THEN
271 IF(TAG(1:2).EQ.'NI') THEN
(See below for the reasons why)
make -j5
You should now have a working binary. This was a real PITA to sort out.
sudo mkdir /opt/cpmd
sudo chown $USER /opt/cpmd
cp cpmd.x /opt/cpmd
Done! Almost...
Time to get some pseudopotential files from http://cpmd.org/download
Download the following files:
http://cpmd.org/downloadable-files/authentication/contributions/pseudo_std.tar.gzAnd some test files:
http://cpmd.org/downloadable-files/authentication/contributions/pseudo_ext.tar.gz
http://cpmd.org/downloadable-files/authentication/contributions/pseudo_vdb.tar.gz
http://cpmd.org/downloadable-files/authentication/contributions/cpmd-test.tar.gzPut them in /opt/cpmd
cd /opt/cpmd
ls *.gz|xargs -I {} tar xvf {}
echo "export PP_LIBRARY_PATH=/opt/cpmd/PP_LIBRARY" >>~/.bashrc
echo "export PATH=$PATH:/opt/cpmd" >>~/.bashrc.
source ~/.bashrc
Note:
For some reason cpmd refuses to read my pseudopotential files unless I remove the .psp ending. Doesn't matter how eagerly I'm trying to point cpmd in the right direction. You can quickly mv all the psp files like this:
cp /opt/cpmd/pseudo_extlib/*.psp /opt/cpmd/PP_LIBRARY/
cd /opt/cpmd/PP_LIBRARY/
ls *.psp|sed 's/\./\t/g'|gawk '{print $1}'|xargs -I {} mv {}.psp {}
mv Cu_MT_PBE_SEMI_2.08.psp Cu_MT_PBE_SEMI_2.08
mv Ga.1_GO_LDA.psp Ga.1_GO_LDA
Not that this will guarantee that it'll work. I often find myself putting the full relative path in the .in file:
&ATOMS
*../../PP_LIBRARY/O_MT_PBE KLEINMAN-BYLANDER
LMAX=P
Anyway
Test your binary
cd /opt/cpmd/CPMD-test/vdw
mpirun -n 4 cpmd.x ch4-ch4.inp
****************************************************************
* *
* TIMING *
* *
****************************************************************
SUBROUTINE CALLS CPU TIME ELAPSED TIME
HIP 12 19.64 21.55
FFT-G/S 734 11.94 12.02
GCENER 12 4.82 4.86
FWFFT 72 4.80 4.85
INVFFTN 100 3.93 3.93
INVFFT 61 3.82 3.86
HOCKNEY 1 3.20 3.31
FFTCOM 133 2.34 2.33
VPSI 21 2.22 2.20
FWFFTN 56 2.14 2.15
XCENER 12 1.70 1.71
RHOOFR 11 1.59 1.61
GRADEN 12 1.58 1.58
PHASE 133 1.46 1.43
LOADPA 1 1.30 1.31
ODIIS 11 0.96 0.99
N-FFTCOM 156 0.80 0.82
VOFRHOH 12 0.61 0.62
VOFRHOB 12 0.57 0.57
EICALC 12 0.43 0.43
NUMPW 1 0.31 0.31
ATRHO 1 0.26 0.29
----------------------------------------------------------------
TOTAL TIME 70.43 72.74
****************************************************************
CPU TIME : 0 HOURS 1 MINUTES 11.75 SECONDS
ELAPSED TIME : 0 HOURS 1 MINUTES 14.21 SECONDS
*** CPMD| SIZE OF THE PROGRAM IS 141512/ 484936 kBYTES ***
PROGRAM CPMD ENDED AT: Thu Jul 19 12:50:17 2012
================================================================
= COMMUNICATION TASK AVERAGE MESSAGE LENGTH NUMBER OF CALLS =
= SEND/RECEIVE 370967. BYTES 51. =
= BROADCAST 10413. BYTES 297. =
= GLOBAL SUMMATION 1153. BYTES 190. =
= GLOBAL MULTIPLICATION 0. BYTES 1. =
= ALL TO ALL COMM 8815282. BYTES 313. =
= PERFORMANCE TOTAL TIME =
= SEND/RECEIVE 3014.256 MB/S 0.006 SEC =
= BROADCAST 271.033 MB/S 0.011 SEC =
= GLOBAL SUMMATION 1.996 MB/S 0.220 SEC =
= GLOBAL MULTIPLICATION 0.000 MB/S 0.001 SEC =
= ALL TO ALL COMM 643.149 MB/S 4.290 SEC =
= SYNCHRONISATION 0.067 SEC =
================================================================
Almost completely and utterly done.
There are a few more things to do for a full install:
1. cpmd2cube
Download cpmd2xyz-scripts.xyz and cpmd2cube.tar.gz and put them in ~/tmp
cd ~/tmp
tar xvf cpmd2cube.tar.gz
cd cpmd2cube/
./Configure Linux-PC-GFORTRAN-FFTW >Makefile
Edit the Makefile
11 #--------------- Configuration
12 FC = mpif77
13 FFLAGS = -ffree-form -D__GFORTRAN -DFFT_FFTW3 -std=gnu -O2 -fno-whole-file
14 LFLAGS =
15 LIBS = -lfftw3
Then compile:
make
cp *.x /opt/cpmd/
2. cpdm2xyz-scripts
tar xvf cpmd2xyz-scripts.tar.gz
mv *xyz.pl /opt/cpmd/
ERRORS I encountered on the way to success:
Why I patched wfnio.F
(K+E1+L+N+X) TOTAL ENERGY = -15.69316800 A.U.
(K) KINETIC ENERGY = 11.26654720 A.U.
(E1=A-S+R) ELECTROSTATIC ENERGY = -14.51438081 A.U.
(S) ESELF = 15.95769122 A.U.
(R) ESR = 0.61466148 A.U.
(L) LOCAL PSEUDOPOTENTIAL ENERGY = -8.04105016 A.U.
(N) N-L PSEUDOPOTENTIAL ENERGY = 1.58864226 A.U.
(X) EXCHANGE-CORRELATION ENERGY = -5.99292649 A.U.
GRADIENT CORRECTION ENERGY = -0.34038005 A.U.
NFI GEMAX CNORM ETOT DETOT TCPU
1 2.271E-02 2.070E-03 -15.693168 0.000E+00 12.31
2 2.968E-03 6.595E-04 -16.042408 -3.492E-01 12.23
3 2.302E-03 2.795E-04 -16.073138 -3.073E-02 12.41
4 1.202E-03 1.028E-04 -16.079251 -6.113E-03 13.11
5 5.911E-04 3.645E-05 -16.080017 -7.655E-04 12.86
6 2.152E-04 1.926E-05 -16.080119 -1.020E-04 12.27
7 7.666E-05 8.749E-06 -16.080151 -3.277E-05 12.24
8 4.416E-05 4.054E-06 -16.080159 -8.141E-06 12.54
9 2.411E-05 1.531E-06 -16.080162 -2.318E-06 12.14
10 1.203E-05 5.610E-07 -16.080162 -3.935E-07 12.22
11 6.744E-06 2.833E-07 -16.080162 -3.662E-08 12.31
At line 2 of file ./wfnio.f
Fortran runtime error: Actual string length is shorter than the declared one for dummy argument 'tag' (2/10)
Line 2 in wfnio.f is SUBROUTINE W_WFNIO(NW,IERROR,NSTATE,C,TAG)
I tried compiling with my own openblas libraries:
Used the following config file base:
IRAT=2
CFLAGS='-c -O1 -Wall'
CPP='/lib/cpp -P -C -traditional'
CPPFLAGS='-D__Linux -D__PGI -D__GNU -DFFT_FFTW -DPOINTER8 -DPARALLEL'
FFLAGS='-c -O2 -fcray-pointer -fno-whole-file -fsecond-underscore'
LFLAGS='-L/opt/fftw/fftw-2.1.5/double/lib -l:libfftw.a -I/opt/fftw/fftw-2.1.5/double/include -I/usr/include -L/opt/openblas/lib -lopenblas'
FFLAGS_GROMOS=' $(FFLAGS)'
FC='mpif90'
CC='mpicc'
LD='mpif90'
Tested with:
! Wavefunction optimization using default setting
!
&CPMD
OPTIMIZE WAVEFUNCTION
&END
&SYSTEM
SYMMETRY
1
CELL
10.2612 1.0 1.0 0.0 0.0 0.0
CUTOFF
13.
&END
&ATOMS
*SI_SGS KLEINMAN-BYLANDER
LMAX=P
8
.00000 .00000 .00000 1
.00000 5.13000 5.13000 1
5.13000 .00000 5.13000 1
5.13000 5.13000 .00000 1
2.56500 2.56500 2.56500 1
2.56500 7.69500 7.69500 1
7.69500 2.56500 7.69500 1
7.69500 7.69500 2.56500 1
&END
And here's the output (NOTE: it wouldn't have worked anyway due to missing pseudopot files. But that's unrelated):
PROGRAM CPMD STARTED AT: Wed Jul 18 20:41:51 2012SETCNST| USING: CODATA 2006 UNITS
[LOTS OF STUFF CUT OUT]EXCHANGE CORRELATION FUNCTIONALSLDA EXCHANGE: NONELDA XC THROUGH PADE APPROXIMATIONS.GOEDECKER, J.HUTTER, M.TETER PRB 54 1703 (1996)Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:#0 0x2B3D01CFA667#1 0x2B3D01CFAC34#2 0x2B3D026F64EF#3 0x2B3CFFD8CBF7Segmentation fault
Valgrind told me:
EXCHANGE CORRELATION FUNCTIONALS
LDA EXCHANGE: NONE
LDA XC THROUGH PADE APPROXIMATION
S.GOEDECKER, J.HUTTER, M.TETER PRB 54 1703 (1996)
==1406== Invalid read of size 8
==1406== at 0x59D0BF7: dcopy_k (in /opt/openblas/lib/libopenblas_barcelona-r0.1.1.so)
==1406== by 0x485152: setsc_ (in /home/me/tmp/CPMD/cpmd.x)
==1406== by 0x4264EE: cpmd_ (in /home/me/tmp/CPMD/cpmd.x)
==1406== by 0x4262EE: main (in /home/me/tmp/CPMD/cpmd.x)
==1406== Address 0x30016a9908 is not stack'd, malloc'd or (recently) free'd
==1406==
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x7737667
#1 0x7737C34
#2 0x81314EF
#3 0x59D0BF7
==1406==
==1406== HEAP SUMMARY:
==1406== in use at exit: 2,152,258 bytes in 2,802 blocks
==1406== total heap usage: 10,893 allocs, 8,091 frees, 16,699,612 bytes allocated
==1406==
==1406== LEAK SUMMARY:
==1406== definitely lost: 567 bytes in 20 blocks
==1406== indirectly lost: 2,973 bytes in 9 blocks
==1406== possibly lost: 0 bytes in 0 blocks
==1406== still reachable: 2,148,718 bytes in 2,773 blocks
==1406== suppressed: 0 bytes in 0 blocks
==1406== Rerun with --leak-check=full to see details of leaked memory
==1406==
==1406== For counts of detected and suppressed errors, rerun with: -v
==1406== Use --track-origins=yes to see where uninitialised values come from
==1406== ERROR SUMMARY: 591 errors from 4 contexts (suppressed: 4 from 4)
Segmentation fault
Ergo, it seems to be a problem with the openblas libs -- but they've work just fine with nwchem and gromacs...who knows? It might be due to mixing debian fftw3 and my own openblas.
If you're having problems with the output below it is because fftw 2 and fftw 3 are not compatible and CPDM uses the fftw 2 iface by default.
mltfft.o: In function `mltfft_fftw_': mltfft.f:(.text+0x958): undefined reference to `fftw_f77_create_plan_' mltfft.f:(.text+0x9af): undefined reference to `fftw_f77_' mltfft.f:(.text+0x9c2): undefined reference to `fftw_f77_destroy_plan_' mltfft.f:(.text+0xaee): undefined reference to `fftw_f77_create_plan_' mltfft.f:(.text+0xb4d): undefined reference to `fftw_f77_' mltfft.f:(.text+0xd03): undefined reference to `fftw_f77_' mltfft.f:(.text+0xdc4): undefined reference to `fftw_f77_' mltfft.f:(.text+0xe6e): undefined reference to `fftw_f77_create_plan_' mltfft.f:(.text+0xe8f): undefined reference to `fftw_f77_create_plan_' collect2: ld returned 1 exit status make: *** [cpmd.x] Error 1
You can either
1. Install fftw2 (sudo apt-get install fftw-dev, fftw2), and put -l:libfftw.so.2 -l:librfftw.so.2 in your LFLAGS.
or
2. Set -DFFT_FFTW3 instead of -DFFT_FFTW.
28 January 2011
3. Compiling nwchem on Ubuntu 10.10 64 bit
** See post on 15/12/2011 for information about Debian 64 bit. It builds fine on Squeeze but not Wheezy. This seems to have to do with the version of mpich2.**
Figuring out how to compile nwchem with mpich support took a little while, but this seems to have worked:
First mpich2 and gfortran need to be installed (since it was not installed on a virgin system there may have been other required packages already installed)
sudo apt-get install mpich2 gfortran
I created a file called myconfig.sh in the nwchem directory, with the following content:
setenv LARGE_FILES TRUE
setenv TCGRSH /usr/local/bin/ssh
setenv NWCHEM_TOP /work/nwchem
setenv NWCHEM_TARGET LINUX64
setenv NWCHEM_MODULES all
setenv USE_MPI y
setenv USE_MPIF y
setenv MPI_LOC /usr
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include/mpich2
setenv LIBMPI "-lfmpich -lmpich"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran >& make.log
do a csh myconfig.sh and you should be good to go.
I then added the following to the end of my ~/.bashrc and sourced it:
PATH=$PATH://work/nwchem/bin/LINUX64
export NWCHEM_EXECUTABLE=/work/nwchem/bin/LINUX64/nwchem
Jobs can then be submitted (assuming that mpd is up) by
mpdrun -n 2 nwchem nameofjob.nw
Figuring out how to compile nwchem with mpich support took a little while, but this seems to have worked:
First mpich2 and gfortran need to be installed (since it was not installed on a virgin system there may have been other required packages already installed)
sudo apt-get install mpich2 gfortran
I created a file called myconfig.sh in the nwchem directory, with the following content:
setenv LARGE_FILES TRUE
setenv TCGRSH /usr/local/bin/ssh
setenv NWCHEM_TOP /work/nwchem
setenv NWCHEM_TARGET LINUX64
setenv NWCHEM_MODULES all
setenv USE_MPI y
setenv USE_MPIF y
setenv MPI_LOC /usr
setenv MPI_LIB $MPI_LOC/lib
setenv MPI_INCLUDE $MPI_LOC/include/mpich2
setenv LIBMPI "-lfmpich -lmpich"
cd $NWCHEM_TOP/src
make nwchem_config
make FC=gfortran >& make.log
do a csh myconfig.sh and you should be good to go.
I then added the following to the end of my ~/.bashrc and sourced it:
PATH=$PATH://work/nwchem/bin/LINUX64
export NWCHEM_EXECUTABLE=/work/nwchem/bin/LINUX64/nwchem
Jobs can then be submitted (assuming that mpd is up) by
mpdrun -n 2 nwchem nameofjob.nw
Subscribe to:
Posts (Atom)