Compiling with NWChem-6.6 with Intel MPI-5.0.3

Here is a write-up of my computing platform and applications:

  1. NWChem 6.6 (Oct 2015)
  2. Intel Compilers 2015 XE (version 15.0.6)
  3. Intel MPI (5.0.3)
  4. Intel MKL (11.2.4)
  5. Infiniband Inteconnect (OFED 1.5.3)
  6. CentOS 6.3 (x86_64)

Step 1: First thing first, source the intel components setting from

source /usr/local/intel_2015/parallel_studio_xe_2015/bin/psxevars.sh intel64
source /usr/local/intel_2015/impi/5.0.3.049/bin64/mpivars.sh intel64
source /usr/local/intel_2015/composerxe/bin/compilervars.sh intel64
source /usr/local/intel_2015/mkl/bin/mklvars.sh intel64

Step 2: Assuming you are done, you may want to download the NWChem 6.6 from NWChem Website. You may also want to take a look at instruction set for Compiling NWChem.

I have less problem running NWCHEM when the installation and the compiling directories are the same. If you recompile, do untar from source. Somehow the “make clean” does not clean the directories properly.

# tar -zxvf Nwchem-6.6.revision27746-src.2015-10-20.tar.gz
# cd nwchem-6.6

Step 3: Apply All the Patches for the 27746 revision of NWChem 6.6

cd $NWCHEM_TOP
wget http://www.nwchem-sw.org/download.php?f=Xccvs98.patch.gz
gzip -d Xccvs98.patch.gz
patch -p0 < Xccvs98.patch

Here is my nwchem_script_Feb2017.sh. For more details information, see Compiling NWChem for details on some of the parameters

export TCGRSH=/usr/bin/ssh
export NWCHEM_TOP=/home/melvin/Downloads/nwchem-6.6
export NWCHEM_TARGET=LINUX64
export NWCHEM_MODULES=all
export LARGE_FILES=TRUE

export ARMCI_NETWORK=OPENIB
export IB_INCLUDE=/usr/include
export IB_LIB=/usr/lib64
export IB_LIB_NAME="-libumad -libverbs -lpthread"

export MSG_COMMS=MPI
export USE_MPI=y
export USE_MPIF=y
export USE_MPIF4=y
export MPI_LOC=/usr/local/RH6_apps/intel_2015/impi_5.0.3/intel64
export MPI_LIB=$MPI_LOC/lib
export MPI_INCLUDE=$MPI_LOC/include
export LIBMPI="-lmpigf -lmpigi -lmpi_ilp64 -lmpi"

export FC=ifort
export CC=icc

export MKLLIB=/usr/local/RH6_apps/intel_2015/mkl/lib/intel64
export MKLINC=/usr/local/RH6_apps/intel_2015/mkl/include

export PYTHONHOME=/usr
export PYTHONVERSION=2.6
export USE_PYTHON64=y
export PYTHONLIBTYPE=so
sed -i 's/libpython$(PYTHONVERSION).a/libpython$(PYTHONVERSION).$(PYTHONLIBTYPE)/g' config/makefile.h

export HAS_BLAS=yes
export BLAS_SIZE=8 
export BLASOPT="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"
export LAPACK_LIBS="-L$MKLLIB -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lpthread -lm"
export SCALAPACK_SIZE=8
export SCALAPACK="-L$MKLLIB -lmkl_scalapack_ilp64 -lmkl_intel_ilp64 -lmkl_core -lmkl_sequential -lmkl_blacs_intelmpi_ilp64 -lpthread -lm"
export USE_64TO32=y

echo "cd $NWCHEM_TOP/src"
cd $NWCHEM_TOP/src

echo "BEGIN --- make realclean "
make realclean
echo "END --- make realclean "

echo "BEGIN --- make nwchem_config "
make nwchem_config
echo "END --- make nwchem_config "

echo "BEGIN --- make"
make CC=icc FC=ifort FOPTIMIZE=-O3 -j4
echo "END --- make "

cd $NWCHEM_TOP/src/util
make CC=icc FC=ifort FOPTIMIZE=-O3 version
make CC=icc FC=ifort FOPTIMIZE=-O3
cd $NWCHEM_TOP/src
make CC=icc FC=ifort FOPTIMIZE=-O3  link

General Site Installation

Determine the local storage path for the install files. (e.g., /usr/local/NWChem).
Make directories

# mkdir /usr/local/nwchem-6.6
# mkdir /usr/local/nwchem-6.6/bin
# mkdir /usr/local/nwchem-6.6/data

Copy binary

# cp $NWCHEM_TOP/bin/ /usr/local/nwchem-6.6/bin
# cd /usr/local/nwchem-6.6/bin
# chmod 755 nwchem

Copy libraries

# cd $NWCHEM_TOP/src/basis
# cp -r libraries /usr/local/nwchem-6.6/data

# cd $NWCHEM_TOP/src/
# cp -r data /usr/local/nwchem-6.6

# cd $NWCHEM_TOP/src/nwpw
# cp -r libraryps /usr/local/nwchem-6.6/data

The Final Lap (From Compiling NWChem)

Each user will need a .nwchemrc file to point to these default data files. A global one could be put in /usr/local/nwchem-6.6/data and a symbolic link made in each users $HOME directory is probably the best plan for new installs. Users would have to issue the following command prior to using NWChem: ln -s /usr/local/nwchem-6.6/data/default.nwchemrc $HOME/.nwchemrc

Contents of the default.nwchemrc file based on the above information should be:

nwchem_basis_library /usr/local/nwchem-6.6/data/libraries/
nwchem_nwpw_library /usr/local/nwchem-6.6/data/libraryps/
ffield amber
amber_1 /usr/local/nwchem-6.6/data/amber_s/
amber_2 /usr/local/nwchem-6.6/data/amber_q/
amber_3 /usr/local/nwchem-6.6/data/amber_x/
amber_4 /usr/local/nwchem-6.6/data/amber_u/
spce    /usr/local/nwchem-6.6/data/solvents/spce.rst
charmm_s /usr/local/nwchem-6.6/data/charmm_s/
charmm_x /usr/local/nwchem-6.6/data/charmm_x/

References:

  1. 470. Very briefly: compiling nwchem 6.3 with ifort and mkl
  2. Compiling NWChem from source
  3. Problem building NWChem version 6.5 on IB cluster with MKL & IntelMPI

Enable Centrify Agent to read UID and GID from Centrify DirectManage Access Manager

We purchased Centrify Standard and setup the DirectManage Access Manager. Next we proceed to install the client agent on the compute node.

After unpacking and installing the agent, when we do a

# getent passwd  |grep kittycool
kittycool:x:1304567321211:1304567321211:kittycool:/home/kittycool:/bin/bash
kittycool:x:10001:10001:kittycool:/home/kittycool:/bin/bash

Apparently, the getent passwd |grep kittycool is pulling both the Active Directory UID and the DirectManage Access and the user UID differs

To resolve this issue, you need to specify the zone which is used by DirectManage Access Manager, so your UID of the user will pick from the DirectManage Access Manager.

# adjoin -z cluster -u OU_Administrator  staff.mycompany.com.sg -c "staff.mycompany.com.sg/HPC/Computers"

To check it is displaying the correct UID and GID,

# getent passwd  |grep kittycool
kittycool:x:10001:10001:kittycool:/home/kittycool:/bin/bash

Restricting SSH Access when using Centrify-Free

To restrict users from accessing the system using Centriy free can be easily managed by using the following files

/etc/centrifydc/users.allow
/etc/centrifydc/groups.allow
/etc/centrifydc/users.deny
/etc/centrifydc/groups.deny

1. You have to manually create the the files accordingly and place it at /etc/centifydc. Next you have to  line 273 and uncomment the line

.....
pam.allow.users: file:/etc/centrifydc/users.allow
.....

If you are blocking by groups, you can likewise uncomment the

.....
pam.allow.groups: file:/etc/centrifydc/groups.allow
.....

2. Flush and Reload Centrify-Free

# adflush
# adreload

3. Add users you wish to have access into the system into /etc/centrifydc/users.allow

Mapping Ad account to Local Linux Group with Centrify Express on CentOS 6

Step 1. In order to add the AD User to the local User Group

adduser command is not nsswitch aware and do not recognize a user not locally defined when adding someone to a group.

# vim /etc/group
.....
.....
users:x:100:user1,user2

Step 2: Edit /etc/centrifydc/centrifydc.conf

Edit /etc/centrifydc/centrifydc.conf, uncomment and change the following parameter to true

# Merge local group membership from /etc/group into the Centrify group
# response for groups with the same name and gid.  This violates the NSS
# interface behavior and may have unexpected side effects, so it is
# disabled by default. You must run adreload to detect changes
# in the local group file.
#
adclient.local.group.merge:true

Step 3: Centrify Load and Flush

# adreload && adflush

Step 4: Check and verify.

Logon as user1

$ id -a
uid=.........gid=.......... 100(users) .....................

References:

  1. How to Add AD user local group

Add an Active Directory (AD) Domain Account to sudo on Centrify Express for CentOS 6

Adding a Active Directory (AD) Domain Account to /etc/sudo is fairly easy on Centrify Express for CentOS 6. Suppose you have an AD Group called “Sys_Admin”, just add it to the

.....
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL
%System_Admin ALL=(ALL)       ALL
.....

If it is an individual user, just get the userid from Active Directory

# getent passwd myuserid
user1:x:122:122:User1 Name:/home/user1:/bin/centrifyda

Update /etc/sudoers

.....
## Allow root to run any commands anywhere
root    ALL=(ALL)       ALL
%user1 ALL=(ALL)       ALL
.....

Do a final check on whether the sudo is working fine by doing

# sudo -l

Matching Defaults entries for user1 on this host:
requiretty, !visiblepw, always_set_home, env_reset, env_keep="COLORS
DISPLAY HOSTNAME HISTSIZE INPUTRC KDEDIR LS_COLORS", env_keep+="MAIL PS1
PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE
LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES", env_keep+="LC_MONETARY
LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL
LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY",
secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin

User user1 may run the following commands on this host:
(ALL) ALL

Enabling Debugging for Linux Systems for Centrify Express on CentOS 6

 

Step 1: Enable Debugging for Centrify

# /usr/share/centrifydc/bin/addebug on
/var/log/centrify_client.log does not exist. Creating now.
Configure /etc/rsyslog.conf
Send HUP signal to rsyslogd
Configure log level in /etc/centrifydc/centrifydc.conf
Reload /etc/centrifydc/centrifydc.conf
Centrify DirectControl debug logging is on

 Step 2: Collect support information via adinfo_extra.sh

# adinfo --support

Collecting stack trace information via stacktrace.pl:

Collecting support information via adinfo_extra.sh:
Collecting information for adinfo --support now...
Collection finished. Compress now.
Successfully gzip /var/centrify/tmp/adinfo_support.tar

Step 3: Disable Debugging for Centrify

# /usr/share/centrifydc/bin/addebug off
Configure /etc/rsyslog.conf
Send HUP signal to rsyslogd
Configure log level in /etc/centrifydc/centrifydc.conf
Reload /etc/centrifydc/centrifydc.conf
Centrify DirectControl debug logging is off

Step 4: Check the file

# ls  -al /var/centrify/tmp/adin_support.tar.gz

References:

  1. Centrify Express: Enabling Debugging for Linux Systems