Getting an account

First, you need to get an account from NOTUR, the webpage for you request is : ​https://www.metacenter.no/user/application/form/notur/.

ssh configuration

You can add the following lines to ~/.ssh/config on your local machine:

Host hexagon hexagon.hpc.uib.no
    HostName hexagon.hpc.uib.no
    User YOURHEXAGONUSERNAME

and replace YOURHEXAGONUSERNAME by your hexagon username.

Once this is done, you can ssh hexagon by simply doing:

ssh hexagon

Password-less ssh

Once you have the account, you can setup a public key authentification in order to avoid having to input your password for each run. You need to have a SSH public/private key pair. If you do not, you can create a SSH public/private key pair by typing the following command and following the prompts (no passphrase necessary):

$your_localhost% ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/username/.ssh/id_rsa):RETURN
Enter passphrase (empty for no passphrase):RETURN
Enter same passphrase again:RETURN
Your identification has been saved in /Users/username/.ssh/id_rsa.
Your public key has been saved in /Users/username/.ssh/id_rsa.pub.

Two files were created: your private key /Users/username/.ssh/id_rsa, and the public key /Users/username/.ssh/id_rsa.pub. The private key is read-only and only for you, it is used to decrypt all correspondence encrypted with the public key. The contents of the public key need to be copied to ~/.ssh/authorized_keys on your hexagon account:

$your_localhost%scp ~/.ssh/id_rsa.pub username@your_remosthost:~

Now on Hexagon, copy the content of id_rsa.pub:

$your_remosthost%cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
$your_remosthost%rm ~/id_rsa.pub

Environment

On Hexagon, add the following lines to ~/.bashrc:

export ISSM_DIR=PATHTOTRUNK
export CRAYPE_LINK_TYPE=dynamic
#
module unload xtpe-interlagos
module unload notur
#
module switch PrgEnv-cray/5.2.40 PrgEnv-gnu
module load craype-barcelona
module load automake
module load cray-petsc
module load cray-tpsl
module load cray-mpich
module load gsl

Log out and log back in to apply this change.

Installing ISSM on Hexagon

You can check out ISSM and install m1qn3. This is the only package we will install, the module loaded from your bashrc are taking care of the rest.

Use the following configuration script (adapt to your needs):

#!/bin/sh
./configure \
  --prefix=/work/bfl022/issm_install \
  --with-m1qn3-dir=$ISSM_DIR/externalpackages/m1qn3/install \
  --with-petsc-dir=$PETSC_DIR \
  --with-metis-dir=$CRAY_TPSL_PREFIX_DIR \
  --with-mumps-dir=$CRAY_TPSL_PREFIX_DIR \
  --with-mpi-include="$CRAY_MPICH2_DIR/include" \
  --with-mpi-libflags="-L$CRAY_MPICH2_DIR/lib -lmpich -lmpl -lfmpich -lmpichcxx -lmpichf90" \
  --with-gsl-dir="/work/apps/gsl/1.16-gnu/" \
  --enable-development

You will also need to change the installation prefix (keep it somewhere on /work to be visible by the compute nodes)

When running autotools you may run into the following error message :

src/c/Makefile.am:9: Libtool library used but `LIBTOOL' is undefined
src/c/Makefile.am:9:   The usual way to define `LIBTOOL' is to add `LT_INIT'
src/c/Makefile.am:9:   to `configure.ac' and run `aclocal' and `autoconf' again.
src/c/Makefile.am:9:   If `LT_INIT' is in `configure.ac', make sure
src/c/Makefile.am:9:   its definition is in aclocal's search path.

running libtoolize should solve this issue.

Due to the fact that the autotool on this machine is getting old you should comment the following line in src/c/main/EnvironmentInit.cpp :

if(!my_rank) printf("(website: %s contact: %s)\n",PACKAGE_URL,PACKAGE_BUGREPORT);

hexagon_settings.py

def hexagon_settings(md):          
    md.login='YOURNOTURLOGIN'
    md.queue='normal'
    md.codepath='YOURCODEPATH'#Should be placed in /work/username
    md.executionpath='YOUREXECPATH'#Should be placed in /work/username
    md.accountname='bjerknes'
    md.numnodes=1
    return md

hexagon_settings.m

We encourage running the python version at Uib but you can clone the python version of hexagon_settings

Running jobs on Hexagon

TO BE UPDATED

Hexagon is aimed to large jobs. As on Vilje, the accounting is done as if you are using 32 processes per nodes even if you are using less. So if you don't need large ammount of memory try to restrict the number of nodes you are using. The scheduler also prioritize jobs with a high CPU counts. More on the queue priorities here: ​https://docs.hpc.uib.no/wiki/Queue_priorities_%28Hexagon%29

There is a batch queue that redirect the jobs in specific queues if they respond to certain requirments:

normal	queue for normal jobs	limited only by job limitations
small	queue for small jobs, jobs will get higher priority	max 512 CPUs, max 1 hour walltime
debug	queue for debugging, jobs will get higher priority	max 64 CPUs, max 20 minutes walltime

There are some limitations to the ressource that one single user can ask for :

• 4096 cpu cores, total number used by all running jobs
• 8-22 running (active) jobs (depending on load)
• 2 idle jobs

If you want to check the ordering of the queue and your jobs position you can use in your active Hexagon session:

showq          # display jobs
showq -u $USER # display jobs for $USER
showq -i      # display only jobs in eligible (idle) queue waiting for execution