== Getting an account ==

First, you need to get an account from NOTUR, the webpage for you request is : https://www.metacenter.no/user/application/form/notur/.

== ssh configuration ==

You can add the following lines to `~/.ssh/config` on your local machine:
{{{
#!sh
Host hexagon hexagon.hpc.uib.no
    HostName hexagon.hpc.uib.no
    User YOURHEXAGONUSERNAME
}}}
and replace `YOURHEXAGONUSERNAME` by your hexagon username.

Once this is done, you can ssh hexagon by simply doing:

{{{
#!sh
ssh hexagon
}}}

== Password-less ssh ==

Once you have the account, you can setup a public key authentification in order to avoid having to input your password for each run. 
You need to have a SSH public/private key pair. If you do not, you can create a SSH public/private key pair by typing the following command and following the prompts (no passphrase necessary):
{{{
#!sh
$your_localhost% ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/username/.ssh/id_rsa):RETURN
Enter passphrase (empty for no passphrase):RETURN
Enter same passphrase again:RETURN
Your identification has been saved in /Users/username/.ssh/id_rsa.
Your public key has been saved in /Users/username/.ssh/id_rsa.pub.
}}}

Two files were created: your private key `/Users/username/.ssh/id_rsa`, and the public key `/Users/username/.ssh/id_rsa.pub`. The private key is read-only and only for you, it is used to decrypt all correspondence encrypted with the public key. The contents of the public key need to be copied to `~/.ssh/authorized_keys` on your hexagon account:

{{{
#!sh
$your_localhost%scp ~/.ssh/id_rsa.pub username@your_remosthost:~
}}}

Now on '''Hexagon''', copy the content of id_rsa.pub:

{{{
#!sh
$your_remosthost%cat ~/id_rsa.pub >> ~/.ssh/authorized_keys
$your_remosthost%rm ~/id_rsa.pub
}}}

== Environment ==

On Hexagon, add the following lines to `~/.bashrc`:
{{{
#!sh
export ISSM_DIR=PATHTOTRUNK
export CRAYPE_LINK_TYPE=dynamic
#
module unload xtpe-interlagos
module unload notur
#
module switch PrgEnv-cray/5.2.40 PrgEnv-gnu
module load craype-barcelona
module load automake
module load cray-petsc
module load cray-tpsl
module load cray-mpich
module load gsl
}}}

''Log out and log back in'' to apply this change.

== Installing ISSM on Hexagon ==

You can check out ISSM and install m1qn3. This is the only package we will install, the module loaded from your bashrc are taking care of the rest.

Use the following configuration script (adapt to your needs):

{{{
#!/bin/sh
./configure \
  --prefix=/work/bfl022/issm_install \
  --with-m1qn3-dir=$ISSM_DIR/externalpackages/m1qn3/install \
  --with-petsc-dir=$PETSC_DIR \
  --with-metis-dir=$CRAY_TPSL_PREFIX_DIR \
  --with-mumps-dir=$CRAY_TPSL_PREFIX_DIR \
  --with-mpi-include="$CRAY_MPICH2_DIR/include" \
  --with-mpi-libflags="-L$CRAY_MPICH2_DIR/lib -lmpich -lmpl -lfmpich -lmpichcxx -lmpichf90" \
  --with-gsl-dir="/work/apps/gsl/1.16-gnu/" \
  --enable-development
}}}

You will also need to change the installation prefix (keep it somewhere on /work to be visible by the compute nodes)

When running autotools you may run into the following error message :
{{{
#!sh
src/c/Makefile.am:9: Libtool library used but `LIBTOOL' is undefined
src/c/Makefile.am:9:   The usual way to define `LIBTOOL' is to add `LT_INIT'
src/c/Makefile.am:9:   to `configure.ac' and run `aclocal' and `autoconf' again.
src/c/Makefile.am:9:   If `LT_INIT' is in `configure.ac', make sure
src/c/Makefile.am:9:   its definition is in aclocal's search path.
}}}
running libtoolize should solve this issue.


Due to the fact that the autotool on this machine is getting old you should comment the following line in  `src/c/main/EnvironmentInit.cpp `:
{{{
#!sh
if(!my_rank) printf("(website: %s contact: %s)\n",PACKAGE_URL,PACKAGE_BUGREPORT);
}}}

== hexagon_settings.py ==
{{{
def hexagon_settings(md):          
    md.login='YOURNOTURLOGIN'
    md.queue='normal'
    md.codepath='YOURCODEPATH'#Should be placed in /work/username
    md.executionpath='YOUREXECPATH'#Should be placed in /work/username
    md.accountname='bjerknes'
    md.numnodes=1
    return md
}}}

== hexagon_settings.m ==

We encourage running the python version at Uib but you can clone the python version of hexagon_settings

== Running jobs on Hexagon ==
TO BE UPDATED

Hexagon is aimed to large jobs. As on Vilje, the accounting is done as if you are using 32 processes per nodes even if you are using less. So if you don't need large ammount of memory try to restrict the number of nodes you are using. The scheduler also prioritize jobs with a high CPU counts. More on the queue priorities here: 
https://docs.hpc.uib.no/wiki/Queue_priorities_%28Hexagon%29

There is a batch queue that redirect the jobs in specific queues if they respond to certain requirments:

|| normal || queue for normal jobs                                  || limited only by job limitations      ||
|| small  || queue for small jobs, jobs will get higher priority    || max 512 CPUs, max 1 hour walltime    ||
|| debug  || queue for debugging, jobs will get higher priority     || max 64 CPUs, max 20 minutes walltime ||

There are some limitations to the ressource that one single user can ask for :
 * 4096 cpu cores, total number used by all running jobs
 * 8-22 running (active) jobs (depending on load)
 * 2 idle jobs

If you want to check the ordering of the queue and your jobs position you can use in your active Hexagon session:
{{{
#!sh
showq          # display jobs
showq -u $USER # display jobs for $USER
showq -i      # display only jobs in eligible (idle) queue waiting for execution
}}}