Changes between Initial Version and Version 1 of eis

Timestamp:

10/26/22 01:44:46 (3 years ago)

Author:

jdquinn

Comment:

Initial page creation

eis

@@ +1 @@
+= Installation on SMCE head node (external packages, binaries only)
+- Added the following to `~/.bashrc`,
+{{{
+# Spack Modules
+source /shared/spack/share/spack/setup-env.sh
+module load intelmpi
+#module load python-3.8.12-gcc-9.4.0-uyaszn4
+module load subversion-1.14.0-gcc-11.1.0-evgneyf
+
+###################
+### Environment ###
+###################
+export CC=/opt/intel/mpi/2021.4.0/bin/mpicc
+export GCC=/opt/intel/mpi/2021.4.0/bin/mpicxx
+export FC=/opt/intel/mpi/2021.4.0/bin/mpifc
+
+export MPI_HOME=/opt/intel/mpi/2021.4.0
+
+######################
+### Path Variables ###
+######################
+
+# ISSM
+export ISSM_DIR="/efs/issm/binaries/repos/trunk-jpl-working"
+export ISSM_EXT_DIR="/efs/issm/binaries/ext"
+}}}
+
+- Checked out a copy of the ISSM development repository with,
+{{{
+mkdir -p /efs/issm/binaries/repos
+svn co https://issm.ess.uci.edu/svn/issm/issm/trunk-jpl $ISSM_DIR
+}}}
+
+- Copied `$ISSM_DIR/jenkins/ross-debian_linux-full` to `$ISSM_DIR/jenkins/eis-smce-binaries`
+- Modified `$ISSM_DIR/jenkins/eis-smce-binaries` with,
+{{{
+#--------------------#
+# ISSM Configuration #
+#--------------------#
+
+ISSM_CONFIG='\
+        --prefix="${ISSM_DIR}" \
+        --disable-static \
+        --with-wrappers=no \
+        --enable-development \
+        --enable-debugging \
+        --with-numthreads=48 \
+        --with-fortran-lib="-L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran" \
+        --with-mpi-include="/opt/intel/mpi/2021.4.0/include" \
+        --with-mpi-libflags="-L/opt/intel/mpi/2021.4.0/lib -lmpi -lmpicxx -lmpifort" \
+        --with-blas-lapack-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-metis-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-parmetis-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-scalapack-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-mumps-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-hdf5-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-petsc-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-gsl-dir="${ISSM_EXT_DIR}/gsl/install" \
+        --with-boost-dir="${ISSM_EXT_DIR}/boost/install" \
+        --with-dakota-dir="${ISSM_EXT_DIR}/dakota/install" \
+        --with-proj-dir="${ISSM_EXT_DIR}/proj/install" \
+        --with-triangle-dir="${ISSM_EXT_DIR}/triangle/install" \
+        --with-chaco-dir="${ISSM_EXT_DIR}/chaco/install" \
+        --with-m1qn3-dir="${ISSM_EXT_DIR}/m1qn3/install" \
+        --with-semic-dir="${ISSM_EXT_DIR}/semic/install"
+'
+
+#-------------------#
+# External Packages #
+#-------------------#
+
+EXTERNALPACKAGES="
+        autotools       install-linux.sh
+        cmake           install.sh
+        petsc           install-3.12-linux.sh
+        gsl                     install.sh
+        boost           install-1.7-linux.sh
+        dakota          install-6.2-linux.sh
+        curl            install-7-linux.sh
+        netcdf          install-4.7-parallel.sh
+        sqlite          install.sh
+        proj            install-6.sh
+        gdal            install-3.sh
+        gshhg           install.sh
+        gmt                     install-6-linux.sh
+        gmsh            install-4-linux.sh
+        triangle        install-linux.sh
+        chaco           install.sh
+        m1qn3           install.sh
+        semic           install.sh
+"
+}}}
+- Committed `$ISSM_DIR/jenkins/eis-smce-head-node-binaries` to repo
+- Installed each of the external packages in `$ISSM_DIR/jenkins/eis-daskhub-python-modules`, after changing each of the installation paths with,
+{{{
+PREFIX=${ISSM_EXT_DIR}/<pkg>/install
+}}}
+  and making the following modifications,
+  - Modified `$ISSM_DIR/externalpackages/petsc/install-3.12-linux.sh` by,
+    - removing,
+    {{{
+--download-mpich=1
+    }}}
+    - adding,
+    {{{
+    --with-mpi-dir=/opt/intel/mpi/2021.4.0
+    --known-mpi-shared-libraries=1
+    }}}
+  - `$ISSM_DIR/externalpackages/dakota/install-6.2-linux.sh` requires Python 2 to build, so either (1) cannot load Spack module for Python 3, or, (2) have to figure out how to override this in the install script (perhaps with `alias python=/usr/bin/python`)
+  - Modified `$ISSM_DIR/externalpackages/gdal/install-3.sh` with,
+  {{{
+--with-curl="${CURL_ROOT}/bin/curl-config"
+  }}}
+
+= Installation on MATLAB node (MATLAB modules)
+TO BE ADDED AFTER FURTHER INSTALLATION AND DEBUGGING
+
+= Installation on Daskhub (Python modules)
+== NOTE: Binaries are also compiled as we do not have a good way of disabling them. We could ignore them, remove them, or let users know that they're available for smaller test runs.
+- Added the following to ~/.bash_profile,
+{{{
+# File system modifications that are not currently part of image
+#
+
+# Create a link to gl.h so that the CMake configuration can find it.
+# Note that we need to add it here because it does not persist.
+if [ ! -f /srv/conda/envs/notebook/include/GL/gl.h ]; then
+        ln -s /srv/conda/envs/notebook/include/gstreamer-1.0/gst/gl/gl.h  /srv/conda/envs/notebook/include/GL/gl.h
+fi
+
+# Set permissions on private key
+chmod 400 ~/.ssh/eis-nasa-smce-head-node
+
+
+## Environment
+#
+export OPENSSL_ROOT_DIR="/srv/conda/envs/notebook"
+
+
+## Path Variables
+#
+
+# ISSM
+export ISSM_DIR="/efs/issm/python-modules/repos/trunk-jpl-working"
+export ISSM_EXT_DIR="/efs/issm/python-modules/ext"
+
+
+## Aliases
+#
+alias issm-python-dev-env="source "\${ISSM_DIR}"/etc/environment.sh; export PYTHONPATH="\${ISSM_DIR}"/src/m/dev; export PYTHONSTARTUP="\${PYTHONPATH}"/devpath.py; export PYTHONUNBUFFERED=1; cd "\${ISSM_DIR}"/test/NightlyRun; LD_PRELOAD=/srv/conda/envs/notebook/lib/libpython3.9.so.1.0:"\${ISSM_EXT_DIR}"/petsc/install/lib/libmpifort.so.12 python" # Then run ./runme.py <options>
+}}}
+
+- Checked out a copy of the ISSM development repository with,
+{{{
+mkdir -p /efs/issm/python-modules/repos
+svn co https://issm.ess.uci.edu/svn/issm/issm/trunk-jpl $ISSM_DIR
+}}}
+
+- Copied `$ISSM_DIR/jenkins/ross-debian_linux-full` to `$ISSM_DIR/jenkins/eis-daskhub-python-modules`
+- Modified `$ISSM_DIR/jenkins/eis-daskhub-python-modules` with,
+{{{
+#--------------------#
+# ISSM Configuration #
+#--------------------#
+
+ISSM_CONFIG='\
+        --prefix="${ISSM_DIR}" \
+        --disable-static \
+        --enable-development \
+        --enable-debugging \
+        --with-numthreads=4 \
+        --with-python-dir="/srv/conda/envs/notebook" \
+        --with-python-version="3.9" \
+        --with-python-numpy-dir="/srv/conda/envs/notebook/lib/python3.9/site-packages/numpy/core/include/numpy" \
+        --with-fortran-lib="-L/usr/lib/x86_64-linux-gnu -lgfortran" \
+        --with-mpi-include="${ISSM_EXT_DIR}/petsc/install/include" \
+        --with-mpi-libflags="-L${ISSM_EXT_DIR}/petsc/install/lib -lmpi -lmpicxx -lmpifort" \
+        --with-blas-lapack-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-metis-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-parmetis-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-scalapack-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-mumps-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-hdf5-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-petsc-dir="${ISSM_EXT_DIR}/petsc/install" \
+        --with-gsl-dir="${ISSM_EXT_DIR}/gsl/install" \
+        --with-boost-dir="${ISSM_EXT_DIR}/boost/install" \
+        --with-dakota-dir="${ISSM_EXT_DIR}/dakota/install" \
+        --with-proj-dir="${ISSM_EXT_DIR}/proj/install" \
+        --with-triangle-dir="${ISSM_EXT_DIR}/triangle/install" \
+        --with-chaco-dir="${ISSM_EXT_DIR}/chaco/install" \
+        --with-m1qn3-dir="${ISSM_EXT_DIR}/m1qn3/install" \
+        --with-semic-dir="${ISSM_EXT_DIR}/semic/install" \
+'
+
+#-------------------#
+# External Packages #
+#-------------------#
+
+EXTERNALPACKAGES="
+        autotools       install-linux.sh
+        cmake           install.sh
+        petsc           install-3.16-linux.sh
+        gsl                     install.sh
+        boost           install-1.7-linux.sh
+        dakota          install-6.2-linux.sh
+        curl            install-7-linux.sh
+        netcdf          install-4.7-parallel.sh
+        sqlite          install.sh
+        proj            install-6.sh
+        gdal            install-3-python.sh
+        gshhg           install.sh
+        gmt                     install-6-linux.sh
+        gmsh            install-4-linux.sh
+        triangle        install-linux.sh
+        chaco           install.sh
+        m1qn3           install.sh
+        semic           install.sh
+"
+}}}
+
+- Committed `$ISSM_DIR/jenkins/eis-daskhub-python-modules` to repo
+- Before installing any of the external packages, needed to disable the Conda environment with `conda deactivate`
+- Installed each of the external packages in `$ISSM_DIR/jenkins/eis-daskhub-python-modules`, after changing each of the installation paths with,
+{{{
+PREFIX=${ISSM_EXT_DIR}/<pkg>/install
+}}}
+  and making the following modifications
+  - Needed to remove,
+{{{
+--FFLAGS="-fallow-argument-mismatch"
+}}}
+    from `$ISSM_DIR/externalpackages/petsc/install-3.16-petsc.sh`
+  - Needed to modify `$ISSM_DIR/externalpackages/gdal/install-3-python.sh` with,
+{{{
+export CPATH="${CPATH}:/srv/conda/envs/notebook/include"
+export LDFLAGS="-L${NETCDF_ROOT}/lib -lnetcdf"
+export LIBRARY_PATH="${LIBRARY_PATH}:/srv/conda/envs/notebook/lib"
+export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/srv/conda/envs/notebook/lib"
+}}}
+    and,
+{{{
+--with-sqlite3="${SQLITE_ROOT}" \
+--with-libjson-c="/srv/conda/envs/notebook"
+}}}
+
+  - Needed to modify `$ISSM_DIR/externalpackages/gmt/install-6-linux.sh` with,
+{{{
+export CC=mpicc
+export CPATH="${CPATH}:/srv/conda/envs/notebook/include"
+export LIBRARY_PATH="${LIBRARY_PATH}:/srv/conda/envs/notebook/lib"
+export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/srv/conda/envs/notebook/lib"
+}}}
+
+- With Conda environment still disabled, ran,
+{{{
+alias envsubst="/srv/conda/envs/notebook/bin/envsubst"
+jenkins/jenkins.sh jenkins/eis-daskhub-python-modules
+}}}
+
+- Ran,
+{{{
+mkdir ~/.ssh
+ssh-keygen -f ~/.ssh/eis-nasa-smce
+}}}
+  then gave contents of public key to SMCE team to add to SMCE head node
+- Ran,
+{{{
+chmod 400 ~/.ssh/eis-nasa-smce
+}}}
+
+- Created `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce.py` with contents,
+{{{
+import os
+import shutil
+import subprocess
+
+try:
+    from eis_nasa_smce_settings import eis_nasa_smce_settings
+except ImportError:
+    print('You need eis_nasa_smce_settings.py to proceed, check presence and sys.path')
+from fielddisplay import fielddisplay
+from helpers import *
+from IssmConfig import IssmConfig
+from issmssh import issmssh
+from MatlabFuncs import *
+from pairoptions import pairoptions
+
+class eis_nasa_smce(object):
+    """EIS_NASA_SMCE cluster class definition
+
+    Usage:
+        cluster = eis_nasa_smce()
+        cluster = eis_nasa_smce('np', 3)
+        cluster = eis_nasa_scme('np', 3, 'login', 'username')
+    """
+
+    def __init__(self, *args):  # {{{
+        self.name = '52.10.233.96'
+        self.login = 'jdquinn1'
+        self.idfile = '~/.ssh/eis-nasa-smce'
+        self.modules = ['intelmpi']
+        self.numnodes = 4
+        self.cpuspernode = 1
+        self.port = 0
+        self.time = 12 * 60 * 60
+        self.processor = 'skylake'
+        self.partition = 'sealevel-c5xl-spot'
+        self.srcpath = '/efs/issm-new/binaries/repos/trunk-jpl-working'
+        self.extpkgpath = '/efs/issm-new/binaries/ext'
+        self.codepath = '/efs/issm-new/binaries/repos/trunk-jpl-working/bin'
+        self.executionpath = '~/issm-exec'
+        self.interactive = 0
+        self.numstreams = 1
+        self.hyperthreading = 0
+        self.email = ''
+
+        # Use provided options to change fields
+        options = pairoptions(*args)
+
+        # Initialize cluster using user settings if provided
+        try:
+            self = eis_nasa_smce_settings(self)
+        except NameError:
+            print('eis_nasa_smce_settings.py not found, using default settings')
+
+        # OK get other fields
+        self = options.AssignObjectFields(self)
+    # }}}
+
+    def __repr__(self):  # {{{
+        # Display the object
+        s = 'class eis_nasa_smce object\n'
+        s += '    name: {}\n'.format(self.name)
+        s += '    login: {}\n'.format(self.login)
+        s += '    idfile: {}\n'.format(self.idfile)
+        s += '    modules: {}\n'.format(strjoin(self.modules, ', '))
+        s += '    numnodes: {}\n'.format(self.numnodes)
+        s += '    cpuspernode: {}\n'.format(self.cpuspernode)
+        s += '    np: {}\n'.format(self.nprocs())
+        s += '    port: {}\n'.format(self.port)
+        s += '    time: {}\n'.format(self.time)
+        s += '    processor: {}\n'.format(self.processor)
+        s += '    partition: {}\n'.format(self.partition)
+        s += '    srcpath: {}\n'.format(self.srcpath)
+        s += '    extpkgpath: {}\n'.format(self.extpkgpath)
+        s += '    codepath: {}\n'.format(self.codepath)
+        s += '    executionpath: {}\n'.format(self.executionpath)
+        s += '    interactive: {}\n'.format(self.interactive)
+        s += '    numstreams: {}\n'.format(self.numstreams)
+        s += '    hyperthreading: {}\n'.format(self.hyperthreading)
+        return s
+    # }}}
+
+    def nprocs(self):  # {{{
+        return self.numnodes * self.cpuspernode
+    # }}}
+
+    def checkconsistency(self, md, solution, analyses):  # {{{
+        # Now, check cluster.cpuspernode according to processor type
+        if self.processor == 'skylake':
+            if self.cpuspernode > 14 or self.cpuspernode < 1:
+                md = md.checkmessage('cpuspernode should be between 1 and 14 for \'skyw\' processors in hyperthreading mode')
+        else:
+            md = md.checkmessage('unknown processor type, should be \'skylake\'')
+
+        # Miscellaneous
+        if not self.login:
+            md = md.checkmessage('login empty')
+        if self.port:
+            md = md.checkmessage('port must be set to 0 as we do not have an SSH tunnel')
+        if not self.codepath:
+            md = md.checkmessage('codepath empty')
+        if not self.executionpath:
+            md = md.checkmessage('executionpath empty')
+
+        return self
+    # }}}
+
+    def BuildQueueScript(self, dirname, modelname, solution, io_gather, isvalgrind, isgprof, isdakota, isoceancoupling):  # {{{
+        if isgprof:
+            print('gprof not supported by cluster, ignoring...')
+
+        issmexec = 'issm.exe'
+        mpiexec = 'mpiexec' # Set to alternative mpiexec if desired
+
+        if isdakota:
+            version = IssmConfig('_DAKOTA_VERSION_')[0:2]
+            version = float(str(version[0]))
+            if version >= 6:
+                issmexec = 'issm_dakota.exe'
+        if isoceancoupling:
+            issmexec = 'issm_ocean.exe'
+
+        # Write queuing script
+        fid = open(modelname + '.queue', 'w')
+
+        fid.write('#!/bin/bash\n')
+        fid.write('#SBATCH --partition={} \n'.format(self.partition))
+        fid.write('#SBATCH -J {} \n'.format(modelname))
+        fid.write('#SBATCH -o {}.outlog \n'.format(modelname))
+        fid.write('#SBATCH -e {}.errlog \n'.format(modelname))
+        fid.write('#SBATCH --nodes={} \n'.format(self.numnodes))
+        fid.write('#SBATCH --ntasks-per-node={} \n'.format(self.cpuspernode))
+        fid.write('#SBATCH --cpus-per-task={} \n'.format(self.numstreams))
+        fid.write('#SBATCH -t {:02d}:{:02d}:00 \n'.format(int(floor(self.time / 3600)), int(floor(self.time % 3600) / 60)))
+        if (self.email.find('@')>-1):
+            fid.write('#SBATCH --mail-user={} \n'.format(self.email))
+            fid.write('#SBATCH --mail-type=BEGIN,END,FAIL \n\n')
+        fid.write('source /etc/profile\n')
+        fid.write('source /shared/spack/share/spack/setup-env.sh\n')
+        for i in range(len(self.modules)):
+             fid.write('module load {} &> /dev/null\n'.format(self.modules[i]))
+        fid.write('export MPI_GROUP_MAX=64\n\n')
+        fid.write('export MPI_UNBUFFERED_STDIO=true\n\n')
+        fid.write('export PATH="$PATH:/opt/slurm/bin"\n')
+        fid.write('export PATH="$PATH:."\n\n')
+        fid.write('export ISSM_DIR="{}"\n'.format(self.srcpath))
+        fid.write('export ISSM_EXT_DIR="{}"\n'.format(self.extpkgpath))
+        fid.write('source $ISSM_DIR/etc/environment.sh\n')
+        fid.write('cd {}/{}/\n\n'.format(self.executionpath, dirname))
+        fid.write('{} -n {} {}/{} {} {}/{} {}\n'.format(mpiexec, self.nprocs(), self.codepath, issmexec, solution, self.executionpath, dirname, modelname))
+
+        if not io_gather: # concatenate the output files
+            fid.write('cat {}.outbin.* > {}.outbin'.format(modelname, modelname))
+        fid.close()
+
+        # In interactive mode, create a run file, and errlog and outlog file
+        if self.interactive:
+            fid = open(modelname + '.run', 'w')
+            if not isvalgrind:
+                fid.write('{} -np {} {}/{} {} {}/{} {}\n'.format(mpiexec, self.nprocs(), self.codepath, issmexec, solution, self.executionpath, dirname, modelname))
+            else:
+                fid.write('{} -np {} valgrind --leak-check=full {}/{} {} {}/{} {}\n'.format(mpiexec, self.nprocs(), self.codepath, issmexec, solution, self.executionpath, dirname, modelname))
+            if not io_gather: # concatenate the output files
+                fid.write('cat {}.outbin.* > {}.outbin'.format(modelname, modelname))
+            fid.close()
+            fid = open(modelname + '.errlog', 'w') # TODO: Change this to system call (touch <file>)?
+            fid.close()
+            fid = open(modelname + '.outlog', 'w') # TODO: Change this to system call (touch <file>)?
+            fid.close()
+    # }}}
+
+    def UploadQueueJob(self, modelname, dirname, filelist):  # {{{
+        # Compress the files into one zip
+        compressstring = 'tar -zcf {}.tar.gz'.format(dirname)
+        for file in filelist:
+            compressstring += ' {}'.format(file)
+        if self.interactive:
+            compressstring += ' {}.run {}.errlog {}.outlog'.format(modelname, modelname, modelname)
+        subprocess.call(compressstring, shell=True)
+
+        print('uploading input file and queueing script')
+        if self.interactive:
+            directory = '{}/Interactive{}'.format(self.executionpath, self.interactive)
+        else:
+            directory = self.executionpath
+
+        # NOTE: Replacement for issmscpout(self.name, directory, self.login, self.port, ['{}.tar.gz'.format(dirname)])
+        copystring = 'cp {}.tar.gz /efs/issm/tmp'.format(dirname)
+        subprocess.call(copystring, shell=True)
+    # }}}
+
+    def LaunchQueueJob(self, modelname, dirname, filelist, restart, batch):  # {{{
+        if self.interactive:
+            if not isempty(restart):
+                launchcommand = 'cd {}/Interactive{}'.format(self.executionpath, self.interactive)
+            else:
+                launchcommand = 'cd {}/Interactive{} && tar -zxf {}.tar.gz'.format(self.executionpath, self.interactive, dirname)
+        else:
+            if not isempty(restart):
+                launchcommand = 'cd {} && cd {} && sbatch {}.queue'.format(self.executionpath, dirname, modelname)
+            else:
+                launchcommand = 'cd {} && rm -rf {} && mkdir {} && cd {} && cp /efs/issm/tmp/{}.tar.gz . && tar -zxf {}.tar.gz && /opt/slurm/bin/sbatch {}.queue'.format(self.executionpath, dirname, dirname, dirname, dirname, dirname, modelname)
+
+        print('launching solution sequence on remote cluster')
+
+        # NOTE: Replacement for issmssh(self.name, self.login, self.port, launchcommand)
+        subprocess.call('ssh -l {} -i {} {} "{}"'.format(self.login, self.idfile, self.name, launchcommand), shell=True)
+    # }}}
+
+    def Download(self, dirname, filelist):  # {{{
+        # Copy files from cluster to current directory
+    
+        # NOTE: Replacement for issmscpin(self.name, self.login, self.port, directory, filelist)
+        directory = '{}/{}/'.format(self.executionpath, dirname)
+        fileliststr = '{' + ','.join([str(x) for x in filelist]) + '}'
+        downloadcommand = 'scp -i {} {}@{}:{} {}/.'.format(self.idfile, self.login, self.name, os.path.join(directory, fileliststr), os.getcwd())
+        subprocess.call(downloadcommand, shell=True) 
+    # }}}
+}}}
+
+- Committed `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce_head_node.py` to repo
+
+NOTE: 
+- It is important that the directory indicated by `self.executionpath` exists on SMCE head node before launch
+- Either in `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce_head_node.py` or runme
+  - `cluster.login` should be changed to your username
+  - `cluster.executionpath` should be changed to desired execution path on SMCE head node
+  - `cluster.partition` should be changed to desired AWS ECS instance type (discuss with SMCE team before changing)
+
+- Created `$ISSM_DIR/test/NightlyRun/test101eisnasasmce.py` with contents,
+{{{
+#Test Name: SquareShelfConstrainedStressSSA2d
+from model import *
+from socket import gethostname
+from triangle import triangle
+from setmask import setmask
+from parameterize import parameterize
+from setflowequation import setflowequation
+from solve import solve
+from massfluxatgate import massfluxatgate
+from generic import generic
+from eis_nasa_smce import eis_nasa_smce
+
+md = triangle(model(), '../Exp/Square.exp', 50000)
+md = setmask(md, 'all', '')
+md = parameterize(md, '../Par/SquareShelfConstrained.py')
+md = setflowequation(md, 'SSA', 'all')
+md.cluster = generic('name', gethostname(), 'np', 2)
+
+if True:
+    cluster = eis_nasa_smce()
+    cluster.partition = 'sealevel-c5xl-spot'
+    md.cluster = cluster
+
+#outputs
+md.stressbalance.requested_outputs = ['default', 'DeviatoricStressxx', 'DeviatoricStressyy', 'DeviatoricStressxy', 'MassFlux1', 'MassFlux2', 'MassFlux3', 'MassFlux4', 'MassFlux5', 'MassFlux6']
+md.outputdefinition.definitions = [massfluxatgate('name', 'MassFlux1', 'profilename', '../Exp/MassFlux1.exp', 'definitionstring', 'Outputdefinition1'),
+                                   massfluxatgate('name', 'MassFlux2', 'profilename', '../Exp/MassFlux2.exp', 'definitionstring', 'Outputdefinition2'),
+                                   massfluxatgate('name', 'MassFlux3', 'profilename', '../Exp/MassFlux3.exp', 'definitionstring', 'Outputdefinition3'),
+                                   massfluxatgate('name', 'MassFlux4', 'profilename', '../Exp/MassFlux4.exp', 'definitionstring', 'Outputdefinition4'),
+                                   massfluxatgate('name', 'MassFlux5', 'profilename', '../Exp/MassFlux5.exp', 'definitionstring', 'Outputdefinition5'),
+                                   massfluxatgate('name', 'MassFlux6', 'profilename', '../Exp/MassFlux6.exp', 'definitionstring', 'Outputdefinition6')]
+
+md = solve(md, 'Stressbalance')
+
+#Fields and tolerances to track changes
+field_names = ['Vx', 'Vy', 'Vel', 'Pressure',
+               'DeviatoricStressxx', 'DeviatoricStressyy', 'DeviatoricStressxy',
+               'MassFlux1', 'MassFlux2', 'MassFlux3', 'MassFlux4', 'MassFlux5', 'MassFlux6']
+field_tolerances = [3e-13, 1e-13, 1e-13, 1e-13,
+                    2e-13, 1e-13, 2e-13,
+                    1e-13, 1e-13, 1e-13,
+                    1e-13, 1e-13, 1e-13]
+field_values = [md.results.StressbalanceSolution.Vx,
+                md.results.StressbalanceSolution.Vy,
+                md.results.StressbalanceSolution.Vel,
+                md.results.StressbalanceSolution.Pressure,
+                md.results.StressbalanceSolution.DeviatoricStressxx,
+                md.results.StressbalanceSolution.DeviatoricStressyy,
+                md.results.StressbalanceSolution.DeviatoricStressxy,
+                md.results.StressbalanceSolution.MassFlux1,
+                md.results.StressbalanceSolution.MassFlux2,
+                md.results.StressbalanceSolution.MassFlux3,
+                md.results.StressbalanceSolution.MassFlux4,
+                md.results.StressbalanceSolution.MassFlux5,
+                md.results.StressbalanceSolution.MassFlux6]
+}}}
+
+- Committed `$ISSM_DIR/test/NightlyRun/test101eisnasasmce.py` to repo
+
+= TODO
+- When okayed by Denis, remove,
+  - `/efs/issm`
+  - `/issm/ext`
+  - `/issm/repos`
+  - `/shared/issm`
+  
+  then, move `/efs/issm-new` to `/efs/issm`
+- Correct link to `libpython` on Daskhub (should not have to `LD_PRELOAD libpython3.9.so.1.0`)
+- Correct link to `libproj` on Daskhub (see `ldd $ISSM_DIR/lib/IssmConfig_python.so`; there is a link to `libproj.so.25`)
+- After SMCE team has created `issm` group, modify `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce.py::LaunchQueueJob` so that tarball is moved rather than copied (look at `/efs/issm/issm/trunk-jpl-denis/src/m/classes/clusters/generic.py` for examples of `cp` as opposed to `scp`)
+
+== For SMCE team
+- Run,
+{{{
+ln -s /srv/conda/envs/notebook/include/gstreamer-1.0/gst/gl/gl.h  /srv/conda/envs/notebook/include/GL/gl.h
+}}}
+  on Daskhub and bake resulting symbolic link onto base image
+- Create group `issm` on SMCE head node
+- Change group ownership of `/efs/issm/tmp` to `issm` group
+- Review `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce.py` on Daskhub, in particular
+  - `self.processor`
+  - `self.interactive`
+  - `self.numstreams`
+  - `self.hyperthreading`
+- Can we support for `self.email`?