= Installation on SMCE head node (external packages, binaries only) - Added the following to `~/.bashrc`, {{{ # Spack Modules source /shared/spack/share/spack/setup-env.sh module load intelmpi #module load python-3.8.12-gcc-9.4.0-uyaszn4 module load subversion-1.14.0-gcc-11.1.0-evgneyf ################### ### Environment ### ################### export CC=/opt/intel/mpi/2021.4.0/bin/mpicc export GCC=/opt/intel/mpi/2021.4.0/bin/mpicxx export FC=/opt/intel/mpi/2021.4.0/bin/mpifc export MPI_HOME=/opt/intel/mpi/2021.4.0 ###################### ### Path Variables ### ###################### # ISSM export ISSM_DIR="/efs/issm/binaries/repos/trunk-jpl-working" export ISSM_EXT_DIR="/efs/issm/binaries/ext" }}} - Checked out a copy of the ISSM development repository with, {{{ mkdir -p /efs/issm/binaries/repos svn co https://issm.ess.uci.edu/svn/issm/issm/trunk-jpl $ISSM_DIR }}} - Copied `$ISSM_DIR/jenkins/ross-debian_linux-full` to `$ISSM_DIR/jenkins/eis-smce-binaries` - Modified `$ISSM_DIR/jenkins/eis-smce-binaries` with, {{{ #--------------------# # ISSM Configuration # #--------------------# ISSM_CONFIG='\ --prefix="${ISSM_DIR}" \ --disable-static \ --with-wrappers=no \ --enable-development \ --enable-debugging \ --with-numthreads=48 \ --with-fortran-lib="-L/usr/lib/gcc/x86_64-linux-gnu/9 -lgfortran" \ --with-mpi-include="/opt/intel/mpi/2021.4.0/include" \ --with-mpi-libflags="-L/opt/intel/mpi/2021.4.0/lib -lmpi -lmpicxx -lmpifort" \ --with-blas-lapack-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-metis-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-parmetis-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-scalapack-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-mumps-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-hdf5-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-petsc-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-gsl-dir="${ISSM_EXT_DIR}/gsl/install" \ --with-boost-dir="${ISSM_EXT_DIR}/boost/install" \ --with-dakota-dir="${ISSM_EXT_DIR}/dakota/install" \ --with-proj-dir="${ISSM_EXT_DIR}/proj/install" \ --with-triangle-dir="${ISSM_EXT_DIR}/triangle/install" \ --with-chaco-dir="${ISSM_EXT_DIR}/chaco/install" \ --with-m1qn3-dir="${ISSM_EXT_DIR}/m1qn3/install" \ --with-semic-dir="${ISSM_EXT_DIR}/semic/install" ' #-------------------# # External Packages # #-------------------# EXTERNALPACKAGES=" autotools install-linux.sh cmake install.sh petsc install-3.12-linux.sh gsl install.sh boost install-1.7-linux.sh dakota install-6.2-linux.sh curl install-7-linux.sh netcdf install-4.7-parallel.sh sqlite install.sh proj install-6.sh gdal install-3.sh gshhg install.sh gmt install-6-linux.sh gmsh install-4-linux.sh triangle install-linux.sh chaco install.sh m1qn3 install.sh semic install.sh " }}} - Committed `$ISSM_DIR/jenkins/eis-smce-head-node-binaries` to repo - Installed each of the external packages in `$ISSM_DIR/jenkins/eis-daskhub-python-modules`, after changing each of the installation paths with, {{{ PREFIX=${ISSM_EXT_DIR}//install }}} and making the following modifications, - Modified `$ISSM_DIR/externalpackages/petsc/install-3.12-linux.sh` by, - removing, {{{ --download-mpich=1 }}} - adding, {{{ --with-mpi-dir=/opt/intel/mpi/2021.4.0 --known-mpi-shared-libraries=1 }}} - `$ISSM_DIR/externalpackages/dakota/install-6.2-linux.sh` requires Python 2 to build, so either (1) cannot load Spack module for Python 3, or, (2) have to figure out how to override this in the install script (perhaps with `alias python=/usr/bin/python`) - Modified `$ISSM_DIR/externalpackages/gdal/install-3.sh` with, {{{ --with-curl="${CURL_ROOT}/bin/curl-config" }}} = Installation on MATLAB node (MATLAB modules) TO BE ADDED AFTER FURTHER INSTALLATION AND DEBUGGING = Installation on Daskhub (Python modules) ---- **NOTE**: Binaries are also compiled as we do not have a good way of disabling them. We could ignore them, remove them, or let users know that they're available for smaller test runs. ---- - Added the following to ~/.bash_profile, {{{ # File system modifications that are not currently part of image # # Create a link to gl.h so that the CMake configuration can find it. # Note that we need to add it here because it does not persist. if [ ! -f /srv/conda/envs/notebook/include/GL/gl.h ]; then ln -s /srv/conda/envs/notebook/include/gstreamer-1.0/gst/gl/gl.h /srv/conda/envs/notebook/include/GL/gl.h fi # Set permissions on private key chmod 400 ~/.ssh/eis-nasa-smce-head-node ## Environment # export OPENSSL_ROOT_DIR="/srv/conda/envs/notebook" ## Path Variables # # ISSM export ISSM_DIR="/efs/issm/python-modules/repos/trunk-jpl-working" export ISSM_EXT_DIR="/efs/issm/python-modules/ext" ## Aliases # alias issm-python-dev-env="source "\${ISSM_DIR}"/etc/environment.sh; export PYTHONPATH="\${ISSM_DIR}"/src/m/dev; export PYTHONSTARTUP="\${PYTHONPATH}"/devpath.py; export PYTHONUNBUFFERED=1; cd "\${ISSM_DIR}"/test/NightlyRun; LD_PRELOAD=/srv/conda/envs/notebook/lib/libpython3.9.so.1.0:"\${ISSM_EXT_DIR}"/petsc/install/lib/libmpifort.so.12 python" # Then run ./runme.py }}} - Checked out a copy of the ISSM development repository with, {{{ mkdir -p /efs/issm/python-modules/repos svn co https://issm.ess.uci.edu/svn/issm/issm/trunk-jpl $ISSM_DIR }}} - Copied `$ISSM_DIR/jenkins/ross-debian_linux-full` to `$ISSM_DIR/jenkins/eis-daskhub-python-modules` - Modified `$ISSM_DIR/jenkins/eis-daskhub-python-modules` with, {{{ #--------------------# # ISSM Configuration # #--------------------# ISSM_CONFIG='\ --prefix="${ISSM_DIR}" \ --disable-static \ --enable-development \ --enable-debugging \ --with-numthreads=4 \ --with-python-dir="/srv/conda/envs/notebook" \ --with-python-version="3.9" \ --with-python-numpy-dir="/srv/conda/envs/notebook/lib/python3.9/site-packages/numpy/core/include/numpy" \ --with-fortran-lib="-L/usr/lib/x86_64-linux-gnu -lgfortran" \ --with-mpi-include="${ISSM_EXT_DIR}/petsc/install/include" \ --with-mpi-libflags="-L${ISSM_EXT_DIR}/petsc/install/lib -lmpi -lmpicxx -lmpifort" \ --with-blas-lapack-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-metis-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-parmetis-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-scalapack-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-mumps-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-hdf5-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-petsc-dir="${ISSM_EXT_DIR}/petsc/install" \ --with-gsl-dir="${ISSM_EXT_DIR}/gsl/install" \ --with-boost-dir="${ISSM_EXT_DIR}/boost/install" \ --with-dakota-dir="${ISSM_EXT_DIR}/dakota/install" \ --with-proj-dir="${ISSM_EXT_DIR}/proj/install" \ --with-triangle-dir="${ISSM_EXT_DIR}/triangle/install" \ --with-chaco-dir="${ISSM_EXT_DIR}/chaco/install" \ --with-m1qn3-dir="${ISSM_EXT_DIR}/m1qn3/install" \ --with-semic-dir="${ISSM_EXT_DIR}/semic/install" \ ' #-------------------# # External Packages # #-------------------# EXTERNALPACKAGES=" autotools install-linux.sh cmake install.sh petsc install-3.16-linux.sh gsl install.sh boost install-1.7-linux.sh dakota install-6.2-linux.sh curl install-7-linux.sh netcdf install-4.7-parallel.sh sqlite install.sh proj install-6.sh gdal install-3-python.sh gshhg install.sh gmt install-6-linux.sh gmsh install-4-linux.sh triangle install-linux.sh chaco install.sh m1qn3 install.sh semic install.sh " }}} - Committed `$ISSM_DIR/jenkins/eis-daskhub-python-modules` to repo - Before installing any of the external packages, needed to disable the Conda environment with `conda deactivate` - Installed each of the external packages in `$ISSM_DIR/jenkins/eis-daskhub-python-modules`, after changing each of the installation paths with, {{{ PREFIX=${ISSM_EXT_DIR}//install }}} and making the following modifications - Needed to remove, {{{ --FFLAGS="-fallow-argument-mismatch" }}} from `$ISSM_DIR/externalpackages/petsc/install-3.16-petsc.sh` - Needed to modify `$ISSM_DIR/externalpackages/gdal/install-3-python.sh` with, {{{ export CPATH="${CPATH}:/srv/conda/envs/notebook/include" export LDFLAGS="-L${NETCDF_ROOT}/lib -lnetcdf" export LIBRARY_PATH="${LIBRARY_PATH}:/srv/conda/envs/notebook/lib" export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/srv/conda/envs/notebook/lib" }}} and, {{{ --with-sqlite3="${SQLITE_ROOT}" \ --with-libjson-c="/srv/conda/envs/notebook" }}} - Needed to modify `$ISSM_DIR/externalpackages/gmt/install-6-linux.sh` with, {{{ export CC=mpicc export CPATH="${CPATH}:/srv/conda/envs/notebook/include" export LIBRARY_PATH="${LIBRARY_PATH}:/srv/conda/envs/notebook/lib" export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/srv/conda/envs/notebook/lib" }}} - With Conda environment still disabled, ran, {{{ alias envsubst="/srv/conda/envs/notebook/bin/envsubst" jenkins/jenkins.sh jenkins/eis-daskhub-python-modules }}} - Ran, {{{ mkdir ~/.ssh ssh-keygen -f ~/.ssh/eis-nasa-smce }}} then gave contents of public key to SMCE team to add to SMCE head node - Ran, {{{ chmod 400 ~/.ssh/eis-nasa-smce }}} - Created `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce.py` with contents, {{{ import os import shutil import subprocess try: from eis_nasa_smce_settings import eis_nasa_smce_settings except ImportError: print('You need eis_nasa_smce_settings.py to proceed, check presence and sys.path') from fielddisplay import fielddisplay from helpers import * from IssmConfig import IssmConfig from issmssh import issmssh from MatlabFuncs import * from pairoptions import pairoptions class eis_nasa_smce(object): """EIS_NASA_SMCE cluster class definition Usage: cluster = eis_nasa_smce() cluster = eis_nasa_smce('np', 3) cluster = eis_nasa_scme('np', 3, 'login', 'username') """ def __init__(self, *args): # {{{ self.name = '52.10.233.96' self.login = 'jdquinn1' self.idfile = '~/.ssh/eis-nasa-smce' self.modules = ['intelmpi'] self.numnodes = 4 self.cpuspernode = 1 self.port = 0 self.time = 12 * 60 * 60 self.processor = 'skylake' self.partition = 'sealevel-c5xl-spot' self.srcpath = '/efs/issm-new/binaries/repos/trunk-jpl-working' self.extpkgpath = '/efs/issm-new/binaries/ext' self.codepath = '/efs/issm-new/binaries/repos/trunk-jpl-working/bin' self.executionpath = '~/issm-exec' self.interactive = 0 self.numstreams = 1 self.hyperthreading = 0 self.email = '' # Use provided options to change fields options = pairoptions(*args) # Initialize cluster using user settings if provided try: self = eis_nasa_smce_settings(self) except NameError: print('eis_nasa_smce_settings.py not found, using default settings') # OK get other fields self = options.AssignObjectFields(self) # }}} def __repr__(self): # {{{ # Display the object s = 'class eis_nasa_smce object\n' s += ' name: {}\n'.format(self.name) s += ' login: {}\n'.format(self.login) s += ' idfile: {}\n'.format(self.idfile) s += ' modules: {}\n'.format(strjoin(self.modules, ', ')) s += ' numnodes: {}\n'.format(self.numnodes) s += ' cpuspernode: {}\n'.format(self.cpuspernode) s += ' np: {}\n'.format(self.nprocs()) s += ' port: {}\n'.format(self.port) s += ' time: {}\n'.format(self.time) s += ' processor: {}\n'.format(self.processor) s += ' partition: {}\n'.format(self.partition) s += ' srcpath: {}\n'.format(self.srcpath) s += ' extpkgpath: {}\n'.format(self.extpkgpath) s += ' codepath: {}\n'.format(self.codepath) s += ' executionpath: {}\n'.format(self.executionpath) s += ' interactive: {}\n'.format(self.interactive) s += ' numstreams: {}\n'.format(self.numstreams) s += ' hyperthreading: {}\n'.format(self.hyperthreading) return s # }}} def nprocs(self): # {{{ return self.numnodes * self.cpuspernode # }}} def checkconsistency(self, md, solution, analyses): # {{{ # Now, check cluster.cpuspernode according to processor type if self.processor == 'skylake': if self.cpuspernode > 14 or self.cpuspernode < 1: md = md.checkmessage('cpuspernode should be between 1 and 14 for \'skyw\' processors in hyperthreading mode') else: md = md.checkmessage('unknown processor type, should be \'skylake\'') # Miscellaneous if not self.login: md = md.checkmessage('login empty') if self.port: md = md.checkmessage('port must be set to 0 as we do not have an SSH tunnel') if not self.codepath: md = md.checkmessage('codepath empty') if not self.executionpath: md = md.checkmessage('executionpath empty') return self # }}} def BuildQueueScript(self, dirname, modelname, solution, io_gather, isvalgrind, isgprof, isdakota, isoceancoupling): # {{{ if isgprof: print('gprof not supported by cluster, ignoring...') issmexec = 'issm.exe' mpiexec = 'mpiexec' # Set to alternative mpiexec if desired if isdakota: version = IssmConfig('_DAKOTA_VERSION_')[0:2] version = float(str(version[0])) if version >= 6: issmexec = 'issm_dakota.exe' if isoceancoupling: issmexec = 'issm_ocean.exe' # Write queuing script fid = open(modelname + '.queue', 'w') fid.write('#!/bin/bash\n') fid.write('#SBATCH --partition={} \n'.format(self.partition)) fid.write('#SBATCH -J {} \n'.format(modelname)) fid.write('#SBATCH -o {}.outlog \n'.format(modelname)) fid.write('#SBATCH -e {}.errlog \n'.format(modelname)) fid.write('#SBATCH --nodes={} \n'.format(self.numnodes)) fid.write('#SBATCH --ntasks-per-node={} \n'.format(self.cpuspernode)) fid.write('#SBATCH --cpus-per-task={} \n'.format(self.numstreams)) fid.write('#SBATCH -t {:02d}:{:02d}:00 \n'.format(int(floor(self.time / 3600)), int(floor(self.time % 3600) / 60))) if (self.email.find('@')>-1): fid.write('#SBATCH --mail-user={} \n'.format(self.email)) fid.write('#SBATCH --mail-type=BEGIN,END,FAIL \n\n') fid.write('source /etc/profile\n') fid.write('source /shared/spack/share/spack/setup-env.sh\n') for i in range(len(self.modules)): fid.write('module load {} &> /dev/null\n'.format(self.modules[i])) fid.write('export MPI_GROUP_MAX=64\n\n') fid.write('export MPI_UNBUFFERED_STDIO=true\n\n') fid.write('export PATH="$PATH:/opt/slurm/bin"\n') fid.write('export PATH="$PATH:."\n\n') fid.write('export ISSM_DIR="{}"\n'.format(self.srcpath)) fid.write('export ISSM_EXT_DIR="{}"\n'.format(self.extpkgpath)) fid.write('source $ISSM_DIR/etc/environment.sh\n') fid.write('cd {}/{}/\n\n'.format(self.executionpath, dirname)) fid.write('{} -n {} {}/{} {} {}/{} {}\n'.format(mpiexec, self.nprocs(), self.codepath, issmexec, solution, self.executionpath, dirname, modelname)) if not io_gather: # concatenate the output files fid.write('cat {}.outbin.* > {}.outbin'.format(modelname, modelname)) fid.close() # In interactive mode, create a run file, and errlog and outlog file if self.interactive: fid = open(modelname + '.run', 'w') if not isvalgrind: fid.write('{} -np {} {}/{} {} {}/{} {}\n'.format(mpiexec, self.nprocs(), self.codepath, issmexec, solution, self.executionpath, dirname, modelname)) else: fid.write('{} -np {} valgrind --leak-check=full {}/{} {} {}/{} {}\n'.format(mpiexec, self.nprocs(), self.codepath, issmexec, solution, self.executionpath, dirname, modelname)) if not io_gather: # concatenate the output files fid.write('cat {}.outbin.* > {}.outbin'.format(modelname, modelname)) fid.close() fid = open(modelname + '.errlog', 'w') # TODO: Change this to system call (touch )? fid.close() fid = open(modelname + '.outlog', 'w') # TODO: Change this to system call (touch )? fid.close() # }}} def UploadQueueJob(self, modelname, dirname, filelist): # {{{ # Compress the files into one zip compressstring = 'tar -zcf {}.tar.gz'.format(dirname) for file in filelist: compressstring += ' {}'.format(file) if self.interactive: compressstring += ' {}.run {}.errlog {}.outlog'.format(modelname, modelname, modelname) subprocess.call(compressstring, shell=True) print('uploading input file and queueing script') if self.interactive: directory = '{}/Interactive{}'.format(self.executionpath, self.interactive) else: directory = self.executionpath # NOTE: Replacement for issmscpout(self.name, directory, self.login, self.port, ['{}.tar.gz'.format(dirname)]) copystring = 'cp {}.tar.gz /efs/issm/tmp'.format(dirname) subprocess.call(copystring, shell=True) # }}} def LaunchQueueJob(self, modelname, dirname, filelist, restart, batch): # {{{ if self.interactive: if not isempty(restart): launchcommand = 'cd {}/Interactive{}'.format(self.executionpath, self.interactive) else: launchcommand = 'cd {}/Interactive{} && tar -zxf {}.tar.gz'.format(self.executionpath, self.interactive, dirname) else: if not isempty(restart): launchcommand = 'cd {} && cd {} && sbatch {}.queue'.format(self.executionpath, dirname, modelname) else: launchcommand = 'cd {} && rm -rf {} && mkdir {} && cd {} && cp /efs/issm/tmp/{}.tar.gz . && tar -zxf {}.tar.gz && /opt/slurm/bin/sbatch {}.queue'.format(self.executionpath, dirname, dirname, dirname, dirname, dirname, modelname) print('launching solution sequence on remote cluster') # NOTE: Replacement for issmssh(self.name, self.login, self.port, launchcommand) subprocess.call('ssh -l {} -i {} {} "{}"'.format(self.login, self.idfile, self.name, launchcommand), shell=True) # }}} def Download(self, dirname, filelist): # {{{ # Copy files from cluster to current directory # NOTE: Replacement for issmscpin(self.name, self.login, self.port, directory, filelist) directory = '{}/{}/'.format(self.executionpath, dirname) fileliststr = '{' + ','.join([str(x) for x in filelist]) + '}' downloadcommand = 'scp -i {} {}@{}:{} {}/.'.format(self.idfile, self.login, self.name, os.path.join(directory, fileliststr), os.getcwd()) subprocess.call(downloadcommand, shell=True) # }}} }}} - Committed `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce_head_node.py` to repo NOTE: - It is important that the directory indicated by `self.executionpath` exists on SMCE head node before launch - Either in `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce_head_node.py` or runme - `cluster.login` should be changed to your username - `cluster.executionpath` should be changed to desired execution path on SMCE head node - `cluster.partition` should be changed to desired AWS ECS instance type (discuss with SMCE team before changing) - Created `$ISSM_DIR/test/NightlyRun/test101eisnasasmce.py` with contents, {{{ #Test Name: SquareShelfConstrainedStressSSA2d from model import * from socket import gethostname from triangle import triangle from setmask import setmask from parameterize import parameterize from setflowequation import setflowequation from solve import solve from massfluxatgate import massfluxatgate from generic import generic from eis_nasa_smce import eis_nasa_smce md = triangle(model(), '../Exp/Square.exp', 50000) md = setmask(md, 'all', '') md = parameterize(md, '../Par/SquareShelfConstrained.py') md = setflowequation(md, 'SSA', 'all') md.cluster = generic('name', gethostname(), 'np', 2) if True: cluster = eis_nasa_smce() cluster.partition = 'sealevel-c5xl-spot' md.cluster = cluster #outputs md.stressbalance.requested_outputs = ['default', 'DeviatoricStressxx', 'DeviatoricStressyy', 'DeviatoricStressxy', 'MassFlux1', 'MassFlux2', 'MassFlux3', 'MassFlux4', 'MassFlux5', 'MassFlux6'] md.outputdefinition.definitions = [massfluxatgate('name', 'MassFlux1', 'profilename', '../Exp/MassFlux1.exp', 'definitionstring', 'Outputdefinition1'), massfluxatgate('name', 'MassFlux2', 'profilename', '../Exp/MassFlux2.exp', 'definitionstring', 'Outputdefinition2'), massfluxatgate('name', 'MassFlux3', 'profilename', '../Exp/MassFlux3.exp', 'definitionstring', 'Outputdefinition3'), massfluxatgate('name', 'MassFlux4', 'profilename', '../Exp/MassFlux4.exp', 'definitionstring', 'Outputdefinition4'), massfluxatgate('name', 'MassFlux5', 'profilename', '../Exp/MassFlux5.exp', 'definitionstring', 'Outputdefinition5'), massfluxatgate('name', 'MassFlux6', 'profilename', '../Exp/MassFlux6.exp', 'definitionstring', 'Outputdefinition6')] md = solve(md, 'Stressbalance') #Fields and tolerances to track changes field_names = ['Vx', 'Vy', 'Vel', 'Pressure', 'DeviatoricStressxx', 'DeviatoricStressyy', 'DeviatoricStressxy', 'MassFlux1', 'MassFlux2', 'MassFlux3', 'MassFlux4', 'MassFlux5', 'MassFlux6'] field_tolerances = [3e-13, 1e-13, 1e-13, 1e-13, 2e-13, 1e-13, 2e-13, 1e-13, 1e-13, 1e-13, 1e-13, 1e-13, 1e-13] field_values = [md.results.StressbalanceSolution.Vx, md.results.StressbalanceSolution.Vy, md.results.StressbalanceSolution.Vel, md.results.StressbalanceSolution.Pressure, md.results.StressbalanceSolution.DeviatoricStressxx, md.results.StressbalanceSolution.DeviatoricStressyy, md.results.StressbalanceSolution.DeviatoricStressxy, md.results.StressbalanceSolution.MassFlux1, md.results.StressbalanceSolution.MassFlux2, md.results.StressbalanceSolution.MassFlux3, md.results.StressbalanceSolution.MassFlux4, md.results.StressbalanceSolution.MassFlux5, md.results.StressbalanceSolution.MassFlux6] }}} - Committed `$ISSM_DIR/test/NightlyRun/test101eisnasasmce.py` to repo = TODO - When okayed by Denis, remove, - `/efs/issm` - `/issm/ext` - `/issm/repos` - `/shared/issm` then, move `/efs/issm-new` to `/efs/issm` - Correct link to `libpython` on Daskhub (should not have to `LD_PRELOAD libpython3.9.so.1.0`) - Correct link to `libproj` on Daskhub (see `ldd $ISSM_DIR/lib/IssmConfig_python.so`; there is a link to `libproj.so.25`) - After SMCE team has created `issm` group, modify `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce.py::LaunchQueueJob` so that tarball is moved rather than copied (look at `/efs/issm/issm/trunk-jpl-denis/src/m/classes/clusters/generic.py` for examples of `cp` as opposed to `scp`) == For SMCE team - Run, {{{ ln -s /srv/conda/envs/notebook/include/gstreamer-1.0/gst/gl/gl.h /srv/conda/envs/notebook/include/GL/gl.h }}} on Daskhub and bake resulting symbolic link onto base image - Create group `issm` on SMCE head node - Change group ownership of `/efs/issm/tmp` to `issm` group - Review `$ISSM_DIR/src/m/classes/clusters/eis_nasa_smce.py` on Daskhub, in particular - `self.processor` - `self.interactive` - `self.numstreams` - `self.hyperthreading` - Can we support for `self.email`?