New cluster setup -- starcluster woes

rlaw

Hi ISSM community,

Changing computing resources have meant I needed to install ISSM again. That bit at least is fairly smooth now.

However, I now have two ISSM installations which I am trying to get to talk to one another, one on my linux laptop using Matlab, and one on the fairly new NREC (Norwegian Research and Education Cloud) cluster with the --no-wrappers flag.

I started constructing a new class for NREC following src/m/classes/clusters/camhpc.m and am learning a fair bit from going through the pre-existing classes.

I get to the issmstscpout.m function called within UploadQueueJob which calls starcluster on the command line (system([ starcluster() ' put ' host ' -u ' login ' ' string ' ' path]);). The ISSM documentation mentions starcluster with reference to Amazon Web Services, but I think it's also essential for other cluster features? However, I am receiving hard to decipher error messages (examples below) and struggling to get a starcluster installation set up on my machine, and it seems the last GitHub update for the package was some years ago (2016).

As starcluster seems to be used to put the .tar.gz onto the cluster, I think the path of least resistance here would be to use an sftp command or similar to transfer files, but I first want to check if this is a known issue and if there is a known workaround.

Thanks!!
Rob

example starcluster installation error output for version 0.91 (the latest version is even further from successfully installing) using pip:

Building wheels for collected packages: starcluster
  Building wheel for starcluster (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/kinak/Documents/svn/ISSM/trunk/cluster-env/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3m25d2ic/starcluster/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3m25d2ic/starcluster/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-jd73wedu
       cwd: /tmp/pip-install-3m25d2ic/starcluster/
  Complete output (6 lines):
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help
  
  error: invalid command 'bdist_wheel'
  ----------------------------------------
  ERROR: Failed building wheel for starcluster
  Running setup.py clean for starcluster
Failed to build starcluster
Installing collected packages: starcluster
    Running setup.py install for starcluster ... done
Successfully installed starcluster-0.91
(cluster-env) kinak@kinak-Latitude-7430:~/Documents/svn/ISSM/trunk$ starcluster
  File "/home/kinak/Documents/svn/ISSM/trunk/cluster-env/bin/starcluster", line 8
    print "Interrupted, exiting."
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Interrupted, exiting.")?

and when I enter starcluster at the command line I recieve

  File "/home/kinak/Documents/svn/ISSM/trunk/cluster-env/bin/starcluster", line 8
    print "Interrupted, exiting."
          ^
SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Interrupted, exiting.")?

justinquinn

Hi Rob,
You might check out the various cluster setup notes compiled here to see if there are any that use starcluster: https://issm.ess.uci.edu/trac/issm/wiki

I personally have not installed that package before, but it looks like it did not build (though it claims it was installed) based on the error message. I would look up error: invalid command 'bdist_wheel'. Perhaps the command needs to be run with sudo, or you need to do some PATH manipulation before running setup.py? Could also be that the script is outdated and intended for Python 2 rather than 3.

rlaw

Hi Justin,

Thanks for the quick reply! I think the problem is that all the classes seem to use starcluster! And yes, I think that starcluster may be in Python2 causing the installation error.

I think I will just go about writing an sftp based function to replace it on my system -- 🤞 should be straightforward!

-------------edit---------------
No, it looks like I was looking at the wrong function, src/m/os/issmscpout.m is the one to use and does not use starcluster. Thanks!
------------end edit-------------

Cheers,
Rob

justinquinn

rlaw
Hi Rob,
Glad to hear that it sounds like you got it worked out. Yes, sftp or scp should both work, generally, so use whatever protocol(s) the head node supports and that you have credentials for.