![(please configure the [header_logo] section in trac.ini)](/trac/issm/chrome/common/trac_banner.png)
Changeset 25896
- Timestamp:
- 12/18/20 16:06:19 (4 years ago)
- Location:
- issm/trunk-jpl/packagers
- Files:
-
- 3 deleted
- 7 edited
- 3 copied
-
linux/complete-issm-linux-binaries-matlab.sh (copied) (copied from issm/trunk-jpl/packagers/mac/complete-issm-mac-binaries-matlab.sh ) (6 diffs)
-
linux/complete-issm-linux-binaries-python.sh (copied) (copied from issm/trunk-jpl/packagers/mac/complete-issm-mac-binaries-matlab.sh ) (6 diffs)
-
linux/package-issm-linux-binaries-matlab.sh (modified) (3 diffs)
-
linux/package-issm-linux-binaries-python.sh (modified) (3 diffs)
-
linux/package-issm-linux-binaries-solid_earth.sh (deleted)
-
linux/package-issm-linux-binaries-with_dakota.sh (deleted)
-
linux/package-issm-linux-binaries.sh (deleted)
-
linux/transfer-issm-linux-binaries.sh (copied) (copied from issm/trunk-jpl/packagers/mac/transfer-issm-mac-binaries.sh ) (2 diffs)
-
mac/complete-issm-mac-binaries-matlab.sh (modified) (2 diffs)
-
mac/complete-issm-mac-binaries-python.sh (modified) (2 diffs)
-
mac/package-issm-mac-binaries-matlab.sh (modified) (1 diff)
-
mac/package-issm-mac-binaries-python.sh (modified) (1 diff)
-
mac/transfer-issm-mac-binaries.sh (modified) (1 diff)
Legend:
- Unmodified
- Added
- Removed
-
issm/trunk-jpl/packagers/linux/complete-issm-linux-binaries-matlab.sh
r25894 r25896 2 2 3 3 ################################################################################ 4 # Wrapper script to build, package, send for signing, and transfer to ISSM Web5 # site ISSM distributable package for macOSwith MATLAB API.4 # Wrapper script to build, package, and transfer to ISSM Web site ISSM 5 # distributable package for Linux with MATLAB API. 6 6 # 7 7 # Normally, we would put this directly into the project configuration under … … 11 11 # When no failures/errors occur, performs the following: 12 12 # - Builds ISSM according to configuration. 13 # - Packages and compressesexecutables and libraries.13 # - Packages executables and libraries. 14 14 # - Runs test suite against package. 15 # - Commits compressed package to repository to be signed by JPL Cybersecurity. 16 # - Retrieves signed package and transmits it to ISSM Web site for 17 # distribution. 15 # - Transmits package to ISSM Web site for distribution. 18 16 # 19 17 # Options: 20 18 # -b/--skipbuild Skip ISSM compilation. 21 # -r/--resign Skip ISSM compilation and packaging. Use to retrigger22 # signing/notarization if it fails but build and package23 # are valid.24 19 # -s/--skiptests Skip ISSM compilation and testing during packaging 25 # step. Use if 20 # step. Use if packaging fails for some reason but build 21 # is valid. 26 22 # -t/--transferonly Transfer package to ISSM Web site only. Use if transfer 27 23 # fails for some reason to skip building, packaging, and 28 24 # signing. 29 # -u/--unlock Remove lock file from signed package repository. Use if30 # build is aborted to allow for subsequent fresh build.31 #32 # Debugging:33 # - Relies on a very tight handshake with project on remote JPL Cybersecurity34 # Jenkins server. Debugging may be perfomed locally by running,35 #36 # packagers/mac/sign-issm-mac-binaries-matlab.sh37 #38 # with "ISSM_BINARIES_USER" and "ISSM_BINARIES_PASS" hardcoded to Apple39 # Developer credentials.40 # - Removing stdout/stderr redirections to null device (> /dev/null 2>&1) can41 # help debug potential SVN issues.42 25 # 43 26 # NOTE: … … 47 30 # workspace. For example, if 'Source Code Management' -> 'Check-out Strategy' 48 31 # select menu is set to "Always check out a fresh copy". 49 # - Assumes that "ISSM_BINARIES_USER" and "ISSM_BINARIES_PASS" are set up in50 # the 'Bindings' section under a 'Username and password (separated)' binding51 # (requires 'Credentials Binding Plugin') with 'Credentials' select menu set52 # to "jenkins/****** (SVN repository for ISSM binaries)".53 32 ################################################################################ 54 33 55 34 ## Constants 56 35 # 57 PKG="ISSM-macOS-MATLAB" # Name of directory to copy distributable files to 58 SIGNED_REPO_COPY="./signed" 59 SIGNED_REPO_URL="https://issm.ess.uci.edu/svn/issm-binaries/mac/matlab/signed" 36 PKG="ISSM-Linux-MATLAB" # Name of directory to copy distributable files to 60 37 61 38 COMPRESSED_PKG="${PKG}.zip" … … 65 42 export COMPRESSED_PKG 66 43 export PKG 67 export SIGNED_REPO_COPY68 export SIGNED_REPO_URL69 44 70 45 ## Parse options … … 80 55 build=1 81 56 package=1 82 sign=183 57 transfer=1 84 58 85 59 if [ $# -eq 1 ]; then 86 60 case $1 in 87 -b|--skipbuild) build=0; shift ;; 88 -r|--resign) build=0; package=0; ;; 89 -s|--skiptests) build=0; ;; 90 -t|--transferonly) build=0; package=0; sign=0; ;; 91 -u|--unlock) build=0; package=0; transfer=0; ;; 92 *) echo "Unknown parameter passed: $1"; exit 1 ;; 61 -b|--skipbuild) build=0; shift ;; 62 -s|--skiptests) build=0; ;; 63 -t|--transferonly) build=0; package=0; ;; 64 *) echo "Unknown parameter passed: $1"; exit 1 ;; 93 65 esac 94 66 fi … … 96 68 # Build 97 69 if [ ${build} -eq 1 ]; then 98 ./jenkins/jenkins.sh ./jenkins/ pine_island-mac-binaries-matlab70 ./jenkins/jenkins.sh ./jenkins/ross-debian_linux-binaries-matlab 99 71 fi 100 72 101 73 # Package 102 74 if [ ${package} -eq 1 ]; then 103 ./packagers/mac/package-issm-mac-binaries-matlab.sh $1 104 shift # Clear $1 so that it is not passed to commit_for_signing script 105 fi 106 107 # Commit for signing 108 if [ ${sign} -eq 1 ]; then 109 ./packagers/mac/commit_for_signing-issm-mac-binaries-matlab.sh $1 75 ./packagers/linux/package-issm-linux-binaries-matlab.sh $1 110 76 fi 111 77 112 78 # Transfer distributable package to ISSM Web site 113 79 if [ ${transfer} -eq 1 ]; then 114 ./packagers/ mac/transfer-issm-mac-binaries.sh80 ./packagers/linux/transfer-issm-linux-binaries.sh 115 81 fi 116 82 -
issm/trunk-jpl/packagers/linux/complete-issm-linux-binaries-python.sh
r25894 r25896 2 2 3 3 ################################################################################ 4 # Wrapper script to build, package, send for signing, and transfer to ISSM Web5 # site ISSM distributable package for macOS with MATLABAPI.4 # Wrapper script to build, package, and transfer to ISSM Web site ISSM 5 # distributable package for Linux with Python API. 6 6 # 7 7 # Normally, we would put this directly into the project configuration under … … 11 11 # When no failures/errors occur, performs the following: 12 12 # - Builds ISSM according to configuration. 13 # - Packages and compressesexecutables and libraries.13 # - Packages executables and libraries. 14 14 # - Runs test suite against package. 15 # - Commits compressed package to repository to be signed by JPL Cybersecurity. 16 # - Retrieves signed package and transmits it to ISSM Web site for 17 # distribution. 15 # - Transmits package to ISSM Web site for distribution. 18 16 # 19 17 # Options: 20 18 # -b/--skipbuild Skip ISSM compilation. 21 # -r/--resign Skip ISSM compilation and packaging. Use to retrigger22 # signing/notarization if it fails but build and package23 # are valid.24 19 # -s/--skiptests Skip ISSM compilation and testing during packaging 25 # step. Use if 20 # step. Use if packaging fails for some reason but build 21 # is valid. 26 22 # -t/--transferonly Transfer package to ISSM Web site only. Use if transfer 27 23 # fails for some reason to skip building, packaging, and 28 24 # signing. 29 # -u/--unlock Remove lock file from signed package repository. Use if30 # build is aborted to allow for subsequent fresh build.31 #32 # Debugging:33 # - Relies on a very tight handshake with project on remote JPL Cybersecurity34 # Jenkins server. Debugging may be perfomed locally by running,35 #36 # packagers/mac/sign-issm-mac-binaries-matlab.sh37 #38 # with "ISSM_BINARIES_USER" and "ISSM_BINARIES_PASS" hardcoded to Apple39 # Developer credentials.40 # - Removing stdout/stderr redirections to null device (> /dev/null 2>&1) can41 # help debug potential SVN issues.42 25 # 43 26 # NOTE: … … 47 30 # workspace. For example, if 'Source Code Management' -> 'Check-out Strategy' 48 31 # select menu is set to "Always check out a fresh copy". 49 # - Assumes that "ISSM_BINARIES_USER" and "ISSM_BINARIES_PASS" are set up in50 # the 'Bindings' section under a 'Username and password (separated)' binding51 # (requires 'Credentials Binding Plugin') with 'Credentials' select menu set52 # to "jenkins/****** (SVN repository for ISSM binaries)".53 32 ################################################################################ 54 33 55 34 ## Constants 56 35 # 57 PKG="ISSM-macOS-MATLAB" # Name of directory to copy distributable files to 58 SIGNED_REPO_COPY="./signed" 59 SIGNED_REPO_URL="https://issm.ess.uci.edu/svn/issm-binaries/mac/matlab/signed" 36 PKG="ISSM-Linux-Python" # Name of directory to copy distributable files to 60 37 61 38 COMPRESSED_PKG="${PKG}.zip" … … 65 42 export COMPRESSED_PKG 66 43 export PKG 67 export SIGNED_REPO_COPY68 export SIGNED_REPO_URL69 44 70 45 ## Parse options … … 80 55 build=1 81 56 package=1 82 sign=183 57 transfer=1 84 58 85 59 if [ $# -eq 1 ]; then 86 60 case $1 in 87 -b|--skipbuild) build=0; shift ;; 88 -r|--resign) build=0; package=0; ;; 89 -s|--skiptests) build=0; ;; 90 -t|--transferonly) build=0; package=0; sign=0; ;; 91 -u|--unlock) build=0; package=0; transfer=0; ;; 92 *) echo "Unknown parameter passed: $1"; exit 1 ;; 61 -b|--skipbuild) build=0; shift ;; 62 -s|--skiptests) build=0; ;; 63 -t|--transferonly) build=0; package=0; ;; 64 *) echo "Unknown parameter passed: $1"; exit 1 ;; 93 65 esac 94 66 fi … … 96 68 # Build 97 69 if [ ${build} -eq 1 ]; then 98 ./jenkins/jenkins.sh ./jenkins/ pine_island-mac-binaries-matlab70 ./jenkins/jenkins.sh ./jenkins/ross-debian_linux-binaries-python 99 71 fi 100 72 101 73 # Package 102 74 if [ ${package} -eq 1 ]; then 103 ./packagers/mac/package-issm-mac-binaries-matlab.sh $1 104 shift # Clear $1 so that it is not passed to commit_for_signing script 105 fi 106 107 # Commit for signing 108 if [ ${sign} -eq 1 ]; then 109 ./packagers/mac/commit_for_signing-issm-mac-binaries-matlab.sh $1 75 ./packagers/linux/package-issm-linux-binaries-python.sh $1 110 76 fi 111 77 112 78 # Transfer distributable package to ISSM Web site 113 79 if [ ${transfer} -eq 1 ]; then 114 ./packagers/ mac/transfer-issm-mac-binaries.sh80 ./packagers/linux/transfer-issm-linux-binaries.sh 115 81 fi 116 82 -
issm/trunk-jpl/packagers/linux/package-issm-linux-binaries-matlab.sh
r25809 r25896 2 2 3 3 ################################################################################ 4 # To be used after running, 5 # 6 # ${ISSM_DIR}/jenkins/jenkins.sh ${ISSM_DIR}/jenkins/ross-debian_linux-binaries-matlab 7 # 8 # in the context of a Jenkins project. 9 # 10 # When no runtime errors occur, performs the following: 11 # - Checks resulting executables and libraries against test suite. 12 # - Packages and compresses executables and libraries. 13 # - Transmits it to ISSM Web site for distribution. 4 # Packages and tests ISSM distributable package for Linux with MATLAB API. 14 5 # 15 6 # Options: 16 # -s/--skiptests Skip tests (use if this script fails for some reason 17 # after tests have successfully passed to save time) 18 # -t/--transferonly Transfer package to ISSM Web site only (use if 19 # transfer fails for some reason to skip testing and 20 # signing) 7 # -s/--skiptests Skip testing during packaging Use if packaging fails 8 # for some reason but build is valid. 9 # 10 # NOTE: 11 # - Assumes that the following constants are defined, 12 # 13 # COMPRESSED_PKG 14 # ISSM_DIR 15 # PKG 16 # 17 # See also: 18 # - packagers/linux/complete-issm-linux-binaries-matlab.sh 21 19 ################################################################################ 22 20 … … 34 32 MATLAB_NROPTIONS="'benchmark','all','exclude',[125,126,234,235,418,420,435,444,445,701,702,703,1101,1102,1103,1104,1105,1106,1107,1108,1109,1110,1201,1202,1203,1204,1205,1206,1207,1208,1301,1302,1303,1304,1401,1402,1601,1602,2002,2003,2005,2006,2010,2020,2021,2051,2052,2053,2101,2102,3001:3200,3201,3202,3300,3480,3481,4001,4002,4003]" # NOTE: Combination of test suites from basic, Dakota, and Solid Earth builds, with tests that require a restart and those that require the JVM excluded 35 33 MATLAB_PATH="/usr/local/MATLAB/R2019b" 36 PKG="ISSM-Linux-MATLAB" # Name of directory to copy distributable files to37 38 COMPRESSED_PKG="${PKG}.tar.gz"39 34 40 35 ## Environment … … 50 45 51 46 skip_tests=0 52 transfer_only=053 47 while [ $# -gt 0 ]; do 54 48 case $1 in 55 -s|--skiptests) skip_tests=1; shift ;; 56 -t|--transferonly) transfer_only=1; shift ;; 57 *) echo "Unknown parameter passed: $1"; exit 1 ;; 49 -s|--skiptests) skip_tests=1; ;; 50 *) echo "Unknown parameter passed: $1"; exit 1 ;; 58 51 esac 59 52 shift 60 53 done 61 54 62 if [ ${transfer_only} -eq 0 ]; then 63 # Check if MATLAB exists 64 if ! [ -d ${MATLAB_PATH} ]; then 65 echo "${MATLAB_PATH} does not point to a MATLAB installation! Please modify MATLAB_PATH variable in $(basename $0) and try again." 55 # Check if MATLAB exists 56 if ! [ -d ${MATLAB_PATH} ]; then 57 echo "${MATLAB_PATH} does not point to a MATLAB installation! Please modify MATLAB_PATH variable in $(basename $0) and try again." 58 exit 1 59 fi 60 61 # Clean up from previous packaging 62 echo "Cleaning up existing assets" 63 cd ${ISSM_DIR} 64 rm -rf ${PKG} ${COMPRESSED_PKG} 65 mkdir ${PKG} 66 67 # Add/modify required binaries and libraries 68 cd ${ISSM_DIR}/bin 69 70 echo "Modify generic" 71 cat generic_static.m | sed -e "s/generic_static/generic/g" > generic.m 72 73 echo "Moving MPICH binaries to bin/" 74 if [ -f ${ISSM_DIR}/externalpackages/petsc/install/bin/mpiexec ]; then 75 cp ${ISSM_DIR}/externalpackages/petsc/install/bin/mpiexec . 76 cp ${ISSM_DIR}/externalpackages/petsc/install/bin/hydra_pmi_proxy . 77 elif [ -f ${ISSM_DIR}/externalpackages/mpich/install/bin/mpiexec ]; then 78 cp ${ISSM_DIR}/externalpackages/mpich/install/bin/mpiexec . 79 cp ${ISSM_DIR}/externalpackages/mpich/install/bin/hydra_pmi_proxy . 80 else 81 echo "MPICH not found" 82 exit 1 83 fi 84 85 echo "Moving GDAL binaries to bin/" 86 if [ -f ${ISSM_DIR}/externalpackages/gdal/install/bin/gdal-config ]; then 87 cp ${ISSM_DIR}/externalpackages/gdal/install/bin/gdalsrsinfo . 88 cp ${ISSM_DIR}/externalpackages/gdal/install/bin/gdaltransform . 89 else 90 echo "GDAL not found" 91 exit 1 92 fi 93 94 echo "Moving GMT binaries to bin/" 95 if [ -f ${ISSM_DIR}/externalpackages/gmt/install/bin/gmt-config ]; then 96 cp ${ISSM_DIR}/externalpackages/gmt/install/bin/gmt . 97 cp ${ISSM_DIR}/externalpackages/gmt/install/bin/gmtselect . 98 else 99 echo "GMT not found" 100 exit 1 101 fi 102 103 echo "Moving Gmsh binaries to bin/" 104 if [ -f ${ISSM_DIR}/externalpackages/gmsh/install/bin/gmsh ]; then 105 cp ${ISSM_DIR}/externalpackages/gmsh/install/bin/gmsh . 106 else 107 echo "Gmsh not found" 108 exit 1 109 fi 110 111 echo "Moving libgfortran to lib/" 112 cp ${LIBGFORTRAN} ${LIBGFORTRAN_DIST} 2> /dev/null 113 114 # Run tests 115 if [ ${skip_tests} -eq 0 ]; then 116 echo "Running tests" 117 cd ${ISSM_DIR}/test/NightlyRun 118 rm matlab.log 2> /dev/null 119 120 # Run tests, redirecting output to logfile and suppressing output to console 121 ${MATLAB_PATH}/bin/matlab -nojvm -nosplash -r "try, addpath ${ISSM_DIR}/bin ${ISSM_DIR}/lib; runme(${MATLAB_NROPTIONS}); exit; catch me,fprintf('%s',getReport(me)); exit; end" -logfile matlab.log &> /dev/null 122 123 # Check that MATLAB did not exit in error 124 matlabExitCode=`echo $?` 125 matlabExitedInError=`grep -E "Activation cannot proceed|Error in matlab_run" matlab.log | wc -l` 126 127 if [[ ${matlabExitCode} -ne 0 || ${matlabExitedInError} -ne 0 ]]; then 128 echo "----------MATLAB exited in error!----------" 129 cat matlab.log 130 echo "-----------End of matlab.log-----------" 131 132 # Clean up execution directory 133 rm -rf ${ISSM_DIR}/execution/* 134 66 135 exit 1 67 136 fi 68 137 69 # Clean up from previous packaging 70 echo "Cleaning up existing assets" 71 cd ${ISSM_DIR} 72 rm -rf ${PKG} ${COMPRESSED_PKG} 73 mkdir ${PKG} 138 # Check that all tests passed 139 numTestsFailed=`cat matlab.log | grep -c -e "FAILED|ERROR"` 74 140 75 # Add/modify required binaries and libraries 76 cd ${ISSM_DIR}/bin 77 78 echo "Modify generic" 79 cat generic_static.m | sed -e "s/generic_static/generic/g" > generic.m 80 81 echo "Moving MPICH binaries to bin/" 82 if [ -f ${ISSM_DIR}/externalpackages/petsc/install/bin/mpiexec ]; then 83 cp ${ISSM_DIR}/externalpackages/petsc/install/bin/mpiexec . 84 cp ${ISSM_DIR}/externalpackages/petsc/install/bin/hydra_pmi_proxy . 85 elif [ -f ${ISSM_DIR}/externalpackages/mpich/install/bin/mpiexec ]; then 86 cp ${ISSM_DIR}/externalpackages/mpich/install/bin/mpiexec . 87 cp ${ISSM_DIR}/externalpackages/mpich/install/bin/hydra_pmi_proxy . 141 if [[ ${numTestsFailed} -ne 0 ]]; then 142 echo "One or more tests FAILED" 143 exit 1 88 144 else 89 echo "MPICH not found" 90 exit 1 145 echo "All tests PASSED" 91 146 fi 92 93 echo "Moving GDAL binaries to bin/" 94 if [ -f ${ISSM_DIR}/externalpackages/gdal/install/bin/gdal-config ]; then 95 cp ${ISSM_DIR}/externalpackages/gdal/install/bin/gdalsrsinfo . 96 cp ${ISSM_DIR}/externalpackages/gdal/install/bin/gdaltransform . 97 else 98 echo "GDAL not found" 99 exit 1 100 fi 101 102 echo "Moving GMT binaries to bin/" 103 if [ -f ${ISSM_DIR}/externalpackages/gmt/install/bin/gmt-config ]; then 104 cp ${ISSM_DIR}/externalpackages/gmt/install/bin/gmt . 105 cp ${ISSM_DIR}/externalpackages/gmt/install/bin/gmtselect . 106 else 107 echo "GMT not found" 108 exit 1 109 fi 110 111 echo "Moving Gmsh binaries to bin/" 112 if [ -f ${ISSM_DIR}/externalpackages/gmsh/install/bin/gmsh ]; then 113 cp ${ISSM_DIR}/externalpackages/gmsh/install/bin/gmsh . 114 else 115 echo "Gmsh not found" 116 exit 1 117 fi 118 119 echo "Moving libgfortran to lib/" 120 cp ${LIBGFORTRAN} ${LIBGFORTRAN_DIST} 2> /dev/null 121 122 123 # Run tests 124 if [ ${skip_tests} -eq 0 ]; then 125 echo "Running tests" 126 cd ${ISSM_DIR}/test/NightlyRun 127 rm matlab.log 2> /dev/null 128 129 # Run tests, redirecting output to logfile and suppressing output to console 130 ${MATLAB_PATH}/bin/matlab -nojvm -nosplash -r "try, addpath ${ISSM_DIR}/bin ${ISSM_DIR}/lib; runme(${MATLAB_NROPTIONS}); exit; catch me,fprintf('%s',getReport(me)); exit; end" -logfile matlab.log &> /dev/null 131 132 # Check that MATLAB did not exit in error 133 matlabExitCode=`echo $?` 134 matlabExitedInError=`grep -E "Activation cannot proceed|license|Error" matlab.log | wc -l` 135 136 if [[ ${matlabExitCode} -ne 0 || ${matlabExitedInError} -ne 0 ]]; then 137 echo "----------MATLAB exited in error!----------" 138 cat matlab.log 139 echo "-----------End of matlab.log-----------" 140 141 # Clean up execution directory 142 rm -rf ${ISSM_DIR}/execution/* 143 144 exit 1 145 fi 146 147 # Check that all tests passed 148 numTestsFailed=`cat matlab.log | grep -c -e "FAILED|ERROR"` 149 150 if [[ ${numTestsFailed} -ne 0 ]]; then 151 echo "One or more tests FAILED" 152 exit 1 153 else 154 echo "All tests PASSED" 155 fi 156 else 157 echo "Skipping tests" 158 fi 159 160 # Create package 161 cd ${ISSM_DIR} 162 svn cleanup --remove-ignored --remove-unversioned test # Clean up test directory (before copying to package) 163 echo "Copying assets to package: ${PKG}" 164 cp -rf bin examples lib scripts test ${PKG}/ 165 mkdir ${PKG}/execution 166 echo "Cleaning up unneeded/unwanted files" 167 rm -f ${PKG}/bin/generic_static.* # Remove static versions of generic cluster classes 168 rm -f ${PKG}/lib/*.a # Remove static libraries from package 169 rm -f ${PKG}/lib/*.la # Remove libtool libraries from package 170 rm -rf ${PKG}/test/SandBox # Remove testing sandbox from package 171 172 # Compress package 173 echo "Compressing package" 174 tar -czf ${COMPRESSED_PKG} ${PKG} 175 ls -lah ${ISSM_DIR}/${COMPRESSED_PKG} 147 else 148 echo "Skipping tests" 176 149 fi 177 150 178 # Transfer package to ISSM Web site 179 echo "Transferring package to ISSM Web site" 180 scp -i ~/.ssh/debian_linux-vm_to_ross ${COMPRESSED_PKG} jenkins@ross.ics.uci.edu:/var/www/html/${COMPRESSED_PKG} 151 # Create package 152 cd ${ISSM_DIR} 153 svn cleanup --remove-ignored --remove-unversioned test # Clean up test directory (before copying to package) 154 echo "Copying assets to package: ${PKG}" 155 cp -rf bin examples lib scripts test ${PKG}/ 156 mkdir ${PKG}/execution 157 echo "Cleaning up unneeded/unwanted files" 158 rm -f ${PKG}/bin/generic_static.* # Remove static versions of generic cluster classes 159 rm -f ${PKG}/lib/*.a # Remove static libraries from package 160 rm -f ${PKG}/lib/*.la # Remove libtool libraries from package 161 rm -rf ${PKG}/test/SandBox # Remove testing sandbox from package 181 162 182 if [ $? -ne 0 ]; then 183 echo "Transfer failed! Verify connection then build this project again." 184 exit 1 185 fi 163 # Compress package 164 echo "Compressing package" 165 tar -czf ${COMPRESSED_PKG} ${PKG} -
issm/trunk-jpl/packagers/linux/package-issm-linux-binaries-python.sh
r25809 r25896 2 2 3 3 ################################################################################ 4 # To be used after running, 5 # 6 # ${ISSM_DIR}/jenkins/jenkins.sh ${ISSM_DIR}/jenkins/ross-debian_linux-binaries-matlab 7 # 8 # in the context of a Jenkins project. 9 # 10 # When no runtime errors occur, performs the following: 11 # - Checks resulting executables and libraries against test suite. 12 # - Packages and compresses executables and libraries. 13 # - Transmits it to ISSM Web site for distribution. 4 # Packages and tests ISSM distributable package for Linux with Python API. 14 5 # 15 6 # Options: 16 # -s/--skiptests Skip tests (use if this script fails for some reason 17 # after tests have successfully passed to save time) 18 # -t/--transferonly Transfer package to ISSM Web site only (use if 19 # transfer fails for some reason to skip testing and 20 # signing) 7 # -s/--skiptests Skip testing during packaging Use if packaging fails 8 # for some reason but build is valid. 9 # 10 # NOTE: 11 # - Assumes that the following constants are defined, 12 # 13 # COMPRESSED_PKG 14 # ISSM_DIR 15 # PKG 16 # 17 # See also: 18 # - packagers/linux/complete-issm-linux-binaries-python.sh 21 19 ################################################################################ 22 20 … … 33 31 LIBGFORTRAN_DIST="${ISSM_DIR}/lib/libgfortran.so.5" # Important the file name matches the SONAME entry in the binaries and other shared libraries which link to it 34 32 PYTHON_NROPTIONS="--benchmark all --exclude 125 126 234 235 418 420 435 444 445 701 702 703 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 1201 1202 1203 1204 1205 1206 1207 1208 1301 1302 1303 1304 1401 1402 1601 1602 2002 2003 2005 2006 2010 2020 2021 2051 2052 2053 2101 2102 3001:3200 3201 3202 3300 3480 3481 4001 4002 4003" # NOTE: Combination of test suites from basic, Dakota, and Solid Earth builds, with tests that require a restart and those that require the JVM excluded 35 PKG="ISSM-Linux-Python" # Name of directory to copy distributable files to36 37 COMPRESSED_PKG="${PKG}.tar.gz"38 33 39 34 ## Environment … … 49 44 50 45 skip_tests=0 51 transfer_only=052 46 while [ $# -gt 0 ]; do 53 47 case $1 in 54 -s|--skiptests) skip_tests=1; shift ;; 55 -t|--transferonly) transfer_only=1; shift ;; 56 *) echo "Unknown parameter passed: $1"; exit 1 ;; 48 -s|--skiptests) skip_tests=1; ;; 49 *) echo "Unknown parameter passed: $1"; exit 1 ;; 57 50 esac 58 51 shift 59 52 done 60 53 61 if [ ${transfer_only} -eq 0 ]; then 62 # Check if MATLAB exists 63 if ! [ -d ${MATLAB_PATH} ]; then 64 echo "${MATLAB_PATH} does not point to a MATLAB installation! Please modify MATLAB_PATH variable in $(basename $0) and try again." 54 # Clean up from previous packaging 55 echo "Cleaning up existing assets" 56 cd ${ISSM_DIR} 57 rm -rf ${PKG} ${COMPRESSED_PKG} 58 mkdir ${PKG} 59 60 # Add/modify required binaries and libraries 61 cd ${ISSM_DIR}/bin 62 63 echo "Modify generic" 64 cat generic_static.py | sed -e "s/generic_static/generic/g" > generic.py 65 66 echo "Moving MPICH binaries to bin/" 67 if [ -f ${ISSM_DIR}/externalpackages/petsc/install/bin/mpiexec ]; then 68 cp ${ISSM_DIR}/externalpackages/petsc/install/bin/mpiexec . 69 cp ${ISSM_DIR}/externalpackages/petsc/install/bin/hydra_pmi_proxy . 70 elif [ -f ${ISSM_DIR}/externalpackages/mpich/install/bin/mpiexec ]; then 71 cp ${ISSM_DIR}/externalpackages/mpich/install/bin/mpiexec . 72 cp ${ISSM_DIR}/externalpackages/mpich/install/bin/hydra_pmi_proxy . 73 else 74 echo "MPICH not found" 75 exit 1 76 fi 77 78 echo "Moving GDAL binaries to bin/" 79 if [ -f ${ISSM_DIR}/externalpackages/gdal/install/bin/gdal-config ]; then 80 cp ${ISSM_DIR}/externalpackages/gdal/install/bin/gdalsrsinfo . 81 cp ${ISSM_DIR}/externalpackages/gdal/install/bin/gdaltransform . 82 else 83 echo "GDAL not found" 84 exit 1 85 fi 86 87 echo "Moving GMT binaries to bin/" 88 if [ -f ${ISSM_DIR}/externalpackages/gmt/install/bin/gmt-config ]; then 89 cp ${ISSM_DIR}/externalpackages/gmt/install/bin/gmt . 90 cp ${ISSM_DIR}/externalpackages/gmt/install/bin/gmtselect . 91 else 92 echo "GMT not found" 93 exit 1 94 fi 95 96 echo "Moving Gmsh binaries to bin/" 97 if [ -f ${ISSM_DIR}/externalpackages/gmsh/install/bin/gmsh ]; then 98 cp ${ISSM_DIR}/externalpackages/gmsh/install/bin/gmsh . 99 else 100 echo "Gmsh not found" 101 exit 1 102 fi 103 104 echo "Moving libgfortran to lib/" 105 cp ${LIBGFORTRAN} ${LIBGFORTRAN_DIST} 2> /dev/null 106 107 108 # Run tests 109 if [ ${skip_tests} -eq 0 ]; then 110 echo "Running tests" 111 cd ${ISSM_DIR}/test/NightlyRun 112 rm python.log 2> /dev/null 113 114 # Set Python environment 115 export PYTHONPATH="${ISSM_DIR}/src/m/dev" 116 export PYTHONSTARTUP="${PYTHONPATH}/devpath.py" 117 export PYTHONUNBUFFERED=1 # We don't want Python to buffer output, otherwise issm.exe output is not captured 118 119 # Run tests, redirecting output to logfile and suppressing output to console 120 ./runme.py ${PYTHON_NROPTIONS} &> python.log 2>&1 121 122 # Check that Python did not exit in error 123 pythonExitCode=`echo $?` 124 pythonExitedInError=`grep -E "Error|Standard exception|Traceback|bad interpreter" python.log | wc -l` 125 126 if [[ ${pythonExitCode} -ne 0 || ${pythonExitedInError} -ne 0 ]]; then 127 echo "----------Python exited in error!----------" 128 cat python.log 129 echo "-----------End of python.log-----------" 130 131 # Clean up execution directory 132 rm -rf ${ISSM_DIR}/execution/* 133 65 134 exit 1 66 135 fi 67 136 68 # Clean up from previous packaging 69 echo "Cleaning up existing assets" 70 cd ${ISSM_DIR} 71 rm -rf ${PKG} ${COMPRESSED_PKG} 72 mkdir ${PKG} 137 # Check that all tests passed 138 numTestsFailed=`cat python.log | grep -c -e "FAILED|ERROR"` 73 139 74 # Add/modify required binaries and libraries 75 cd ${ISSM_DIR}/bin 76 77 echo "Modify generic" 78 cat generic_static.py | sed -e "s/generic_static/generic/g" > generic.py 79 80 echo "Moving MPICH binaries to bin/" 81 if [ -f ${ISSM_DIR}/externalpackages/petsc/install/bin/mpiexec ]; then 82 cp ${ISSM_DIR}/externalpackages/petsc/install/bin/mpiexec . 83 cp ${ISSM_DIR}/externalpackages/petsc/install/bin/hydra_pmi_proxy . 84 elif [ -f ${ISSM_DIR}/externalpackages/mpich/install/bin/mpiexec ]; then 85 cp ${ISSM_DIR}/externalpackages/mpich/install/bin/mpiexec . 86 cp ${ISSM_DIR}/externalpackages/mpich/install/bin/hydra_pmi_proxy . 140 if [[ ${numTestsFailed} -ne 0 ]]; then 141 echo "One or more tests FAILED" 142 exit 1 87 143 else 88 echo "MPICH not found" 89 exit 1 144 echo "All tests PASSED" 90 145 fi 91 92 echo "Moving GDAL binaries to bin/" 93 if [ -f ${ISSM_DIR}/externalpackages/gdal/install/bin/gdal-config ]; then 94 cp ${ISSM_DIR}/externalpackages/gdal/install/bin/gdalsrsinfo . 95 cp ${ISSM_DIR}/externalpackages/gdal/install/bin/gdaltransform . 96 else 97 echo "GDAL not found" 98 exit 1 99 fi 100 101 echo "Moving GMT binaries to bin/" 102 if [ -f ${ISSM_DIR}/externalpackages/gmt/install/bin/gmt-config ]; then 103 cp ${ISSM_DIR}/externalpackages/gmt/install/bin/gmt . 104 cp ${ISSM_DIR}/externalpackages/gmt/install/bin/gmtselect . 105 else 106 echo "GMT not found" 107 exit 1 108 fi 109 110 echo "Moving Gmsh binaries to bin/" 111 if [ -f ${ISSM_DIR}/externalpackages/gmsh/install/bin/gmsh ]; then 112 cp ${ISSM_DIR}/externalpackages/gmsh/install/bin/gmsh . 113 else 114 echo "Gmsh not found" 115 exit 1 116 fi 117 118 echo "Moving libgfortran to lib/" 119 cp ${LIBGFORTRAN} ${LIBGFORTRAN_DIST} 2> /dev/null 120 121 122 # Run tests 123 if [ ${skip_tests} -eq 0 ]; then 124 echo "Running tests" 125 cd ${ISSM_DIR}/test/NightlyRun 126 rm python.log 2> /dev/null 127 128 # Set Python environment 129 export PYTHONPATH="${ISSM_DIR}/src/m/dev" 130 export PYTHONSTARTUP="${PYTHONPATH}/devpath.py" 131 export PYTHONUNBUFFERED=1 # We don't want Python to buffer output, otherwise issm.exe output is not captured 132 133 # Run tests, redirecting output to logfile and suppressing output to console 134 ./runme.py ${PYTHON_NROPTIONS} &> python.log 2>&1 135 136 # Check that Python did not exit in error 137 pythonExitCode=`echo $?` 138 pythonExitedInError=`grep -E "Error|Standard exception|Traceback|bad interpreter" python.log | wc -l` 139 140 if [[ ${pythonExitCode} -ne 0 || ${pythonExitedInError} -ne 0 ]]; then 141 echo "----------Python exited in error!----------" 142 cat python.log 143 echo "-----------End of python.log-----------" 144 145 # Clean up execution directory 146 rm -rf ${ISSM_DIR}/execution/* 147 148 exit 1 149 fi 150 151 # Check that all tests passed 152 numTestsFailed=`cat python.log | grep -c -e "FAILED|ERROR"` 153 154 if [[ ${numTestsFailed} -ne 0 ]]; then 155 echo "One or more tests FAILED" 156 exit 1 157 else 158 echo "All tests PASSED" 159 fi 160 else 161 echo "Skipping tests" 162 fi 163 164 # Create package 165 cd ${ISSM_DIR} 166 svn cleanup --remove-ignored --remove-unversioned test # Clean up test directory (before copying to package) 167 echo "Copying assets to package: ${PKG}" 168 cp -rf bin examples lib scripts test ${PKG}/ 169 mkdir ${PKG}/execution 170 ${ISSM_DIR}/scripts/py_to_pyc.sh ${PKG}/bin # Compile Python source files 171 echo "Cleaning up unneeded/unwanted files" 172 rm -f ${PKG}/bin/*.py # Remove all Python scripts 173 rm -f ${PKG}/bin/generic_static.* # Remove static versions of generic cluster classes 174 rm -f ${PKG}/lib/*.a # Remove static libraries from package 175 rm -f ${PKG}/lib/*.la # Remove libtool libraries from package 176 rm -rf ${PKG}/test/SandBox # Remove testing sandbox from package 177 178 # Compress package 179 echo "Compressing package" 180 tar -czf ${COMPRESSED_PKG} ${PKG} 181 ls -lah ${ISSM_DIR}/${COMPRESSED_PKG} 146 else 147 echo "Skipping tests" 182 148 fi 183 149 184 # Transfer package to ISSM Web site 185 echo "Transferring package to ISSM Web site" 186 scp -i ~/.ssh/debian_linux-vm_to_ross ${COMPRESSED_PKG} jenkins@ross.ics.uci.edu:/var/www/html/${COMPRESSED_PKG} 150 # Create package 151 cd ${ISSM_DIR} 152 svn cleanup --remove-ignored --remove-unversioned test # Clean up test directory (before copying to package) 153 echo "Copying assets to package: ${PKG}" 154 cp -rf bin examples lib scripts test ${PKG}/ 155 mkdir ${PKG}/execution 156 ${ISSM_DIR}/scripts/py_to_pyc.sh ${PKG}/bin # Compile Python source files 157 echo "Cleaning up unneeded/unwanted files" 158 rm -f ${PKG}/bin/*.py # Remove all Python scripts 159 rm -f ${PKG}/bin/generic_static.* # Remove static versions of generic cluster classes 160 rm -f ${PKG}/lib/*.a # Remove static libraries from package 161 rm -f ${PKG}/lib/*.la # Remove libtool libraries from package 162 rm -rf ${PKG}/test/SandBox # Remove testing sandbox from package 187 163 188 if [ $? -ne 0 ]; then 189 echo "Transfer failed! Verify connection then build this project again." 190 exit 1 191 fi 164 # Compress package 165 echo "Compressing package" 166 tar -czf ${COMPRESSED_PKG} ${PKG} -
issm/trunk-jpl/packagers/linux/transfer-issm-linux-binaries.sh
r25894 r25896 2 2 3 3 ################################################################################ 4 # Transfers ISSM distributable package to ISSM Web site. 4 # Transfers ISSM distributable package for Linux with MATLAB API to ISSM Web 5 # site. 5 6 # 6 7 # NOTE: … … 8 9 # 9 10 # COMPRESSED_PKG 10 # ISSM_BINARIES_REPO_PASS11 # ISSM_BINARIES_REPO_USER12 # SIGNED_REPO_COPY13 # SIGNED_REPO_URL14 11 # 15 12 # See also: 16 # - packagers/ mac/complete-issm-mac-binaries-matlab.sh17 # - packagers/ mac/complete-issm-mac-binaries-python.sh13 # - packagers/linux/complete-issm-linux-binaries-matlab.sh 14 # - packagers/linux/complete-issm-linux-binaries-python.sh 18 15 ################################################################################ 19 16 20 # Expand aliases within the context of this script 21 shopt -s expand_aliases 22 23 # From https://developer.apple.com/documentation/macos-release-notes/macos-catalina-10_15-release-notes, 24 # 25 # Command line tool support for Subversion — including svn, git-svn, and 26 # related commands — is no longer provided by Xcode. (50266910) 27 # 28 # which results in, 29 # 30 # svn: error: The subversion command line tools are no longer provided by 31 # Xcode. 32 # 33 # when calling svn, even when subversion is installed via Homebrew and its path 34 # is available in PATH. 35 # 36 # NOTE: May be able to remove this after updating macOS. 37 # 38 alias svn='/usr/local/bin/svn' 39 40 ## Override certain other aliases 41 # 42 alias cp=$(which cp) 43 alias grep=$(which grep) 44 45 ## Functions 46 # 47 checkout_signed_repo_copy(){ 48 echo "Checking out copy of repository for signed packages" 49 50 # NOTE: Get empty copy because we do not want to have to check out package 51 # from previous signing. 52 # 53 svn checkout \ 54 --trust-server-cert \ 55 --non-interactive \ 56 --depth empty \ 57 --username ${ISSM_BINARIES_REPO_USER} \ 58 --password ${ISSM_BINARIES_REPO_USER} \ 59 ${SIGNED_REPO_URL} \ 60 ${SIGNED_REPO_COPY} > /dev/null 2>&1 61 } 62 validate_signed_repo_copy(){ 63 # Validate copy of repository for signed binaries (e.g. 64 # 'Check-out Strategy' was set to 'Use 'svn update' as much as possible'; 65 # initial checkout failed) 66 if [[ ! -d ${SIGNED_REPO_COPY} || ! -d ${SIGNED_REPO_COPY}/.svn ]]; then 67 rm -rf ${SIGNED_REPO_COPY} 68 checkout_signed_repo_copy 69 fi 70 } 71 72 # Check if working copy of repository for signed packages is missing 73 validate_signed_repo_copy 74 75 # Retrieve signed and notarized package 76 svn update \ 77 --username ${ISSM_BINARIES_REPO_USER} \ 78 --password ${ISSM_BINARIES_REPO_PASS} \ 79 ${SIGNED_REPO_COPY}/${COMPRESSED_PKG} 80 81 # Transfer signed package to ISSM Web site 82 echo "Transferring signed package to ISSM Web site" 83 scp -i ~/.ssh/pine_island_to_ross ${SIGNED_REPO_COPY}/${COMPRESSED_PKG} jenkins@ross.ics.uci.edu:/var/www/html/${COMPRESSED_PKG} 17 # Transfer package to ISSM Web site 18 echo "Transferring package to ISSM Web site" 19 scp -i ~/.ssh/debian_linux-vm_to_ross ${COMPRESSED_PKG} jenkins@ross.ics.uci.edu:/var/www/html/${COMPRESSED_PKG} 84 20 85 21 if [ $? -ne 0 ]; then 86 echo "Transfer failed! Verify connection then build this project again (with -t/--transferonly option to skip building , packaging, and signing)."22 echo "Transfer failed! Verify connection then build this project again (with -t/--transferonly option to skip building and packaging)." 87 23 exit 1 88 24 fi -
issm/trunk-jpl/packagers/mac/complete-issm-mac-binaries-matlab.sh
r25893 r25896 11 11 # When no failures/errors occur, performs the following: 12 12 # - Builds ISSM according to configuration. 13 # - Packages and compressesexecutables and libraries.13 # - Packages executables and libraries. 14 14 # - Runs test suite against package. 15 15 # - Commits compressed package to repository to be signed by JPL Cybersecurity. … … 23 23 # are valid. 24 24 # -s/--skiptests Skip ISSM compilation and testing during packaging 25 # step. Use if 25 # step. Use if packaging fails for some reason but build 26 # is valid. 26 27 # -t/--transferonly Transfer package to ISSM Web site only. Use if transfer 27 28 # fails for some reason to skip building, packaging, and -
issm/trunk-jpl/packagers/mac/complete-issm-mac-binaries-python.sh
r25893 r25896 11 11 # When no failures/errors occur, performs the following: 12 12 # - Builds ISSM according to configuration. 13 # - Packages and compressesexecutables and libraries.13 # - Packages executables and libraries. 14 14 # - Runs test suite against package. 15 15 # - Commits compressed package to repository to be signed by JPL Cybersecurity. … … 23 23 # are valid. 24 24 # -s/--skiptests Skip ISSM compilation and testing during packaging 25 # step. Use if 25 # step. Use if packaging fails for some reason but build 26 # is valid. 26 27 # -t/--transferonly Transfer package to ISSM Web site only. Use if transfer 27 28 # fails for some reason to skip building, packaging, and -
issm/trunk-jpl/packagers/mac/package-issm-mac-binaries-matlab.sh
r25884 r25896 5 5 # 6 6 # Options: 7 # -s/--skiptests Skip tests and package only. 7 # -s/--skiptests Skip testing during packaging Use if packaging fails 8 # for some reason but build is valid. 8 9 # 9 10 # NOTE: -
issm/trunk-jpl/packagers/mac/package-issm-mac-binaries-python.sh
r25893 r25896 5 5 # 6 6 # Options: 7 # -s/--skiptests Skip tests and package only. 7 # -s/--skiptests Skip testing during packaging Use if packaging fails 8 # for some reason but build is valid. 8 9 # 9 10 # NOTE: -
issm/trunk-jpl/packagers/mac/transfer-issm-mac-binaries.sh
r25893 r25896 2 2 3 3 ################################################################################ 4 # Transfers ISSM distributable package to ISSM Web site. 4 # Transfers ISSM distributable package for macOS with MATLAB API to ISSM Web 5 # site. 5 6 # 6 7 # NOTE:
Note:
See TracChangeset
for help on using the changeset viewer.
