== Introduction ==

ISSM relies on regression tests to check that no bug is introduced during code development. All the tests are located in `$ISSM_DIR/test/test/NightlyRun` and all the tests whose number are less than 1000 are run after each change pushed to the svn repository (through jenkins: https://ross.ics.uci.edu/jenkins/). See [wiki:jenkins] for more information.

If a new capability is being added to ISSM, it is *critical* that a new test is created.

These nightly runs are not meant to be realistic. They rely on very coarse meshes. We only want to check that the numerics has not changed, the results are obviously meaningless. Each test should take less than 10 seconds to run (ideally 1 second).

 == Running a test ==
Tests are organized as follows:
- 1-999: "Nightly" series:
  - 100s: Square ice shelf constrained (no ice front)
  - 200s: Square ice shelf with ice front
  - 300s: Square grounded ice constrained (no ice front)
  - 400s: Square grounded ice with ice front (some floating ice)
  - 500s: Pine Island Glacier (West Antarctica)
  - 600s: 79 North (Greenland)
  - 700s: flow band models
  - 800s: valley glaciers

- 1000-1999 "validation" series:
  - 1000s: ISMIP
  - 1200s: EISMINT
  - 1300s: Thermal validation
  - 1400s: Mesh
  - 1500s: Transient forcings
  - 1600s: Thermal validation

- 2000-2999: "sea level" series
- 3000-3999: "ADOLC" series

Go to `test/NightlyRun`. To run a test without checking whether it worked or not, just type: `testxxx` in MATLAB, where `xxx` is the test number. To check whether the results of that test are still consistent with the archive:

{{{
#!m
>> runme('id',XXX)
}}}

For tests that are >1000, you will need to specify that you want to use all the benchmarks:

{{{
#!m
>> runme('id',XXX,'benchmark','all')
}}}

If you would like to test a specific feature, you can use the function `IdFromString`. For example, if you would like to run all the tests that involve dakota:

{{{
#!m
>> runme('id',IdFromString('Dakota'))
}}}

You should see a list of `SUCCESS`. In very rare cases, you get an `ERROR` that requires to update the tolerance. Check with developers before changing the tolerance though.

 == Adding a new test ==

 === Files to change/create ===

You must create a new test file in `test/NightlyRun`. Make sure to be consistent with the series above.

Create a `testXXX.m` file with the simulation test. The very first line should be a comment with the test name (which should be unique):
{{{
#!m
%Test Name: TEST_NAME_HERE
}}}
For example test328.m: 
{{{
#!m
%Test Name: SquareSheetConstrainedSmbGradients2d 
}}}
at the end of the file make sure to include `field_names`, `field_tolerances` and `field_values` the names (no space) of the fields you want to check for your test with the tolerances associated (because every computer is different). 

Run the following command to create the archive (the archive gives the reference values for all the fields that are checked),
{{{
#!m
>> runme('id',XXX,'procedure','update')
}}}

'''NOTE''': The default benchmark used by the test driver is 'nightly', which includes only tests in the range 1-999. If you are manually running a test outside of this range, make sure to pass in the appropriate benchmark option.

This archive is located in `$ISSM_DIR/test/Archives` in binary format (.arch), and is the same for the Python test if you create one as well. Now if you run 

{{{
#!m
>> runme('id',XXX)
}}}

you should see SUCCESS everywhere. If not, you have to adjust the tolerances slightly.

'''Note:''' Do the same thing for `testXXX.py` without creating the archive this time. 

 === Checking in the new test ===
In the case of a brand new test
{{{
svn add test/NightlyRun/testXXX.m test/NightlyRun/testXXX.py test/Archives/ArchiveXXX.arch
}}}
then,
{{{
svn commit –m "NEW: Added test to check ...”`
}}}
with an appropriate message. Otherwise you can simply update the archive and commit the changes.