NaN found in solution vectorloading results from cluster

tyl

Hi,

I use my observation velocity to process ISSM on Windows7 OS generally follow tutorial1, but I encountered some problem in solve step.


md.cluster=generic('name',oshostname,'np',4);
md=solve(md,StressbalanceSolutionEnum);

and there show error messages:

checking model consistency

Warning: no ice front provided

> In mask/checkconsistency (line 54)

In ismodelselfconsistent (line 31)

In solve (line 52)

marshalling file russell.bin

Warning: parallel runs not allowed yet in Windows. Defaulting to 1 cpus

> In generic/BuildQueueScript (line 146)

In solve (line 99)

Ice Sheet System Model (ISSM) version 4.9

(website: http://issm.jpl.nasa.gov contact: issm@jpl.nasa.gov)

call computational core:

preparing initial solution

x | Cost function f(x) | List of contributions

====================== step 1/20 ===============================

x = 0 | computing velocities

computing velocities

Error message: NaN found in solution vectorloading results from cluster

Error using parseresultsfromdisk>parseresultsfromdiskioserial (line 29)

no results found in binary file russell.outbin

Error in parseresultsfromdisk (line 10)

results=parseresultsfromdiskioserial(filename);

Error in loadresultsfromdisk (line 26)

structure=parseresultsfromdisk(filename,~md.settings.io_gather);

Error in loadresultsfromcluster (line 30)

md=loadresultsfromdisk(md,);

Error in solve (line 146)

md=loadresultsfromcluster(md);

>> oshostname() 
ans =
AllenTsai-PC
>> md.cluster
ans = 
class 'generic' object 'ans' = 
    name: AllenTsai-PC
    login: 
    np: 4
    port: 0
    codepath: C:\ISSM\bin
    executionpath: C:\ISSM\execution
    etcpath: C:\ISSM\etc
    valgrind: C:\ISSM/externalpackages/valgrind/install/bin/valgrind
    valgrindlib: C:\ISSM/externalpackages/valgrind/install/lib/libmpidebug.so
    valgrindsup: C:\ISSM/externalpackages/valgrind/issm.supp
    verbose: 
    shell: /bin/sh

any suggestion to solve this?

Thank you

glperez

Hello,

Could you provide us with the full script that you are using when running into this issue?

tyl


disp('	Step 1: Mesh creation');
md=triangle(model,'russellareaAMPT01.exp',2000); %%%%%%
ncdata='amplt_Edisp3s_o.nc';
x1 = ncread(ncdata,'x');
y1 = ncread(ncdata,'y');
y1=flipud(y1); 
velx = ncread(ncdata,'amplt_edisp_o');
velx(velx==min(min(velx)))=0;
max(max(velx))
min(min(velx))
vx = InterpFromGridToMesh(x1,y1,velx',md.mesh.x,md.mesh.y,0);

ncdata2='amplt_Ndisp3s_o.nc';
x2 = ncread(ncdata2,'x');
y2 = ncread(ncdata2,'y');
y2=flipud(y2); 
vely = ncread(ncdata2,'amplt_ndisp_o');
vely(vely==min(min(vely)))=0; 
vy = InterpFromGridToMesh(x2,y2,vely',md.mesh.x,md.mesh.y,0);

vel  = sqrt(vx.^2+vy.^2);

md=bamg(md,'hmin',500,'hmax',2000,'field',vel,'err',8); 
plotmodel(md,'data','mesh');
=xy2ll(md.mesh.x,md.mesh.y,+1,39,71); 
plotmodel (md,'data','mesh');
save RUSSELL.mdl md


disp('	Step 2: Parameterization');
md=loadmodel('RUSSELL.mdl');
md=setmask(md,'','');
md=parameterize(md,'AMTdata.par'); 
md=parameterize(md,'Jks_noV.par'); 

weakb=ContourToMesh(md.mesh.elements,md.mesh.x,md.mesh.y,'shear2.exp','node',2); 
pos=find(weakb); 
md.materials.rheology_B(pos)=.3*md.materials.rheology_B(pos);
plotmodel(md,'data',weakb,'edgecolor','w');
save RUSSELL.mdl md 

disp('	Step 3: Control method friction');
md=loadmodel('RUSSELL.mdl');
md=setflowequation(md,'SSA','all');
%       Control general
md.inversion.iscontrol=1;
md.inversion.nsteps=20;
md.inversion.step_threshold=0.99*ones(md.inversion.nsteps,1);
md.inversion.maxiter_per_step=5*ones(md.inversion.nsteps,1);
md.verbose=verbose('solution',true,'control',true);
%       Cost functions
md.inversion.cost_functions=;
md.inversion.cost_functions_coefficients=ones(md.mesh.numberofvertices,2);
md.inversion.cost_functions_coefficients(:,1)=40;
md.inversion.cost_functions_coefficients(:,2)=1;
%       Controls
md.inversion.control_parameters={'FrictionCoefficient'};
md.inversion.gradient_scaling(1.inversion.nsteps)=30;
md.inversion.min_parameters=1*ones(md.mesh.numberofvertices,1);
md.inversion.max_parameters=200*ones(md.mesh.numberofvertices,1);
%       Additional parameters
md.stressbalance.restol=0.01;
md.stressbalance.reltol=0.1;
md.stressbalance.abstol=NaN;

%       Go solve
md.cluster=generic('name',oshostname,'np',4);
md=solve(md,StressbalanceSolutionEnum);

%%%     Error message: NaN found in solution vectorloading results from cluster
%%%     Error using parseresultsfromdisk>parseresultsfromdiskioserial (line 29)
%%%     no results found in binary file russell.outbin
%%%     Error in parseresultsfromdisk (line 10)
%%%     	results=parseresultsfromdiskioserial(filename);
%%%     Error in loadresultsfromdisk (line 26)
%%%     	structure=parseresultsfromdisk(filename,~md.settings.io_gather);
%%%     Error in loadresultsfromcluster (line 30)
%%%     md=loadresultsfromdisk(md,);
%%%     Error in solve (line 146)
%%%     	md=loadresultsfromcluster(md); 

%%%     >> md.cluster
%%%     class 'generic' object 'ans' = 
%%%         name: AllenTsai-PC
%%%         login: 
%%%         np: 4
%%%         port: 0
%%%         codepath: C:\ISSM\bin
%%%         executionpath: C:\ISSM\execution
%%%         etcpath: C:\ISSM\etc
%%%         valgrind: C:\ISSM/externalpackages/valgrind/install/bin/valgrind
%%%         valgrindlib: C:\ISSM/externalpackages/valgrind/install/lib/libmpidebug.so
%%%         valgrindsup: C:\ISSM/externalpackages/valgrind/issm.supp
%%%         verbose: 
%%%         shell: /bin/sh

glperez

Thanks for the response.

Unfortunately, I do not see anything glaring that would cause the issue you are seeing. The next step is to attempt to reproduce the problem you are having our end. Could you provide us with your model data?

Also, if you access to a Linux or OSX machine, then I recommend trying to run your model on one of them. At the very least we can find out whether this an issue specific to the Windows build or whether it is more widespread. I will attempt to do the same once I have the appropriate data.

-Gil

tyl

Thank you for response, attached link is my used files.

Thank you

https://goo.gl/vkVxQi

glperez

Great!

I have downloaded the 'rar' file and will try it out on my end to attempt to recreate the issue.

As a heads up, I am at AGU all this week, along with most of the ISSM team, so responses might be a bit slow until we get back.

tyl

Thank you, now I am also at AGU, maybe can I ask you directly?

Also I would like to ask whether the observed velocity input must be Vx Vy? or can directly input the LOS direction which DInSAR made?

Thank you

mathieumorlighem

you have several Inf values in md.initialization.vx/vy and md.stressbalance.vx/vy. The consistency check does not check for infinite values, but I will add that. If you remove the Infs, the model runs:

>> pos=find(isinf(md.stressbalance.spcvx));
>> md.initialization.vx(pos)=0;
>> md.initialization.vy(pos)=0;
>> md.stressbalance.spcvx(pos) = 0;
>> md.stressbalance.spcvy(pos) = 0;
>> md.inversion.vx_obs(pos)=0;
>> md.inversion.vy_obs(pos)=0;
>> md.inversion.iscontrol=0;
>> md=solve(md,StressbalanceSolutionEnum);
checking model consistency
marshalling file russell.bin
uploading input file and queueing script
launching solution sequence on remote cluster

Ice Sheet System Model (ISSM) version  4.9
(website: http://issm.jpl.nasa.gov contact: issm@jpl.nasa.gov)

call computational core:
   computing new velocity
   computing velocities
   saving results
write lock file:

   FemModel initialization elapsed time:   0.0100969
   Core solution elapsed time:             0.255884

   Total elapsed time: 0 hrs 0 min 0 sec
closing PETSc
WARNING! There are options you set that were not used!
WARNING! could be spelling mistake, etc!
Option left: name:-pc_factor_shift_positive_definite value: true
Option left: name:-toolkit value: petsc
closing MPI
loading results from cluster

Mathieu

tyl

OK, it does work now!

Really thank you for helping!