ISSM matlab version taking in another an MPI communicator

briankyanjo

Is there a way for ISSM matlab version to take in an MPI communicator? I am coupling ISSM with my code and am using MPI sub communicators for parallelization, synchronizing the two codes will be eaiser if i can insert my own subcommunicator into ISSM.

mathieumorlighem

Hello Brian
it's going to be really complicated to pass an MPI communicator through MATLAB.... depending on what you are doing it may be worth "skipping" MATLAB altogether and do this on the C++ side. If that's too complicated, we could "hack" the code so that the communicator is written on a file that would be loaded by ISSM (in issm.cpp, using femmodel = new FemModel(argc,argv,comm_init);). What do you think?
Mathieu

briankyanjo

Hi Mathieu,

I like the idea of passing a communicator into femmodel = new FemModel(argc, argv, comm_init);—it seems like a cleaner and potentially more scalable approach. That said, I’d still like to ensure that users can stay entirely on the MATLAB side without having to interact with C++ when implementing new examples in ICESEE.

To give you more context, ICESEE is a Python-based data assimilation package I’m developing. I’ve already set up a MATLAB server that sits idle and listens for Python commands, which lets me launch multiple MATLAB instances—e.g., four instances for four ensemble members—each tied to its own processor. Currently, users can specify both the number of processors for ICESEE and the number to allocate to ISSM from the MATLAB side using a command like:

mpiexec -n 4 python run_da_issm.py --Nens=4 --model_nprocs=2

Here, -n 4 is for ICESEE, and --model_nprocs=2 is passed to MATLAB for ISSM. This avoids needing to pass the MPI communicator directly.

Still, I’m very interested in supporting the Python-based version of ISSM in the future, and I see how directly passing a communicator into issm.cppcould reduce the overhead of launching multiple MATLAB processes, which is one of my current concerns.

How do you envision this working in practice? I assume we’d need to compile issm.cpp into an executable that accepts an external communicator. But I’m wondering how this would affect the simulation setup process—especially in terms of usability. Would users still be able to configure and run experiments easily, or would this shift introduce added complexity?

I’m happy to try prototyping this on my end to explore the trade-offs.

-brian

mathieumorlighem

Hello Brian
sorry for the slow response, I am just not quite sure how to deal with a communicator if we do this through MATLAB since MATLAB does not have a specific type for that. I don't really know what you are thinking about changing between different ISSM calls, but we could potentially just manipulate the *.bin file and call issm.exe again with a different input. It may not be as flexible as using MATLAB but it would be much more efficient. Anyway, if you are happy with the current setup, we can leave it as is for now. What do you think?
Mathieu

briankyanjo

Hi Mathieu,

Thanks for your message. I’m definitely open to the idea of manipulating the *.bin files and calling issm.exe directly—efficiency is my top priority at this stage, even if it comes at the cost of user flexibility.

I’m actually considering supporting both approaches: keeping the current setup for users who prefer more flexibility via MATLAB and offering this new streamlined method for those prioritizing performance.

How would you suggest we move forward with implementing this? Would it involve writing out modified *.bin files between forecast steps (calls to ISSM) and then reloading them via issm.exe?

mathieumorlighem

The most important part is that you understand how the bin file works. It stores each field of md in binary format (using the marshall function of each subclass, which calls src/m/solve/WriteData.m). The order of the fields does not matter. Each field starts with:

field name (e.g., md.initialization.vx)
length of the field
the field itself

You should take a look at scripts/BinRead.py to understand how the file is written. For example, if you want to add a new field to the bin file, you could do this:

from struct import error, pack

fid = open("inputfile.bin", "ab")

# 1: Write field name
name="md.NewField"
fid.write(pack('i', len(name)))
fid.write(pack('{}s'.format(len(name)), name.encode()))

# 2: Write length of record
fid.write(pack('q', 4 + 4))  #1 integer + code
fid.write(pack('i', 2)) #2 means Integer for ISSM

# 3: Write data (here 20, but could be anything)
fid.write(pack('i', int(20)))

#done!
fid.close()

You can check that it is there by running:

$ISSM_DIR/scripts/BinRead.py -f inputfile.bin

I hope this helps you get started! Maybe a natural next step would be to change a field, like md.timestepping.final_time, you will need to find where “md.timestepping.final_time” is (with seek), and then replace its value (the “Code” for “Double” is 3, I think you used 2 for an integer:

# 2: Write length of record
fid.write(pack('q', 8 + 4))  #1 double + code
fid.write(pack('i', 3)) #3 means Double for ISSM

# 3: Write data
fid.write(pack(‘d', NEWFINALTIMEVALUE))

Cheers
Mathieu

briankyanjo

Hi Mathieu,

This is definitely a good start. Let me give it a try, and I will let you know if I get any issues. Thanks