Parallel performance

wadewei

I am getting my hands on a test server with 64-core AMD EPYC gen2 processor but would like to set my expectation straight, as I understand that PETSc performance is more memory-limited than cpu-limited. Do you guys have any whitepaper-like material as how ISSM performance scales within a single node and beyond? Any tuning tips are also appreciated.

mathieumorlighem

Hi Wade,

this is a bit old: https://gmd.copernicus.org/articles/10/155/2017/gmd-10-155-2017.html if you are using the latest version of ISSM, the bottleneck is the solver (which depends on your PETSc options in md.toolkits). MUMPS is very robust and reliable but does not scale. You can try this one instead (biconjugate gradient with block Jacobi preconditioner), see if it works for you:

md.toolkits.DefaultAnalysis=bcgslbjacobioptions();

Best,
Mathieu

wadewei

mathieumorlighem
Hi Mathieu,
It does seem that MUMPS would not scale beyond tens of cores on a mesh of _20k elements. In fact it slows down when increasing np from 32 to 64 on EYPC 7502*2 with hyperthreading off and I highly doubt it would gain any performance increase with 7702.
It's kind of comprehensible that direct solvers would be limited by the rate of which each core exchanges data with others (correct me if I am wrong). However I still want to double check if it's normal that after increasing memory channel from 4 per CPU to 8 channels per CPU, the speed of runs using 32 and 64 cores remains the same.

Best,
Wade