Question about multiple runtimes on MPI for MCNP6

[Alexander Camargo]

When I increase the number of nodes in the simulation (e.g., -N 3), the number of runtpe files and outputs generated equals the number of nodes; in this case, three runtpe files and three outputs appear. Each output has a different simulation time, and sometimes even different results. Is this normal when running MCNP in MPI mode? The command I'm using is:

srun -N 3 mpirun -np 48 mcnp6.mpich i=input

 

[pete94857]

what you’re experiencing is normal behavior for MCNP running in MPI mode, but understanding the reasons behind the variations can help you manage and interpret the outputs more effectively

 

[Alexander Camargo]

what you’re experiencing is normal behavior for MCNP running in MPI mode, but understanding the reasons behind the variations can help you manage and interpret the outputs more effectively

Hi Pete94857.

Thank you for the help. I've seen some MPI script examples similar to mine, but they didn't generate this many runtpe files. I thought I might be doing something wrong or missing a command.

 

[Alex A]

@pete94857, careful with that and have a read of the rules.
[Mentor Note: The post by Pete that Alex is referring to has been deleted]

@Alexander Camargo, are you sure you need srun AND mpirun in the command? I don't know if it will make a difference but I think srun may just replace mpirun. What is srun, is it slurm and if so what is slurm?

 

[Alexander Camargo]

@pete94857, careful with that and have a read of the rules.

@Alexander Camargo, are you sure you need srun AND mpirun in the command? I don't know if it will make a difference but I think srun may just replace mpirun. What is srun, is it slurm and if so what is slurm?

The srun is the cluster system scheduler, used to configure the number of nodes and CPUs to be used. The MCNP manual states that the command to use multiple CPUs is 'mpirun -np X', where X is the number of CPUs, followed by mcnp6.mpi (MCNP6 compiled for MPI) and input/output file parameters. I have 3 nodes, each with 16 CPUs.

It's working; I can utilize nodes and CPUs. However, it always creates a separate simulation for each node used. If I use only 1 node, it runs faster than using 3 (since processing is divided when using 3), generating 3 runtpe and 3 output files.

My question is: is this normal behavior for MCNP6 MPI version or do I need to configure something better?

 

[Alex A]

I've never tried to make MPI work before so this is a bit of a guess. If you are getting multiple output files, that seems unhelpful. OMP generates multiple threads in a shared memory system, MPI runs a separate copy of the whole program and problem in each node.

I have read that srun replaces the mpirun command in a slurm system so I wonder if by feeding srun with an MPI split, but still running mpirun without MPI (srun -N 3 mpirun -np 48) into mcnp that also isn't being told to use MPI, creates 3 separate MCNP instances that don't know about the others.

I don't really know what a slurm system is or that you are using one. I'm drawing from this document. The example command for OMP+MPI is,

srun –n X mcnp6.mpi i=athena1_hex.txt tasks 36

That is for one MPI manager and X-1 MPI worker nodes with 36 threads per worker node. X must be 3 or more.