The Pn form of the Neutron Transport Problem Achieves Linear Scalability Through Domain Decomposition

• Paper
• Conference

Due to strong coupling between the angular moments, the spherical harmonics (Pn) formulation of the neutron transport equation has been neglected in favor of the discrete ordinates (Sn) form.

In this work, we target large-scale neutron transport simulation using a combined discontinuous Galerkin (DG) - spherical harmonics approximation. We leverage the benefits of DG discretization to wrap the previous developed solver, called NYMO, in a canonical ghost-mesh based domain decomposition framework. The developed solver handles unstructured, curved and non-conforming meshes with vacuum and reflection as boundary conditions.

Robustness, strong and weak scalability experiments have been conducted on the CEA's pre-exascale system Topaze. We reach and maintain a strong scaling efficiency of 100% up to 4096 cores and 80% up to 8192 cores. In particular, a calculation with 913 million degrees of freedom is performed in 101 seconds. Thus outperforming previously published results for Pn transport as well as many Sn and SPn solvers.

Subdomains coupling

The general Matrix-Vector product from the point of view of a subdomain

for(const auto& subdomain : domain_neighbors)
{
  mpi::request_in[i] = mpi::world.irecv(x_upwind); // async
  // Copy outgoing part of x in the x_out buffer
  mpi::request_out[i] = mpi::world.isend(x_out); // async
  i++;
}

y += A_diag * x;

mpi::world.waitall(request_in);
mpi::world.waitall(request_out);

y += A_offd * x_upwind;

Weak scaling experiment

Strong scaling experiment

Strong scaling experiment on up to 128 domains using a total of 8192 CPU-cores. 0 means shared memory only calculation.

Thanks for reading! Go home for more.