# The Pn form of the Neutron Transport Problem Achieves Linear Scalability Through Domain Decomposition

Large-scale neutron transport simulation

Author: Kenneth Assogba
Authors: Kenneth Assogba, Lahbib Bourhrara
Date: 2023
Topic: Conference proceeding
URL: https://kennethassogba.github.io/publications/mc-2023

- [Paper](https://kennethassogba.github.io/assets/docs/mc_2023.pdf)
- [Conference](https://mc2023.com/)

![Distributed matrix-vector product](https://kennethassogba.github.io/assets/img/matrix_distributed-spmv.svg "Distributed matrix-vector product")

Due to strong coupling between the angular moments, the spherical harmonics (Pn) formulation of the neutron transport equation has been neglected in favor of the discrete ordinates (Sn) form.

In this work, we target large-scale neutron transport simulation using a combined discontinuous Galerkin (DG) - spherical harmonics approximation. We leverage the benefits of DG discretization to wrap the previous developed solver, called NYMO, in a canonical ghost-mesh based domain decomposition framework. The developed solver handles unstructured, curved and non-conforming meshes with vacuum and reflection as boundary conditions.

Robustness, strong and weak scalability experiments have been conducted on the CEA's pre-exascale system Topaze. We reach and maintain a strong scaling efficiency of 100% up to 4096 cores and 80% up to 8192 cores. In particular, a calculation with 913 million degrees of freedom is performed in 101 seconds. Thus outperforming previously published results for Pn transport as well as many Sn and SPn solvers.

## Subdomains coupling

The general Matrix-Vector product from the point of view of a subdomain

```cpp
for(const auto& subdomain : domain_neighbors)
{
  mpi::request_in[i] = mpi::world.irecv(x_upwind); // async
  // Copy outgoing part of x in the x_out buffer
  mpi::request_out[i] = mpi::world.isend(x_out); // async
  i++;
}

y += A_diag * x;

mpi::world.waitall(request_in);
mpi::world.waitall(request_out);

y += A_offd * x_upwind;
```

## Weak scaling experiment

| Partitioning |     |   |   | Simple              |                 |   | Geometric              |                 |
|------:|-------:|------:|--:|--------------------:|----------------:|--:|-----------------------:|----------------:|
| $n_d$ | \#core | \#dof |   |            time (s) | efficiency (\%) |   |               time (s) | efficiency (\%) |
|    16 |   1024 | 228M  |   |                 145 |               - |   |                    142 |               - |
|    32 |   2048 | 456M  |   |                 154 |              94 |   |                    157 |              90 |
|    64 |   4096 | 913M  |   |                 146 |              99 |   |                    147 |              96 |

## Strong scaling experiment

Strong scaling experiment on up to 128 domains using a total of 8192 CPU-cores. 0 means shared memory only calculation.

| Partitioning   |        |   | Simple              |         |                 |   | Geometric              |         |                 |
|---------------:|-------:|---|--------------------:|--------:|----------------:|--:|-----------------------:|--------:|----------------:|
|          $n_d$ | \#core |   |             time(s) | speedup | efficiency(\%)  |   |                time(s) | speedup |  efficiency(\%) |
|              0 |     64 |   |               10597 |       - |               - |   |                  10597 |       - |               - |
|              4 |    256 |   |                2557 |     4.1 |             104 |   |                   2658 |       4 |             100 |
|              8 |    512 |   |                1242 |     8.5 |             107 |   |                   1245 |     8.5 |             106 |
|             16 |   1024 |   |                 667 |    15.9 |              99 |   |                    628 |    16.9 |             106 |
|             32 |   2048 |   |                 302 |    35.1 |             110 |   |                    305 |    34.7 |             109 |
|             64 |   4096 |   |                 146 |    72.6 |             113 |   |                    147 |    72.1 |             113 |
|            128 |   8192 |   |                 101 |   104.9 |              82 |   |                    108 |    98.1 |              77 |