# Hierarchical Parallelism with Kokkos in Sacado

Sacado and its integration with Kokkos is designed to "just work" in as many types of thread parallel contexts as possible without special configuration of Sacado or custom logic in the thread-parallel code.  This means that Sacado's dervative computations are entirely local to each thread.  However, in situations where the parallel kernel does not expose enough parallelism to fully utilize all of the parallel capabilities of the architecture and the derivative dimension is large enough, substantially improved computation performance can be obtained on GPU architectures by mapping Kokkos vector parallelism across the derivative dimension.  Sacado supports this in two mutually exclusive ways, which we call Hierarchical and Hierarchical-DFad.  For a thorough description of how these work, please see the following article:

> Eric Phipps, Roger Pawlowski, and Christian Trott, "Automatic Differentiation of C++ Codes on Emerging Manycore Architectures with Sacado," ACM Trans. Math. Softw. 48, 4, Article 43, 2022, https://doi.org/10.1145/3560262.

Here we describe some of the basics of how to use this capability and its implications.  We assume the reader is already familiar with using Sacado and multi-level parallelism in Kokkos.

## Hierarchical Mode in Sacado

To enable Hierarchical mode in Sacado, one must turn it on by either configuring Sacado with `Sacado_ENABLE_HIERARCHICAL=ON` or adding `#define SACADO_VIEW_CUDA_HIERARCHICAL` before including any Sacado header files (the use of `CUDA` here is a misnomer and the capability also works for HIP, and in the future, SYCL).  Then hierarchical parallelism is triggered by using a custom view specialization that constrains derivative components to always be contiguous in memory through the use of the custom memory layout `Kokkos::LayoutContiguous<Layout,N>` where `Layout` is the chosen Kokkos memory layout (e.g., `Kokkos::LayoutLeft` or `Kokkos::LayoutRight`) and `N` will be the chosen vector parallelism size.  View access operations with this layout then produce temporary Fad handle objects with a strided derivative array that is strided by `N`.  For example, given a view `Kokkos::View<Sacado::Fad::SFad<double,50>**,Kokkos::LayoutContiguous<Kokkos::LayoutLeft,32>,Kokkos::Cuda> v` and a team policy with vector size of 32, `v(i,j)` will produce Fad handle objects with a derivative array of length 2 for Cuda threads with `0 <= threadIdx.x < 18` and length 1 for Cuda threads with `18 <= threadIdx.x < 32`.  This means the derivative array length is now different than the one specified in the scalar type for types such as SFad that encode this length in the type.  Therefore, the code must also use a different scalar type for thread-local temporaries to match.  Sacado provides a helper trait `typename Sacado::ThreadLocalScalarType<ViewType>::type` where `ViewType` is the type of the view above to correctly determine this type at compile-time.  Furthermore, one can use the function `Sacado::partition_scalar<stride>(a)` to partition a given Fad object `a` across threads, where `stride = Sacado::ViewScalarStride<ViewType>::stride` is the derivative stride.  An example of using these tools in a simple parallel kernel can be found [here](https://github.com/trilinos/Trilinos/blob/master/packages/sacado/test/UnitTests/Fad_KokkosTests.hpp#L249-L323).

## Hierarchical-DFad mode in Sacado

The above approach works for any Fad type supported by Sacado and Kokkos, but introduces the complication of a thread-local scalar type to handle the shorter derivative length.  For DFad, where the derivative length is chosen dynamically at run-time, a simpler approach is possible which can be enabled with `Sacado_ENABLE_HIERARCHICAL_DFAD=ON` or adding `#define SACADO_VIEW_CUDA_HIERARCHICAL_DFAD` before any Sacado headers.  In this case, the striding across derivative components is handled within the Sacado overloaded operators themselves, and therefore a new thread-local scalar type is not necessary, nor is even the use of the contiguous view specialization (however, to achieve good performance, the user must ensure a Kokkos layout is chosen that would result in contiguous derivative components in memory).  Thus the user is free to choose the vector dimension at runtime.

## Implications

Both approaches introduce several complications that must be respected for correct operation:
* If Hierarchical is on, any Kokkos kernel involving a Fad view with `Kokkos::LayoutContiguous<Layout,N>` cannot use any vector parallelism directly, and the vector size must be equal to `N`.  However, views with a standard layout can still be used in multilevel kernels, since the striding across derivative components is only triggered in the contiguous view specialization.
* If Hierarchical-DFad is on, DFad cannot be used in any Kokkos parallel kernel that uses vector parallelism directly.  However, the other Fad types can be used in multilevel kernels as normal.
* Hierarchical and Hierarchical-DFad cannot both be enabled at the same time.
* Even if either is enabled, Fad views can always be used in flat kernels using a range policy without any special logic.
* With regards to Sacado testing, turning either on does change the behavior of some of the tests in terms of what code paths are actually tested, however both modes of hierarchical parallelism are tested regardless of whether they are were turned on at configure time.