ExaCA

An exascale-capable cellular automaton for nucleation and grain growth ExaCA is a cellular automata (CA) code for grain growth under additive manufacturing conditions, created by ExaAM within the Exascale Computing Project.

License

ExaCA is distributed under an MIT license.

Citing ExaCA

If you use ExaCA in your work, please cite the following paper. In addition, cite the current release or version used from Zenodo.

Publications using ExaCA

In addition to the primary ExaCA citation, this list of articles use ExaCA in their work.

Contributing

We encourage you to contribute to ExaCA. Please check the contribution guidelines.

Build

ExaCA uses Kokkos and MPI for parallelism, JSON for input files, and CMake to build.

Package managers

ExaCA is integrated with the following package managers. Detailed build instructions are available below as well.

• spack: see the spack documentation for installation and use
  • spack install exaca will install with the default options
  • all ExaCA options and hardware details are available through spack, e.g. spack install exaca +finch +cuda cuda_arch=90
• nix: see NIX.md for building ExaCA using Nix
  • the Nix installation currently only supports the Serial Kokkos backend and does not support Finch

Dependencies

|Dependency | Version | Required | Details| |---------- | ------- |-------- |------- | |CMake | 3.11+ | Yes | Build system |Kokkos | 4.0+ | Yes | Portable on-node parallelism |MPI | GPU Aware if enabled | Yes | Multi-node parallelism |json | 3.10+| Yes | Input parsing |GoogleTest | 1.10+ | No | Unit test framework |CUDA | 9+ | No | Programming model for NVIDIA GPUs |HIP | 3.5+ | No | Programming model for AMD GPUs |Finch | main | No | Heat transport model for coupled simulation

CMake must be available to build ExaCA and Kokkos. The underlying parallel programming models and MPI are available on most systems and can generally be found automatically by CMake. Note these dependencies must all be installed first, if not available. Kokkos is also available on many systems; if not, obtain the desired version:

git clone https://github.com/kokkos/kokkos.git --branch 4.0.00

Obtaining ExaCA

ExaCA is available on GitHub, by default starting from the current master branch:

git clone https://github.com/LLNL/ExaCA.git

Backend options

Note that ExaCA runs with the default enabled Kokkos backend (see https://kokkos.org/kokkos-core-wiki/get-started/configuration-guide.html#backend-selection).

ExaCA has been tested with the Serial, OpenMP, Threads, CUDA, and HIP backends.

Building Kokkos

Host build

If Kokkos is not already installed on your system, configure, build, and install Kokkos:

# Change this path to desired Kokkos installation location
export KOKKOS_INSTALL_DIR=`pwd`/install/kokkos
# Change this path to Kokkos source
cd ./kokkos
# Configure Kokkos
cmake \
  -B build \
  -D CMAKE_BUILD_TYPE="Release" \
  -D CMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR \
  -D Kokkos_ENABLE_OPENMP=ON
# Build Kokkos
cmake --build build
# Install Kokkos
cmake --install build
cd ../

Note that there are other host backends available. Kokkos architecture flags can also be set to improve performance and must match the hardware you run on (e.g. -DKokkos_ARCH_POWER9=ON); see Kokkos architecture flag details.

CUDA build

Similar to the CPU build above, Kokkos can instead be configured and built for NVIDIA GPUs:

# Change this path to desired Kokkos installation location
export KOKKOS_INSTALL_DIR=`pwd`/install/kokkos
# Change this path to Kokkos source
cd ./kokkos
# Check the GPU architecture flag matches the hardware
# Configure Kokkos
cmake \
  -B build \
  -D CMAKE_BUILD_TYPE="Release" \
  -D CMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR \
  -D Kokkos_ENABLE_CUDA=ON \
  -D Kokkos_ARCH_VOLTA70=ON
# Build Kokkos
cmake --build build
# Install Kokkos
cmake --install build
cd ../

Note the two flags needed for the Kokkos::Cuda backend. The Kokkos architecture flag must match the hardware you run on. Kokkos will automatically redirect the (default) host compiler to nvcc in the example above. By default, the host will use Kokkos::Serial; other parallel host backends can also be used, e.g. by adding -D Kokkos_ENABLE_OPENMP as was done above.

HIP Build

To build Kokkos for HIP the hipcc compiler must be explicitly passed, along with architecture and backend flags analogous to the previous examples:

# Change this path to desired Kokkos installation location
export KOKKOS_INSTALL_DIR=`pwd`/install/kokkos
cd ./kokkos
# Configure Kokkos
cmake \
  -B build \
  -D CMAKE_BUILD_TYPE="Release" \
  -D CMAKE_CXX_COMPILER=hipcc \
  -D CMAKE_INSTALL_PREFIX=$KOKKOS_INSTALL_DIR$ \
  -D Kokkos_ENABLE_HIP=ON \
  -D Kokkos_ARCH_VEGA908=ON
# Build Kokkos
cmake --build build
# Install Kokkos
cmake --install build
cd ../

---

Build ExaCA

Once Kokkos and MPI are installed, ExaCA can be built:

# Change this path to desired ExaCA installation location
export EXACA_INSTALL_DIR=`pwd`/install/exaca
# Change this path to Kokkos installation location
export KOKKOS_INSTALL_DIR=`pwd`/install/kokkos

# Change this path to ExaCA source
# Configure ExaCA
cd ./ExaCA
cmake \
  -B build \
  -D CMAKE_BUILD_TYPE="Release" \
  -D CMAKE_PREFIX_PATH=$KOKKOS_INSTALL_DIR \
  -D CMAKE_INSTALL_PREFIX=$EXACA_INSTALL_DIR
# Build ExaCA
cmake --build build
# Install ExaCA
cmake --install build
cd ../

Kokkos will forward the compilation flags and details to ExaCA automatically.

Building with external JSON

By default, ExaCA will download the JSON library dependency used for input files. This automatic download does not work on all systems; a separate build of this library can be done instead. As with the dependencies described above, first obtain the source:

git clone https://github.com/nlohmann/json

And then build the json library (header only):

# Change this path to desired JSON installation location
export JSON_INSTALL_DIR=`pwd`/install/json

cd json
# Configure json
cmake \
  -B build \
  -D CMAKE_BUILD_TYPE="Release" \
  -D CMAKE_INSTALL_PREFIX=$JSON_INSTALL_DIR \
  -D JSON_BuildTests=OFF
# Build json
cmake --build build
# Install json
cmake --install build
cd ../

Then add this install path to the ExaCA configuration (example above) together with the path to Kokkos -D CMAKE_PREFIX_PATH="$KOKKOS_INSTALL_DIR;$JSON_INSTALL_DIR" and build ExaCA. Note that quotes are necessary for multiple paths.

Building with Finch

ExaCA can be compiled with Finch, a finite difference-based heat transport solver, for coupled heat transport and solidification simulation without the need to read time-temperature history data from file(s). The Finch source code and build instructions are available at https://github.com/ORNL-MDF/Finch. To compile ExaCA with Finch, include the path to the Finch install in the CMAKE_INSTALL_PREFIX. To require that ExaCA is compiled with Finch, add ExaCA_REQUIRE_FINCH=ON.

Testing ExaCA

Unit tests can be run if the ExaCA_ENABLE_TESTING CMake option is enabled in the build and if the GoogleTest framework is available on the system or built locally with the install path passed to ExaCA (see the previous section describing the JSON build and pointing ExaCA to the installation).

After building, tests can be run with cmake --build build --target test from the source directory (presuming build/ is the relative location of the build folder). Tests are automatically generated for all enabled Kokkos backend.

Running ExaCA

ExaCA runs using an input file, passed on the command line. Example problems are provided in the examples/ directory - a separate examples/README.md file goes into more detail on the problem types, the optional and required arguments needed for each problem type, and additional files used by ExaCA. The example input files present in this repository are:

• Inp_DirSolidification.json: simulates grain growth from a surface with a fixed thermal gradient and cooling rate
• Inp_SmallDirSolidification.json: a smaller and simpler version of the previous
• Inp_SpotMelt.json: simulates overlapping spot melts with fixed a fixed thermal gradient and cooling rate
• Inp_SmallSpotMelt.json: a smaller and simpler version of the previous
• Inp_SmallSpotMelt_Steel2phase.json: a modified version of the small spot melt problem for stainless steel 316L where two solidification phases can nucleate and grow - the initial substrate is set to the primary phase (austenite), but nucleation of new grains is assigned one of the phase (austenite and ferrite) based on the faster-solidifying phase at the nucleation undercooling. Each phase solidifies using a unique interfacial response function relating solidification velocity to local undercooling.
• Inp_SmallSpotMelt_Steel1phase.json: a control version of the previous problem where only the austenite phase for stainless steel 316L is considered.

Example problems only possible with external data:

• Inp_SingleLine.json: simulates melting and solidification of a single line of melt pool data
• Inp_SingleLineTranslate.json: using the same single line data as in Inp_SingleLine.json, create 3 overlapping line melt pools using specified bounds in the Y direction
• Inp_TwoLineTwoLayer.json: simulates two layers consisting of segments of two o