Installation

StarSmasher is compiled against whatever MPI and CUDA your machine provides, so it is built from source rather than installed from a package.

What you need

  • Linux, or macOS on Apple Silicon

  • An MPI implementation providing mpif90. OpenMPI and MPICH both work

  • A Fortran compiler. Tested on gfortran 10 and later, and on ifort

  • On Linux, optionally an NVIDIA card and CUDA. On Apple Silicon, the integrated GPU is supported through Metal. Without a GPU backend, make cpu builds a working executable

Apple Silicon users should follow the Apple Silicon installation guide. The build detects Darwin/arm64 and selects Metal automatically; CUDA is not required and no source or Makefile edits are needed.

The repository’s own documentation/installation.md covers setting these up, including installing CUDA and OpenMPI and dealing with module environments on clusters. What follows is the short version.

Getting the code

$ git clone https://github.com/jalombar/starsmasher.git

Clone rather than downloading an archive: the code is compiled against your own libraries, and a clone can be updated later.

Checking what the build will use

From parallel_bleeding_edge/src:

$ make config

which prints something like:

Platform
  OS               Linux
  architecture     x86_64
  GPU backend      cuda
MPI
  mpif90          /usr/lib64/openmpi/bin/mpif90
  wrapping        GNU
  fixed-form flag -ffixed-line-length-132 -fallow-argument-mismatch
CUDA (used only when GPU backend is cuda)
  nvcc            /usr/local/cuda/bin/nvcc
  CUDAPATH        /usr/local/cuda
  runtime libdir  /usr/local/cuda/lib64
Compilation
  FFLAGS          -O4 -mcmodel=medium
  code model      -mcmodel=medium
Output
  GPU executable  ../parallel_bleeding_edge_gpu_sph
  CPU executable  ../parallel_bleeding_edge_cpu_sph

Everything there is detected, not hardcoded. If a line is wrong, override it on the command line rather than editing the Makefile:

If this is wrong

Set

the mpif90 path, or the compiler behind it

make MPIF90=/full/path/to/mpif90

CUDAPATH, or the runtime library directory

make CUDAPATH=/opt/cuda

the optimisation flags

make OLEVEL=-O2

the compute capability, if nvidia-smi cannot report it

make COMPUTE_CAPABILITY=120

Building

One command, from parallel_bleeding_edge/src:

$ make

That builds the gravity library too, so there is no need to build it separately. To rebuild only the library, which is occasionally useful when chasing a CUDA problem, use make -C SPHgrav_lib2.

On Apple Silicon the same command builds SPHgrav_lib_metal instead. The CPU-only target remains available with make cpu, but the Metal build is recommended for normal runs. Current M1 Pro measurements are about five times faster than CPU-only runs for the tested simulations. As a rough estimate for gravity-dominated work, Pro-class chips should be about five times faster with Metal than with CPU alone, and Max-class chips about ten times faster.

The library works out your card’s compute capability for itself, so there is normally nothing to set.

Setting the compute capability by hand

Where the machine you compile on differs from the machine you run on, such as a login node with no card in it or a cluster with mixed hardware, give the value explicitly:

$ make COMPUTE_CAPABILITY=120

Write it as the capability times ten, with no decimal point: 9.0 is 90, 12.0 is 120. The setting carries from src down into the gravity library, so it works on either make line.

If you are unsure of the number, __nvcc_device_query, which ships with CUDA, prints it. If you are unsure which of several cards to target, choose the lowest: code built for a lower capability still runs on a higher card, but not the other way round.

A successful build ends with:

***MADE VERSION THAT USES GPUS***  ->  ../parallel_bleeding_edge_gpu_sph

The executable is named after the directory containing src, and is copied one level up. make cpu produces ..._cpu_sph in the same place.

Note

nmax, nnmax and ntab are generated by the Makefile. Their defaults can be overridden without editing source files, for example with make NMAX=200000 NNMAX=128 NTAB=400000. Apple Silicon uses smaller defaults suitable for unified-memory Macs; other platforms keep the original values. kdm remains a compile-time constant in starsmasher.h.

Checking that it works

From the top of the repository, not from src:

$ cd ../..
$ python3 tests/run_tests.py

The suite builds its own copy of the code and runs seven short calculations, each checking something that has to be true rather than comparing against stored output. It relaxes a polytrope and checks it against the virial theorem, verifies energy conservation, confirms the star stays put once relaxed, checks that the answer does not depend on the number of MPI ranks or on whether the GPU or CPU build is used, and confirms that two input guards still work.

Expect about seven minutes on a GPU, and longer without one. For a quicker check, --quick uses a smaller star and finishes in well under a minute:

$ python3 tests/run_tests.py --quick

Nothing about what is asserted changes, only the size of the model. Use --list to see the individual tests, and name one to run it alone.