       /§§             /§§
      | §§            | §§
  /§§§§§§§  /§§§§§§  /§§§§§§    /§§§§§§  /§§§§§§/§§§§   /§§§§§§§
 /§§__  §§ /§§__  §§|_  §§_/   /§§__  §§| §§_  §§_  §§ /§§_____/
| §§  | §§| §§§§§§§§  | §§    | §§  \ §§| §§ \ §§ \ §§| §§
| §§  | §§| §§_____/  | §§ /§§| §§  | §§| §§ | §§ | §§| §§
|  §§§§§§§|  §§§§§§§  |  §§§§/|  §§§§§§§| §§ | §§ | §§|  §§§§§§§
 \_______/ \_______/   \___/   \____  §§|__/ |__/ |__/ \_______/
                                    | §§
                                    | §§       generic C++ replica exchange
                                    |__/       determinantal quantum Monte Carlo code

The determinantal quantum Monte Carlo (DQMC) method, also known as auxiliary field QMC, is the premier method for the numerically exact, unbiased solution of strongly correlated itinerant electron systems in condensed matter physics.

Here I present a quite generic high-performance code in modern C++ that implements this method at finite temperatures. While it is commented and this document provides some overview documentation, understanding it requires some engagement with physics and mathematical rigor. A good starting point can be found in my PhD thesis, particularly in chapters 4 and 5.

The code has been used successfully for the simulation of a two-dimensional metal coupled to a spin-density wave (SDW) order parameter (publications: [PRL], [supplementary], [PRB]). Correspondingly, the related class DetSDW is the most advanced, but there is also a more basic implementation for the repulsive half-filled Hubbard model in class DetHubbard.

Features:

effective model-independent numerical stabilization
efficient linear algebra using checkerboard break ups
delayed local updates
several types of global updates
MPI-parallelized replica exchange mechanism
control of finite size effects via artificial magnetic fluxes

Also included is mrpt, an implementation of the powerful multiple histogram reweighting method for parallel tempering / replica exchange Monte Carlo data. A detailed account of this method is given in my diploma thesis.

Table of Contents

Setup
Usage
- Single replica simulation
- Replica exchange simulation
Structure of DQMC code
Executables
Overview of source code files
Limitations

Setup

Prerequisites

This code has only been tested on Linux. You will need the following to get started:

A reasonably modern C++ compiler and standard library. These are known to work well:
- g++ 4.9.4
- Clang 3.4
- Intel 15.0 in conjunction with the libraries coming with a recent g++
Earlier versions might do the job (Intel 13 should be fine; g++ 4.6 may compile with some tinkering to work around its C++11 shortcomings, while some intermediate versions < 4.9.4 fail due to a bug).
CMake version 2.8.5+
Implementations of BLAS and LAPACK
- OpenBlas and LAPACK packages work well
- Intel MKL is a fast commercial alternative (commonly installed on compute clusters)
FFTW 3.x
- you can link to Intel MKL instead
For replica exchange simulations: an implementation of MPI
- Open MPI works well
- Intel MPI is fine too

All of these should be available via your distribution's package manager or in the form of modules on your HPC cluster.

Included libraries

For simplicity (portions of) these libraries are included with this package:

Armadillo 7.600.2 for high-level linear algebra
dSFMT 2.1 for pseudo-random number generation
Boost 1.51
Dlib, used in the mrpt reweighting code
cnpy, used in sdwcorr

License information is included within the respective subdirectories.

Compilation

Download a relase of the code or clone the git repository. To compile an optimized full build with default settings do:

cd Release
./runcmake.sh
make -j

After having run CMake you can alternatively build only select targets. For example, run make -j detqmcptsdwo2 to build all requirements for the executable to run parallelized replica exchange DQMC simulations of the O(2) SDW model.

Go to the directory Debug instead of Release for a build with debug symbols, additional error checks, and no optimization.

If this does not work immediately, some tweaking may be necessary.

You can change the invocation of cmake in Release/runcmake.sh. For instance, to use clang instead of g++, edit it to read

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_EXPORT_COMPILE_COMMANDS=ON -DUSE_GNU_COMPILER=OFF -DUSE_CLANG_COMPILER=ON  ../src

There are a couple more options defined in src/CMakeList.txt.

For more specific configuration you will have to directly edit the src/CMakeList.txt file. Its inital section contains logic for system specific build settings. It contains some example sections for the "Cheops" and "Jureca" HPC clusters, and for some machines running outdated system installations: the "l71" work station and the computers in a domain "thp". The parts of the CMake script that enable these specific settings are currently commented out. Use them as example guidelines to adapt the setup to your own machines.

Depending on your BLAS and LAPACK installation you might also have to uncomment some preprocessor directives in the file src/armadillo/armadillo_bits/config.hpp. Alternatively, you could just delete the entire directory src/armadillo and use your own, independent installation of Armadillo.

Usage

Single replica simulation

An example single replica simulation run is prepared in example/sdw-o2-single-replica/simulation.conf. From the project root do the following to execute it:

$ cd example/sdw-o2-single-replica
$ ../../Release/detqmcsdwopdim

This will set up the simulation according to the parameter values specified in simulation.conf. You can get an annotated list of all possible command line or config file options with detqmcsdwopdim --help.

Once the simulation has started, some information is printed to the screen. After it has finished, you will find some new files in the same directory: info.dat summarizes data about the simulation. results.values contains expectation values with error bars for some measured observables. The time series of these observable measurements are contained in files *.series. Then there is a file configs-phi.binarystream containing the raw system configurations sampled in the simulation to be evaluated subsequently. Finally, simulation.state contains all the checkpointing information to resume a simulation that has been interrupted previously. By running detqmcsdwopdim --sweeps 200 it is also possible to continue a previously finished simulation until a higher target sweep count is reached.

Replica exchange simulation

A slightly more involved example for a replica exchange simulation with eight different values of the tuning parameter r is prepared in example/sdw-o2-replica-exchange/simulation.conf. Typically, you would run it like this:

$ cd example/sdw-o2-replica-exchange
$ mpirun -n 8 ../../Release/detqmcptsdwopdim

Depending on your environment, you may need to adapt the invocation of MPI, with the SLURM scheduler you would use srun instead of mpirun for instance.

This will produce a common info.dat for all replicas, three files exchange-*.values with statistics on the replica exchange process, and a file simulation.*.state for each replica such that the entire simulation can be resumed again. Also, a number of subdirectories has appeared, one for each value of the tuning parameter r. In these subdirectories the measurements corresponding to that value of r are collected.

See scripts/detptsubmit.py for some inspiration how to set up a number of large scale simulation jobs more easily, taking a file like example/simulation.job as input.

Structure of DQMC code

Non-model specific aspects of the simulation are handled by class DetQMC (detqmc.h) for single replica simulations and class DetQMCPT ([detqmcpt.h

Detqmc

Install / Use

README