Galaxy: an asynchronous distributed parallel framework

master: dev:

Galaxy is an asynchronous parallel visualization framework for ray-based operations, including ray tracing, sampling and volume integration. Galaxy has been designed for performance in distributed computing environments, including in situ use. Galaxy builds upon [Intel OSPRay][1] and [Intel Embree][2] and includes ray queueing and sending logic inspired by [TACC GraviT][3].

Galaxy has been developed primarily by Greg Abram and Paul Navratil with funding from the US Department of Energy's Office of Science ASCR program (Drs. Lucy Nowell and Laura Biven, program managers) through sub-award from Los Alamos National Laboratory; from Intel Corporation through Intel Parallel Computing Center awards; and from the Texas Advanced Computing Center at the University of Texas at Austin. A full list of contributors can be found in the CONTRIBUTORS file.

Galaxy is licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. A copy of the License is included with this software in the file LICENSE. If your copy does not contain the License, you may obtain a copy of the License at: [Apache License Version 2.0][15]. Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Galaxy dependencies

Galaxy has two types of dependencies: (1) components assumed to be already installed in an accessible spot; and (2) components that are associated with the Galaxy repository in the third-party subdirectory.

Galaxy assumes the following are already installed on your system in an accessible spot (these should all be available via package manager for your OS):

a modern MPI (Message Passing Interface) implementation
[a recent VTK][4] (at least version 7.x)
[Intel Thread Building Blocks (TBB)][6]
[PNG image format library][7]
[zlib compression library][8]
[CMake cross-platform build utility][9]
[GLFW][14] (required by OSPRay build)
[Python][18] (required for testing and using python scripts in scripts)
[Qt5][20] (at least v5.9, tested against v5.14)

Galaxy has the following components associated in the third-party subdirectory:

[Intel ISPC][10]
[Intel Embree][12]
[Intel OSPRay][13]
[rapidjson][16]

NOTE: the third-party component locations are intended for development builds of Galaxy. If you intend to install Galaxy for general use, you should install Embree, OSPRay and rapidjson into a generally-accessible location and use those locations in the Galaxy build.

Installing Galaxy associated dependencies

Prior to building Galaxy itself, you should ensure that all the general dependencies are installed (we recommend your humble OS package manager). Once those are in place, you're ready to install the dependencies in third-party. There are two steps to this process: 1) downloading and patching the submodules, and 2) building and installing the submodules.

Scripted build

We recommend using the Galaxy-provided build script scripts/install-third-party.sh, which will configure and build all submodules with a single command (if this script does not work for your configuration, please open a [GitHub issue][19]). If you need to customize your build, please see the next section.

To download and build the submodules: cd to the root directory of your local Galaxy repository and run the script ./scripts/install-third-party.sh, which will (1) init and update the git submodules, (2) download ispc to where Galaxy expects to find it, and (3) build each in a build subdirectory and install into an install subdirectory. This script will populate the third-party/ispc, third-party/embree, third-party/ospray, and third-pary/rapidjson subdirectories. Once built, each third-party component will be installed to a single centralized location in third-party/install.

Custom build

The third-party dependency build can also be done by hand, for example, to specify custom software locations or to change the configuration of Embree or OSPRay. Note that even with a custom build, the submodules still should be patched as described below.

To download and patch the submodules: cd to the root directory of your local Galaxy repository and run the script ./scripts/prep-third-party.sh, which will (1) init and update the git submodules, and (2) download ispc to where Galaxy expects to find it. This script will populate the third-party/ispc, third-party/embree, third-party/ospray, and third-pary/rapidjson subdirectories.

To build and install the submodules, do the following in third-party/embree, then third-part/ospray, then third-party/rapidjson (third-party/ispc is already installed via the script):

mkdir build
cd build
cmake <cmake flags> .. && make && make install

NOTE: for the CMake flags used for each third-party library, see the CMAKE_FLAGS variables defined in scripts/install-third-party.sh.

Local installs of Galaxy associated dependencies

If you prefer to use local installs of any of the dependencies, you can follow instructions below. Make sure to issue git submodule init <path> and git submodule update <path> for the third-party dependencies you do NOT have locally installed. For example, to use the rapidjson submodule, type:

git submodule init third-party/rapidjson
git submodule update third-party/rapidjson

ISPC

The third-party/ispc directory contains a script to download a recent ISPC binary for MacOS or Linux. ISPC installed using this script should be detected automatically by the Galaxy CMake configuration. If you already have a recent ISPC installed (at least version 1.9.1) you are free to use it, though you might need to specify its location by hand in the CMake configurations for Embree, OSPRay, and Galaxy.

From the root directory of your Galaxy repository, issue the following commands:

cd third-party/ispc
./get-ispc.sh

This will download ISPC (currently version 1.12.0) into third-party/ispc/ispc-v<version>-<OS type> (e.g., ispc-v1.10.0-macOS), and it will install the ispc executable in third-party/install/bin. If this binary does not work on your system, you will need to build ISPC by hand following the directions at the [ISPC website][10].

Embree

After updating the git submodules as described above, the third-party/embree directory should contain the Embree source tree. We recommend building in third-party/embree/build and installing to third-party/install, as doing so should enable Galaxy to find Embree. The recommended install directory is configured as part of the Embree cmake flags in scripts/install-third-party.sh.

First, create a build directory and build Embree. Assuming CMake can find all required dependencies, you can use the cmake command to configure the makefile for the Embree build:

mkdir build
cd build
cmake <cmake flags> .. && make && make install

If cmake complains about missing dependencies, you can specify or change their locations using cmake -D<CMAKE VAR> command-line syntax or using the interactive ccmake interface with ccmake .. in the build directory. See scripts/install-third-party.sh for recommended cmake flags.

OSPRay

Before installing OSPRay, make sure you have updated the Galaxy git submodules and successfully built Embree, as described above. Once the git submodules have been updated, the third-party/ospray directory should contain the OSPRay source tree. We recommend building in third-party/ospray/build and installing to third-party/install, as doing so should enable Galaxy to find OSPRay automatically. The recommended install directory is configured as part of the OSPRay cmake flags in scripts/install-third-party.sh.

First, create a build directory and build OSPRay. Assuming CMake can find all required dependencies, you can use the cmake command to configure the makefile for the OSPRay build:

mkdir build
cd build
cmake <cmake flags> .. && make && make install

rapidjson

Before installing rapidjson, make sure you have updated the Galaxy git submodules as described above. Once the git submodules have been updated, the third-party/rapidjson directory should contain the rapidjson source tree. We recommend building in third-party/rapidjson/build and installing to third-party/install, as doing so should enable Galaxy to find rapidjson automatically. The recommended install directory is configured as part of the RapidJSON cmake flags in scripts/install-third-party.sh.

First, create a build directory and build rapidjson. Assuming CMake can find all required dependencies, you can use the cmake command to configure the makefile for the rapidjson build:

mkdir build
cd build
cmake <cmake flags> .. && make && make install

Galaxy

Install / Use

README