Morphogenesis

Acknowledgement

"2014, Hoetzlein, Rama Karl. Fast Fixed-Radius Nearest Neighbors: Interactive Million-Particle Fluids. GPU Technology Conference, 2014. San Jose, CA. 2010-2014. Online at http://fluids3.com"

Morphogenesis started as a minimal variant of Rama Hoetzlein's Fluids SPH, cut down from gFluidSurface in gvdb-voxels, https://github.com/ramakarl/gvdb-voxels, which was in turn developed from Fluids-v3.

Master branch

This is the cut down version of gFluidSurface. Dependence on gvdb-voxels library has been removed, and CMakeLists.txt has been rewritten. New output has been written to provide ascii .ply files for viewing in MeshLab.

This code compiles and runs with cmake 3.10, vtk-9.0, and Cuda 11.2 on Ubuntu 20.04 with GTX 980m. and on Suse Linux cluster with Cuda 9.1 and Tesla P100.

Morphogenesis branch

A morphogenesis simulator (in progress) , with soft-matter elasticity, diffusion of heat/chemicals/morphogens, epi-genetics and particle automata behaviour.

The notes below are rough working notes, and will change with development.

Dependencies

VTK is used for .vtp file output for visualization in Parraview. NB vtk-dev is broken in Ubuntu 18.04 LTS. You may need to build VTK from source. This code has been developed with VTK-9.0.1 .

***NB if you need to build VTK, please remember to ***

sudo make install
sudo ldconfig

so that the new library will be found at runtime.

Build instructions

This project uses Cmake. Create a build subdirectory. In the build subdirectory

cmake ../
make
make install

Within Morphogenesis branch executables (so far) include:

make_demo

usage:

cd data
make_demo  num_particles  spacing  x_dim  y_dim  z_dim  demoType  simSpace

where demoType(0:free falling, 1:remodelling & actuation, 2:diffusion & epigenetics.)

and simSpace{0:regression test, 1:Tower(256,128,256), 2:Wave pool(400,200,400), 3:Small dam break(80,60,80), 4:Dual-Wave pool(200,100,30), 5: Microgravity(160,100.160) }

demoType sets individual particle properties in "demo/particles_pos_vel_color100001.csv", especially epigenetic states, from 3D positions.

simSpace sets parameters in "SimParams." , especially gravity, and wavepool actuation.

e.g.

../build/install/bin/make_demo 125 1  6 6 6  0 5       // free falling

../build/install/bin/make_demo 120 1  2 2 30  1 5
../build/install/bin/make_demo 600 1  4 4 30  1 5      // remodelling & actuation, with fixed, bone, tendon, muscle, elastic, mesenchyme, external actuation
../build/install/bin/make_demo 2000 1  6 6 30  1 5     //  "" , with reserve particles for growth.

../build/install/bin/make_demo 400 1  10 10 3  1 5     // diffusion & epigenetics, with reserve particles for growth.


../build/install/bin/make_demo 10000 1  100 10 10  1 1
../build/install/bin/make_demo 100000 1  100 10 100  1 1
../build/install/bin/make_demo 100000 1  100 100 10  1 1
../build/install/bin/make_demo 1000000 1  100 100 100  1 1

NB memory usage

Factors affecting memory requrement for Morphogenesis:

0/ Available GPU memory depends on the device + other sofftware using the GPU, especially yuor desktop, GUIs, graphics and video.
To see your current GPU memory useage and total GPU memory, call "nvidia-smi" from the commandline.

    NB When running a GUI onthe GPU it is advised to use models with fewer than 100k bins and particles.

1/ The number of bins in the simulation volume. i.e. volume/particle_radius^3
Dense arrays of bins are maintained for sorting particles. genes, remodelling.
These arrays are use to profuce the dense lists of particles for each kernel, for efficient computation.

2/ The number of particles in the simulation

3/ The number of genes in the simulation will increase the size of the bin arrays(1), and the data per particle(2).

4/ The number of active genes among the particles will increase the size of the dense lists of particles for each active gene (1).

5/ The number of particles subject to remodelling per time step, will incease the size of the dense lists for each remodelling kernel.


    NB the simulation volume and particle radius are set in SimParams.txt by :
    m_Param[PSIMSCALE], m_Param[PGRID_DENSITY], m_Param[PSMOOTHRADIUS],  
    m_Vec[PVOLMIN], m_Vec[PVOLMAX].
    
    Where: 
    
    m_GridRes = (PVOLMAX-PVOLMIN)*PGRID_DENSITY*PSIMSCALE / (2*PSMOOTHRADIUS)
    
    Total number of bins = m_GridRes.x * m_GridRes.y * m_GridRes.z
    
    
    For default PGRID_DENSITY=2, PSIMSCALE=0.005, PSMOOTHRADIUS=0.015,
    This gives  m_GridRes = (PVOLMAX-PVOLMIN)/3
    So 
    10x10x10 vol -> 333 bins
    30x30x30 vol -> 9,000 bins
    60x60x60 vol -> 72,000 bins
    100x100x100 vol -> 333,3333 bins
    150x150x150 vol -> 1,125,000 bins
    1000x1000x1000 vol -> 333,333,333 bins. Too big for most GPUs, would need multi-GPU on cluster.

Processing speed

Factors affecting prcoessing speed include:

0/ Available GPU blocks and cores per block, after space taken by other GPU contxts for other programs.

1/ The number of bins. These have to be processesd every time step to sort the particle data and generate the dense lists.

2/ The number of particles. 
NB there is the possiblity of "solid" and "living" particles into separate lists (using genes), to avoid running elastic and gene kernels on fluid and non-living tissue.

3/ The length of the "active gene" and "remodelling" particle lists per time step.
NB the system of dense lists, provides faster running for models with few active genes per particle, and few particles remodelling per time step.

CPU-only test program to generate example "SimParams.txt" and "particles_pos_vel_color100001.csv" files for specifying models, and a "particles_pos100001.ply" for viewing a model in e.g. Meshlab.

check_demo

usage:

cd data
check_demo  simulation_data_folder  output_folder

e.g.

../build/install/bin/check_demo  demo  check 

CPU-only test program to verify the ability to read and re-output models.

load_sim

New launch program to load data from files, and run simulation on GPU.

usage:

cd data
../build/install/bin/load_sim  demo  out   num_files   steps_per_file   freeze_steps      save_ply(y/n)  save_csv(y/n)  save_vtp(y/n)      debug(y/n)  gene_activity(y/n)  remodelling(y/n)

usage: load_sim  simulation_data_folder  output_folder  num_files  steps_per_file  freeze_steps save_ply(y/n)  save_csv(y/n)  save_vtp(y/n)  debug(0-5) gene_activity(y/n)  remodelling(y/n) 

debug: 0=full speed, 1=current special output,  2=host cout, 3=device printf, 4=SaveUintArray(), 5=save .csv after each kernel.

e.g.

cd data/test
./load_sim ../demo/ ../out/  10 3 1 y y y
./load_sim ../demo/ ../out/  10 1 6 n y y

./load_sim ../demo/ ../out/  1000 10 0 n y y y y y                   // NB Now '100' frames per timestep x02-x20 snapshot after each kernel. x91 end of timestep. x00 begining of simulation.
./load_sim ../demo/ ../out/  1000 1 10 n y y y n y                  // NB now 'freeze_steps' delay the start of particle movement, while heal() forms initial bonds.
./load_sim ../demo/ ../out/  1000 1 10 n y y y n n
./load_sim ../demo/ ../out/  1000 1 10 n y y n y y

./load_sim ../demo/ ../out/  200 3 1 n n y 1 y y                    // Good launch option for profiling

./load_sim ../demo_10000_1_100_10_10/ ../out/  100 30 1 n n y
./load_sim ../demo_100000_1_100_100_10/ ../out/  100 30 1 n n y
./load_sim ../demo_1000000_1_100_100_100/ ../out/  100 30 1 n n y

profiling the code

If you have Caua-toolkit installed, enter "nsight-sys" at the commandline to open "Nvidia Nsight Systems".

See https://developer.nvidia.com/nsight-systems , https://docs.nvidia.com/nsight-systems/index.html

Follow the nsight-systems InstallationGuide.
NB especially wrt setting /proc/sys/kernel/perf_event_paranoid    
https://docs.nvidia.com/nsight-systems/InstallationGuide/index.html#linux-requirements

For a an introduction to how to use Nsight Systems, see "Blue Waters Webinar: Introduction to NVIDIA Nsight Systems"   https://www.youtube.com/watch?v=WA8C48FJi3c 

Note, setting the "debug" value (0-5) when launching "load_sim" will have a big effect on the memory transfers, flies saved and console output, and therefore the performance.

viewing with Meshlab

The .ply files output can be viewed in MeshLab. It is recommended to select the following MeshLab options: NB need old MeshLab (Ubuntu18.04). Version on Ubuntu 20.04 does not work.

Render - Show vertex dots
Render - Render Mode - Wireframe

viewing with Paraview

The .vtp files output can be viewed in Paraview. This allows visualization of all the parameters, and is especially relevant for diffusion of morphogens, epigenetic state, and material properties.

ParaView is a widely used scientific data visualization tool.
ParaView can be downloaded from https://www.paraview.org/

NB GPU conflict:
It is advised to exit Paraview before launching Morphogenesis.
Sometimes other runtime errors arrise if Paraview is still running when Morphogenesis is launched.

Both Paraview and Morphogenesis will both try to use your GPU. 
This may result in an _"invalid device context"_ or _"There is no device supporting CUDA"_ error.
This seems to happen specifically where the computer suspends while ParaView is open.
If ParaView fails to release the GPU after being shut down, then it may be necessary to reboot.

This does not arise where the two