Bayesian Algorithm for Luminosity Function Fitting (BALFF)

README for the Bayesian Algorithm for Luminosity Function Fitting (BALFF) presented in Schmidt et al. (2014) and extended to include magnification bias in Mason et al. (2015)

If you find this code useful please cite

• Schmidt et al. (2014), ApJ, 786:57, http://adsabs.harvard.edu/abs/2014ApJ...786...57S
• Mason et al. (2015), ApJ, 805:79, http://adsabs.harvard.edu/abs/2015ApJ...805...79M

Table of Content

• Description
• Script Overview
• Dependencies and Requirements
  • Standard Packages
  • Special Packages
• Running BALFF
  • Default Run
  • Accounting for Magnification Bias
• Main Keywords in BALFF
• References

Description

The Bayesian Algorithm for Luminosity Function Fitting (BALFF) presented in Schmidt et al. (2014) is an algorithm that fits the (UV) luminosity function to a sample of galaxies (e.g., photometrically selected Lyman Break Galaxies, LBGs) selected from a set of observations. BALFF uses the Schechter Function shown to be a good approximation for the underlying distribution at low and high redshift, as its luminosity function model. The Bayesian formalism which BALFF is build on, avoids binning (and thus smearing) of object samples, includes a likelihood based on the formally correct binomial distribution as opposed to the often-used approximate Poisson distribution, and models the photometric uncertainties of each object in the sample directly, making full use of the full information providing more rigorous results. In Mason et al. (2015) the default BALFF setup, was extended to account for the magnification bias of galaxies in the input sample from both strong and weak gravitational lensing by foreground sources.

Luminosity functions obtained with BALFF (or any other fitting method) can be used to discuss the physics of reionization as also presented in Schmidt et al. (2014). Scripts accompanying the core of the BALFF tool provides means for such a discussion and comparison. The fraction of neutral hydrogren at the targeted redshift is estimated assuming theoretically motivated priors on the clumping factor and the photon escape fraction.

Note that in the current BALFF framework the p(z) prior on the redshift of the individual objects (LBG candidates) is not explicitly taken into account.

For detailed information on the Bayesian framework and the formal derivation and description of the terms in the Bayesian expressions, please refer to Schmidt et al. (2014) and Mason et al. (2015).

Script Overview

The following gives and overview of the scripts provided with the BALFF code

• balff_run_commands.sh
  • Shell script with default commands to run all necessary steps in a full BALFF analysis.
• balff_run.py
  • Wrapper setting up the pymc MCMC sample used to obtain the full BALFF luminosity function estimate and uncertainties.
• balff_mpd.py
  • The main class used to load the input, setting up the framework, and calculating the marginal posterior distribution at the heart of the BALFF luminosity function fitting framework.
• balff_estimatePhiStar.py
  • Estimating the luminosity function normalization (phi*) for the luminosity function determined for the input data by BALFF.
• balff_estimateEpsilon.py
  • Estimating the luminosity density (epsilon) for a given integration limit, corresponding to the luminosity function determined for the input data by BALFF.
• balff_plot_epsilonQCf.py
  • Estimating the neutral hydrogen fraction of the IGM assuming default values (distributions) for the escape fraction, clumping factor, and conversion between luminosity density and ionizing photons. Diagnostic plots are generated.
• balff_plot.py
  • Generating diagnostic plots of the samples and luminosity function resulting from running BALFF on an input data array.
• balff_createDataArray.py
  • Generating a data array on the fits format expected by BALFF from provided object names, field names, luminosities, and limiting luminosities.
• balff_createDataArray_sim.py
  • Generating a data array for a simulated sample of data. I.e., data with a known input Schechter Function distribution. Useful for testing and de-bugging.
• balff_createLookupTable.py
  • Generating a look-up tables of tabulated values on the binary .npz format expected by BALFF. This look-up table can be used to speed up the BALFF run via interpolation of values, rather than running the full integration for each MCMC step.
• balff_createSelectionFunctionFiles.py
  • Storing selection and completeness functions in the .npz file format expected by BALFF. Can also be used to plot and estimate values of these. If the completeness C(m) is included in the selection function S(m,z), completeness function files of constant 1 can be generated for BALFF input.
• balff_utilities.py
  • Various utilities used in the BALFF code.

Dependencies and Requirements

The code is written in python and uses a wide range of default packages included in standard installations of python. A few special packages as outlined below, needs to be installed on top of that to get BALFF running.

Standard Packages

The following standard packages are imported in on or more of the BALFF scripts: os, sys, pdb, time, glob, numpy, pylab, scipy, types, mpmath, pyfits, commands, argparse, datetime, cosmocalc, matplotlib, mpl_toolkits, multiprocessing,

Special Packages

• pymc: The default MCMC sampler is pymc which needs to be installed. To run pymc a FORTRAN and C compiler needs to be available. E.g., gfortran and gcc both available at http://hpc.sourceforge.net. Alternatively, compilers can be installed on MacOSX via the Xcode app. The pymc instructions can be found at https://pymc-devs.github.io/pymc/INSTALL.html.
• pymc_steps: A robust adaptive metropils algorithm (RAM) for stepping in parameters space in the pymc chains. Available at https://github.com/brandonckelly/pymc_steps
• astropysics: Suite of astronomy related utilities. Available here at https://pythonhosted.org/Astropysics/
• fast_kde: Used for plotting balff_plot_epsilonQCf.confcontours(). Provided in BALFF GitHub repository.

Running BALFF

In this section the default run of BALFF (Schmidt et al. 2014) and a run of BALFF accounting for the magnification bias of sources (Mason et al. 2015) are described.

To run BALFF and generate the diagnostic plots and output files, a set of data inputs are required as described below. BALFF assumes that data are stored in the directory ./balff_data/. The data output and plots generated by running BALFF will be stored in ./balff_output and ./balff_plots. Both of these directories will be created if they do not exist.

Default Run

The default run of BALFF corresponds to running the code described by Schmidt et al. (2014). This run requires the following data input (the names used below only refer to the default set of files provide in the BALFF GitHub directory):

• fields_info.txt: This files contain a minimal set of information needed for BALFF. The columns of the file are:
  • fieldname: String contain the unique name of the field
  • filter: Filter which the field was observed in (corresponding to the fileter used to assemble the luminosity function)
  • maglim5sigma: 5sigma limiting magnitude (including Galactic extinction) of the field
  • Av: A_V value for the field
  • fieldarea: Field area in arc minutes
  • magmedfield: Median observed magnitude error over field
• objects_info.fits: This binary fits table contain information on the individual objects that a luminosity functions should be fitted to. In the case of the BoRG sample analyzed by Schmidt et al. (2014)) and Mason et al. 2015 these are photometrically selected LBGs. The fits format expected by BALFF can be generated using balff_createDataArray.py. The columns in the binary table are:
  • OBJNAME: String containing the unique name of the object
  • FIELD: Name of field (from fields_info.txt) the object is located in
  • LOBJ: The luminosity of the object in units of []1e-44 erg/s]. I.e., this includes the redshift information when converting the apparent magnitude to absolute magnitude and then to luminosity (These conversion can be done using the tools in balff_utilities.py)
  • LOBJERR: The error on the object's luminosity
  • LFIELDLIM: The Nsigma limiting luminosity of the field the object was found in (corresponds to maglimNsigma from fields_info.txt)
  • LFIELDLIMERR: The error on the limiting luminosity of the field
• selectionfunctions/Szm*SN*.npz: Files containing selection function for each individual field. The selection function is assumed to be dependent on both the apparent magnitude and the redshift, i.e., S(z,m). By default S(z,m) does not include the completeness function. This is provided with C*SN*.npz. However, S(z,m) does include the completeness function of constant 1 can be provided. Known selection functions can be converted into the .npz format expected by BALFF with balff_createSelectionFunctionFiles.
• selectionfunctions/C*SN*.npz: Files containing the completeness function for each indiv