PADLL: Programmable and Adaptable I/O for Dynamically Loaded Libraries

PADLL is a storage middleware that enables system administrators to proactively and holistically control the rate of data and metadata workflows to achieve QoS in HPC storage systems. Its design if build under the following core principles:

Application and file system agnostic: PADLL does not require code changes to any core layer of the HPC I/O stack, being applicable over multiple applications and cross-compatible with POSIX-compliant file systems. PADLL achieves this using LD_PRELOAD.
Fine-grained I/O control: PADLL classifies, differentiates and controls requests at different levels of granularity, including operation type (open, read, mkdir), operation class (metadata, data, extended attributes), user, and job.
Global visibility: PADLL ensures holistic control of all I/O workflows and coordinated access to the file system, preventing contention and unfair usage or shared storage resources. This is achieved using Cheferd, a hierarchical Software-Defined Storage control plane.
Custom QoS specification: PADLL enables system administrators to create custom QoS policies for rate limiting jobs running at the cluster (e.g., uniform and priorty-based rate distribution, proportion sharing, DRF), protecting the file system from greedy jobs and I/O burstiness.

The storage middleware follows a Software-Defined Storage approach, being composed of two main components:

Data plane (PADLL): The data plane is a multi-stage component that provides the building blocks for differentiating and rate limiting I/O workflows. The data plane corresponds to this repository.
Control plane (Cheferd): The control plane is a global coordinator that manages all data plane stages to ensure that storage QoS policies are met over time and adjusted according to workload variations. The control plane repository can be found at dsrhaslab/cheferd.

Please cite the following paper if you use PADLL:

Taming Metadata-intensive HPC Jobs Through Dynamic, Application-agnostic QoS Control. Ricardo Macedo, Mariana Miranda, Yusuke Tanimura, Jason Haga, Amit Ruhela, Stephen Lien Harrell, Richard Todd Evans, José Pereira, João Paulo. 23rd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGrid 2023).

@inproceedings {Macedo2023Padll,
    title     = {Taming Metadata-intensive HPC Jobs Through Dynamic, Application-agnostic QoS Control},
    author    = {Ricardo Macedo and Mariana Miranda and Yusuke Tanimura and Jason Haga and Amit Ruhela and Stephen Lien Harrell and Richard Todd Evans and Jos{\'e} Pereira and Jo{\~a}o Paulo},
    booktitle = {23rd IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing},
    year      = {2023}
}

If you use PADLL, give us some feedback on how to further improve it!

Getting started with PADLL

This tutorial will guide on how to set up, benchmark, and use PADLL.

Requirements and Dependencies

PADLL is written with C++17 and was built and tested with g++-9.3.0 and cmake-3.16. The core library was built using the PAIO storage data plane framework (install instructions below). It also uses the spdlog v1.8.1 logging library (installed at compile time).

Further, PADLL uses the following third party libraries, which are embedded as single-header files: Xoshiro-cpp (pseudorandom number generator library), tabulate (library for printing aligned, formatted, and colorized tables), better-enums (compile-time enum library).

Install PAIO

$ cd /path/to/dir   # select the path to clone the PAIO github repository
$ git clone https://github.com/dsrhaslab/paio.git
$ cd paio
$ git checkout v1.0.0
$ mkdir build; cd build
$ cmake ..; cmake --build .
$ export CPATH="/path/to/dir/paio/include:${CPATH}"
$ export LD_LIBRARY_PATH="/path/to/dir/paio/build/:${LD_LIBRARY_PATH}"

Setup PADLL

$ cd /path/to/dir   # select the path to clone the PADLL github repository
$ $ git clone https://github.com/dsrhaslab/padll.git
$ cd padll
$ git checkout v1.0.0
$ vi CMakeLists.txt   # line19: set path to the PAIO build
$ mkdir build; cd build;
$ cmake ..
$ cmake --build .
$ export PATH_PADLL=$PWD

Configuring and tuning PADLL

PADLL provides two sets of configurations:

options.hpp: configurations related to the data plane stage are placed in the options header file.
libc_calls.hpp: configurations related to which POSIX operations should be intercepted and handled with PADLL.

options header

# include/padll/options/options.hpp

Mount point differentiation
- option_default_remote_mount_point : "/tmp"  # IMPORTANT: define the default mount point to control I/O requests

Statistic collection
- option_default_statistic_collection : true  # simple statistic collection to validate which requests were successfully handled

Data plane stage configuration
- option_default_stage_name : "padll-stage" # default name of the data plane stage
- option_paio_environment_variable_env : "paio_env" # environment variable to set additional information for the stage

Connection to control plane and standalone mode
- option_sync_with_controller : true  # IMPORTANT: defines if the data plane stage should execute standalone, or connect to the control plane  
- option_default_connection_address_env : "cheferd_local_address" # environment variable to define the connection address to the control plane local controller
- main_path : "/path/to/padll/files" # if running on standalone mode, define that path for housekeeping, differentiation, and enforcement files
- option_default_hsk_rules_file : "hsk-simple-test" # if running on standalone mode, define the path to the housekeeping rules file (will define PAIO's channels and enforcement objects)

Logging and Debugging
- option_default_log_path : "/tmp/padll-info" # default path for PADLL logging files
- option_default_statistics_report_path : "/tmp"  # main path to store statistic reports
- OPTION_DETAILED_LOGGING : false # detailed logging (mainly used for debugging)

libc calls header

# include/padll/configurations/libc_calls.hpp

# to replicate the micro-benchmarking experiments of the PADLL paper (§5.1), 
# one should enable the following system calls.
- padll_intercept_open_var : true
- padll_intercept_open : true
- padll_intercept_close : true
- padll_intercept_rename : true
- padll_intercept_getxattr : true
- padll_intercept_read : true
- padll_intercept_write : true
# remainder configurations should be set at false

Configuring and tuning PAIO

PADLL's core internals are built using the PAIO data plane framework, namely the request differentiation and rate limiting. As such, depending on the use case that you may want to use PADLL, there are a few tuning knobs in PAIO that should be properly configured. These configurations are in PAIO options header file (options.hpp).

# include/paio/options/options.hpp

Debugging and ld_preload settings
- option_default_debug_log : true   # minor logging messages do not introduce significant overhead
- option_default_ld_preload_enabled : true  # when using LD_PRELOAD, this should be set to prevent cyclic libc.so dependencies

Statistic collection
- option_default_channel_statistic_collection : true  # per-channel statistic collection
- option_default_statistic_metric : StatisticMetric::counter  # simple statistic counter (optionally use StatisticMetric::throughput)

Channel-level differentiation
- option_default_channel_differentiation_workflow : true  # classify requests based on their workflow-id
- option_default_channel_differentiation_operation_type : false # classify requests based on their operation type
- option_default_channel_differentiation_operation_context : false  # classify requests based on their operation context

Context identifier
- option_default_context_type : ContextType::POSIX  # class of operations to consider
- option_default_statistic_classifier : ClassifierType::operation_type  # type of classifier to be used on statistic counting

Simple test

A straightforward example on how to use and test PADLL is with the padll_scalability_bench.

$ cd path/to/padll
$ ./build/padll_scalability_bench 1 1 1000000  # <number-of-runs> <number-of-working-threads (same as workflows)> <number-of-operations>

This is a simple benchmark that generates POSIX open calls in a closed-loop, based on a configurable number of threads and total of operations. The expected result (depicted below) shows that the benchmark can submit approximately 531kops (only open operations).

Date:      2023-02-05 11:53:47
CPU:       6 * Intel(R) Core(TM) i5-9500 CPU @ 3.00GHz
CPUCache:  9216 KB
------------------------------------
Executing ./padll_scalability_bench: 1 runs -- 1 threads -- 1000000 ops
Starting worker thread #1 (140453677

Padll

Install / Use

README