Ppl
Parallelo Parallel Library (PPL) is a small parallel framework that brings Structured Parallel Programming in Rust.
Install / Use
/learn @valebes/PplREADME
Parallelo Parallel Library (PPL): Unlock the Power of Parallel Computing in Rust
🌟 Welcome to Parallelo Parallel Library (PPL) – a small, but powerful, parallel framework written in Rust. 🚀
Parallelo Parallel Library (PPL) is an under-development parallel framework written in Rust. The main goal of PPL is to provide a structured approach to parallel programming in Rust, allowing developers to unleash the power of parallelism without having to deal with low-level concurrency management mechanisms. PPL empowers your Rust programs by unlocking the immense potential of parallelism, making your computations faster and more efficient. Whether you're working on large-scale data processing, simulations, or any computationally intensive task, PPL has got you covered.
Table of Contents
Features
-
Parallel Computing: Unleash the full power of parallelism in Rust with the Parallelo Parallel Library (PPL). Tap into the potential of multiple cores to turbocharge your computations, making them faster and more efficient than ever before.
-
Task, Stream, and Data Parallelism: PPL offers tools to express task, stream, and data parallelism models. Harness the power of task parallelism to break down your computations into smaller tasks that can be executed in parallel. Handle continuous data streams and process structured datasets in parallel, enabling efficient data processing and analysis.
-
Work-Stealing Thread Pool: PPL includes a work-stealing thread pool that elevates parallel execution to new heights. Thanks to the work-stealing thread pool, tasks are dynamically distributed among available threads, ensuring load balancing and efficient utilization of computational resources. This feature boosts the efficiency of parallel computations, enabling superior performance and scalability in your Rust applications.
-
Efficient Resource Management: Optimal resource management is vital in the realm of parallel programming. PPL grants you fine-grained control over resource allocation, including choose the number of threads to use, set up thread wait policies, and enable CPU pinning. This level of control ensures the optimal utilization of your hardware resources, resulting in enhanced performance and efficiency for your parallel computations.
-
Multiple Channel Backends: PPL provides you with the freedom to choose from a range of channel backends for seamless communication and coordination between parallel tasks. Take advantage of popular channel implementations like crossbeam-channel, flume, and kanal. Each backend possesses its own unique performance characteristics, memory usage, and feature set, allowing you to select the one that perfectly aligns with your specific requirements. These versatile channel backends facilitate smooth data flow and synchronization within your parallel computations.
-
Flexibility and Customization: With PPL you have the flexibility to create custom nodes for your pipeline, allowing you to express more complex parallel computations. This empowers you to tailor your parallel pipelines to suit your precise needs, creating highly customizable parallel workflows. Stateful nodes can store intermediate results and maintain crucial context information, enabling efficient data sharing between different stages of the computation. This flexibility and customizability augment the expressiveness of PPL, enabling you to tackle a wide range of parallel programming scenarios.
-
Intuitive API: Whether you're a seasoned parallel computing expert or a parallelism novice, PPL simplifies parallel programming with its intuitive API based on Structured Parallel Programming. Developers of all skill levels can effortlessly harness the power of parallel computing in Rust. The intuitive API ensures a smooth learning curve, allowing you to dive straight into parallelizing your computations with ease.
Installation
Parallelo Parallel Library (PPL) is currently available on both GitHub and Crates.io.
Please make sure you have Rust and Cargo installed on your system before proceeding. If you don't have them installed, you can get them from the official Rust website: https://www.rust-lang.org/tools/install
To use PPL in your Rust project, you can add the following in your Cargo.toml:
[dependencies]
ppl = { git = "https://github.com/valebes/ppl.git" }
Usage
A Simple (but long) Example: Fibonacci pipeline
Here a simple example that show how create a pipeline that computes the first 20 numbers of Fibonacci.
This pipeline is composed by:
- Source: emits number from 1 to 20.
- Stage: computes the i-th number of Fibonacci.
- Sink: accumulates and return a Vec with the results.
Here a snipped of code to show how build this pipeline:
use ppl::{prelude::*, templates::misc::{SourceIter, Sequential, SinkVec}};
fn main() {
let mut p = pipeline![
SourceIter::build(1..21),
Sequential::build(fib), // fib is a method that computes the i-th number of Fibonacci
SinkVec::build()
];
p.start();
let res = p.wait_end().unwrap().len();
assert_eq!(res, 20)
}
In this example we're building a simple pipeline, so why not use clousure instead than the library templates?
use ppl::prelude::*;
fn main() {
let mut p = pipeline![
{
let mut counter = 0;
move || {
if counter < 20 {
counter += 1;
Some(counter)
} else {
None
}
}
},
|input| Some(fib(input)),
|input| println!("{}", input)
];
p.start();
p.wait_end();
}
The run-time supports of the framework will call the clousure representing the Source till it returns None, when the Source returns None then a termination message will be propagated to the other stages of the pipeline.
If we want to parallelize the computation we must find a part of this algorithm that can be parallelized. In this case the stage in the middle is a good candidate, we replicate that stage introducing a Farm.
Now the code is as follows:
use ppl::{prelude::*, templates::misc::{SourceIter, Parallel, SinkVec}};
fn main() {
let mut p = pipeline![
SourceIter::build(1..21),
Parallel::build(8, fib), // We create 8 replicas of the stage seen in the previous code.
SinkVec::build()
];
p.start();
let res = p.wait_end().unwrap().len();
assert_eq!(res, 20)
}
If we don't want to use the templates offered by the library, there is the possibility to create custom stage. By creating custom stages it is possible to build stateful and more complex nodes for our pipeline.
Here the same example but using custom defined stages:
use ppl::prelude::*;
struct Source {
streamlen: usize,
counter: usize,
}
impl Out<usize> for Source {
// This method must be implemented in order to implement the logic of the node
fn run(&mut self) -> Option<usize> {
if self.counter < self.streamlen {
self.counter += 1;
Some(self.counter)
} else {
None
}
}
}
#[derive(Clone)]
struct Worker {}
impl InOut<usize, usize> for Worker {
// This method must be implemented in order to implement the logic of the node
fn run(&mut self, input: usize) -> Option<usize> {
Some(fib(input))
}
// We can override this method to specify the number of replicas of the stage
fn number_of_replicas(&self) -> usize {
8
}
}
struct Sink {
counter: usize,
}
impl In<usize, usize> for Sink {
// This method must be implemented in order to implement the logic of the node
fn run(&mut self, input: usize) {
println!("{}", input);
self.counter += 1;
}
// If at the end of the stream we want to return something, we can do it by implementing this method.
fn finalize(self) -> Option<usize> {
println!("End");
Some(self.counter)
}
}
fn main() {
let mut p = pipeline