EtlFlow

EtlFlow is an ecosystem of functional libraries in Scala based on ZIO for running complex Auditable jobs/workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases and more.

Below are some important features of this library, some of which come from ZIO.

• Universal. It provides a consistent way to interact with different services/products across cloud platforms like GCP, AWS, Azure, and On-Premises systems.
• Functional. Rapidly compose complex jobs/workflows from simple tasks.
• Auditable. Build jobs/workflows that provides auditability by default for multiple backends.
• Resource-safe. Build jobs/workflows that never leak resources (including threads!), even when they fail.
• Compile Time DI. Build jobs/workflows that allows resolving dependencies at compile time.
• Fibers. Built on non-blocking fibers that never waste or leak resources, which lets you build scalable, resilient, and reactive applications
• Concurrent and Asynchronous. Easily build concurrent asynchronous or synchronous jobs/workflows without deadlocks, race conditions, or complexity.
• Type-safe. Use the full power of the Scala compiler to catch bugs at compile time.
• Testable. Inject test services into your job/workflow for fast, deterministic, and type-safe testing.
• Resilient. Build jobs/workflows that never lose errors, and which respond to failure locally and flexibly.

Examples

• Core Module:
  In this example project, you can explore core features of etlflow, Task and Audit API.
• GCP Module (GCS, DataProc, BigQuery tasks):
  In this example project, you can explore GCP tasks.
• K8S Module (K8S tasks):
  In this example project, you can explore different Kubernetes tasks.
• Spark Module (Spark tasks):
  In this example project, you can explore different Apache Spark tasks.

Modules Dependency Graph

| Module | Latest Version | Documentation | Scala Versions | Java Version | |--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| | Core | | | | Java 8 + | | GCP | | | | Java 8 + | | JDBC | | | | Java 8 + | | Http | | | | Java 11 + | | K8S | | | | Java 8 + | | Email | | | | Java 8 + | | AWS | | | | Java 8 + | | FTP | | | | Java 8 + | | Redis | | | | Java 8 + | | Spark | | | [](https://index.scala-lang.org/tharwaninitin/e