Kestra
Event Driven Orchestration & Scheduling Platform for Mission Critical Applications
Install / Use
/learn @kestra-io/KestraREADME
🌟 What is Kestra?
Kestra is an open-source, event-driven orchestration platform that makes both scheduled and event-driven workflows easy. By bringing Infrastructure as Code best practices to data, process, and microservice orchestration, you can build reliable workflows directly from the UI in just a few lines of YAML.
Key Features:
- Everything as Code and from the UI: keep workflows as code with a Git Version Control integration, even when building them from the UI.
- Event-Driven & Scheduled Workflows: automate both scheduled and real-time event-driven workflows via a simple
triggerdefinition. - Declarative YAML Interface: define workflows using a simple configuration in the built-in code editor.
- Rich Plugin Ecosystem: hundreds of plugins built in to extract data from any database, cloud storage, or API, and run scripts in any language.
- Intuitive UI & Code Editor: build and visualize workflows directly from the UI with syntax highlighting, auto-completion and real-time syntax validation.
- Scalable: designed to handle millions of workflows, with high availability and fault tolerance.
- Version Control Friendly: write your workflows from the built-in code Editor and push them to your preferred Git branch directly from Kestra, enabling best practices with CI/CD pipelines and version control systems.
- Structure & Resilience: tame chaos and bring resilience to your workflows with namespaces, labels, subflows, retries, timeout, error handling, inputs, outputs that generate artifacts in the UI, variables, conditional branching, advanced scheduling, event triggers, backfills, dynamic tasks, sequential and parallel tasks, and skip tasks or triggers when needed by setting the flag
disabledtotrue.
🧑💻 The YAML definition gets automatically adjusted any time you make changes to a workflow from the UI or via an API call. Therefore, the orchestration logic is always managed declaratively in code, even if you modify your workflows in other ways (UI, CI/CD, Terraform, API calls).
<p align="center"> <img src="https://kestra.io/adding-tasks.gif" alt="Adding new tasks in the UI"> </p>🚀 Quick Start
Launch on AWS (CloudFormation)
Deploy Kestra on AWS using our CloudFormation template:
Launch on Google Cloud (Terraform deployment)
Deploy Kestra on Google Cloud Infrastructure Manager using our Terraform module.
Get Started Locally in 5 Minutes
Launch Kestra in Docker
Make sure that Docker is running. Then, start Kestra in a single command:
docker run --pull=always -it -p 8080:8080 --user=root \
--name kestra --restart=always \
-v kestra_data:/app/storage \
-v /var/run/docker.sock:/var/run/docker.sock \
-v /tmp:/tmp \
kestra/kestra:latest server local
If you're on Windows and use PowerShell:
docker run --pull=always -it -p 8080:8080 --user=root `
--name kestra --restart=always `
-v "kestra_data:/app/storage" `
-v "/var/run/docker.sock:/var/run/docker.sock" `
-v "C:/Temp:/tmp" `
kestra/kestra:latest server local
If you're on Windows and use Command Prompt (CMD):
docker run --pull=always -it -p 8080:8080 --user=root ^
--name kestra --restart=always ^
-v "kestra_data:/app/storage" ^
-v "/var/run/docker.sock:/var/run/docker.sock" ^
-v "C:/Temp:/tmp" ^
kestra/kestra:latest server local
If you're on Windows and use WSL (Linux-based environment in Windows):
docker run --pull=always -it -p 8080:8080 --user=root \
--name kestra --restart=always \
-v kestra_data:/app/storage \
-v "/var/run/docker.sock:/var/run/docker.sock" \
-v "/mnt/c/Temp:/tmp" \
kestra/kestra:latest server local
Check our Installation Guide for other deployment options (Docker Compose, Podman, Kubernetes, AWS, GCP, Azure, and more).
Access the Kestra UI at http://localhost:8080 and start building your first flow!
Your First Hello World Flow
Create a new flow with the following content:
id: hello_world
namespace: dev
tasks:
- id: say_hello
type: io.kestra.plugin.core.log.Log
message: "Hello, World!"
Run the flow and see the output in the UI!
🧩 Plugin Ecosystem
Kestra's functionality is extended through a rich ecosystem of plugins that empower you to run tasks anywhere and code in any language, including Python, Node.js, R, Go, Shell, and more. Here's how Kestra plugins enhance your workflows:
-
Run Anywhere:
- Local or Remote Execution: Execute tasks on your local machine, remote servers via SSH, or scale out to serverless containers using Task Runners.
- Docker and Kubernetes Support: Seamlessly run Docker containers within your workflows or launch Kubernetes jobs to handle compute-intensive workloads.
-
Code in Any Language:
- Scripting Support: Write scripts in your preferred programming language. Kestra supports Python, Node.js, R, Go, Shell, and others, allowing you to integrate existing codebases and deployment patterns.
- Flexible Automation: Execute shell commands, run SQL queries against various databases, and make HTTP requests to interact with APIs.
-
Event-Driven and Real-Time Processing:
- Real-Time Triggers: React to events from external systems in real-time, such as file arrivals, new messages in message buses (Kafka, Redis, Pulsar, AMQP, MQTT, NATS, AWS SQS, Google Pub/Sub, Azure Event Hubs), and more.
- Custom Events: Define custom events to trigger flows based on specific conditions or external signals, enabling highly responsive workflows.
-
Cloud Integrations:
- AWS, Google Cloud, Azure: Integrate with a variety of cloud services to interact with storage solutions, messaging systems, compute resources, and more.
- Big Data Processing: Run big data processing tasks using tools like Apache Spark or interact with analytics platforms like Google BigQuery.
-
Monitoring and Notifications:
- Stay Informed: Send messages to Slack channels, email notifications, or trigger alerts in PagerDuty to keep your team updated on workflow statuses.
Kestra's plugin ecosystem is continually expanding, allowing you to tailor the platform to your specific needs. Whether you're orchestrating complex data pipelines, automating scripts across multiple environments, or integrating with cloud services, there's likely a plugin to assist. And if not, you can always build your own plugins to extend Kestra's capabilit
