Bacalhau
Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.
Install / Use
/learn @bacalhau-project/BacalhauREADME
What is Bacalhau?
Bacalhau is an open-source distributed compute orchestration framework designed to bring compute to the data. Instead of moving large datasets around networks, Bacalhau makes it easy to execute jobs close to the data's location, drastically reducing latency and resource overhead.
Why Bacalhau?
- ⚡ Fast job processing: Jobs in Bacalhau are processed where the data was created and all jobs are parallel by default
- 💰 Low cost: Reduce (or eliminate) ingress/egress costs since jobs are processed closer to the source
- 🔒 Secure: Data scrubbing and security can happen before migration, with a granular, code-based permission model
- 🚛 Large-scale data: Process petabytes of data efficiently without massive data transfers
- 🏢 Data sovereignty: Process sensitive data within security boundaries without requiring it to leave your premises
- 🤝 Cross-organizational computation: Allow specific vetted computations on protected datasets without exposing raw data
Key Features
-
Single Binary Simplicity: Bacalhau is a single self-contained binary that functions as a client, orchestrator, and compute node—making it incredibly easy to set up and scale
-
Modular Architecture: Support for multiple execution engines (Docker, WebAssembly) and storage providers through clean interfaces
-
Orchestrator-Compute Model: A dedicated orchestrator coordinates job scheduling, while compute nodes run tasks
-
Flexible Storage Integrations: Integrates with S3, HTTP/HTTPS, IPFS, and local storage systems
-
Multiple Job Types: Support for batch, ops, daemon, and service job types for different workflow requirements
-
Declarative & Imperative Submissions: Define jobs in YAML (declarative) or pass arguments via CLI (imperative)
-
Publisher Support: Output results to local volumes, S3, or other storage backends
Getting Started
Quick Installation
# Install Bacalhau CLI (Linux/macOS)
curl -sL https://get.bacalhau.org/install.sh | bash
# Verify installation
bacalhau version
For the complete quick start guide, including running your first job, see our Quick Start Documentation.
Use Cases
Bacalhau's distributed compute framework enables a wide range of applications:
- Log Processing: Process logs efficiently at scale by running distributed jobs directly at the source
- Distributed Data Warehousing: Query and analyze data across multiple regions without moving large datasets
- Fleet Management: Efficiently manage distributed nodes across multiple environments
- Distributed Machine Learning: Train and deploy ML models across a distributed compute fleet
- Edge Computing: Run compute tasks closer to the data source for applications requiring low latency
Documentation
📚 Read the Bacalhau docs guide here! 📚
The Bacalhau documentation contains all the information you need to get started:
Community & Contributing
Bacalhau has a very friendly community, and we are always happy to help:
- Join the Slack Community and go to the
#generalchannel - it is the easiest way to engage with other members in the community and get help
If you are interested in contributing to the Bacalhau project:
- Set up your local environment
- Check out our Contributing Guide
- For issues and feature requests, please open a GitHub issue
We are excited to hear your feedback!
Open Source
This repository contains the Bacalhau software, covered under the Apache-2.0 license, except where noted (any Bacalhau logos or trademarks are not covered under the Apache License, and should be explicitly noted by a LICENSE file.)
Bacalhau is a product produced from this open source software, exclusively by Expanso, Inc. It is distributed under our commercial terms.
Others are allowed to make their own distribution of the software, but they cannot use any of the Bacalhau trademarks, cloud services, etc.
We explicitly grant permission for you to make a build that includes our trademarks while developing Bacalhau software itself. You may not publish or share the build, and you may not use that build to run Bacalhau software for any other purpose.
We have borrowed the above Open Source clause from the excellent System Initiative
Related Skills
feishu-drive
335.2k|
things-mac
335.2kManage Things 3 via the `things` CLI on macOS (add/update projects+todos via URL scheme; read/search/list from the local Things database)
clawhub
335.2kUse the ClawHub CLI to search, install, update, and publish agent skills from clawhub.com
SchoolAnalytics
Skill: IB MYP Analytics & Grading Activation Trigger - Any task involving grade calculations, student flagging, or analytics dashboarding. - Questions about Criteria A, B, C, or D. Knowledge
