Teal
Teal is an open-source data transformation and workflow (DAG) management tool in the Go stack.
Install / Use
/learn @go-teal/TealREADME
Teal
- Teal
In the changing field of data engineering, having strong, scalable, and user-friendly tools is essential. We introduce Teal, a new open-source ETL tool designed to improve your data transformation and orchestration.
Teal combines the best features of tools like dbt, Dagster, and Airflow, while solving common problems found in traditional Python-based solutions. Our goal is to provide data engineers and analysts with a powerful, easy-to-use platform that simplifies complex workflows and increases productivity.
Why Choose Teal?
- Flexible Integration: Integrate smoothly with various data sources and destinations, offering great flexibility and connectivity.
- Optimized Performance with Go: Teal uses Go's concurrency model with goroutines and channels to maximize performance and efficiency. This ensures your data pipelines run quickly and reliably, making the best use of your system's resources.
- Go Stack Advantage: Built on the efficient Go stack, Teal offers high performance, low latency, and excellent scalability. The simplicity and power of Go provide a solid foundation for managing complex ETL workflows.
QuickStart
Installation
go install github.com/go-teal/teal/cmd/teal@latest
CLI Commands Reference
Teal CLI provides the following commands to manage your data pipeline projects:
teal init <!-- omit from toc -->
Creates a basic Teal project structure with default configuration files.
teal init
This command initializes a new Teal project with:
config.yaml(database connections)profile.yaml(project configuration)assets/directory structure with example models and testsstore/directory with sample CSV data
No flags required.
teal gen <!-- omit from toc -->
Generates Go code from SQL asset model files.
teal gen [flags]
Flags:
--project-path string- Project directory (default:.)--config-file string- Path to config.yaml (default:config.yaml)--model string- Name of target model to generate (optional, generates all if not specified)
Examples:
teal gen # Generate all models in current directory
teal gen --project-path ./my-project # Generate in specific directory
teal gen --model staging.customers # Generate specific model only
teal gen --config-file custom-config.yaml # Use custom config file
teal clean <!-- omit from toc -->
Cleans generated files from the project.
teal clean [flags]
Flags:
--project-path string- Project directory (default:.)--model string- Models for cleaning (default:*for all)--clean-main- Delete production main.go incmd/<project-name>/--clean-main-ui- Delete UI debug main.go incmd/<project-name>-ui/--clean-dockerfile- Delete Dockerfile--clean-go-mod- Delete go.mod and go.sum--clean-all- Delete ALL generated files (prompts for confirmation)
Examples:
teal clean # Clean all models (with confirmation)
teal clean --model staging.customers # Clean specific model
teal clean --clean-main # Clean production main.go only
teal clean --clean-main-ui # Clean UI main.go only
teal clean --clean-dockerfile # Clean Dockerfile only
teal clean --clean-go-mod # Clean go.mod and go.sum
teal clean --clean-all # Clean ALL generated files
teal clean --project-path ./my-project # Clean in specific directory
Note:
- When cleaning all models (
*), you will be prompted for confirmation. --clean-allwill delete ALL generated files including go.mod, Dockerfile, and main files.
Files NOT Overwritten by teal gen:
The following files are generated only once and will NOT be overwritten on subsequent teal gen executions:
Dockerfile- Container configuration (skip if exists)go.mod- Go module definition (skip if exists)cmd/<project-name>/<project-name>.go- Production binary main file (skip if exists)cmd/<project-name>-ui/<project-name>-ui.go- UI debug binary main file (skip if exists)
All other files (assets, tests, configs, docs) are regenerated on every teal gen run.
To regenerate these protected files, use the appropriate --clean-* flags before running teal gen.
teal ui <!-- omit from toc -->
Starts the UI development server with hot-reload for debugging and monitoring.
teal ui [flags]
Flags:
--port int- Port for API server (default:8080). UI Dashboard runs on port+1.--log-level string- Log level:debug,info,warn,error(default:debug)--project-path string- Project directory (default:.)
Examples:
teal ui # Start on default port 8080 (Dashboard on 8081)
teal ui --port 9090 # Start on port 9090 (Dashboard on 9091)
teal ui --log-level info # Start with info log level
teal ui --project-path ./my-project # Start for specific project
The UI provides:
- DAG Visualization: Interactive graph showing all assets and dependencies
- Execution Control: Trigger DAG runs and monitor task status
- Test Results: View test execution results and data quality checks
- Asset Inspection: Examine asset data and execution results
- Real-time Logs: View logs for specific task executions
Access: Open http://localhost:8081 (or custom port + 1) in your browser.
teal version <!-- omit from toc -->
Shows the current version of Teal CLI.
teal version
No flags required.
Getting Help <!-- omit from toc -->
teal --help # Show all commands and their flags
teal [command] --help # Show detailed help for specific command
Creating your project <!-- omit from toc -->
mkdir my_test_project
cd my_test_project
Init your project from scratch <!-- omit from toc -->
teal init
❯ ls -al
total 16
drwxr-xr-x@ 6 wwtlf wwtlf 192 24 Jun 21:23 .
drwxr-xr-x 5 wwtlf wwtlf 160 24 Jun 21:21 ..
drwxr-xr-x@ 3 wwtlf wwtlf 96 24 Jun 07:46 assets
-rw-r--r--@ 1 wwtlf wwtlf 302 24 Jun 07:51 config.yaml
drwxr-xr-x@ 2 wwtlf wwtlf 64 24 Jun 20:03 docs
-rw-r--r--@ 1 wwtlf wwtlf 137 24 Jun 07:46 profile.yaml
Update config.yaml <!-- omit from toc -->
version: '1.0.0'
module: github.com/my_user/my_test_project
connections:
- name: default
type: duckdb
config:
path: ./store/test.duckdb
extensions:
- postgres
- httpfs
# extraParams:
# - name: "name"
# value: "value"
moduleparam will be used as a module in go.mod- Make sure the dir from the
pathexists.
Update profile.yaml <!-- omit from toc -->
version: '1.0.0'
name: 'my-test-project'
connection: 'default'
models:
stages:
- name: staging
- name: dds
- name: mart
namewill be used as a name for the binary file
Generate go project <!-- omit from toc -->
teal gen
You'll see the following output
project-path: .
config-file: ./config.yaml
Building: staging.stg_airports.sql
Building: staging.stg_crew_assignments.sql
Building: staging.stg_employees.sql
Building: staging.stg_flights.sql
Building: staging.stg_routes.sql
Building: dds.dim_airports.sql
Building: dds.dim_employees.sql
Building: dds.dim_routes.sql
Building: dds.fact_crew_assignments.sql
Building: dds.fact_flights.sql
Building: mart.mart_airport_statistics.sql
Building: mart.mart_crew_utilization.sql
Building: mart.mart_flight_performance.sql
Files 28
./cmd/hello-world/hello-world.go ...................................... [OK]
./cmd/hello-world-ui/hello-world-ui.go ................................ [OK]
./go.mod .............................................................. [OK]
./Makefile ............................................................ [OK]
./Dockerfile .......................................................... [OK]
./internal/assets/staging.stg_airports.go ............................. [OK]
./internal/assets/staging.stg_crew_assignments.go ..................... [OK]
./internal/assets/staging.stg_employees.go ............................ [OK]
./internal/assets/staging.stg_flights.go .............................. [OK]
./internal/assets/staging.stg_routes.go ............................... [OK]
./internal/assets/dds.dim_airports.go ................................. [OK]
./internal/assets/dds.dim_employees.go ................................ [OK]
./internal/assets/dds.dim_routes.go ................................... [OK]
./in
