Dagster

Dagster is an open-source data orchestrator designed to help teams build, schedule, and monitor reliable data pipelines. It provides a unified framework for defining data assets, transformations, and dependencies as code, enabling reproducible and testable workflows. Dagster’s primary purpose is to improve the quality, observability, and maintainability of data systems across analytics, machine learning, and production data applications.

Dagster introduces a typed, Python-based programming model that lets you define “ops” and “jobs” with clear inputs, outputs, and dependencies, making pipelines more modular and testable. Its asset-based approach allows you to model data as first-class objects, track lineage, and reason about how changes propagate through your ecosystem. The platform includes a web-based UI (Dagster UI) for visualizing pipelines, monitoring runs, inspecting logs, and managing schedules and sensors. It integrates with tools such as dbt, Spark, Snowflake, BigQuery, Kubernetes, and cloud storage, and supports versioning, configuration management, and environment isolation.