TrilogyT and Dagster

We love the modern design of dagster.

It's one of the two core orchestration engines we currently support in TrilogyT, alongside DBT.

Integration

TrilogyT Dagster support is based on assets.

Your persist statements will be turned into assets; any optimized CTEs will be created as assets, and dependencies will be defined between them.

An example generated asset for a duckdb model.

from dagster import asset
from dagster_duckdb import DuckDBResource

from assets.optimization.dsdd968c4c1215b184ec36e1ed881d193d3e8e2ad062dd6750257f78115dccdfd7_gen_model import (
    dsdd968c4c1215b184ec36e1ed881d193d3e8e2ad062dd6750257f78115dccdfd7,
)


@asset(
    deps=[
        dsdd968c4c1215b184ec36e1ed881d193d3e8e2ad062dd6750257f78115dccdfd7,
    ]
)
def dim_splits(duck_db: DuckDBResource) -> None:
    with duck_db.get_connection() as conn:
        conn.execute(
            """ 
CREATE OR REPLACE TABLE dim_splits AS

WITH 
quizzical as (
SELECT
    cast(get_current_timestamp() as datetime) as "_trilogyt__created_at"
),
dynamic as (
SELECT
    dsdd968c4c1215b184ec36e1ed881d193d3e8e2ad062dd6750257f78115dccdfd7."generic_split" as "generic_split"
FROM
    dsdd968c4c1215b184ec36e1ed881d193d3e8e2ad062dd6750257f78115dccdfd7
GROUP BY 
    dsdd968c4c1215b184ec36e1ed881d193d3e8e2ad062dd6750257f78115dccdfd7."generic_split")
SELECT
    dynamic."generic_split" as "generic_split",
    quizzical."_trilogyt__created_at" as "_trilogyt__created_at"
FROM
    dynamic
    FULL JOIN quizzical on 1=1 """
        )

Adoption

To integrate trilogy into a project, we recommend incremental adoption starting with building trilogy models into a subset of your existing dagster environment.

Trilogy does not support all features of dagster, focusing primarily on SQL orchestration. Please open feature requests for missing issues.