Construct information pipelines with dbt in Amazon Redshift utilizing Amazon MWAA and Cosmos

Efficient collaboration and scalability are important for constructing environment friendly information pipelines. Nevertheless, information modeling groups usually face challenges with advanced extract, rework, and cargo (ETL) instruments, requiring programming experience and a deep understanding of infrastructure. This complexity can result in operational inefficiencies and challenges in sustaining information high quality at scale.

dbt addresses these challenges by offering an easier strategy the place information groups can construct sturdy information fashions utilizing SQL, a language they’re already acquainted with. When built-in with trendy growth practices, dbt tasks can use model management for collaboration, incorporate testing for information high quality, and make the most of reusable elements via macros. dbt additionally robotically manages dependencies, ensuring information transformations execute within the appropriate sequence.

On this submit, we discover a streamlined, configuration-driven strategy to orchestrate dbt Core jobs utilizing Amazon Managed Workflows for Apache Airflow (Amazon MWAA) and Cosmos, an open supply bundle. These jobs run transformations on Amazon Redshift, a completely managed information warehouse that permits quick, scalable analytics utilizing normal SQL. With this setup, groups can collaborate successfully whereas sustaining information high quality, operational effectivity, and observability. Key steps lined embody:

• Making a pattern dbt undertaking
• Enabling auditing inside the dbt undertaking to seize runtime metrics for every mannequin
• Making a GitHub Actions workflow to automate deployments
• Establishing Amazon Easy Notification Service (Amazon SNS) to proactively alert on failures

These enhancements allow model-level auditing, automated deployments, and real-time failure alerts. By the top of this submit, you’ll have a sensible and scalable framework for operating dbt Core jobs with Cosmos on Amazon MWAA, so your group can ship dependable information workflows sooner.

Resolution overview

The next diagram illustrates the answer structure.

The workflow comprises the next steps:

1. Analytics engineers handle their dbt undertaking of their model management device. On this submit, we use GitHub for example.
2. We configure an Apache Airflow Directed Acyclic Graph (DAG) to make use of the Cosmos library to create an Airflow process group that comprises all of the dbt fashions as a part of the dbt undertaking.
3. We use a GitHub Actions workflow to sync the dbt undertaking recordsdata and the DAG to an Amazon Easy Storage Service (Amazon S3) bucket.
4. In the course of the DAG run, dbt converts the fashions, assessments, and macros to Amazon Redshift SQL statements, which run immediately on the Redshift cluster.
5. If a process within the DAG fails, the DAG invokes an AWS Lambda operate to ship out a notification utilizing Amazon SNS.

Stipulations

It’s essential to have the next stipulations:

Create a dbt undertaking

A dbt undertaking is structured to facilitate modular, scalable, and maintainable information transformations. The next code is a pattern dbt undertaking construction that this submit will comply with:

MY_SAMPLE_DBT_PROJECT
├── .github
│   └── workflows
│       └── publish_assets.yml
└── src
    ├── dags
    │   └── dbt_sample_dag.py
    └── my_sample_dbt_project
        ├── macros
        ├── fashions
        └── dbt_project.yml

dbt makes use of the next YAML recordsdata:

• dbt_project.yml –  Serves as the principle configuration in your undertaking. Objects on this undertaking will inherit settings outlined right here except overridden on the mannequin degree. For instance:

# Title your undertaking! Mission names ought to comprise solely lowercase characters
# and underscores. 
identify: 'my_sample_dbt_project'
model: '1.0.0'

# These configurations specify the place dbt ought to search for various kinds of recordsdata.
# The `model-paths` config, for instance, states that fashions on this undertaking might be
# discovered within the "fashions/" listing. 
model-paths: ["models"]
macro-paths: ["macros"]

# Configuring fashions
# Full documentation: https://docs.getdbt.com/docs/configuring-models
# On this instance config, we inform dbt to construct fashions within the instance/
# listing as views. These settings might be overridden within the particular person mannequin
# recordsdata utilizing the `{{ config(...) }}` macro.
fashions:
  my_sample_dbt_project:
    # Config indicated by + and applies to recordsdata below fashions/instance/
    instance:
      +materialized: view
      
on-run-end:
# add run outcomes to audit desk 
  - "{{ log_audit_table(outcomes) }}" 

• sources.yml – Defines the exterior information sources that your dbt fashions will reference. For instance:

sources:
  - identify: sample_source
    database: sample_database
    schema: sample_schema
    tables:
      - identify: sample_table

• schema.yml – Outlines the schema of your fashions and information high quality assessments. Within the following instance, we’ve outlined two columns, full_name for the mannequin model1 and sales_id for model2. Now we have declared them as the first key and outlined information high quality assessments to examine if the 2 columns are distinctive and never null.

model: 2

fashions:
  - identify: model1
    config: 
      contract: {enforced: true}

    columns:
      - identify: full_name
        data_type: varchar(100)
        constraints:
          - sort: primary_key
        assessments:
          - distinctive
          - not_null

  - identify: model2
    config: 
      contract: {enforced: true}

    columns:
      - identify: sales_id
        data_type: varchar(100)
        constraints:
          - sort: primary_key
        assessments:
          - distinctive
          - not_null

Allow auditing inside dbt undertaking

Enabling auditing inside your dbt undertaking is essential for facilitating transparency, traceability, and operational oversight throughout your information pipeline. You may seize run metrics on the mannequin degree for every execution in an audit desk. By capturing detailed run metrics resembling load identifier, runtime, and variety of rows affected, groups can systematically monitor the well being and efficiency of every load, rapidly determine points, and hint adjustments again to particular runs.

The audit desk consists of the next attributes:

• load_id – An identifier for every mannequin run executed as a part of the load
• database_name – The identify of the database inside which information is being loaded
• schema_name – The identify of the schema inside which information is being loaded
• identify – The identify of the thing inside which information is being loaded
• resource_type – The kind of object to which information is being loaded
• execution_time – The time period taken for every dbt mannequin to finish execution as a part of every load
• rows_affected – The variety of rows affected within the dbt mannequin as a part of the load

Full the next steps to allow auditing inside your dbt undertaking:

1. Navigate to the fashions listing (src/my_sample_dbt_project/fashions) and create the audit_table.sql mannequin file:

{%- set run_date = "CURRENT_DATE" -%}
{{
    config(
        materialized='incremental',
        incremental_strategy='append',
        tags=["audit"]
    )
}}

with empty_table as (
    choose
        'test_load_id'::varchar(200) as load_id,
        'test_invocation_id'::varchar(200) as invocation_id,
        'test_database_name'::varchar(200) as database_name,
        'test_schema_name'::varchar(200) as schema_name,
        'test_model_name'::varchar(200) as identify,
        'test_resource_type'::varchar(200) as resource_type,
        'test_status'::varchar(200) as standing,
        solid('12122012' as float) as execution_time,
        solid('100' as int) as rows_affected,
        {{run_date}} as model_execution_date
)

choose * from empty_table
-- This can be a filter so we are going to by no means really insert these values
the place 1 = 0

2. Navigate to the macros listing (src/my_sample_dbt_project/macros) and create the parse_dbt_results.sql macro file:

{% macro parse_dbt_results(outcomes) %}
    -- Create an inventory of parsed outcomes
    {%- set parsed_results = [] %}
    -- Flatten outcomes and add to listing
    {% for run_result in outcomes %}
        -- Convert the run outcome object to a easy dictionary
        {% set run_result_dict = run_result.to_dict() %}
        -- Get the underlying dbt graph node that was executed
        {% set node = run_result_dict.get('node') %}
        {% set rows_affected = run_result_dict.get(
        'adapter_response', {}).get('rows_affected', 0) %}
        {%- if not rows_affected -%}
            {% set rows_affected = 0 %}
        {%- endif -%}
        {% set parsed_result_dict = {
                'load_id': invocation_id ~ '.' ~ node.get('unique_id'),
                'invocation_id': invocation_id,
                'database_name': node.get('database'),
                'schema_name': node.get('schema'),
                'identify': node.get('identify'),
                'resource_type': node.get('resource_type'),
                'standing': run_result_dict.get('standing'),
                'execution_time': run_result_dict.get('execution_time'),
                'rows_affected': rows_affected
                }%}
        {% do parsed_results.append(parsed_result_dict) %}
    {% endfor %}
    {{ return(parsed_results) }}
{% endmacro %}

3. Navigate to the macros listing (src/my_sample_dbt_project/macros) and create the log_audit_table.sql macro file:

{% macro log_audit_table(outcomes) %}
    -- depends_on: {{ ref('audit_table') }}
    {%- if execute -%}
        {{ print("Operating log_audit_table Macro") }}
        {%- set run_date = "CURRENT_DATE" -%}
        {%- set parsed_results = parse_dbt_results(outcomes) -%}
         size  > 0 -%
            {% set allowed_columns = ['load_id', 'invocation_id', 'database_name', 
            'schema_name', 'name', 'resource_type', 'status', 'execution_time', 
            'rows_affected', 'model_execution_date'] -%}
            {% set insert_dbt_results_query -%}
                insert into {{ ref('audit_table') }}
                    (
                        load_id,
                        invocation_id,
                        database_name,
                        schema_name,
                        identify,
                        resource_type,
                        standing,
                        execution_time,
                        rows_affected,
                        model_execution_date
                ) values
                    {%- for parsed_result_dict in parsed_results -%}
                        (
                            {%- for column, worth in parsed_result_dict.gadgets() %}
                                {% if column not in allowed_columns %}
                                    {{ exceptions.raise_compiler_error("Invalid
                                     column") }}
                                {% endif %}
                                 exchange("'", "''") %
                                '{{ sanitized_value }}'
                                {%- if not loop.final %}, {% endif %}
                            {%- endfor -%}
                        )
                        {%- if not loop.final %}, {% endif %}
                    {%- endfor -%}
            {%- endset -%}
            {%- do run_query(insert_dbt_results_query) -%}
        {%- endif -%}
    {%- endif -%}
    {{ return ('') }}
{% endmacro %}

4. Append the next strains to the dbt_project.yml file:

on-run-end:
  - "{{ log_audit_table(outcomes) }}" 

Create a GitHub Actions workflow

This step is non-compulsory. In the event you want, you’ll be able to skip it and as a substitute add your recordsdata on to your S3 bucket.

The next GitHub Actions workflow automates the deployment of dbt undertaking recordsdata and DAG file to Amazon S3. Substitute the placeholders {s3_bucket_name}, {account_id}, {role_name}, and {area} along with your S3 bucket identify, account ID, IAM function identify, and AWS Area within the workflow file.

To reinforce safety, it’s really useful to make use of OpenID Join (OIDC) for authentication with IAM roles in GitHub Actions as a substitute of counting on long-lived entry keys.

identify: Sync dbt Mission with S3

on:
  workflow_dispatch:
  push:
    branches: [ main ]
    paths:
      - "src/**"

permissions:
  id-token: write   # That is required for requesting the JWT
  contents: learn    # That is required for actions/checkout
  pull-requests: write

jobs:
  sync-dev:
    runs-on: ubuntu-latest
    surroundings: dev
    defaults:
      run:
        shell: bash
    steps:
      - makes use of: actions/checkout@v4
      - identify: Assume AWS IAM Position
        makes use of: aws-actions/[email protected]
        with:
          aws-region: {area}
          role-to-assume: arn:aws:iam::{account_id}:function/{role_name}
          role-session-name: my_sample_dbt_project_${{ github.run_id }}
          role-duration-seconds: 3600 # 1 hour

      - run: aws sts get-caller-identity

      - identify: Sync dbt Mannequin recordsdata
        id: dbt_project_files
        working-directory: src/my_sample_dbt_project
        run: aws s3 sync . s3://{s3_bucket_name}/dags/dbt/my_sample_dbt_project 
        --delete
        continue-on-error: false

      - identify: Sync DAG recordsdata
        id: dag_file
        working-directory: src/dags
        run: aws s3 sync . s3://{s3_bucket_name}/dags

GitHub has the next safety necessities:

• Department safety guidelines – Earlier than continuing with the GitHub Actions workflow, be sure that department safety guidelines are in place. These guidelines implement required standing checks earlier than merging code into protected branches (resembling predominant).
• Code overview pointers – Implement code overview processes to ensure adjustments bear overview. This will embody requiring at the least one approving overview earlier than code is merged into the protected department.
• Incorporate safety scanning instruments – This will help detect vulnerabilities in your repository.

Ensure you are additionally adhering to dbt-specific safety greatest practices:

• Take note of dbt macros with variables and validate their inputs.
• When including new packages to your dbt undertaking, consider their safety, compatibility, and upkeep standing to ensure they don’t introduce vulnerabilities or conflicts into your undertaking.
• Assessment dynamically generated SQL to safeguard in opposition to points like SQL injection.

Replace the Amazon MWAA occasion

Full the next steps to replace the Amazon MWAA occasion:

1. Set up the Cosmos library on Amazon MWAA by including astronomer-cosmos within the necessities.txt file. Make sure that to examine for model compatibility for Amazon MWAA and the Cosmos library.
2. Add the next entries in your startup.sh script:
   1. Within the following code, DBT_VENV_PATH specifies the placement the place the Python digital surroundings for dbt shall be created. DBT_PROJECT_PATH factors to the placement of your dbt undertaking inside Amazon MWAA.

      #!/bin/sh
      export DBT_VENV_PATH="${AIRFLOW_HOME}/dbt_venv"
      export DBT_PROJECT_PATH="${AIRFLOW_HOME}/dags/dbt"

   2. The next code creates a Python digital surroundings on the path ${DBT_VENV_PATH} and installs the dbt-redshift adapter to run dbt transformations on Amazon Redshift:

      python3 -m venv "${DBT_VENV_PATH}"
      ${DBT_VENV_PATH}/bin/pip set up dbt-redshift

Create a dbt person in Amazon Redshift and retailer credentials

To create dbt fashions in Amazon Redshift, you have to arrange a local Redshift person with the mandatory permissions to entry supply tables and create new tables. It’s important to create separate database customers with minimal permissions to comply with the precept of least privilege. The dbt person shouldn’t be granted admin privileges, as a substitute, it ought to solely have entry to the precise schemas required for its duties.

Full the next steps:

1. Open the Amazon Redshift console and join as an admin (for extra particulars, confer with Connecting to an Amazon Redshift database).
2. Run the next command within the question editor v2 to create a local person, and observe down the values for dbt_user_name and password_value:

   create person {dbt_user_name} password 'sha256|{password_value}';

3. Run the next instructions within the question editor v2 to grant permissions to the native person:
   1. Hook up with the database the place you wish to supply tables from and run the next instructions:

      grant utilization on schema {schema_name} to {dbt_user_name};
      grant choose on all tables in schema {schema_name} to {dbt_user_name};

   2. To permit the person to create tables inside a schema, run the next command:

      grant create on schema {schema_name} to {dbt_user_name};

4. Optionally, create a secret in AWS Secrets and techniques Supervisor and retailer the values for dbt_user_name and password_value from the earlier step as plaintext:

{
    "username":"dbt_user_name",
    "password":"password_value"
}

Making a Secrets and techniques Supervisor entry is non-compulsory, however really useful for securely storing your credentials as a substitute of hardcoding them. To study extra, confer with AWS Secrets and techniques Supervisor greatest practices.

Create a Redshift connection in Amazon MWAA

We create one Redshift connection in Amazon MWAA for every Redshift database, ensuring that every information pipeline (DAG) can solely entry one database. This strategy gives distinct entry controls for every pipeline, serving to forestall unauthorized entry to information. Full the next steps:

1. Log in to the Amazon MWAA UI.
2. On the Admin menu, select Connections.
3. Select Add a brand new document.
4. For Connection Id, enter a reputation for this connection.
5. For Connection Sort, select Amazon Redshift.
6. For Host, enter the endpoint of the Redshift cluster with out the port and database identify (for instance, redshift-cluster-1.xxxxxx.us-east-1.redshift.amazonaws.com).
7. For Database, enter the database of the Redshift cluster.
8. For Port, enter the port of the Redshift cluster.

Arrange an SNS notification

Establishing SNS notifications is non-compulsory, however they could be a helpful enhancement to obtain alerts on failures. Full the next steps:

1. Create an SNS matter.
2. Create a subscription to the SNS matter.
3. Create a Lambda operate with the Python runtime.
4. Modify the operate code in your Lambda operate, and exchange {topic_arn} along with your SNS matter Amazon Useful resource Title (ARN):

import json

sns_client = boto3.shopper('sns')

def lambda_handler(occasion, context):
     strive:
        # Extract DAG identify from occasion
        failed_dag = occasion['dag_name']
        
        # Ship notification 
        sns_client.publish(
            TopicArn={topic_arn}, 
            Topic="Knowledge modelling dags - WARNING", 
            Message=json.dumps({'default': json.dumps(f"Knowledge modelling DAG - 
            {failed_dag} has failed, please inform the information modelling group")}),
            MessageStructure="json"
        )
        
    besides KeyError as e:
        # Deal with lacking 'dag_name' within the occasion
        logger.error(f"KeyError: invalid payload - dag_name not current")

Configure a DAG

The next pattern DAG orchestrates a dbt workflow for processing and auditing information fashions in Amazon Redshift. It retrieves credentials from Secrets and techniques Supervisor, runs dbt duties in a digital surroundings, and sends an SNS notification if a failure happens. The workflow consists of the next steps:

1. It begins with the audit_dbt_task process group, which creates the audit mannequin.
2. The transform_data process group executes the opposite dbt fashions, excluding the audit-tagged one. Contained in the transform_data group, there are two dbt fashions, model1 and model2, and every is adopted by a corresponding check process that runs information high quality assessments outlined within the schema.yml file.
3. To correctly detect and deal with failures, the DAG features a dbt_check Python process that runs a customized operate, check_dbt_failures. That is vital as a result of when utilizing DbtTaskGroup, particular person model-level failures contained in the group don’t robotically propagate to the duty group degree. Consequently, downstream duties (such because the Lambda operator sns_notification_for_failure) configured with trigger_rule="one_failed" won’t be triggered except a failure is explicitly raised.

The check_dbt_failures operate addresses this by inspecting the outcomes of every dbt mannequin and check, and elevating an AirflowException if a failure is discovered. When an AirflowException is raised, the sns_notification_for_failure process is triggered.

4. If a failure happens, the sns_notification_for_failure process invokes a Lambda operate to ship an SNS notification. If no failures are detected, this process is skipped.

The next diagram illustrates this workflow.

Configure DAG variables

To customise this DAG in your surroundings, configure the next variables:

• project_name – Make sure that the project_name matches the S3 prefix of your dbt undertaking
• secret_name – Present the identify of the key that shops dbt person credentials
• target_database and target_schema – Replace these variables to mirror the place you wish to land your dbt fashions in Amazon Redshift
• redshift_connection_id – Set this to match the connection configured in Amazon MWAA for this Redshift database
• sns_lambda_function_name – Present the Lambda operate identify to ship SNS notifications
• dag_name – Present the DAG identify that shall be handed to the SNS notification Lambda operate

import os
import json
import boto3
from airflow import DAG
from cosmos import (
    DbtTaskGroup, ProfileConfig, ProjectConfig,
    ExecutionConfig, RenderConfig
)
from cosmos.constants import ExecutionMode, LoadMode
from cosmos.profiles import RedshiftUserPasswordProfileMapping
from pendulum import datetime
from airflow.operators.python_operator import PythonOperator
from airflow.suppliers.amazon.aws.operators.lambda_function import (
    LambdaInvokeFunctionOperator
)
from airflow.exceptions import AirflowException

# undertaking identify - ought to match the s3 prefix of your dbt undertaking
project_name = "my_sample_dbt_project"
# identify of the key that shops dbt person credentials 
secret_name = "dbt_user_credentials_secret"
# goal database to land dbt fashions
target_database = "sample_database"
# goal schema to land dbt fashions
target_schema = "sample_schema"
# Redshift connection identify from MWAA
redshift_connection_id = "my_sample_dbt_project_connection"
# sns lambda operate identify
sns_lambda_function_name = "sns_notification"
# dag identify - this shall be handed to SNS for notification
payload = json.dumps({
            "dag_name": "my_sample_dbt_project_dag"
        })

Incorporate DAG elements

After setting the variables, now you can incorporate the next elements to finish the DAG.

Secrets and techniques Supervisor

The DAG retrieves dbt person credentials from Secrets and techniques Supervisor:

sm_client = boto3.shopper('secretsmanager')

def get_secret(secret_name):
    strive:
        get_secret_value_response = sm_client.get_secret_value(SecretId=secret_name)
        return json.hundreds(get_secret_value_response["SecretString"])
    besides Exception as e:
        increase

secret_value = get_secret(secret_name)
username = secret_value["username"]
password = secret_value["password"]

Redshift connection configuration

It makes use of RedshiftUserPasswordProfileMapping to authenticate:

profile_config = ProfileConfig(
    profile_name="redshift",
    target_name=target_database,
    profile_mapping=RedshiftUserPasswordProfileMapping(
        conn_id=redshift_connection_id,
        profile_args={"schema": target_schema,
                      "person": username, "password": password}
    ),
)

dbt execution setup

This code comprises the next variables:

• dbt executable path – Makes use of a digital surroundings
• dbt undertaking path – Is positioned within the surroundings variable DBT_PROJECT_PATH below your undertaking

execution_config = ExecutionConfig(
    dbt_executable_path=f"{os.environ['DBT_VENV_PATH']}/bin/dbt",
    execution_mode=ExecutionMode.VIRTUALENV,
)

project_config = ProjectConfig(
    dbt_project_path=f"{os.environ['DBT_PROJECT_PATH']}/{project_name}",
)

Duties and execution circulate

This step consists of the next elements:

• Audit dbt process group (audit_dbt_task) – Runs the dbt mannequin tagged with audit
• dbt process group (transform_data) – Runs the dbt fashions tagged with operations, excluding the audit mannequin

In dbt, tags are labels that you could assign to fashions, assessments, seeds, and different dbt sources to prepare and selectively run subsets of your dbt undertaking. In your render_config, you’ve exclude=["tag:audit"]. This implies dbt will exclude fashions which have the tag audit, as a result of the audit mannequin runs individually.

• Failure examine (dbt_check) – Checks for dbt mannequin failures, raises an AirflowException if upstream dbt duties fail
• SNS notification on failure (sns_notification_for_failure) – Invokes a Lambda operate to ship an SNS notification upon a dbt process failure (for instance, a dbt mannequin within the process group)

def check_dbt_failures(**kwargs):
    if kwargs['ti'].state == 'failed':
        increase AirflowException('Failure in dbt process group')

with DAG(
    dag_id="my_sample_dbt_project_dag",
    start_date=datetime(2025, 4, 2),
    schedule_interval="@day by day",
    catchup=False,
    tags=["dbt"]
):

    audit_dbt_task = DbtTaskGroup(
        group_id="audit_dbt_task",
        execution_config=execution_config,
        profile_config=profile_config,
        project_config=project_config,
        operator_args={
            "install_deps": True,
        },
        render_config= RenderConfig(
            choose=["tag:audit"],
            load_method=LoadMode.DBT_LS
        )
    )

    transform_data = DbtTaskGroup(
        group_id="transform_data",
        execution_config=execution_config,
        profile_config=profile_config,
        project_config=project_config,
        operator_args={
            "install_deps": True,
            # set up mandatory dependencies earlier than operating dbt command
        },
        render_config= RenderConfig(
            exclude=["tag:audit"],
            load_method=LoadMode.DBT_LS
        )
    )

    dbt_check = PythonOperator(
        task_id='dbt_check', 
        python_callable=check_dbt_failures,
        provide_context=True,
    )

    sns_notification_for_failure = LambdaInvokeFunctionOperator(
        task_id="sns_notification_for_failure",
        function_name=sns_lambda_function_name,
        payload=payload,
        trigger_rule="one_failed"
    )

    audit_dbt_task >> transform_data >> dbt_check >> sns_notification_for_failure

The pattern dbt orchestrates a dbt workflow in Amazon Redshift, beginning with an audit process and adopted by a process group that processes information fashions. It features a failure dealing with mechanism that checks for failures and raises an exception to set off an SNS notification utilizing Lambda if a failure happens. If no failures are detected, the SNS notification process is skipped.

Clear up

In the event you not want the sources you created, delete them to keep away from further expenses. This consists of the next:

• Amazon MWAA surroundings
• S3 bucket
• IAM function
• Redshift cluster or serverless workgroup
• Secrets and techniques Supervisor secret
• SNS matter
• Lambda operate

Conclusion

By integrating dbt with Amazon Redshift and orchestrating workflows utilizing Amazon MWAA and the Cosmos library, you’ll be able to simplify information transformation workflows whereas sustaining sturdy engineering practices. The pattern dbt undertaking construction, mixed with automated deployments via GitHub Actions and proactive monitoring utilizing Amazon SNS, gives a basis for constructing dependable information pipelines. The addition of audit logging facilitates transparency throughout your transformations, so groups can keep excessive information high quality requirements.

You should utilize this resolution as a place to begin in your personal dbt implementation on Amazon MWAA. The strategy we outlined emphasizes SQL-based transformations whereas incorporating important operational capabilities like deployment automation and failure alerting. Get began by adapting the configuration to your surroundings, and construct upon these practices as your information wants evolve.

For extra sources, confer with Handle information transformations with dbt in Amazon Redshift and Redshift setup.

In regards to the authors

Cindy Li is an Affiliate Cloud Architect at AWS Skilled Providers, specialising in Knowledge Analytics. Cindy works with prospects to design and implement scalable information analytics options on AWS. When Cindy shouldn’t be diving into tech, you’ll find her out on walks together with her playful toy poodle Mocha.

Akhil B is a Knowledge Analytics Guide at AWS Skilled Providers, specializing in cloud-based information options. He companions with prospects to design and implement scalable information analytics platforms, serving to organizations rework their conventional information infrastructure into trendy, cloud-based options on AWS. His experience helps organizations optimize their information ecosystems and maximize enterprise worth via trendy analytics capabilities.

Joao Palma is a Senior Knowledge Architect at Amazon Internet Providers, the place he companions with enterprise prospects to design and implement complete information platform options. He makes a speciality of serving to organizations rework their information into strategic enterprise belongings and enabling data-driven choice making.

Harshana Nanayakkara is a Supply Guide at AWS Skilled Providers, the place he helps prospects deal with advanced enterprise challenges utilizing AWS Cloud know-how. He makes a speciality of information and analytics, information governance, and AI/ML implementations.