Introducing Apache Airflow 3 on Amazon MWAA: New options and capabilities

In the present day, Amazon Internet Companies (AWS) introduced the overall availability of Apache Airflow 3 on Amazon Managed Workflows for Apache Airflow (Amazon MWAA). This launch transforms how organizations use Apache Airflow to orchestrate information pipelines and enterprise processes within the cloud, bringing enhanced safety, improved efficiency, and fashionable workflow orchestration capabilities to Amazon MWAA prospects.

Amazon MWAA introduces Airflow 3 options that modernize workflow administration for AWS prospects. Following the April 2025 launch of Airflow 3 by the Apache group, AWS has included these capabilities into Amazon MWAA. Airflow now encompasses a fully redesigned, intuitive UI that simplifies workflow orchestration for customers throughout expertise ranges. With the Process Execution Interface (Process API), duties can run each inside Airflow and as standalone Python scripts, bettering code portability and testing. Scheduler-managed Backfill strikes operations from the CLI to the scheduler, offering centralized management and visibility via the Airflow UI. CLI safety enhancements substitute direct database entry with API calls, sustaining constant safety throughout interfaces. Airflow now helps event-driven workflows, enabling triggers from AWS providers and exterior sources. Amazon MWAA additionally provides help for Python 3.12, bringing the most recent language capabilities to workflow growth.

This publish explores the options of Airflow 3 on Amazon MWAA and descriptions enhancements that enhance your workflow orchestration capabilities. The service maintains the Amazon MWAA pay-as-you-go pricing mannequin with no upfront commitments. You’ll be able to start instantly by visiting the Amazon MWAA console, launching new Apache Airflow environments via the AWS Administration Console, AWS Command Line Interface (AWS CLI), AWS CloudFormation, or AWS SDK inside minutes.

Architectural developments in Airflow 3 on Amazon MWAA

Airflow 3 on Amazon MWAA introduces vital architectural enhancements that improve safety, efficiency, and suppleness. These developments create a extra strong basis for workflow orchestration whereas sustaining backward compatibility with current workflows.

Enhanced safety

Amazon MWAA with Airflow 3 adjustments the safety mannequin by making part isolation a regular apply slightly than non-compulsory. In Airflow 2, the DAG processor (the part that parses and processes DAG information) runs inside the scheduler course of by default, however can optionally be separated into its personal course of for higher scalability and safety isolation. Airflow 3 makes this separation normal, sustaining constant safety practices throughout deployments.

API server and Process API

Constructing on this safety basis, a brand new API server part is launched in Amazon MWAA with Airflow 3, which serves as an middleman between job cases and the Airflow metadata database. This alteration improves your workflows’ safety posture by minimizing direct entry to the Airflow metadata database from duties. Duties now function with least privilege database entry, lowering the danger of 1 job affecting others and bettering total system stability via fewer direct database connections.

The standardized communication via well-defined API endpoints creates a basis for safer, scalable, and versatile workflow orchestration. The Process Execution Interface (Process API) helps duties run each inside Airflow and as standalone Python scripts, bettering code portability and testing capabilities.

From data-aware to event-driven scheduling

Airflow’s evolution towards event-driven scheduling started with the introduction of data-aware scheduling in Airflow 2.4, so DAGs might be triggered primarily based on information availability slightly than time schedules alone. Amazon MWAA with Airflow 3 builds on this basis via a transition that features the renaming of datasets to belongings and introduces superior capabilities, together with asset partitions, exterior occasion integration, and asset-centric workflow design.

The transition from datasets to belongings represents greater than a easy rename. A knowledge asset is a group of logically associated information that may signify numerous information merchandise, together with database tables, continued ML fashions, embedded dashboards, or directories containing information.

Amazon MWAA with Airflow 3 introduces a brand new asset-centric syntax that represents an essential shift in how workflows could be designed. The @asset decorator helps builders put information belongings on the middle of their workflow design, creating extra intuitive asset-driven pipelines.

The next code is an instance of asset-aware DAG scheduling:

from airflow.sdk import DAG, Asset
from airflow.suppliers.normal.operators.python import PythonOperator

# Outline the asset
customer_data_asset = Asset(identify="customer_data", uri="s3://my-bucket/customer-data.csv")

def process_customer_data():
    """Course of buyer information..."""
    # Implementation right here

# Create the DAG and job
with DAG(dag_id="process_customer_data", schedule="@day by day"):
    PythonOperator(
        task_id="process_data", 
        shops=[customer_data_asset], 
        python_callable=process_customer_data
    )

The next code reveals an asset-centric method with the @asset decorator:

from airflow.sdk import asset

@asset(uri="s3://my-bucket/customer-data.csv", schedule="@day by day")
def customer_data():
    """Course of buyer information..."""
    # Implementation right here

The @asset decorator robotically creates an asset with the operate identify, a DAG with the identical identifier, and a job that produces the asset. This reduces code complexity and facilitates automated DAG creation, the place every asset turns into a self-contained workflow unit.

Exterior event-driven scheduling with Asset Watchers

A major development in Amazon MWAA with Airflow 3 is the introduction of Asset Watchers, which assist Airflow react to occasions taking place exterior of the Airflow system itself. Whereas earlier variations supported inner cross-DAG dependencies, Asset Watchers lengthen this functionality to exterior information methods and message queues via the AssetWatcher class.

Amazon MWAA with Airflow 3 consists of help for Amazon Easy Queue Service (Amazon SQS) via Asset Watchers. This enables your workflows to be triggered by exterior messages and facilitates extra event-driven scheduling. Airflow now helps event-driven workflows, enabling triggers from AWS providers and exterior sources. Asset Watchers monitor exterior methods asynchronously and set off workflow execution when particular occasions happen, enabling workflows to reply to enterprise occasions, information updates, or system notifications with out the overhead of conventional sensor-based polling mechanisms.

Trendy React-based UI

Amazon MWAA with Airflow 3 encompasses a fully redesigned, intuitive UI constructed with React and FastAPI that simplifies workflow orchestration for customers throughout expertise ranges. The brand new interface offers extra intuitive navigation and workflow visualization, with an enhanced grid view that gives higher visibility into job standing and historical past. Customers will respect the addition of darkish mode help, which reduces eye pressure throughout prolonged use, and the general quicker efficiency that’s particularly noticeable when working with giant DAGs.

The brand new UI maintains acquainted workflows whereas offering a extra fashionable and environment friendly expertise for DAG administration and monitoring, making day by day operations extra productive for each builders and operators. The legacy UI has been fully eliminated, providing a cleaner, extra constant expertise throughout the system. The muse for the brand new UI is constructed on REST APIs and a set of inner APIs for UI operations, each of which are actually primarily based on FastAPI, making a extra cohesive and safe structure for each programmatic entry and UI operations.

Scheduler optimizations

Amazon MWAA with Airflow 3’s enhanced scheduler delivers efficiency enhancements for job execution and workflow administration. The redesigned scheduling engine processes duties extra effectively, lowering the time between job submissions and executions. This optimization advantages information pipeline operations that require speedy job processing and well timed workflow completion.

The scheduler now manages computing assets extra successfully, enabling secure efficiency whilst workloads scale. When working a number of DAGs concurrently, the improved useful resource allocation system helps forestall bottlenecks and maintains constant execution speeds. This development is especially helpful for organizations working complicated workflows with various useful resource necessities. The brand new scheduler additionally handles concurrent operations with elevated precision, so groups can run a number of DAG cases concurrently whereas sustaining system stability and predictable efficiency.

Enhanced scheduler backfill operations

Scheduler-managed backfill (the method of working DAGs for historic dates) strikes operations from the CLI to the scheduler, offering centralized management and visibility via the Airflow UI. Amazon MWAA with Airflow 3 delivers essential upgrades to the scheduler’s backfill capabilities, serving to information groups course of historic information extra effectively. The backfill course of has been optimized for higher efficiency, lowering the database load throughout these operations and ensuring backfills could be accomplished extra shortly, minimizing the impression on close to real-time workflow execution.

Amazon MWAA with Airflow 3 additionally improves the administration of backfill operations, with the scheduler offering higher isolation between backfill jobs and supporting extra environment friendly processing of historic datasets. Operators now have higher monitoring instruments to trace the progress and standing of their backfill jobs, leading to more practical administration of those important information processing duties.

Developer-focused enhancements

Airflow 3 on Amazon MWAA delivers a number of enhancements designed to enhance the developer expertise, from simplified job definition to higher workflow administration capabilities.

Process SDK

The Process SDK offers a extra intuitive strategy to outline duties and DAGs:

# Instance utilizing the Process SDK
from airflow.sdk import dag, job
from datetime import datetime

@dag(
    start_date=datetime(2023, 1, 1),
    schedule="@day by day",
    catchup=False
)
def modern_etl_workflow():
    
    @job
    def extract():
        # Extract information from supply
        return {"information": [1, 2, 3, 4, 5]}
    
    @job
    def rework(input_data):
        # Rework the information
        return [x * 10 for x in input_data]
    
    @job
    def load(transformed_data):
        # Load information to vacation spot
        print(f"Loading information: {transformed_data}")
    
    # Outline the workflow
    extracted_data = extract()
    transformed_data = rework(extracted_data["data"])
    load(transformed_data)

# Instantiate the DAG
etl_dag = modern_etl_workflow()

This method gives extra intuitive information circulate between duties, higher built-in growth atmosphere (IDE) help with improved kind hinting, and extra easy unit testing of job logic. The result’s cleaner, extra maintainable code that higher represents the precise information circulate of your pipelines. Groups adopting this sample typically discover their DAGs develop into extra readable and easier to keep up over time, particularly as workflows develop in complexity.

DAG versioning

Amazon MWAA with Airflow 3 consists of fundamental DAG versioning capabilities that come by default with Airflow 3. Every time a DAG is modified and deployed, Airflow serializes and shops the DAG definition to protect historical past. This automated model monitoring minimizes the necessity for guide record-keeping and ensures each modification is documented.

By the Airflow UI, groups can entry and assessment the historical past of their DAGs. This visible illustration reveals model numbers (v1, v2, v3, and many others.) and helps groups perceive how their workflows have advanced over time.

The DAG versioning supported in Amazon MWAA offers the aptitude to see totally different DAG variations that had been run within the Airflow UI, providing improved workflow visibility and enhanced collaboration for information engineering groups managing complicated, evolving information pipelines.

Python 3.12 help

Amazon MWAA provides help for Python 3.12, bringing the most recent language capabilities to workflow growth. This improve offers entry to the most recent Python language enhancements, efficiency enhancements, and library updates, protecting your information pipelines fashionable and environment friendly.

Options not presently supported in Amazon MWAA

Though we’re launching a lot of the Airflow 3 options on Amazon MWAA on this launch, some options usually are not supported presently:

• DAG versioning (AIP-63) – Superior versioning options past fundamental model monitoring
• Substitute Flask AppBuilder (AIP-79) – Full substitute capabilities
• Edge Executor and job isolations (AIP-69) – Distant execution capabilities
• Multi-language help (AIP-72) – Help for languages apart from Python

We plan to help these options in subsequent variations of Airflow on Amazon MWAA.

Conclusion

Airflow 3 on Amazon MWAA delivers enhanced workflow automation capabilities. The architectural enhancements, enhanced safety mannequin, and developer-friendly options present a strong basis for constructing extra dependable and maintainable information pipelines.The introduction of Asset Watchers adjustments how workflows can reply to exterior occasions, enabling really event-driven scheduling. This functionality, mixed with the brand new asset-centric workflow design, makes Airflow 3 a extra highly effective and versatile orchestration service.

The scheduler optimizations ship efficiency enhancements for job execution and workflow administration, and the improved backfill capabilities make historic information processing extra environment friendly. The DAG versioning system improves workflow stability and collaboration, and Python 3.12 help retains your information pipelines fashionable and environment friendly.

Organizations can now make the most of these new options and enhancements in Airflow 3 on Amazon MWAA to reinforce their workflow orchestration capabilities. To get began, go to the Amazon MWAA product web page.

Concerning the authors

Anurag Srivastava works as a Senior Large Knowledge Cloud Engineer at Amazon Internet Companies (AWS), specializing in Amazon MWAA. He’s captivated with serving to prospects construct scalable information pipelines and workflow automation options on AWS.

Kamen Sharlandjiev is a Sr. Large Knowledge and ETL Options Architect, Amazon MWAA and AWS Glue ETL skilled. He’s on a mission to make life simpler for patrons who’re going through complicated information integration and orchestration challenges. His secret weapon? Totally managed AWS providers that may get the job finished with minimal effort. Comply with Kamen on LinkedIn to maintain updated with the most recent Amazon MWAA and AWS Glue options and information!

Ankit Sahu brings over 18 years of experience in constructing revolutionary digital services and products. His numerous expertise spans product technique, go-to-market execution, and digital transformation initiatives. Presently, Ankit serves as Senior Product Supervisor at Amazon Internet Companies (AWS), the place he leads the Amazon MWAA service.

Mohammad Sabeel works as a Senior Cloud Help Engineer at Amazon Internet Companies (AWS), specializing in AWS Analytics providers together with AWS Glue, Amazon MWAA, and Amazon Athena. With over 14 years of IT expertise, he’s captivated with serving to prospects construct scalable information processing pipelines and optimize their analytics options on AWS.

Satya Chikkala is a Options Architect at Amazon Internet Companies. Based mostly in Melbourne, Australia, he works carefully with enterprise prospects to speed up their cloud journey. Past work, he’s very captivated with nature and images.

Sriharsh Adari is a Senior Options Architect at Amazon Internet Companies (AWS), the place he helps prospects work backward from enterprise outcomes to develop revolutionary options on AWS. Over time, he has helped a number of prospects on information system transformations throughout business verticals. His core space of experience embrace know-how technique, information analytics, and information science. In his spare time, he enjoys enjoying sports activities, binge-watching TV reveals, and enjoying Tabla.