Automate knowledge lineage in Amazon SageMaker utilizing AWS Glue Crawlers supported knowledge sources

The subsequent technology of Amazon SageMaker is the middle for all of your knowledge, analytics, and AI. Bringing collectively broadly adopted Amazon Internet Providers (AWS) machine studying (ML) and analytics capabilities, it delivers an built-in expertise for analytics and AI with unified entry to all of your knowledge. From Amazon SageMaker Unified Studio, a single knowledge and AI growth surroundings, you’ll be able to entry your knowledge and use a collection of highly effective instruments for knowledge processing, SQL analytics, mannequin growth, coaching and inference, and generative AI growth.

With knowledge lineage, now a part of Amazon SageMaker Catalog, you’ll be able to centralize lineage metadata of your knowledge property in a single place. You possibly can observe the circulation of knowledge over time, figuring out a transparent understanding of the place it originated, the way it has modified, and its utilization throughout the enterprise. By offering this stage of transparency, knowledge lineage helps knowledge customers achieve belief that the info is appropriate and compliant for his or her use circumstances. With knowledge lineage captured on the desk, column, and job stage, knowledge producers can conduct influence evaluation of adjustments of their knowledge pipelines and reply to knowledge points when wanted, for instance, when a column within the ensuing dataset is lacking the standard required by the enterprise.

Knowledge lineage is a strong device that may remodel how organizations perceive and handle their knowledge flows. On this put up, we discover its real-world influence by way of the lens of an ecommerce firm striving to spice up their backside line.

As an example this sensible software, we stroll you thru how you need to use the prebuilt integration between SageMaker Catalog and AWS Glue crawlers to routinely seize lineage for knowledge property saved in Amazon Easy Storage Service (Amazon S3) and Amazon DynamoDB. Utilizing this workflow, you’ll be able to seize lineage routinely from further knowledge sources utilizing AWS Glue crawlers. Consult with the Knowledge lineage help matrix within the SageMaker Unified Studio Person Information for supported sources. We additionally use SageMaker Unified Studio to navigate these knowledge property and find out about their origin, transformations, and dependencies, because of the lineage metadata captured utilizing the AWS Glue crawlers.

Key options of the SageMaker Catalog lineage graph

In SageMaker Unified Studio, you’ll be able to discover and uncover knowledge property of your group suited to your use case. As you dive into these knowledge property, you’ll be able to study extra about its enterprise context, schema, high quality, and lineage. While you determine to work with a subset of those property, you’ll be able to subscribe to them in a self-service style and begin working with them. For extra element, go to Knowledge discovery, subscription, and consumption within the SageMaker Unified Studio Person Information.

SageMaker Studio supplies a visible lineage graph that exhibits how a knowledge asset has developed from its supply by way of transformations to its closing state. This helps knowledge scientists, engineers, and analysts reply key questions resembling:

• The place did this knowledge come from?
• What transformations has it gone by way of?
• Which downstream property will probably be impacted by a change?

With this stage of visibility, groups can carry out quicker influence evaluation, discover the basis trigger of knowledge high quality points, and guarantee fashions are constructed on trusted knowledge. It additionally helps higher collaboration so customers can confidently use and share knowledge throughout the group. The next screenshot exhibits how SageMaker Unified Studio visualizes knowledge lineage, making it easy to hint knowledge circulation and perceive dependencies.

• Column-level lineage – You possibly can develop column-level lineage when accessible in dataset nodes. This routinely exhibits relationships with upstream or downstream dataset nodes if supply column data is accessible.
• Column search – If the dataset has greater than 10 columns, the node presents pagination to navigate to columns not initially introduced. To rapidly view a selected column, you’ll be able to search on the dataset node that lists solely the searched column.
• Particulars pane – Every lineage node captures and shows the next particulars:
  • Each dataset node has three tabs: LINEAGE, SCHEMA, and HISTORY. The HISTORY tab lists the completely different variations of lineage occasion captured for that node.
  • The job node has a particulars pane to show job particulars with the tabs Job information and Historical past. The small print pane additionally captures queries or expressions run as a part of the job.
• View dataset nodes solely – If you wish to filter out the job nodes, you’ll be able to select the open view management icon within the graph viewer and toggle the show dataset nodes solely, which is able to take away all of the job nodes from the graph and allow you to navigate solely the dataset nodes.
• Model tabs – All lineage nodes in Amazon DataZone knowledge lineage could have versioning, captured as historical past, based mostly on lineage occasions captured. You possibly can view lineage at a specific timestamp that opens a brand new tab on the lineage web page to assist evaluate or distinction between the completely different timestamps.

You possibly can strive a few of these options as you discover the info property of this put up. To study extra on knowledge lineage in SageMaker, we encourage you to dive deep into the Knowledge lineage in Amazon SageMaker Unified Studio.

Answer overview

Think about a state of affairs the place an ecommerce firm goals to optimize conversion charges and improve buyer expertise by gaining deeper insights into the client journey. They should join the dots between consumer interactions and precise purchases, however with knowledge scattered throughout a number of sources, the place do they start? That is the place knowledge lineage turns into invaluable. To carry out their evaluation, they want knowledge from two major sources:

• Clickstream knowledge saved in Amazon S3 (in JSON or Parquet format)
• Transactional order knowledge saved as gadgets in Amazon DynamoDB

To make these datasets discoverable throughout the enterprise, it’s essential to:

1. Create a undertaking in SageMaker Unified Studio that will probably be used to supply and handle the datasets
2. Allow knowledge lineage seize within the SageMaker Unified Studio undertaking
3. Arrange the assets for this use case, which incorporates an AWS Glue knowledge supply (arrange in SageMaker Unified Studio) and AWS Glue crawler (arrange in AWS Glue)
4. Run the AWS Glue crawler to catalog the datasets in AWS Glue Knowledge Catalog
5. Supply the metadata of the info property into the SageMaker Catalog by working the info supply
6. Use SageMaker Unified Studio to navigate by way of the lineage of the info property and visualize their origin
7. Perceive how schema evolution is captured within the knowledge asset’s lineage

Conditions

To finish the steps on this put up, you want an SageMaker Unified Studio area already deployed in your AWS account. To get began rapidly in a testing surroundings, we propose creating your SageMaker area utilizing the fast setup possibility as defined in Create an Amazon SageMaker Unified Studio area – fast setup.

Answer steps

To seize knowledge lineage for AWS Glue tables managed with AWS Glue crawlers utilizing SageMaker Unified Studio, full the steps within the following sections.

Arrange a SageMaker undertaking with SQL functionality

In SageMaker Unified Studio, a undertaking profile defines an uber template for tasks in your Amazon SageMaker unified area. By establishing a undertaking with the fitting tooling (undertaking profile), you’ll provision assets you need to use to work with knowledge, which could embody cataloging it in SageMaker, remodeling it into new knowledge property, analyzing it to drive enterprise worth, and even use it for ML or AI purposes.

To exhibit knowledge lineage successfully, we use SageMaker SQL analytics undertaking profile for a streamlined setup. Though this profile provides complete knowledge analytics capabilities, we focus particularly on two key parts:

• AWS Glue database – A lakehouse for storing and managing technical metadata
• Knowledge supply job – Robotically collects and tracks metadata into SageMaker Catalog

We’ve chosen this profile to bypass advanced guide configurations so we are able to concentrate on the core ideas of knowledge lineage.

To create a brand new undertaking in your SageMaker area utilizing the SQL analytics undertaking profile, comply with the steps detailed in SQL analytics undertaking profile. Hold all default configurations when creating the undertaking.

After creating your undertaking in SageMaker Studio, you’ll unlock highly effective knowledge lineage capabilities that make monitoring and understanding your knowledge flows intuitive. By means of the info sourcing function, you’ll be able to simply monitor how knowledge strikes from supply to the AWS Glue database. This visibility turns into significantly invaluable when debugging knowledge points—you’ll be able to rapidly hint knowledge again to its supply, perceive how adjustments influence downstream processes, and determine affected analyses or reviews. Subsequent, populate the AWS Glue database with pattern knowledge to watch these options in motion and exhibit how they will streamline your knowledge operations.

For additional steering on methods to entry the main points of the brand new SageMaker undertaking, discuss with Get undertaking particulars. After you entry the info supply particulars, within the Database title discipline, be aware of the AWS Glue database title related to the SageMaker undertaking.

Allow knowledge lineage seize within the SageMaker undertaking’s knowledge supply

To allow lineage seize, comply with these steps:

1. Develop the Actions menu, then select Edit knowledge supply.
2. Go to the connections and choose Import knowledge lineage to configure lineage seize from the supply, as proven within the following screenshot.
3. Make different adjustments to the info supply fields as desired, then select Save.

Enabling lineage will be sure the info supply job will seize lineage within the subsequent run.

Deploy assets for the use case

Observe these steps:

1. To deploy the assets required for this put up, obtain the AWS CloudFormation template amazon-datazone-examples within the AWS Samples GitHub repository. Deploy it in your AWS account.

For additional steering on methods to deploy a CloudFormation stack, discuss with Create a stack from the CloudFormation console. You must present a Stack title and the title of the AWS GlueDatabaseName related to the undertaking of your SageMaker area, as proven within the following screenshot.

2. Select Subsequent.

The template will deploy the next assets:

• A S3 bucket with a pattern file of clickstream knowledge. The bucket title and placement of the file will comply with the trail sample s3://ecomm-analytics--/clickstream///
  /knowledge.json

  . The file will comprise a pattern report with the next construction:

{
    "session_id": "abc123",
    "user_id": "u789",
    "event_type": "product_view",
    "product_id": "prod456",
    "timestamp": "2025-06-04T09:23:12Z"
}

• A DynamoDB desk with a pattern merchandise of order knowledge (transactions). The desk will probably be named OrderTransactionTable. The pattern merchandise could have the next construction:

{
    "order_id": "ord789",
    "user_id": "u789",
    "product_id": "prod456",
    "order_total": 79.99,
    "order_timestamp": "2025-06-04T09:27:10Z"
}

• An AWS Glue crawler configured to crawl the S3 bucket and DynamoDB desk deployed as a part of the stack and retailer the metadata within the AWS Glue database related to the SageMaker undertaking. You possibly can entry the crawler’s particulars within the AWS console, as proven within the following screenshot.

Run the AWS Glue crawler

The AWS Glue crawler deployed within the earlier step will permit you to seize metadata from the 2 knowledge sources, Amazon S3 and DynamoDB, and retailer it in AWS Glue Knowledge Catalog, particularly within the database related to the SageMaker undertaking. After the metadata is saved, it will likely be accessible to SageMaker.

Earlier than working the crawler, it’s essential to present AWS Lake Formation permissions to the IAM function that the AWS Glue crawler will use to work together together with your knowledge supply and goal AWS Glue database. The next command will grant the permissions wanted for the crawler to retailer metadata into the AWS Glue database of the SageMaker undertaking.

To invoke this command, we advocate utilizing AWS CloudShell on the AWS console as defined in AWS CloudShell Ideas. Replace the , and  placeholders with the fitting values to your AWS Area, AWS account ID, and title of the AWS Glue database related to the SageMaker undertaking.

aws lakeformation grant-permissions 
  --region  
  --principal DataLakePrincipalIdentifier=arn:aws:iam:::function/glue-crawler-role  
  --permissions CREATE_TABLE 
  --resource '{ "Database": { "Identify": "" } }'
  

Subsequent, run the AWS Glue Crawler on the AWS console. After the crawler efficiently finishes, two new tables, clickstream and ordertransactiontable, will probably be created within the AWS Glue database related to the SageMaker undertaking. Consult with Viewing crawler outcomes and particulars to study extra about AWS Glue crawler outcomes.

Supply metadata from the AWS Glue database into SageMaker

To supply metadata from knowledge property within the AWS Glue database, together with their lineage, into SageMaker, use the info supply that was deployed as a part of the SageMaker undertaking creation.

1. To run the info supply, go to the info supply particulars web page.
2. Select Run. (Knowledge sources may be scheduled to run as effectively, nevertheless, for this demonstration we set off a guide run).

After the info supply run is full, metadata from each knowledge property within the AWS Glue database will probably be imported into the SageMaker area because the undertaking’s stock property. Yow will discover the main points of the info supply run from inside SageMaker Unified Studio, which embody:

• The information property from the AWS Glue database that had been ingested into SageMaker.
• The standing of the info lineage import for every knowledge asset, which incorporates an occasion ID for traceability. This lineage occasion ID can be utilized to debug inconsistencies within the ensuing lineage graph. You should utilize the GetLineageEvent API to retrieve the uncooked payload of the lineage occasion.

Visualizing the info lineage graph of the info property in SageMaker Unified Studio

With SageMaker Unified Studio, you have got a single place to handle and uncover knowledge property. When accessing a knowledge asset printed within the SageMaker central catalog or in your undertaking’s personal stock, you’ll be able to dive into the asset’s metadata, which incorporates its schema, enterprise description, customized metadata types, high quality, lineage, and extra. To visualise the lineage graph of every knowledge asset of this put up, comply with these steps:

1. In SageMaker Studio, navigate to the Belongings part of the SageMaker undertaking particulars web page and select INVENTORY
2. Choose the asset that you just wish to discover. You too can entry the asset instantly from the info supply run by choosing the asset title.
3. To view the lineage graph of the info asset as much as its origin, proven within the following screenshots, select the LINEAGE tab.
   • For clickstream desk (Sourced from S3)

• • For order transactions desk (Sourced from DynamoDB)

With lineage, now you can affirm that the info originated from sources resembling Amazon S3 and Amazon DynamoDB and perceive the way it has been reworked alongside the way in which. Due to this end-to-end visibility, you’ll be able to belief the info, make knowledgeable choices, and supply compliance with confidence. The lineage graph captures important metadata that types the inspiration of lineage monitoring.

• This consists of desk schemas, column definitions and their knowledge sorts.
• Column-level lineage turns into significantly highly effective on this context. Think about your clickstream’s AWS Glue desk powers an Amazon QuickSight dashboard analyzing buyer buy patterns and spot discrepancies in your income reviews. With column lineage, you’ll be able to immediately hint the supply of these columns.
• This granular visibility not solely accelerates debugging but in addition proves invaluable throughout schema adjustments, as we present within the following part by altering the supply schema.
• The crawler particulars resembling crawlerRunId (current within the supply identifier of the lineage node) and crawler begin and finish occasions can be utilized to debug which crawler runs up to date the desk.

Understanding your knowledge asset’s schema evolution by way of lineage in SageMaker Unified Studio

Think about the order transactions supply in DynamoDB was up to date with new data. As a result of this supply powers an Amazon QuickSight report for the client utilizing the AWS Glue database desk, it’s essential for customers to know what adjustments within the knowledge pipeline up to date the report.

1. Edit the DynamoDB desk merchandise with further columns to learn the way lineage graph can be utilized to view historic updates:

{
    "order_id": "ord789",
    "user_id": "u789",
    "product_id": "prod456",
    "order_total": 79.99,
    "order_timestamp": "2025-06-04T09:27:10Z",
	"customerSegment": "new-customer",
    "conversionSource": "primeDayEmailCampaign"
}

2. Enter the OrderTransactionsCrawler Glue crawler once more on the AWS console. After completion, you’ll discover that it up to date the ordertransactiontable AWS Glue desk, as proven within the following screenshot.

3. Run once more the info supply related to the undertaking in SageMaker Unified Studio to import the newest metadata into the SageMaker Catalog. After completion, you’ll discover the info supply up to date the ordertransactiontable knowledge asset within the SageMaker Catalog, as proven within the following screenshot.

This part explores how lineage may be helpful to trace the updates.

Navigate to the ordertransactiontable knowledge asset in SageMaker Catalog by choosing it from the info supply run and select the LINEAGE tab, as proven within the following screenshot.

Discover how the brand new columns can be found within the lineage graph. A brand new crawler run ID is current because the supply identifier of the crawler lineage node. The historical past tab exhibits a number of crawler runs. You possibly can navigate to verify the state of the system throughout the first run.

Cleanup

After you’re performed, we advocate to cleansing up the assets created for this put up to keep away from unintended fees:

1. Delete the stock property that had been cataloged within the SageMaker undertaking’s stock, as defined in Delete an Amazon SageMaker Unified Studio asset.
2. Delete the SageMaker undertaking that was created as a part of this put up, as defined in Delete a undertaking.
3. Delete the CloudFormation stack that was deployed as a part of this put up, as defined in Delete a stack from the CloudFormation console.
4. The S3 bucket created as a part of the CloudFormation stack will stay after its deletion as a result of it incorporates a knowledge file in it. Empty and delete the bucket, as defined in Deleting a basic objective bucket.

Conclusion

On this put up, you had been in a position to discover the info lineage capabilities of Amazon SageMaker, particularly when working with AWS Glue crawlers. You discovered how one can arrange an AWS Glue crawler to deduce metadata from knowledge property in a number of sources resembling Amazon S3 and DynamoDB and retailer it the AWS Glue Knowledge Catalog. You additionally imported this metadata, together with knowledge lineage, into Amazon SageMaker by way of the info supply functionality of a SageMaker undertaking. Lastly, you explored the ensuing lineage graph of knowledge property in SageMaker Unified Studio and noticed a number of the functionalities accessible to know the origin path of them, perceive how columns are reworked, and what influence appears like when performing adjustments to any step of the pipeline.We encourage you to now check the capabilities you explored on this put up with your individual knowledge. By following the sample introduced on this put up, many shoppers have been in a position to obtain governance of their knowledge lake and lakehouse platforms on prime of Amazon SageMaker with knowledge lineage and extra.

In regards to the authors

Mohit Dawar is a Senior Software program Engineer at Amazon Internet Providers (AWS) engaged on Amazon DataZone. Over the previous 3 years, he has led efforts across the core metadata catalog, generative AI–powered metadata curation, and lineage visualization. He enjoys engaged on large-scale distributed methods, experimenting with AI to enhance consumer expertise, and constructing instruments that make knowledge governance really feel easy. Join with him on LinkedIn: Mohit Dawar.

Jose Romero is a Senior Options Architect for Startups at Amazon Internet Providers (AWS) based mostly in Austin, TX, US. He’s obsessed with serving to prospects architect trendy platforms at scale for knowledge, AI, and ML. As a former senior architect in AWS Skilled Providers, he enjoys constructing and sharing options for frequent advanced issues in order that prospects can speed up their cloud journey and undertake finest practices. Join with him on LinkedIn: Jose Romero.