Breaking down knowledge silos: Volkswagen’s strategy with Amazon DataZone

October 8, 2025

9

Through the years, organizations have invested in constructing purpose-built cloud-based knowledge warehouses which are siloed from each other. One of many main challenges these organizations encounter at the moment is enabling cross-organization discovery and entry to knowledge throughout these siloed knowledge warehouses constructed utilizing totally different know-how stacks. The knowledge mesh sample addresses these points, based in 4 rules: domain-oriented decentralized knowledge possession and structure, treating knowledge as a product, offering self-serve knowledge infrastructure as a platform, and implementing federated governance. The info mesh sample helps organizations mimic their organizational construction into knowledge domains and makes it attainable to share the information throughout the group and past to enhance their enterprise fashions.

In 2019, Volkswagen AG and Amazon Internet Companies (AWS) began their collaboration to co-develop the Digital Manufacturing Platform (DPP), with the aim of enhancing manufacturing and logistics effectivity by 30% whereas decreasing manufacturing prices by the identical margin. The DPP was developed to streamline entry to knowledge from store ground gadgets and manufacturing methods by dealing with integrations and offering a spread of standardized interfaces. Nonetheless, as functions and use circumstances advanced on the platform, a big problem emerged: the flexibility to share knowledge throughout functions saved in remoted knowledge warehouses (inside Amazon Redshift in remoted AWS accounts designated for particular use circumstances), with out the necessity to consolidate knowledge right into a central knowledge warehouse. One other problem was discovering all of the out there knowledge saved throughout a number of knowledge warehouses and facilitating a workflow to request entry to knowledge throughout enterprise domains inside every plant. The widespread methodology used was largely handbook, counting on emails and basic communication (by tickets and emails). The handbook strategy not solely elevated the overhead but additionally diverse from one use case to a different by way of knowledge governance.

On this publish, we introduce Amazon DataZone and discover how Volkswagen used Amazon DataZone to construct their knowledge mesh, sort out the challenges encountered, and break the information silos. A key facet of the answer was enabling knowledge suppliers to mechanically publish their knowledge merchandise to Amazon DataZone, serving as a central knowledge mesh for enhanced knowledge discoverability. Moreover, we offer code to information you thru the deployment and implementation course of.

Introduction to Amazon DataZone

Amazon DataZone is an information administration service that makes it sooner and easy to catalog, uncover, share, and govern knowledge saved throughout AWS, on-premises, and third-party sources. Key options of Amazon DataZone embrace the enterprise knowledge catalog, with which customers can seek for revealed knowledge, request entry, and begin engaged on knowledge in days as a substitute of weeks. As well as, the service facilitates collaboration throughout groups and helps them handle and monitor knowledge property throughout totally different organizational models. The service additionally contains the Amazon DataZone portal, which gives a customized analytics expertise for knowledge property by a web-based utility or API. Lastly, Amazon DataZone gives ruled knowledge sharing, which makes positive the best knowledge is accessed by the best person for the best function with a ruled workflow.

Resolution overview

The next structure diagram represents a high-level design that’s constructed on high of the information mesh sample. It separates supply methods, knowledge area producers (knowledge publishers), knowledge area subscribers (knowledge shoppers), and central governance to focus on the important thing features. This knowledge mesh structure is specifically tailor-made for cross-AWS account utilization. The target of this strategy is to create a basis for constructing knowledge governance on a scale, supporting the targets of information producers and shoppers with robust and constant governance.

This structure permits for the mixing of a number of knowledge warehouses right into a centralized governance account that shops all of the metadata from every surroundings.

A knowledge area producer makes use of Amazon Redshift as their analytical knowledge warehouse to retailer, course of, and handle structured and semi-structured knowledge. The info area producers load knowledge into their respective Amazon Redshift clusters by extract, remodel, and cargo (ETL) pipelines they handle, personal, and function. The producers preserve management over their knowledge by Amazon Redshift safety features, together with column-level entry controls and dynamic knowledge masking, supporting knowledge governance on the supply. A knowledge area producer makes use of Amazon Redshift ETL and Amazon Redshift Spectrum to course of and remodel uncooked knowledge into consumable knowledge merchandise. The info merchandise might be Amazon Redshift tables, views, or materialized views.

Knowledge area producers expose datasets to the remainder of the group by registering them to Amazon DataZone service, which acts as a central knowledge catalog. They will select what knowledge property to share, for the way lengthy, and the way shoppers can work together with these. They’re additionally answerable for sustaining the information and ensuring it’s correct and present.

The info property from the producers are then revealed utilizing the information supply run to Amazon DataZone within the central governance account. This course of populates the technical metadata into the enterprise knowledge catalog for every knowledge asset. The enterprise metadata might be added by enterprise customers (knowledge analysts) to supply enterprise context, tags, and knowledge classification for the datasets. This strategy supplies the required options to permit producers to create catalog entries with Amazon Redshift from all their knowledge warehouses in-built with Redshift clusters. As well as, the central knowledge governance account is used to share datasets securely between producers and shoppers. It’s essential to notice that sharing is finished by metadata linking alone. No knowledge (besides logs) exists within the governance account. The info isn’t copied to the central account; only a reference to the information is used, in order that the information possession stays with the producer.

Amazon DataZone supplies a streamlined approach to seek for knowledge. The Amazon DataZone knowledge portal supplies a customized view for customers to find and search knowledge property. An Amazon DataZone person (client) with permissions to entry the information portal can seek for property and submit requests for subscription of information property utilizing a web-based utility. An approver can then approve or reject the subscription request.

When an information area client has entry to an asset within the catalog, they’ll devour it (question and analyze) utilizing the Amazon Redshift question editor. Every client runs their very own workload primarily based on their use case. On this means, the group can select the instruments for the job to carry out analytics and machine studying actions in its AWS client surroundings.

Publishing and registering knowledge property to Amazon DataZone

To publish an information asset from the producer account, every asset have to be registered in Amazon DataZone for client subscription. For extra data, discuss with Create and run an Amazon DataZone knowledge supply for Amazon Redshift. Within the absence of an automatic registration course of, required duties have to be accomplished manually for every knowledge asset.

Utilizing the automated registration workflow, the handbook steps might be automated for the Amazon Redshift knowledge asset (Redshift desk or view) that must be revealed in an Amazon DataZone area or when there’s a schema change in an already revealed knowledge asset.

The next structure diagram represents how knowledge property from Amazon Redshift knowledge warehouses have been mechanically revealed to the information mesh created with Amazon DataZone.

The method consists of the next steps:

Within the producer account (Account B), the information to be shared resides in a Redshift cluster.
The producer account (Account B) makes use of a mechanism to set off the dataset registration AWS Lambda operate with a particular payload containing the data and identify of the database, schema, desk, or view that has a change in metadata.
The Lambda operate performs the steps to mechanically register and publish the dataset in Amazon DataZone:
1. Get the Amazon Redshift clusterName, dbName, schemas, and tables from the JSON payload, which is used because the occasion to set off the Lambda operate.
2. Get the Amazon DataZone knowledge warehouse blueprint ID.
3. Allow the blueprint within the knowledge producer account.
4. Establish the Amazon DataZone Area ID and mission ID for the producer through assuming function in Amazon DataZone account (Account A).
5. Verify if an surroundings already exists within the mission. If not, create an surroundings.
6. Create a brand new Redshift knowledge supply by offering the right Redshift database data within the newly created surroundings.
7. Provoke an information supply run request within the knowledge supply to make the Redshift tables or views out there in Amazon DataZone.
8. Publish the tables or views within the Amazon DataZone catalog.

Stipulations

The next conditions are required earlier than beginning:

Two AWS accounts to implement the answer have been described on this publish. Nonetheless, you may as well use Amazon DataZone to publish knowledge inside a single account or throughout a number of accounts.
- Amazon DataZone account (Account A) – That is the central knowledge governance account, which may have the Amazon DataZone area and mission.
- Knowledge area producer account (Account B) – This account acts as the information area producer. It has been added as an related account to Account A.

Stipulations in knowledge area producer account (Account B)

As a part of this publish, we wish to publish property and subscribe to property from a Redshift cluster that already exists. Full the next prerequisite steps to arrange Account B:

Arrange the Redshift cluster, together with database, schema, tables, and views (optionally available). The node kind have to be from the RA3 household. For extra data, see Amazon Redshift provisioned clusters.
Create a superuser in Amazon Redshift for Amazon DataZone. For the Redshift cluster, the database person you present in AWS Secrets and techniques Supervisor should have superuser permissions. For reference please see the be aware part on this QuickStart information with pattern Amazon Redshift knowledge
Retailer the person’s credentials in Secrets and techniques Supervisor. Choose the credential kind, enter the credential values, and select the AWS Key Administration Service (AWS KMS) key with which to encrypt the key.
Add the tags to the Secret Supervisor secret to permit Amazon DataZone to seek out this secret and restrict the entry to a selected Amazon DataZone area and Amazon DataZone mission. The Redshift cluster Amazon Useful resource Identify (ARN) have to be added as a tag so it may be utilized by Amazon Redshift as a legitimate credential. For reference please see the be aware part on this QuickStart information with pattern Amazon Redshift knowledge

Add an Amazon DataZone provisioning IAM function and Amazon Redshift handle entry IAM function within the secret’s useful resource coverage. The AWS Id and Entry Administration (IAM) roles are created as a part of the AWS Cloud Growth Equipment (AWS CDK) deployment (mentioned later on this publish). The next code exhibits an instance of the Secrets and techniques Supervisor secret’s useful resource coverage. Retailer the key ARN in an AWS Programs Supervisor parameter.

{
  "Model" : "2012-10-17",
  "Assertion" : [ {
    "Effect" : "Allow",
    "Principal" : "*",
    "Action" : "secretsmanager:GetSecretValue",
    "Resource" : "*",
    "Condition" : {
      "ArnEquals" : {
        "aws:PrincipalArn" : [ 
          "arn:aws:iam::>:role/DzRedshiftAccess->->",
          "arn:aws:iam::>:role/DataZoneProvisioning->"
        ]
      }
    }
  } ]
}

In case your secret is encrypted with a customized KMS key, append the important thing coverage with the next assertion and add a tag to the important thing: AmazonDatazoneEnvironment = All. You possibly can skip this step in case you’re utilizing an AWS managed KMS key.

{
    "Impact": "Enable",
    "Principal": {
        "Service": "logs.>.amazonaws.com",
        "AWS": "arn:aws:iam::>:root"
    },
    "Motion": [
        "kms:Decrypt",
        "kms:Encrypt",
        "kms:GenerateDataKey*",
        "kms:ReEncrypt*"
    ],
    "Useful resource": "*"
 },
 {
    "Sid": "AllowDatazoneRoles-DEV",
    "Impact": "Enable",
    "Principal": {
        "AWS": "*"
    },
    "Motion": [
        "kms:Decrypt",
        "kms:Describe*",
        "kms:Get*",
        "kms:Encrypt",
        "kms:GenerateDataKey",
        "kms:ReEncrypt*",
        "kms:CreateGrant"
    ],
    "Useful resource": "*",
    "Situation": {
        "StringLike": {
            "aws:PrincipalArn": [
                "arn:aws:iam::>:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift",
                "arn:aws:iam::>:role/datazone_*",
                "arn:aws:iam::>:role/>",
                "arn:aws:iam::>:role/service-role/AmazonDataZoneRedshiftAccess->-*"
            ]
         }
     }
 }

Place a mechanism to generate the next payload to set off the dataset registration Lambda operate. The payload should comprise the related Redshift database, schema, and desk or view that you simply wish to publish within the Amazon DataZone area. The next instance code assumes you have got three databases in your Redshift cluster and inside these databases you have got totally different schemas, tables, and views. You must alter the payload primarily based in your use case.

{
    "supply": "redshift-user-initiated",
    "detail-type": "Amazon Redshift dataset registration in Amazon DataZone",
    "datasets": [
        {
            "clusterName": ">",
            "dbName":">",
            "schemas": [
                {
                    "schemaName":">",
                    "addAllTables":false,
                    "addAllViews":false,
                    "tables":[
                        ">",
                        ">"
                    ],
                    "views":[
                        ">"
                    ]
                }
            ]
        },
        {
            "clusterName": ">",
            "dbName":">",
            "schemas": [
                {
                    "schemaName":">",
                    "addAllTables":true,
                    "addAllViews":true,
                    "tables":[],
                    "views":[]
                }
            ]
        },
        {
            "clusterName": ">",
            "dbName":">",
            "schemas": [
                {
                    "schemaName":">",
                    "addAllTables":true,
                    "addAllViews":false,
                    "tables":[],
                    "views":[
                        ">"
                    ]
                }
            ]
        }
    ]
}

Stipulations in Amazon DataZone account (Account A)

Full the next steps to arrange your Amazon DataZone account (Account A):

Register to Account A and ensure you have already deployed an Amazon DataZone area and a mission inside that area. Discuss with Create Amazon DataZone domains for directions to create a website.
In case your Amazon DataZone area is encrypted with a KMS key, add the information area account (Account B) to the KMS key coverage with the next actions:
```
"Motion": [
    "kms:Encrypt",
    "kms:Decrypt",
    "kms:ReEncrypt*",
    "kms:GenerateDataKey*",
    "kms:DescribeKey"
]
```

Create an IAM function that’s assumable by Account B and ensure the function has a following coverage connected and is a member (as contributor) of your Amazon DataZone mission. For this publish, we name the function dz-assumable-env-dataset-registration-role. By including this function, you possibly can efficiently run the registration Lambda operate.

Within the following coverage, present the AWS Area and account ID equivalent to the place your Amazon DataZone area is created, and the KMS key ARN used to encrypt the area:

  {
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Action": [
                "datazone:CreateDataSource",
                "datazone:CreateEnvironment",
                "datazone:CreateEnvironmentProfile",
                "datazone:GetDataSource",
                "datazone:GetDataSourceRun",
                "datazone:GetEnvironment",
                "datazone:GetEnvironmentProfile",
                "datazone:GetIamPortalLoginUrl",
                "datazone:ListDataSources",
                "datazone:ListDomains",
                "datazone:ListEnvironmentProfiles",
                "datazone:ListEnvironments",
                "datazone:ListProjectMemberships",
                "datazone:ListProjects",
                "datazone:StartDataSourceRun",
                "datazone:UpdateDataSource",
                "datazone:SearchUserProfiles"
            ],
            "Useful resource": "*",
            "Impact": "Enable"
        },
        {
            "Motion": [
                "kms:Decrypt",
                "kms:DescribeKey",
                "kms:GenerateDataKey"
            ],
            "Useful resource": "arn:aws:kms:>:>


}:key/${DataZonekmsKey}",
            "Impact": "Enable"
        }
    ]
}

Add Account B within the belief relationship of this function with the next belief relationship:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::>:root",
                    "arn:aws:iam::>:root",
                ]
            },
            "Motion": "sts:AssumeRole"
        }
    ]
}

Add the function as a member of the Amazon DataZone mission by which you wish to register your knowledge sources. For extra data, see Add members to a mission.

Extra instruments

The next instruments are wanted to deploy the answer utilizing the AWS CDK:

Deploy the answer

After you full the conditions, use the AWS CDK stack offered on the GitHub repo to deploy the answer for computerized registration of information property into the Amazon DataZone area. Full the next steps:

Clone the repository from GitHub to your most well-liked built-in improvement surroundings (IDE) utilizing the next instructions:

git clone https://github.com/aws-samples/sample-how-to-automate-amazon-redshift-cluster-data-asset-publish-to-amazon-datazone

$ cd sample-how-to-automate-amazon-redshift-cluster-data-asset-publish-to-amazon-datazone

On the base of the repository folder, run the next instructions to construct and deploy sources to AWS:
Register to Account B (the information area producer account) utilizing the AWS CLI along with your profile identify.
Be sure to have configured the Area in your credential’s configuration file.
Bootstrap the AWS CDK surroundings with the next instructions on the base of the repository folder. Present the profile identify of your deployment account (Account B). Bootstrapping is a one-time exercise and isn’t wanted in case your AWS account is already bootstrapped.
Change the placeholder parameters (marked with the suffix _PLACEHOLDER) within the file config/DataZoneConfig.ts:
1. Amazon DataZone area and mission identify of your Amazon DataZone occasion. Ensure all names are in lowercase.
2. The AWS account ID of the Amazon DataZone account (Account A).
3. The assumable IAM function from the conditions.
4. The AWS Programs Supervisor parameter identify containing the Secrets and techniques Supervisor secret ARN of the Amazon Redshift credentials.
Use the next command within the base folder to deploy the AWS CDK answer. Throughout deployment, enter y if you wish to deploy the modifications for some stacks whenever you see the immediate Do you want to deploy these modifications (y/n)?
After the deployment is full, register to Account B and open the AWS CloudFormation console to confirm that the infrastructure was deployed.

Check computerized knowledge registration to Amazon DataZone

Full the next steps to check the answer:

Register to Account B (producer account).
On the Lambda console, open the datazone-redshift-dataset-registration operate.
Below TEST EVENTS, select Create new check occasion.

For Occasion identify, enter Redshift, and for Occasion JSON, enter the next JSON construction (change the cluster, schema, database, and desk names based on your surroundings):

{
  "supply": "redshift-user-initiated",
  "detail-type": "Amazon Redshift dataset registration in Amazon DataZone",
  "datasets": [
    {
      "clusterName": "YOUR_REDSHIFT_CLUSTER_NAME",
      "dbName": "DATABASE_NAME",
      "schemas": [
        {
          "schemaName": "SCHEMA_NAME_1",
          "addAllTables": false,
          "addAllViews": false,
          "tables": [
            "TABLE_NAME"
          ],
          "views": []
        },
        {
          "schemaName": "SCHEMA_NAME_2",
          "addAllTables": false,
          "addAllViews": false,
          "tables": [],
          "views": [
            "VIEW_NAME"
          ]
        }
      ]
    }
  ]
}

Select Save.
Select Invoke.
Open the Amazon DataZone console in Account A the place you deployed the sources.
Select Domains within the navigation pane, then open your area.
On the area particulars web page, find the Amazon DataZone knowledge portal URL within the Abstract part. Select the hyperlink to the information portal.
For extra particulars about accessing Amazon DataZone, discuss with How can I entry Amazon DataZone?
Within the knowledge portal, open your mission and select the Knowledge tab.
Within the navigation pane, select Knowledge sources and discover the newly created knowledge supply for Amazon Redshift.
Confirm that the information supply has been efficiently revealed.

After the information sources are revealed, customers can uncover the revealed knowledge and submit a subscription request. The info producer can approve or reject requests. Upon approval, customers can devour the information by querying the information within the Amazon Redshift question editor. The next screenshot illustrates knowledge discovery within the Amazon DataZone knowledge portal.

Clear up

Full the next steps to scrub up the sources deployed by the AWS CDK:

Register to Account B, go to the Amazon DataZone area portal, and verify there isn’t any subscription on your revealed knowledge asset. If there’s a subscription, both ask the subscriber to unsubscribe or revoke the subscription request.
Delete the revealed knowledge property that have been created within the Amazon DataZone mission by the dataset registration Lambda operate.
Delete the remaining sources created utilizing the next command within the base folder:
```
npm run cdk destroy –all
```

Conclusion

Amazon DataZone gives a seamless integration with AWS companies, offering a robust answer for organizations like Volkswagen to interrupt down their knowledge silos and implement efficient knowledge mesh architectures by a simple implementation highlighted on this publish. Through the use of Amazon DataZone, Volkswagen addressed its speedy knowledge sharing hurdles and laid the groundwork for a extra agile, data-driven future in automotive manufacturing. The automated knowledge publishing from numerous warehouses, coupled with standardized governance workflows, has considerably diminished the handbook overhead that when slowed down Volkswagen’s knowledge engineering groups. Now, as a substitute of navigating a labyrinth of emails, tickets, and communication, Volkswagen’s knowledge engineers and knowledge scientists can shortly uncover and entry the information they want, all whereas sustaining their safety and compliance requirements.

Through the use of Amazon DataZone, organizations can convey their remoted knowledge collectively in ways in which make it less complicated for groups to collaborate whereas sustaining safety and compliance at scale. This strategy not solely addresses present knowledge governance challenges but additionally creates a extremely scalable basis for future data-driven improvements. For steering on establishing your group’s knowledge mesh with Amazon DataZone, contact your AWS group at the moment.