Create and replace Apache Iceberg tables with partitions within the AWS Glue Information Catalog utilizing the AWS SDK and AWS CloudFormation

Lately, we’ve witnessed a major shift in how enterprises handle and analyze their ever-growing information lakes. On the forefront of this transformation is Apache Iceberg, an open desk format that’s quickly gaining traction amongst large-scale information customers.

Nevertheless, as enterprises scale their information lake implementations, managing these Iceberg tables at scale turns into difficult. Information groups typically must handle desk schema evolution, its partitioning, and snapshots variations. Automation streamlines these operations, gives consistency, reduces human error, and helps information groups deal with higher-value duties.

The AWS Glue Information Catalog now helps Iceberg desk administration utilizing the AWS Glue API, AWS SDKs, and AWS CloudFormation. Beforehand, customers needed to create Iceberg tables within the Information Catalog with out partitions utilizing CloudFormation or SDKs and later add partitions from Amazon Athena or different analytics engines. This prevents the desk lineage from being tracked in a single place and provides steps outdoors automation within the steady integration and supply (CI/CD) pipeline for desk upkeep operations. With the launch, AWS Glue prospects can now use their most popular automation or infrastructure as code (IaC) instruments to automate Iceberg desk creation with partitions and use the identical instruments to handle schema updates and kind order.

On this put up, we present easy methods to create and replace Iceberg tables with partitions within the Information Catalog utilizing the AWS SDK and CloudFormation.

Answer overview

Within the following sections, we illustrate the AWS SDK for Python (Boto3) and AWS Command Line Interface (AWS CLI) utilization of Information Catalog APIs—CreateTable() and UpdateTable()—for Amazon Easy Storage Service (Amazon S3) based mostly Iceberg tables with partitions. We additionally present the CloudFormation templates to create and replace an Iceberg desk with partitions.

Stipulations

The Information Catalog API modifications are made accessible within the following variations of the AWS CLI and SDK for Python:

• AWS CLI model of two.27.58 or above
• SDK for Python model of 1.39.12 or above

AWS CLI utilization

Let’s create an Iceberg desk with one partition, utilizing CreateTable() within the AWS CLI:

aws glue create-table --cli-input-json file://createicebergtable.json

The createicebergtable.json is as follows:

{
    "CatalogId": "123456789012",
    "DatabaseName": "bankdata_icebergdb",
    "Title": "transactiontable1",
    "OpenTableFormatInput": { 
      "IcebergInput": { 
         "MetadataOperation": "CREATE",
         "Model": "2",
         "CreateIcebergTableInput": { 
            "Location": "s3://sampledatabucket/bankdataiceberg/transactiontable1/",
            "Schema": {
                "SchemaId": 0,
                "Kind": "struct",
                "Fields": [ 
                    { 
                        "Id": 1,
                        "Name": "transaction_id",
                        "Required": true,
                        "Type": "string"
                    },
                    { 
                        "Id": 2,
                        "Name": "transaction_date",
                        "Required": true,
                        "Type": "date"
                    },
                    { 
                        "Id": 3,
                        "Name": "monthly_balance",
                        "Required": true,
                        "Type": "float"
                    }
                ]
            },
            "PartitionSpec": { 
                "Fields": [ 
                    { 
                        "Name": "by_year",
                        "SourceId": 2,
                        "Transform": "year"
                    }
                ],
                "SpecId": 0
            },
            "WriteOrder": { 
                "Fields": [ 
                    { 
                        "Direction": "asc",
                        "NullOrder": "nulls-last",
                        "SourceId": 1,
                        "Transform": "none"
                    }
                ],
                "OrderId": 1
            }  
        }
      }
   }
}

The previous AWS CLI command creates the metadata folder for the Iceberg desk in Amazon S3, as proven within the following screenshot.

You may populate the desk with values as follows and confirm the desk schema utilizing the Athena console:

SELECT * FROM "bankdata_icebergdb"."transactiontable1" restrict 10;
insert into bankdata_icebergdb.transactiontable1 values
    ('AFTERCREATE1234', DATE '2024-08-23', 6789.99),
    ('AFTERCREATE5678', DATE '2023-10-23', 1234.99);
SELECT * FROM "bankdata_icebergdb"."transactiontable1";

The next screenshot reveals the outcomes.

After populating the desk with information, you’ll be able to examine the S3 prefix of the desk, which is able to now have the information folder.

The information folders partitioned in response to our desk definition and Parquet information recordsdata created from our INSERT command can be found underneath every partitioned prefix.

Subsequent, we replace the Iceberg desk by including a brand new partition, utilizing UpdateTable():

aws glue update-table --cli-input-json file://updateicebergtable.json

The updateicebergtable.json is as follows.

{
  "CatalogId": "123456789012",
  "DatabaseName": "bankdata_icebergdb",
  "Title": "transactiontable1",
  "UpdateOpenTableFormatInput": {
    "UpdateIcebergInput": {
      "UpdateIcebergTableInput": {
        "Updates": [
          {
            "Location": "s3://sampledatabucket/bankdataiceberg/transactiontable1/",
            "Schema": {
              "SchemaId": 1,
              "Type": "struct",
              "Fields": [
                {
                  "Id": 1,
                  "Name": "transaction_id",
                  "Required": true,
                  "Type": "string"
                },
                {
                  "Id": 2,
                  "Name": "transaction_date",
                  "Required": true,
                  "Type": "date"
                },
                {
                  "Id": 3,
                  "Name": "monthly_balance",
                  "Required": true,
                  "Type": "float"
                }
              ]
            },
            "PartitionSpec": {
              "Fields": [
                {
                  "Name": "by_year",
                  "SourceId": 2,
                  "Transform": "year"
                },
                {
                  "Name": "by_transactionid",
                  "SourceId": 1,
                  "Transform": "identity"
                }
              ],
              "SpecId": 1
            },
            "SortOrder": {
              "Fields": [
                {
                  "Direction": "asc",
                  "NullOrder": "nulls-last",
                  "SourceId": 1,
                  "Transform": "none"
                }
              ],
              "OrderId": 2
            }
          }
        ]
      }
    }
  }
}

UpdateTable() modifies the desk schema by including a metadata JSON file to the underlying metadata folder of the desk in Amazon S3.

We insert values into the desk utilizing Athena as follows:

insert into bankdata_icebergdb.transactiontable1 values
    ('AFTERUPDATE1234', DATE '2025-08-23', 4536.00),
    ('AFTERUPDATE5678', DATE '2022-10-23', 23489.00);
SELECT * FROM "bankdata_icebergdb"."transactiontable1";

The next screenshot reveals the outcomes.

Examine the corresponding modifications to the information folder within the Amazon S3 location of the desk.

This instance has illustrated easy methods to create and replace Iceberg tables with partitions utilizing AWS CLI instructions.

SDK for Python utilization

The next Python scripts illustrate utilizing CreateTable() and UpdateTable() for an Iceberg desk with partitions:

CloudFormation utilization

Use the next CloudFormation templates for CreateTable() and UpdateTable(). After the CreateTable template is full, replace the identical stack with the UpdateTable template by creating a brand new changeset to your stack and executing it.

Clear up

To keep away from incurring prices on the Iceberg tables created utilizing the AWS CLI, delete the tables from the Information Catalog.

Conclusion

On this put up, we illustrated easy methods to use the AWS CLI to create and replace Iceberg tables with partitions within the Information Catalog. We additionally offered the SDK for Python and CloudFormation pattern code and templates. We hope this helps you automate the creation and administration of your Iceberg tables with partitions in your CI/CD pipelines and manufacturing environments. Strive it out to your personal use case and share your suggestions within the feedback part.

In regards to the authors

Acknowledgements: A particular because of everybody who contributed to the event and launch of this characteristic – Purvaja Narayanaswamy, Sachet Saurabh, Akhil Yendluri and Mohit Chandak.