Introducing Apache Spark improve agent for Amazon EMR

For organizations operating Apache Spark workloads, model upgrades have lengthy represented a major operational problem. What needs to be a routine upkeep activity typically evolves into an engineering challenge spanning a number of months, consuming useful sources that would drive innovation as a substitute of managing technical debt. Engineering groups should typically manually analyze API deprecation, resolve behavioral adjustments within the engine, tackle shifting dependency necessities, and re-validate each performance and knowledge high quality, all whereas maintaining manufacturing workloads operating easily. This complexity delays entry to efficiency enhancements, new options, and important safety updates.

At re:Invent 2025, we introduced the AI-powered improve agent for Apache Spark on Amazon EMR. Working immediately inside your IDE, this agent handles the heavy lifting of model upgrades that includes analyzing code, making use of fixes, and validating outcomes, when you preserve management over each change. What as soon as took months can now be accomplished in hours.

On this submit, you’ll learn to:

• Assess your current Amazon EMR Spark purposes
• Use the Spark improve agent immediately from the Kiro IDE
• Improve a pattern e-commerce order analytics Spark utility challenge (construct configs, supply code, exams, knowledge high quality validation)
• Evaluate code adjustments after which roll them out via your CI/CD pipeline

Spark improve agent structure

The Apache Spark improve agent for Amazon EMR is a conversational AI functionality designed to speed up Spark model upgrades for EMR purposes. By an MCP-compatible consumer, such because the Amazon Q Developer CLI, the Kiro IDE, or any customized agent constructed with frameworks like Strands, you possibly can work together with a Mannequin Context Protocol (MCP) server utilizing pure language.

Determine 1: A diagram of the Apache Spark improve agent workflow.

Working as a completely managed, cloud-hosted MCP server, the agent removes the necessity to preserve any native infrastructure. All device calls and AWS useful resource interactions are ruled by your AWS Id and Entry Administration (IAM) permissions, making certain the agent operates solely inside the entry you authorize. Your utility code stays in your machine, and solely the minimal info required to diagnose and repair improve points is transmitted. Each device invocation is recorded in AWS CloudTrail, offering full auditability all through the method.

Constructed on years of expertise serving to EMR prospects improve their Spark purposes, the improve agent automates the end-to-end modernization workflow, decreasing handbook effort and eliminating a lot of the trial-and-error usually concerned in main model upgrades. The agent guides you thru six phases:

1. Planning: The agent analyzes your challenge construction, identifies compatibility points, and generates an in depth improve plan. You evaluate and customise this plan earlier than execution begins.
2. Atmosphere setup: The agent configures construct instruments, updates language variations, and manages dependencies. For Python tasks, it creates digital environments with appropriate package deal variations.
3. Code transformation: The agent updates construct recordsdata, replaces deprecated APIs, fixes kind incompatibilities, and modernizes code patterns. Adjustments are defined and proven earlier than being utilized.
4. Native validation: The agent compiles your challenge and runs your take a look at suite. When exams fail, it analyzes errors, applies fixes, and retries. This continues till all exams move.
5. EMR validation: The agent packages your utility, deploys it to EMR, screens execution, and analyzes logs. Runtime points are mounted iteratively.
6. Knowledge high quality checks: The agent can run your utility on each supply and goal Spark variations, evaluate outputs, and report variations in schemas, values, or statistics.

All through the method, the agent explains its reasoning and collaborates with you on choices.

Getting began

(Elective) Assessing your accounts for EMR Spark Upgrades

Earlier than starting a Spark improve, it’s useful to know the present state of your setting. Many shoppers run Spark purposes throughout a number of Amazon EMR clusters and variations, making it difficult to know which workloads needs to be prioritized for modernization. If in case you have already recognized the Spark purposes that you just want to improve or have already got a dashboard, you possibly can skip this evaluation step and transfer to the subsequent part to get began with the Spark improve agent.

Constructing an Evaluation Dashboard

To simplify this discovery course of, we offer a light-weight Python-based evaluation device that scans your EMR setting and generates an interactive dashboard summarizing your Spark utility footprint. The device critiques EMR steps, extracts utility metadata, and computes EMR lifecycle timelines that will help you to:

• Perceive your Spark purposes and their executions distribution over completely different EMR variations.
• Evaluate days remaining till every EMR model reaches finish of help (EOS) for all Spark purposes.
• Consider what purposes needs to be prioritized emigrate to newer EMR model.

Key insights from the evaluation

Determine 2: A graph of EMR variations per utility.

This dashboard reveals what number of Spark purposes are operating on legacy EMR variations, serving to you determine which workloads emigrate first.

Determine 3: a graph of utility use and present variations.

This dashboard identifies your most steadily used purposes and their present EMR variations. Purposes marked in purple point out high-impact workloads that needs to be prioritized for migration.

Determine 4: a utilization and EMR model graph.

This dashboard highlights high-usage purposes operating on older EMR variations. Bigger bubbles symbolize extra steadily used purposes, and the Y-axis reveals the EMR model. Collectively, these dimensions make it simple to identify which purposes needs to be prioritized for improve.

Determine 5: a graph highlighting purposes nearing Finish of Help.

The dashboard identifies purposes approaching EMR Finish of Help, serving to you prioritize migrations earlier than updates and technical help are discontinued. For extra details about help timelines, see Amazon EMR customary help.

After getting recognized the purposes that have to be upgraded, you should use any IDE akin to VS Code, Kiro IDE, or some other setting that helps putting in an MCP server to start the improve.

Getting began with Spark improve agent utilizing Kiro IDE

Stipulations

System necessities

IAM permissions

Your AWS IAM profile should embrace permissions to invoke the MCP server and entry your Spark workload sources. The CloudFormation template offered within the setup documentation creates an IAM function with these permissions, together with supporting sources such because the Amazon S3 staging bucket the place the improve artifacts shall be uploaded. You may as well customise the template to regulate which sources are created or skip sources you like to handle manually.

1. Deploy the template inside the similar area you run your workloads in.
2. Open the CloudFormation Outputs tab and replica the 1-line instruction ExportCommand, then execute it in your native setting.

   export SMUS_MCP_REGION= && export IAM_ROLE=arn:aws:iam::111122223333:function/spark-upgrade-role-xxxxxx && export STAGING_BUCKET_PATH=

3. Configure your AWS CLI profile:

   aws configure set profile.smus-mcp-profile.role_arn ${IAM_ROLE}
   aws configure set profile.smus-mcp-profile.source_profile default
   aws configure set profile.smus-mcp-profile.area ${SMUS_MCP_REGION}

Arrange Kiro IDE and hook up with the Spark improve agent

Kiro IDE gives a visible improvement setting with built-in AI help for interacting with the Apache Spark improve agent.

Set up and configuration:

1. Set up Kiro IDE
2. Open the command palette utilizing Ctrl + Shift + P (Linux) or Cmd + Shift + P (macOS) and Seek for Kiro: Open MCP Config
   
   
   
   Determine 6: the Kiro command palette.
3. Add the Spark improve agent configuration

   "mcpServers": {
       "upgrade-server": {
         "command": "uvx",
         "args": [
           "mcp-proxy-for-aws@latest",
           "https://sagemaker-unified-studio-mcp.${SMUS_MCP_REGION}.api.aws/spark-upgrade/mcp",
           "--service",
           "sagemaker-unified-studio-mcp",
           "--profile",
           "smus-mcp-profile",
           "--region",
           "${SMUS_MCP_REGION}",
           "--read-timeout",
           "180"
         ],
         "timeout": 180000
       }
     }
   }

4. As soon as saved, the Kiro sidebar shows a profitable connection to the improve server.
   
   
   
   Determine 7: Kiro IDE displaying a profitable connection to the MCP server.

Upgrading a pattern Spark utility utilizing Kiro IDE

To display upgrading from EMR 6.1.0 (Spark 3.0.0) to EMR 7.11.0 (Spark 3.5.6), we now have ready a pattern e-commerce order processing utility. This utility fashions a typical analytics pipeline that processes order knowledge to generate enterprise insights, together with buyer income metrics, supply date calculations, and multi-dimensional gross sales reviews. The workload incorporates struct operations, date/interval math, grouping semantics, and aggregation logic patterns generally present in manufacturing knowledge pipelines.

Obtain the pattern challenge

Clone the pattern challenge from the Amazon EMR utilities GitHub repository:

git clone https://github.com/aws-samples/aws-emr-utilities.git
cd aws-emr-utilities/purposes/spark-upgrade-assistant/demo-spark-application

Open the challenge in Kiro IDE

Launch Kiro IDE and open the demo-spark-application folder. Take a second to discover the challenge construction, which incorporates the Maven configuration (pom.xml), the principle Scala utility, unit exams, and pattern knowledge.

Beginning an improve

After getting the challenge loaded within the Kiro IDE, choose the Chat tab on the right-hand facet of the IDE and kind the next immediate to start out the improve of the pattern income analytics utility:

Assist me improve my utility  from Spark 3.0 to Spark 3.5 
Use EMR-EC2 cluster j-9XXXXXXXXXX  with Spark 3.5 for validation. 
Retailer up to date artifacts at s3://
Allow knowledge high quality checks.

Observe: Change j-XXXXXXXXXXXXX along with your EMR cluster ID and along with your S3 bucket title.

How the improve agent works

Step 1: Analyze and plan

After you submit the immediate, the agent analyzes your challenge construction, construct system, and dependencies to create an improve plan. You possibly can evaluate the proposed plan and counsel modifications earlier than continuing.

Determine 8: the proposed improve plan from the agent, prepared for evaluate.

Step 2: Improve dependencies

The agent will analyze all challenge dependencies and makes the mandatory adjustments to improve the variations for compatibility with the goal Spark model. It then compiles the challenge, builds the applying, and runs exams to confirm all the things works appropriately with the goal Spark model.

Determine 9: Kiro IDE upgrading dependency variations.

Step 3: Code transformation

Alongside dependency updates, the agent identifies and fixes code adjustments in supply and take a look at recordsdata arising from deprecated APIs, modified dependencies, or backward incompatible habits. The agent validates these modifications via unit, integration, and distant validation on Amazon EMR on Amazon EC2 or EMR Serverless relying in your deployment mode, iterating till profitable execution.

Determine 10: the improve agent iterating via change testing.

Step 4: Validation

As a part of validation, the agent submits jobs to EMR to confirm the applying runs efficiently with precise knowledge. It additionally compares the output from the brand new Spark model in opposition to the output from the earlier Spark model and gives an information high quality abstract.

Determine 11: the improve agent validating adjustments with actual knowledge.

Step 5: Abstract

As soon as the agent completes the complete automation workflow, it generates a complete improve abstract. This abstract allows you to evaluate the dependency adjustments, code modifications with diffs and file references, related migration guidelines utilized, job configuration updates required for the improve, and knowledge high quality validation standing. After reviewing the abstract and confirming the adjustments meet your necessities, you possibly can then proceed with integrating them into your CI/CD pipeline.

Determine 12: the ultimate improve abstract offered by the Spark improve agent.

Integrating along with your current CI/CD framework

As soon as the Spark improve agent completes the automated improve course of, you possibly can seamlessly combine the adjustments into your improvement workflow.

Pushing adjustments to distant repository

After the improve completes, ask Kiro to create a function department and push the upgraded code

Immediate to Kiro

Create a function department 'spark-upgrade-3.5' and push these adjustments to distant repository.

Kiro executes the mandatory Git instructions to create a clear function department, enabling correct code evaluate workflows via pull requests.

CI/CD pipeline integration

As soon as the adjustments are pushed, your current CI/CD pipeline can routinely set off validation workflows. Well-liked CI/CD platforms akin to GitHub Actions, Jenkins, GitLab CI/CD, or Azure DevOps could be configured to run builds, exams, and deployments upon detecting adjustments to improve branches.

Determine 14: the improve agent submitting a brand new function department with detailed commit message.

Conclusion

Beforehand, maintaining Apache Spark present meant selecting between innovation and months of migration work. By automating the advanced evaluation and transformation work that historically consumed months of engineering effort, the Spark improve agent removes a barrier that may stop you from maintaining your knowledge infrastructure present. Now you can preserve up to date Spark environments with out the useful resource constraints that pressured tough trade-offs between innovation and upkeep. Taking the above Spark utility upgrading expertise for instance, what beforehand required 8 hours of handbook work, together with updating construct configs, resolving construct/compile failures, fixing runtime points, and reviewing knowledge high quality outcomes, now takes simply half-hour with the automated agent.

As knowledge workloads proceed to develop in complexity and scale, staying present with the newest Spark capabilities turns into more and more vital for sustaining aggressive benefit. The Apache Spark improve agent makes this achievable by remodeling upgrades from high-risk, resource-intensive tasks into manageable workflows that match inside regular improvement cycles.

Whether or not you’re operating a handful of purposes or managing a big Spark property throughout Amazon EMR on EC2 and EMR Serverless, the agent gives the automation and confidence wanted to improve quicker.Able to improve your Spark purposes? Begin by deploying the evaluation dashboard to know your present EMR footprint, then configure the Spark improve agent in your most well-liked IDE to start your first automated improve.

For extra info, go to the Amazon EMR documentation or discover the EMR utilities repository for added instruments and sources. Refer for particulars on which variations are supported are listed right here in Amazon EMR documentation.

Particular thanks

A particular due to everybody who contributed from Engineering and Science to the launch of the Spark improve agent and the Distant MCP Service: Chris Kha, Chuhan Liu, Liyuan Lin, Maheedhar Reddy Chappidi, Raghavendhar Thiruvoipadi Vidyasagar, Rishabh Nair, Tina Shao, Wei Tang, Xiaoxi Liu, Jason Cai, Jinyang Li, Mingmei Yang, Hirva Patel, Jeremy Samuel, Weijing Cai, Kartik Panjabi, Tim Kraska, Kinshuk Pahare, Santosh Chandrachood, Paul Meighan, and Rick Sears.

A particular due to all our companions who contributed to the launch of the Spark improve agent and the Distant MCP Service: Karthik Prabhakar, Mark Fasnacht, Suthan Phillips, Arun AK, Shoukat Ghouse, Lydia Kautsky, Larry Weber, Jason Berkovitz, Sonika Rathi, Abhinay Reddy Bonthu, Boyko Radulov, Ishan Gaur, Raja Jaya Chandra Mannem, Rajesh Dhandhukia, Subramanya Vajiraya, Kranthi Polusani, Jordan Vaughn, and Amar Wakharkar.

Concerning the authors