On this submit, you’ll learn to use Terraform to automate and streamline your Apache Flink utility lifecycle administration on Amazon Managed Service for Apache Flink. We’ll stroll you thru the entire lifecycle together with deployment, updates, scaling, and troubleshooting frequent points.
Managing Apache Flink functions by means of their total lifecycle from preliminary deployment to scaling or updating might be complicated and error-prone when carried out manually. Groups usually wrestle with inconsistent deployments throughout environments, problem monitoring configuration adjustments over time, and complicated rollback procedures when points come up.
Infrastructure as Code (IaC) addresses these challenges by treating infrastructure configuration as code that may be versioned, examined, and automatic. Whereas there are totally different IaC instruments obtainable together with AWS CloudFormation or AWS Cloud Improvement Package (AWS CDK), we concentrate on HashiCorp Terraform to automate the entire lifecycle administration of Apache Flink functions on Amazon Managed Service for Apache Flink.
Managed Service for Apache Flink means that you can run Apache Flink jobs at scale with out worrying about managing clusters and provisioning assets. You possibly can concentrate on creating your Apache Flink utilizing your Built-in Improvement Atmosphere (IDE) of alternative, constructing and packaging the applying utilizing normal construct and CI/CD instruments. As soon as your utility is packaged and uploaded to Amazon S3, you possibly can deploy and run it with a serverless expertise.
Whilst you can management your Managed Service for Apache Flink functions instantly utilizing the AWS Console, CLI, or SDKs, Terraform offers key benefits similar to model management of your utility configuration, consistency throughout environments, and seamless CI/CD integration. This submit builds upon our two-part weblog sequence “Deep dive into the Amazon Managed Service for Apache Flink utility lifecycle – Half 1” and “Half 2” that discusses the overall lifecycle ideas of Apache Flink functions.
We use the pattern code revealed on the GitHub repository to exhibit the lifecycle administration. Word that this isn’t a production-ready resolution.
Organising your Terraform setting
Earlier than you possibly can handle your Apache Flink functions with Terraform, it is advisable arrange your execution setting. On this part, we’ll cowl easy methods to configure Terraform state administration and credential dealing with. The Terraform AWS supplier helps Managed Service for Apache Flink by means of the aws_kinesis_analyticsv2_application useful resource (utilizing the legacy title “Kinesis Analytics V2“).
Terraform state administration
Terraform makes use of a state file to trace the assets it manages. In Terraform, storing the state file in Amazon S3 is a finest follow for groups working collaboratively as a result of it offers a centralised, sturdy, and safe location for monitoring infrastructure adjustments. Nevertheless, since a number of engineers or CI/CD pipelines could run Terraform concurrently, state locking is important to forestall race situations the place concurrent executions might corrupt the state. S3 as backend is often used for state storage and locking, making certain that just one Terraform course of can modify the state at a time, thus sustaining infrastructure consistency and avoiding deployment conflicts.
Passing credentials
To run Terraform inside a Docker container whereas making certain that it has entry to the required AWS credentials and infrastructure code, we observe a structured strategy. This course of entails exporting AWS credentials, mounting required directories, and executing Terraform instructions inside a Docker container. Let’s break this down step-by-step. Earlier than operating Terraform, we have to guarantee that our Docker container has entry to the required AWS credentials. Since we’re utilizing short-term credentials, we generate them utilizing the AWS CLI with the next command:
aws configure export-credentials --profile $AWS_PROFILE --format env-no-export > .env.dockerThis command does the next:
- It exports AWS credentials from a selected AWS profile (
$AWS_PROFILE). - The credentials are saved in
.env.dockerin a format appropriate for Docker. - The
--format env-no-exportpossibility shows credentials as non-exported shell variables
This file (.env.docker) will later be used to go credentials into the Docker container
Working Terraform in Docker
Working Terraform inside a Docker container offers a constant, moveable, and remoted setting for managing infrastructure with out requiring Terraform to be put in instantly on the native machine. This strategy ensures that Terraform runs in a managed setting, decreasing dependency conflicts and bettering safety. To execute Terraform inside a Docker container, we use a docker run command that mounts the required directories and passes AWS credentials, permitting Terraform to use infrastructure adjustments seamlessly.
The Terraform configuration recordsdata are saved in an area terraform folder, which is nearly hooked up to the container utilizing the -v flag. This permits the containerised Terraform occasion to entry and modify infrastructure code as if it had been operating regionally.
To run Terraform in Docker, the next command is executed:
Breaking down this command step-by-step:
--env-file .env.dockeroffers the AWS credentials required for Terraform to authenticate.--rm -itruns the container interactively and is eliminated after execution to forestall muddle.-v ./terraform:/house/flink-project/terraformmounts the Terraform listing into the container, making the configuration recordsdata accessible.-v ./construct.sh:/house/flink-project/construct.shmounts theconstruct.shscript, which incorporates the logic to construct JAR file for flink and execute Terraform instructions.msf-terraformis the Docker picture used, which has Terraform pre-installed.bash construct.sh applyruns theconstruct.shscript contained in the container, passingapplyas an argument to set off the Terraform apply course of.
Contained in the container, construct.sh sometimes consists of instructions similar to terraform init to initialise the Terraform working listing and terraform apply to use infrastructure adjustments. For the reason that Terraform execution occurs fully inside the container, there isn’t a want to put in Terraform regionally, and the method stays constant throughout totally different methods. This technique is especially helpful for groups working in collaborative environments, because it standardises Terraform execution and permits for reproducibility throughout improvement, staging, and manufacturing environments.
Managing utility lifecycle with Terraform
On this part, we stroll by means of every section of the Apache Flink utility lifecycle and perceive how one can implement these operations utilizing Terraform. Whereas these operations are normally absolutely automated as a part of a CI/CD pipeline, you’ll execute the person steps manually from the command line for demonstration functions. There are various methods to run Terraform relying in your group’s tooling and infrastructure setup, however for this demonstration, we run Terraform in a container alongside the applying construct to simplify dependency administration. In real-world eventualities, you’ll sometimes have separate CI/CD levels for constructing your utility and deploying with Terraform, with distinct configurations for every setting. Since each group has totally different CI/CD tooling and approaches, we hold these implementation particulars out of scope and concentrate on the core Terraform operations.
For a complete deep dive into Apache Flink utility lifecycle operations, seek advice from our earlier two-part weblog sequence.
Create and begin a brand new utility
To get began you wish to create your Apache Flink utility operating on Managed Service for Apache Flink. It is best to execute the next Docker command:
This command will full the next operations by executing the bash script construct.sh:
- Constructing the Java ARchive (JAR) file out of your Apache Flink utility
- Importing the JAR file to S3
- Setting the config variables in your Apache Flink utility in
terraform/config.tfvars.json - Create and deploy the Apache Flink utility to Managed Service for Apache Flink utilizing
terraform apply
Terraform absolutely covers this operation. You possibly can verify the operating Apache Flink utility utilizing AWS CLI or contained in the Managed Apache Flink Console after Terraform completes with Apply Full! Terraform is anticipating the Apache Flink artifact, i.e. the JAR file to be packaged and copied to S3. This operation is normally a part of the CI/CD pipeline and executed earlier than invoking the terraform apply. Right here, the operation is specified within the construct.sh script.
Deploy code change to an utility
You could have efficiently created and began the Flink utility. Nevertheless, you understand that you must make a change to the Flink utility code. Let’s make a code change to the applying code in flink/ and see easy methods to construct and deploy it. After making the required adjustments, you merely need to run the next Docker command once more that builds the JAR file, uploads it to S3 and deploys the Apache Flink utility utilizing Terraform:
This section of the lifecycle is absolutely supported by Terraform so long as each functions are state appropriate, that means that the operators of the upgraded Apache Flink utility are capable of restore the state from the snapshot that’s taken from the previous utility model, earlier than Managed Service for Apache Flink stops and deploys the change. For instance, eradicating a stateful operator with out enabling the allowNonRestoredState flag or altering an operator’s UID might forestall the brand new utility from restoring from the snapshot. For extra info on state compatibility, seek advice from Upgrading Purposes and Flink Variations. For an instance of state incompatibility, and techniques for dealing with state incompatibility, seek advice from Introducing the brand new Amazon Kinesis supply connector for Apache Flink.
When deploying a code change goes improper – An issue prevents the applying code from being deployed
You additionally should be cautious with deploying code adjustments that include bugs stopping the Apache Flink job from beginning. For extra info, seek advice from failure mode (a) – an issue prevents the applying code from being deployed beneath When beginning or updating the applying goes improper. As an illustration, this may be simulated by setting the mainClass in flink/pom.xml mistakenly to com.amazonaws.providers.msf.WrongJob. Much like earlier than you construct the JAR, add it and run the terraform apply by operating the Docker command from above. Nevertheless, Terraform now fails to accurately apply the adjustments and throws an error message because the Apache Flink utility fails to accurately replace. Lastly, the utility standing strikes to READY.
To treatment the problem, you must change the worth of mainClass again to the unique one and deploy the adjustments to Managed Service for Apache Flink. The Apache Flink utility stays in READY standing and doesn’t begin routinely, as this was its state earlier than making use of the repair. Word that Terraform doesn’t attempt to begin the applying once you deploy a change. You’ll have to manually begin the Flink utility utilizing the AWS CLI or by means of the Managed Apache Flink Console.
As detailed in Half 2 of the companion weblog, there’s a second failure state of affairs the place the applying begins efficiently, however the job turns into caught in a steady fail-and-restart loop. A code change may trigger this failure mode. We are going to cowl the second error state of affairs once we cowl deploying configuration adjustments.
Handbook rollback utility code to earlier utility code
As a part of the lifecycle administration of your Apache Flink utility, it’s possible you’ll must explicitly rollback to a earlier operating utility model. That is significantly helpful when a newly deployed utility model with utility code adjustments reveals surprising behaviour and also you wish to explicitly rollback the applying. Presently, Terraform doesn’t help specific rollbacks of your Apache Flink utility operating in Managed Service for Apache Flink. You’ll have to resort to therollbackApplication API by means of the AWS CLI or the Managed Service for Apache Flink Console to revert the applying to the earlier operating model.
While you carry out the express rollback, Terraform will initially not concentrate on the adjustments. Extra particularly, the S3 path to the JAR file within the Managed Service for Apache Flink service (see left a part of the picture beneath) is totally different to the S3 path denoted within the terraform.tfstate file saved in Amazon S3 (see the appropriate a part of the picture beneath). Luckily, Terraform will at all times carry out refreshing actions that embrace studying the present settings from all managed distant objects and updating the Terraform state to match as a part of making a plan in each terraform plan and terraform apply instructions.
In abstract, whereas you cannot carry out a handbook rollback utilizing Terraform, Terraform will routinely refresh the state when deploying a change utilizing terraform apply.
Deploy config change to utility
You could have already made adjustments to the applying code of your Apache Flink utility. What about making adjustments to the config of the applying, e.g., altering runtime parameters? Think about you wish to change the utility logging stage of your operating Apache Flink utility. To vary the logging stage from ERROR to INFO, you must change the worth for flink_app_monitoring_metrics_level within the terraform/config.tfvars.json to INFO. To deploy the config adjustments, it is advisable run the docker run command once more as carried out within the earlier sections. This state of affairs works as anticipated and is absolutely coated by Terraform.
What occurs when the Apache Flink utility deploys efficiently however fails and restarts throughout execution? For extra info, please seek advice from failure mode (b) – the applying is began, the job is caught in a fail-and-restart loop beneath When beginning or updating the applying goes improper. Word that this failure mode can occur when making code adjustments as nicely.
When deploying config change goes improper – The applying is began, the job is caught in a fail-and-restart loop
Within the following instance, we apply a improper configuration change stopping the Kinesis connector from initialising accurately, finally placing the job in a fail-and-restart loop. To simulate this failure state of affairs, you’ll want to change the Kinesis stream configuration by altering the stream title to a non-existent one. This variation is made within the terraform/config.tfvars.json file, particularly altering the stream.title worth beneath flink_app_environment_variables. While you deploy with this invalid configuration, the preliminary deployment will seem profitable, displaying an Apply Full! message. The Flink utility standing may even present as RUNNING. Nevertheless, the precise behaviour reveals issues. Should you verify the Flink Dashboard, you’ll see the applying is repeatedly failing and restarting. Additionally, you will note a warning message in regards to the utility requiring consideration within the AWS Console.
As detailed within the part Monitoring Apache Flink utility operations within the companion weblog (half 2), you possibly can monitor the FullRestarts metric to detect the fail-and-restart loop.
Reverting the adjustments made to the setting variable and deploying the adjustments will lead to Terraform displaying the next error message: Didn’t take snapshot for the applying flink-terraform-lifecycle at this second. The applying is at present experiencing downtime.
You must force-stop with no snapshot and restart the applying with a snapshot to get your Flink utility again to a correctly functioning state. It is best to continuously monitor the applying state of your Apache Flink utility to detect any points.
Different frequent operations
Manually scaling the applying
One other frequent operation within the lifecycle of your Apache Flink utility is scaling the applying up or down by adjusting the parallelism. This operation adjustments the variety of Kinesis Processing Items (KPUs) allotted to your utility. Let’s have a look at two totally different scaling eventualities and the way they’re dealt with by Terraform.
Within the first state of affairs, you wish to change the parallelism of your operating Apache Flink utility inside the default parallelism quota. To do that, it is advisable modify the worth for flink_app_parallelism within the terraform/config.tfvars.json file. After updating the parallelism worth, you deploy the adjustments by operating the Docker command as carried out within the earlier sections:
This state of affairs works as anticipated and is absolutely coated by Terraform. The applying can be up to date with the brand new parallelism setting, and Managed Service for Apache Flink will modify the allotted KPUs accordingly. Word that there’s a default quota of 64 KPUs for a single Managed Service for Apache Flink utility, which should be raised proactively through a quota enhance request if it is advisable scale your Managed Service for Apache Flink utility past 64 KPUs. For extra info, seek advice from Managed Service for Apache Flink quota.
Much less frequent change deployments which require particular dealing with On this part we analyze some much less frequent change deployment eventualities which require some particular dealing with.
Deploy code change that removes an operator
Eradicating an operator out of your Apache Flink utility requires particular consideration, significantly relating to state administration. While you take away an operator, the state from that operator nonetheless exists within the newest snapshot, however there’s not a corresponding operator to revive it. Let’s take a more in-depth have a look at this state of affairs and perceive how one can deal with it correctly. First, it is advisable guarantee that the parameter AllowNonRestoredState is about to True. This parameter specifies whether or not the runtime is allowed to skip a state that can not be mapped to the brand new program, when restoring from a snapshot. Permitting non-restored state is required to efficiently replace an Apache Flink utility once you dropped an operator. To allow the AllowNonRestoredState, it is advisable set the configuration worth for flink_app_allow_non_restored_state to true in terraform/config.tfvars.json. Then, you possibly can go forward and take away an operator: For instance, you possibly can instantly have the sourceStream write to the sink connector in flink/src/most important/java/com/amazonaws/providers/msf/StreamingJob.java. Change code line 146 from windowedStream.sinkTo(sink).uid("kinesis-sink")to sourceStream.sinkTo(sink).uid("kinesis-sink"). Just be sure you have commented out the total windowedStream code block (traces 103 to 140).
This variation will take away the windowed computation and instantly join the supply stream to the sink, successfully eradicating the stateful operation. After eradicating the operator out of your Flink utility code, you deploy the adjustments utilizing the Docker command as beforehand carried out. Nevertheless, the deployment fails with the next error message: Couldn’t execute utility. Because of this, the Apache Flink utility strikes to the READY state. To get well from this example, it is advisable restart the Apache Flink utility utilizing the newest snapshot for the applying to efficiently begin and transfer to RUNNING standing. Importantly, it is advisable guarantee that AllowNonRestoredState is enabled. In any other case, the applying will fail to begin because it can’t restore the state for the eliminated operator.
Deploy change that breaks state compatibility with system rollback enabled
Throughout the lifecycle administration of your Apache Flink utility, you may encounter eventualities the place code adjustments break state compatibility. This sometimes occurs once you modify stateful operators in ways in which forestall them from restoring their state from earlier snapshots.
A typical instance of breaking state compatibility is altering the UID of a stateful operator (similar to an aggregation or windowing operator) in your utility code. To safeguard towards such breaking adjustments, you possibly can allow the automated system rollback function in Managed Service for Apache Flink as described within the subsection Rollback beneath Lifecycle of an utility in Managed Service for Apache Flink beforehand. This function is disabled by default and might be enabled utilizing the AWS Administration Console or invoking the UpdateApplication API operation. There is no such thing as a means in Terraform to allow system rollback.
Subsequent, let’s exhibit this by breaking the state compatibility of your Apache Flink utility by altering the UID of a stateful operator, e.g., the string windowed-avg-price in line 140 of flink/src/most important/java/com/amazonaws/providers/msf/StreamingJob.java to windowed-avg-price-v2 and deploy the adjustments as earlier than. You’ll encounter the next error:
Error: ready for Kinesis Analytics v2 Software (flink-terraform-lifecycle) operation (*) success: surprising state ‘FAILED’, needed goal ‘SUCCESSFUL’. final error: org.apache.flink.runtime.relaxation.handler.RestHandlerException: Couldn’t execute utility.
At this level, Managed Service for Apache Flink routinely rolls again the applying to the earlier snapshot with the earlier JAR file, sustaining your utility’s availability as you will have enabled system-rollback functionality. Terraform will initially be not conscious of the carried out rollback. Luckily, as we now have already witnessed in subsection Handbook rollback utility code to earlier utility code, Terraform will routinely refresh the state once we change UID to the earlier worth and deploy the adjustments.
In-place improve of Apache Flink runtime model
Managed Service for Apache Flink helps in-place improve to new Flink runtime variations. See the documentation for extra particulars. Updating the applying dependencies and any required code adjustments is a accountability of the person. After you have up to date the code artifact, the service is ready to improve the runtime of your operating utility in-place, with out information loss. Let’s look at how Terraform handles Flink model upgrades.
To improve your Apache Flink utility from model 1.19.1 to 1.20, it is advisable:
- Replace the Flink dependencies in your
flink/pom.xmlto model1.20.0(flink.modelto1.20.1andflink.connector.modelto5.0.0-1.20in - Replace the
flink_app_runtime_environmenttoFLINK-1_20interraform/config.tfvars.json - Construct and deploy the adjustments utilizing the acquainted
docker runcommand
Terraform efficiently performs an in-place improve of your Flink utility. You’ll obtain the next message: Apply full! Sources: 0 added, 1 modified, 0 destroyed.
Operations at present not supported by Terraform
Let’s take a more in-depth have a look at operations which might be at present not supported by Terraform.
Beginning or stopping the applying with none configuration change
Terraform offers the start_application parameter, indicating whether or not to begin or cease the applying. You possibly can set this parameter utilizing flink_app_start in config.tfvars.json to cease your operating Apache Flink utility. Nevertheless, this can solely work if the present configuration worth is about to true. In different phrases, Terraform solely responds to the change within the parameter worth, not absolutely the worth itself. After Terraform applies this transformation, your Apache Flink utility will cease and its utility standing will transfer to READY. Equally, restarting the applying requires altering the flink_app_start worth again to true, however this can solely take impact if the present configuration worth is false. Terraform will then restart your utility, shifting it again to the RUNNING state.
In abstract, you can’t begin or cease your Apache Flink utility with out making any configuration change in Terraform. You must use AWS CLI, AWS SDK or AWS Console to begin or cease your utility.
Restarting utility from an older snapshot or no snapshot with none configuration change
Much like the earlier part, Terraform requires an precise configuration change of application_restore_type to set off a restart with totally different snapshot settings. Merely reapplying the identical configuration values received’t provoke a restart from a special snapshot or no snapshot. You must use AWS CLI, AWS SDK or AWS Console to restart your utility from an older snapshot.
Performing rollback triggered manually or by system-rollback function
Terraform doesn’t help performing a handbook rollback nor computerized system rollback. As well as, Terraform may even not bear in mind when such a rollback is happening. The state info can be outdated, e.g. S3 path info. Nevertheless, Terraform routinely performs refreshing actions to learn settings from all managed distant objects and updates the Terraform state to match. Consequently, you possibly can have Terraform refresh the Terraform state by efficiently operating a terraform apply command.
Conclusion
On this submit, we demonstrated easy methods to use Terraform to automate the lifecycle administration of your Apache Flink functions on Managed Service for Apache Flink. We walked by means of elementary operations together with creating, updating, and scaling functions, explored how Terraform handles varied failure eventualities and examined superior eventualities similar to eradicating operators and performing in-place runtime upgrades. We additionally recognized operations which might be at present not supported by Terraform.
For extra info, see Run a Managed Service for Apache Flink utility and our two-part weblog on Deep dive into the Amazon Managed Service for Apache Flink utility lifecycle.
