Implement safe hybrid and multicloud log ingestion with Amazon OpenSearch Ingestion

Operating functions throughout hybrid or multicloud environments creates a standard problem: fragmented logs scattered throughout totally different platforms. This fragmentation complicates monitoring, slows troubleshooting, and reduces operational visibility. To handle this, many organizations search to implement safe log ingestion from all environments right into a centralized platform.

Amazon OpenSearch Service gives a unified resolution for real-time search, analytics, and log administration throughout your whole infrastructure. Amazon OpenSearch Ingestion, a totally managed information collector, simplifies information processing with built-in capabilities to filter, rework, and enrich your logs earlier than evaluation.

Nevertheless, securely sending logs from non-AWS environments presents a problem. Each request to OpenSearch Ingestion requires AWS Signature Model 4 (AWS SigV4) authentication, historically requiring long-term credentials that introduce safety dangers. AWS Identification and Entry Administration Roles Anyplace solves this drawback by offering short-term credentials for workloads operating outdoors AWS.

On this put up, we exhibit tips on how to configure Fluent Bit, a quick and versatile log processor and router supported by varied working methods, to securely ship logs from any surroundings to OpenSearch Ingestion utilizing IAM Roles Anyplace. This strategy alleviates the necessity for long-term credentials whereas offering a complete view of your software logs throughout all environments—bettering safety, simplifying operations, and enhancing your capability to shortly resolve points.

Options overview

The answer on this put up makes use of Fluent Bit to gather logs, retrieve short-term credentials from IAM Roles Anyplace, and signal HTTP log ingestion requests with AWS SigV4 earlier than sending them to the OpenSearch Ingestion pipeline. The next diagram reveals the structure.

This resolution provisions the next key parts:

• Certificates authority – For this put up, we use AWS Personal Certificates Authority (AWS Personal CA) because the certificates authority (CA) supply. Alternatively, you’ll be able to combine with an exterior CA; for extra particulars, see IAM Roles Anyplace with an exterior certificates authority. Certificates issued from public CAs can’t be used as belief anchors for IAM Roles Anyplace.
• X.509 Certificates – We use a pattern non-public certificates saved in AWS Certificates Supervisor (ACM) and issued by AWS Personal CA.
• IAM Roles Anyplace configuration – This consists of the next:
  • Belief anchor – Establishes belief between IAM Roles Anyplace and the required CA.
  • IAM function – Grants permissions for log ingestion and trusts the IAM Roles Anyplace service principal. At minimal, this function have to be granted permission for the osis:Ingest motion.
  • Profile – Defines which roles IAM Roles Anyplace can assume and the utmost permissions granted with the short-term credentials.
• OpenSearch Service area – For this put up, we use an OpenSearch Service area, which is an AWS provisioned equal of an open supply OpenSearch cluster. We create the area inside a digital non-public cloud (VPC); see VPC versus public domains for extra info. Alternatively, you should use an Amazon OpenSearch Serverless assortment, which is an OpenSearch cluster that scales compute capability primarily based in your software’s wants.
• OpenSearch Ingestion – That is configured to obtain logs over HTTP because the pipeline supply and ahead them to the OpenSearch Service area because the pipeline sink.

Connectivity between AWS and your hybrid or multicloud environments

You possibly can entry your OpenSearch Ingestion pipelines utilizing an interface VPC endpoint with push-based HTTP supply, which gives non-public IP tackle connectivity. For manufacturing environments, we suggest utilizing these non-public connections by way of interface endpoints for enhanced safety.

Organising this connectivity requires extra configuration, resembling creating an AWS Website-to-Website VPN connection along with your hybrid and multicloud community. Though this put up focuses on the log ingestion resolution, yow will discover detailed steering on community connectivity within the following sources:

How Fluent Bit retrieves short-term credentials utilizing IAM Roles Anyplace

Utilizing the HTTP output plugin, Fluent Bit can ship logs to the OpenSearch Ingestion pipeline. The next diagram is a simplified view of how Fluent Bit retrieves AWS credentials.

On Linux methods, Fluent Bit can use an AWS Command Line Interface (AWS CLI) profile that makes use of the credential_process parameter to set off an exterior course of. This exterior course of is invoked to generate or retrieve credentials indirectly supported by the AWS CLI.

The next are two widespread mechanisms for the exterior course of:

Though each choices are viable, this put up focuses on IAM Roles Anyplace. On this setup, the AWS IAM Roles Anyplace Credential Helper is executed to deal with the signing course of for the CreateSession API. This returns credentials in a JSON format that Fluent Bit can eat.

As of this writing, the Fluent Bit aws_profile configuration is supported solely on Linux. It’s untested on different Unix-based methods (resembling macOS) and isn’t carried out for Home windows.

Conditions

Earlier than you start this walkthrough, ensure you have the next:

• AWS account necessities – This consists of:
  • An AWS account with permissions to deploy AWS CloudFormation templates.
  • Entry to AWS CloudShell for exporting a pattern non-public certificates we’ll create utilizing AWS CloudFormation in a later step.
• Distant (hybrid or multicloud) surroundings – You will need to have a distant machine with Linux-based working system. This resolution was examined on Ubuntu 24.04 with the next extra tooling put in:

Deploy AWS sources with AWS CloudFormation

Observe these steps to deploy AWS sources required for this resolution:

1. Select Launch Stack:
   
   
   
2. Enter a novel title for Stack title. The default worth is osis-with-iamra.
3. Configure the stack parameters. Default values are supplied within the following desk.

4. Choose the acknowledgement test field and select Create Stack.
   Stack deployment takes about half-hour to finish.
5. When stack creation is full, navigate to the Outputs tab on the AWS CloudFormation console and be aware down the values for the sources created.
   The next desk summarizes the output values.

Export a pattern non-public certificates utilizing CloudShell

Observe these steps to export the pattern non-public certificates created by the CloudFormation stack:

1. Open CloudShell. For extra particulars, see Navigating the AWS CloudShell interface.
2. Export the certificates ARN from the CloudFormation outputs. In the event you modified the stack title within the earlier step, use that worth for , in any other case use the default worth osis-with-iamra.

export CERT_ARN=$(aws cloudformation describe-stacks 
    --stack-name  
    --query 'Stacks[0].Outputs[?OutputKey==`ACMCertificateArn`].OutputValue' 
    --output textual content)

3. Extract the certificates and personal key recordsdata:

# Generate and save the passphrase
export PASSPHRASE=$(openssl rand -base64 32)

# Export certificates utilizing surroundings variables
aws acm export-certificate 
    --certificate-arn $CERT_ARN 
    --passphrase $(echo -n "$PASSPHRASE" | base64) 
    > cert_export.json

# Extract parts to separate recordsdata
jq -r '.Certificates' cert_export.json > certificates.pem
jq -r '.PrivateKey' cert_export.json > encrypted_private_key.pem

# Decrypt the non-public key
openssl rsa -in encrypted_private_key.pem -out private_key.pem -passin move:"$PASSPHRASE"

# Clear surroundings variables
unset PASSPHRASE CERT_ARN

4. Obtain the extracted certificates and personal key recordsdata from CloudShell:
   1. /house/cloudshell-user/certificates.pem
   2. /house/cloudshell-user/private_key.pem

Configure an AWS CLI profile

Observe these steps to configure an AWS CLI profile on your log ingestion surroundings:

1. Retailer the downloaded certificates and personal key to your surroundings. For an automatic strategy to generate and rotate certificates, see Arrange AWS Personal Certificates Authority to situation certificates to be used with IAM Roles Anyplace.
2. Create a brand new profile named osis-pipeline-credentials that invokes the credential course of. Exchange the placeholders along with your particular values. Discover the values for trusted-anchor-arn, profile-arn, and ingestion-role-arn in your CloudFormation stack outputs.

aws configure set profile.osis-pipeline-credentials.credential_process " credential-process 
      --certificate  
      --private-key  
      --trust-anchor-arn  
      --profile-arn  
      --role-arn "

[profile osis-pipeline-credentials]
credential_process =  credential-process       --certificate        --private-key        --trust-anchor-arn        --profile-arn        --role-arn 

cat  ~/fluent-bit.conf
[INPUT]
  title                  tail
  path                  /var/log/syslog
  read_from_head        true
  refresh_interval      5
[OUTPUT]
  title 			http
  match	   		*
  aws_service 		osis
  host 			
  port 			443
  uri 			 
  format 		json
  aws_auth	 	true
  aws_region 		   
  aws_profile 		osis-pipeline-credentials
  tls 			On
EOF

pip set up awscurl

export OPENSEARCH_DOMAIN_ENDPOINT=https://

# Checklist indices matching logs-%{yyyy.MM.dd} format and get most up-to-date one to question
export INDEX=$(awscurl --service es "$OPENSEARCH_DOMAIN_ENDPOINT/_cat/indices?v" | grep -E "logs-[0-9]{4}.[0-9]{2}.[0-9]{2}" | type -r | head -1 | awk '{print $3}')

awscurl --service es $OPENSEARCH_DOMAIN_ENDPOINT/$INDEX/_search 
        -X GET -H "Content material-Kind: software/json" 
        -d '{
            "dimension": 10,
            "type": [
              {"@timestamp": {"order": "desc"}}
            ],
            "question": { "match_all": {} }
          }' | jq '.hits.hits[]._source'

{
  "date": 1732039662.399506,
  "log": "2024-11-19T18:07:42.399375+00:00 test-server fluent-bit[9986]: 200 OK",
  "@timestamp": "2024-11-19T18:07:42.812Z"
}
{
  "date": 1732039662.399501,
  "log": "2024-11-19T18:07:42.399224+00:00 test-server fluent-bit[9986]: [2024/11/19 18:07:42] [ info] [output:http:http.0] test-pipeline-123456789012.us-east-2.osis.amazonaws.com:443, HTTP standing=200",
  "@timestamp": "2024-11-19T18:07:42.812Z"
}
...