Saying cross-account ingestion for Amazon OpenSearch Service

Amazon OpenSearch Ingestion is a strong information ingestion pipeline that AWS prospects use for a lot of completely different functions, akin to observability, analytics, and zero-ETL search. Many shoppers at this time push logs, traces, and metrics from their functions to OpenSearch Ingestion to retailer and analyze this information.

At the moment, we’re completely happy to announce that OpenSearch Ingestion pipelines now assist cross-account ingestion for push-based sources akin to HTTP and OpenTelemetry (OTel). Organizations can now use this characteristic to effortlessly share information throughout groups. For instance, many organizations have central observability groups—now these groups can create OpenSearch Ingestion pipelines and share them with different groups of their group. You may as well use this characteristic to ingest information into Amazon OpenSearch Service domains or Amazon OpenSearch Serverless collections in different accounts.

Beforehand, sharing OpenSearch Ingestion pipelines throughout accounts required groups to make use of digital personal cloud (VPC) options to share entry. For instance, groups may use VPC peering, which isn’t all the time possible, or AWS Transit Gateway. The brand new cross-account ingestion options in OpenSearch Ingestion can simplify your deployment and scale back price for sharing pipelines.

Answer overview

Let’s have a look at the right way to share a pipeline from a central logging account with two different improvement accounts (A and B). The central logging account can create an OpenSearch Ingestion pipeline utilizing a push-based supply, for instance, HTTP. After creating the pipeline, a member of the central logging workforce can grant entry to the opposite groups. They’ll use a useful resource coverage that provides permissions to the 2 different workforce accounts to create pipeline endpoints. After making this variation, the OpenSearch Ingestion pipeline is out there to be used by the opposite groups.

The next diagram illustrates this configuration.

Within the following sections, we show the right way to implement this answer.

Conditions

First, the central logging account will need to have a VPC with two choices enabled.

• enableDnsSupport have to be set to true
• enableDnsHostnames have to be set to true

The central logging account should additionally create a push-based OpenSearch Ingestion pipeline within the VPC. This is usually a pipeline receiving logs from FluentBit or OpenTelemetry telemetry.

The event accounts which are going to connect with the pipeline additionally will need to have VPCs in the identical area with the identical DNS choices enabled.

• enableDnsSupport have to be set to true
• enableDnsHostnames have to be set to true

Create useful resource coverage

Because the proprietor of the pipeline, you’ll be able to create a useful resource coverage that enables the 2 improvement accounts to create pipeline endpoints in opposition to your pipeline.

The next is an instance useful resource coverage for this situation:

{
  "Model": "2012-10-17",
  "Assertion": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "000000000000",
          "999999999999"
        ]
      },
      "Motion": "osis:CreatePipelineEndpoint",
      "Useful resource": "arn:aws:osis:us-west-2:123456789012:pipeline/central-logging"
    }
  ]
}

The OpenSearch Ingestion console makes it easy to create these insurance policies, as proven within the following screenshot.

Create pipeline endpoint

Now that the central logging account has shared permissions on their pipeline, the event accounts can create pipeline endpoints. A pipeline endpoint is a connection from one VPC to an OpenSearch Ingestion pipeline.

The event accounts are liable for creating the pipeline endpoints within the VPCs they wish to join from. They create this within the subnets they want and supply a safety group. The safety group ought to have an inbound rule permitting entry port HTTPS over port 443 from any supply that the event accounts must ingest logs.

Improvement workforce A can create a pipeline endpoint utilizing a command just like the next:

aws --region us-west-2 osis create-pipeline-endpoint 
--pipeline-arn arn:aws:osis:us-west-2:123456789012:pipeline/central-logging 
--vpc-options '{"SubnetIds":["subnet-123456789012345678","subnet-012345678912345678"],"SecurityGroupIds":["sg-123456789012345678"]}'

Improvement workforce A can even use the OpenSearch Ingestion console to create the pipeline endpoint.

After performing this variation, the VPC for improvement workforce A can have a pipeline endpoint. This pipeline endpoint now permits for ingesting information into the central logging pipeline. Now, Amazon Elastic Compute Cloud (Amazon EC2) situations, Amazon Elastic Container Service (Amazon ECS) duties, Kubernetes pods, and different compute working within the VPC can ingest their log information into the pipeline utilizing instruments akin to FluentBit.

On the similar time or at a later time, improvement workforce B can create a pipeline endpoint as effectively. This workforce will create it for their very own VPC.

After this, the pipeline will now have two pipeline endpoints, so each groups can ingest their log information into the central logging VPC.

Clear up

After a pipeline endpoint is created, both account can take away it. The event groups in our situation can use the DeletePipelineEndpoint API to delete it from their accounts. Moreover, if the central logging account must take away a pipeline endpoint from a pipeline, it could possibly use the RevokePipelineEndpointConnections API. Each choices can be found on the OpenSearch Ingestion console as effectively.

After the pipeline endpoints are eliminated, the central logging workforce can even take away the pipeline in the event that they now not want it.

Conclusion

The brand new pipeline endpoint characteristic for OpenSearch Ingestion simplifies how one can share pipelines for cross-account ingestion. This will help groups use the highly effective options of OpenSearch Ingestion and open up new prospects for groups or organizations utilizing a number of accounts and VPCs. The brand new pipeline endpoint characteristic is out there at this time in AWS Areas the place OpenSearch Ingestion is out there.

To get began with cross-account ingestion in OpenSearch Ingestion, discuss with OpenSearch Ingestion documentation or attempt creating your first cross-account pipeline on the OpenSearch Ingestion console.

Concerning the authors