Workload administration in OpenSearch-based multi-tenant centralized logging platforms

Trendy architectures use many alternative applied sciences to realize their targets. Service-oriented architectures, cloud providers, distributed tracing, and extra create streams of telemetry and different sign knowledge. Every of those knowledge streams turns into a tenant in your logging backend. If your organization runs multiple software, the IT group will incessantly centralize the storage and processing of log knowledge, making every software a tenant within the general observability system.

Whenever you use Amazon OpenSearch Service to retailer and analyze log knowledge, whether or not as a developer or an IT admin, you should stability these tenants to ensure you ship the assets to every tenant to allow them to ingest, retailer, and question their knowledge. On this publish, we current a multi-layered workload administration framework with a rules-based proxy and OpenSearch workload administration that may successfully deal with these challenges.

Instance use case

On this publish, we focus on GlobalLog, a fictional firm supporting healthcare, finance, retail, safety, and inside tenants, that constructed a centralized logging system with OpenSearch Service. Every tenant has distinctive logging patterns primarily based on their enterprise necessities. Monetary tenants generate advanced, high-volume queries, healthcare tenants concentrate on compliance with average quantity logs and queries, and retail tenants expertise seasonal spikes with heavy dashboard utilization. Inside operation has regular, low-volume logs and rare, easy queries. Safety monitoring has a relentless, high-volume presence all through the system.

Because the GlobalLog’s tenants scaled, operational challenges emerged: high-priority tenant efficiency suffered throughout peak hours, resource-intensive queries induced node crashes, and unpredictable site visitors created instability. Restricted visibility into tenant useful resource utilization sophisticated troubleshooting and cross-domain safety investigations. The platform required sturdy dealing with of various workload patterns and peak utilization occasions, sturdy efficiency isolation to forestall tenant interference, and scalability to handle 30% annual knowledge development.

Answer overview

GlobalLog carried out a complete workload administration technique to deal with the varied calls for of its tenants. The answer manages the tenancy with a tiered tenant placement, a rule-based proxy layer that shapes incoming site visitors primarily based on the tenant profile and the standing of the OpenSearch cluster, and an OpenSearch workload administration plugin that gives granular useful resource governance, allocating assets corresponding to CPU and reminiscence proportionally to every tenant’s tier. The monitoring element gives the intelligence that the answer must do its evaluation and make reactive and proactive scaling and performance-related selections by adjusting the site visitors governance guidelines and insurance policies in a well timed method.

The next diagram illustrates the structure.

GlobalLog multi tier workload administration

Tenant tiering and placement

GlobalLog categorized tenants into 4 tiers primarily based on their logging necessities (quantity, retention, question frequency) and allotted assets accordingly. The tiering system, enforced by the built-in proxy layer and OpenSearch workload administration, prevents useful resource over-allocation whereas ensuring service ranges match enterprise priorities. The specification for every tier is detailed within the following desk.

GlobalLog’s built-in structure streamlines its price allocation and useful resource distribution mannequin. Monetary {industry} tenants pay premium charges for his or her assured high-performance assets, successfully subsidizing the infrastructure that helps extra variable workloads. These tenants are categorized into Tier 1. Healthcare tenants profit from isolation that enforces compliance with out bearing the total price of devoted infrastructure. These tenants are categorized into Tier 2. Retail tenants are categorized into Tier 3 as a result of they admire the elastic capability throughout peak seasons with out sustaining extra capability year-round. Tier 4 consists of the executive tenants with entry to enterprise-grade logging at reasonably priced charges by environment friendly useful resource sharing.

This balanced ecosystem helps GlobalLog keep profitability whereas delivering acceptable service ranges to each tenant no matter their industry-specific workload traits.

Within the subsequent sections, we focus on GlobalLog’s workload administration system.

Proxy layer

GlobalLog’s steady suggestions loop structure creates a dynamic ecosystem that optimizes useful resource allocation throughout various tenant workloads in OpenSearch Service. Quite than relying on static configurations, the structure displays efficiency metrics and tenant utilization patterns to drive scaling and remediation selections. This makes positive the system evolves as workloads change over time.

The proxy layer core element is the OpenSearch Visitors Gateway, which capabilities as an middleman between shoppers and OpenSearch clusters. It options the next key capabilities:

• Rule-based site visitors shaping by sample matching for request paths and parameters
• Metrics for useful resource price allocation
• Visitors replay

GlobalLog expanded the capabilities of their OpenSearch Visitors Gateway by a complete set of enhancements centered on centralization, dynamism, and adaptableness. On the core of this evolution, they used Amazon DynamoDB because the centralized repository for crucial gateway knowledge. This central database homes the entire ecosystem of guidelines, insurance policies, and tenant profiles, alongside essential operational knowledge together with metrics, utilization patterns, SLA necessities, tier configurations, and real-time cluster standing info.

Past this centralization effort, GlobalLog reworked the gateway with a dynamic mechanism able to real-time changes and responsive decision-making. This architectural shift permits the gateway to react intelligently to altering circumstances moderately than following predetermined pathways.

Moreover, GlobalLog carried out an adaptive rule system with refined contextual consciousness. The system now prompts particular guidelines primarily based on present cluster states and tenant utilization patterns, enabling exact useful resource allocation and safety mechanisms that reply to precise circumstances moderately than hypothetical eventualities. The system implements time-based rule scheduling, offering flexibility by permitting completely different limits and insurance policies to mechanically have interaction throughout particular durations corresponding to upkeep home windows. This gives optimum efficiency whereas accommodating essential system operations.

The answer implements a steady suggestions loop between the monitoring system, the OpenSearch cluster, and the proxy layer, the place the move of efficiency metrics and tenant utilization patterns drive automated, rule-based scaling and optimization selections, serving to the system evolve as workloads change over time. On this structure, Amazon EventBridge triggers an AWS Lambda operate when predefined standards are met (for instance, an anomaly is detected in OpenSearch Service), ensuing within the Lambda operate taking steps to remediate the problems by adjusting the site visitors shaping guidelines and importing them to the OpenSearch Visitors Gateway. To stabilize the suggestions loop, GlobalLog took the next steps:

• Added dampening mechanisms to forestall speedy rule modifications
• Applied gradual adjustment patterns as an alternative of binary switches
• Created circuit breakers for computerized fallback to baseline guidelines

OpenSearch workload administration layer

GlobalLog carried out tenant-level admission management and reactive question administration by OpenSearch workload administration. The system makes use of workload administration to outline useful resource limits, primarily based on tenant criticality, offering environment friendly useful resource allocation and stopping bottlenecks.

A key element of OpenSearch’s workload administration is its workload teams. A workload group refers to a logical grouping of queries, sometimes used for managing assets and prioritizing workloads. GlobalLog makes use of workload teams to handle useful resource allocation primarily based on the beforehand outlined tenant tiers. Enterprise-critical workloads obtain substantial CPU and reminiscence ensures, offering constant efficiency for monetary operations. Enterprise Essential tenants function with average useful resource ensures, and Customary and Primary tiers operate with extra constrained assets, reflecting their decrease precedence standing. The next instance exhibits the workload group setup for Enterprise Essential and Enterprise Essential tiers:

PUT _wlm/workload_group
{
  “identify”: “Enterprise Essential”,
  “resiliency_mode”: “enforced”,
  “resource_limits”: {
    “cpu”: 0.5,
    “reminiscence”: 0.5
  }

PUT _wlm/workload_group
{
  “identify”: “Enterprise Essential”,
  “resiliency_mode”: “enforced”,
  “resource_limits”: {
    “cpu”: 0.3,
    “reminiscence”: 0.25
  }

OpenSearch responds with the set useful resource limits and the ID for the workload group for Enterprise Essential tier tenants:

{
"_id":"preXpc67RbKKeCyka72_Gw",
  "identify":"analytics",
 "resiliency_mode":"enforced",
 "resource_limits":{
"cpu":0.5,
 "reminiscence":0.5
  },
 "updated_at":1726270204642
}

To make use of a workload group, use the next code:

GET finindex/_search
Host: localhost:9200
Content material-Kind: software/json
workloadGroupId: preXpc67RbKKeCyka72_Gw
{
 "question": {
      "match": {
             "field_name": "worth"
     }
}
}

Actual-world use circumstances

On this part, we focus on two eventualities the place GlobalLog’s workload administration system helped the corporate overcome numerous challenges.

Situation 1: Safety incident response

Throughout a crucial safety incident, GlobalLog confronted a fancy problem of managing simultaneous log entry requests from a number of enterprise models, every with completely different precedence ranges. On the highest tier had been safety and monetary operations (Tier 1), adopted by healthcare operations (Tier 2), retail operations (Tier 3), and inside operations (Tier 4).

On the proxy layer, GlobalLog gave priority to safety and monetary tenant queries whereas implementing particular limitations for different models. Healthcare operations had been capped at 15 concurrent queries, retail operations had been restricted to five queries per minute, and inside operations had their date ranges narrowed.

OpenSearch workload administration and the proxy layer performed an important function by sustaining the safety group’s question precedence whereas managing useful resource strain, together with the cancellation of advanced retail queries throughout excessive CPU utilization.

Situation 2: Finish-of-month reporting

Throughout month-end reporting durations, GlobalLog efficiently dealt with intensive analytical workloads from a number of tenants. The implementation of time-based guidelines proved notably efficient, with prioritizing Tier 4 tenants for batch reporting throughout common end-of-month off-peak enterprise hours. The next code exhibits an instance of GlobalLog guidelines on this context. The primary rule permits Tier 4 tenants to run stories throughout off-peak enterprise hours, and the second rule denies Tier 4 tenants’ requests throughout enterprise hours:

monthlyReportAllowRule",
"ruleConfig": {
"tenantTier": "tier4$",
"timeWindow": {
     		"dayOfMonth": "25-30",
      		"hours": "18:00-8:00"
    	      }
               }
monthlyReportDenyRule",
"ruleConfig": {
"tenantTier": "^tier4$",
"timeWindow": {
     	       "dayOfMonth": "25-30",
      	       "hours": "9:00-18:00"
    	      }
               }

The system dynamically adjusted useful resource allocation for Tier 4 tenants for the off-peak hours (6:00 PM – 8:00 AM) utilizing the OpenSearch workload administration API.

This complete strategy proved extremely profitable in managing peak reporting durations, facilitating each system stability and optimum efficiency throughout all tenant tiers.

Conclusion

The combination of proxy-layer site visitors shaping with the OpenSearch workload administration plugin in a steady suggestions loop structure achieved resiliency, secure efficiency, and truthful useful resource allocation whereas supporting various enterprise priorities. The implementation mentioned on this publish demonstrates that large-scale, multi-tenant logging environments can successfully serve various enterprise wants on shared infrastructure whereas sustaining efficiency and cost-efficiency.

Check out these workload administration methods in your personal use case and share your suggestions and questions within the feedback.

Concerning the Authors

Ezat Karimi is a Senior Options Architect at AWS, primarily based in Austin, TX. Ezat focuses on designing and delivering modernization options and methods for database functions. Working intently with a number of AWS groups, Ezat helps clients migrate their database workloads to the AWS Cloud.

Jon Handler is a Senior Principal Options Architect at AWS primarily based in Palo Alto, CA. Jon works intently with OpenSearch and Amazon OpenSearch Service, offering assist and steerage to a broad vary of shoppers who’ve vector, search, and log analytics workloads that they wish to transfer to the AWS Cloud. Previous to becoming a member of AWS, Jon’s profession as a software program developer included 4 years of coding a large-scale, ecommerce search engine. Jon holds a Bachelor’s of the Arts from the College of Pennsylvania, and a Grasp’s of Science and a PhD in Laptop Science and Synthetic Intelligence from Northwestern College.