Doing Extra With Your Present Kafka

July 30, 2025

10

(Quardia/Shutterstock)

Apache Kafka has grow to be the inspiration for real-time knowledge pipelines throughout industries. From processing monetary transactions to monitoring IoT sensor knowledge, Kafka is a key constructing block of enterprise structure. Regardless of its usefulness, organizations and builders alike nonetheless battle to unlock the total worth of their Kafka investments.

The problem, nonetheless, isn’t Kafka itself. It’s all the things round it. From the custom-built proxies to the restricted entry, restricted governance and operational complexity, customers have limitations that stop real-time knowledge from being totally leveraged throughout groups. For a lot of developer groups, Kafka stays highly effective however inaccessible, and scalable however costly to handle.

In accordance with IDC, 90% of the world’s largest firms will use real-time intelligence to enhance providers and buyer expertise by this 12 months. Gartner predicts that 68% of IT leaders plan to extend their use of event-driven structure (EDA). Given these statistics, organizations can’t afford for his or her Kafka pipelines to sit down underutilized.

The transformation right into a real-time enterprise isn’t only a technical shift, it’s a strategic one. In accordance with MIT’s Middle for Info Programs Analysis (CISR), firms within the high quartile of real-time enterprise maturity report 62% increased income progress and 97% increased revenue margins than these within the backside quartile. These organizations use real-time knowledge not solely to energy techniques however to tell choices, personalize buyer experiences and streamline operations. Kafka is vital on this technique, however solely when its knowledge streams are totally accessible and actionable.

Navigating the Complexities of Kafka

Many groups battle with exposing Kafka subjects in a safe, discoverable and managed means. Inside builders usually want specialised data to entry or work together with it, which slows growth and creates bottlenecks. In the meantime, safety and compliance groups face challenges imposing constant authentication and authorization insurance policies. This may be compounded by organizations with a number of Kafka clusters or cases.

To bridge the hole, organizations usually construct {custom} proxies or integration layers to reveal Kafka to exterior groups or companions. Whereas practical, these DIY options can break simply and are arduous to take care of and scale. Kafka will not be a full-stack governance or API resolution, and that is the place the complications start.

Take into account an organization with a number of gross sales and product techniques producing dwell utilization knowledge. With no standardized gateway layer, every integration between these techniques and their Kafka clusters requires {custom} engineering effort, one API for the CRM, one other for the billing platform and a 3rd for the analytics device. Over time, this patchwork method turns into fragile and troublesome to audit or increase.

Reframing Kafka as an API

Organizations are beginning to consider Kafka in a different way, although, and are treating it as an extension of the broader API ecosystem. New applied sciences, like Kong Occasion Gateway, permit organizations to reveal Kafka subjects and occasion streams as managed APIs. This brings built-in governance, observability and safety.

(BezierMagic/Shutterstock)

There are sensible implications to this reframing, together with:

Kafka subjects could be printed in an inner or exterior developer portal, similar to REST APIs, permitting for simpler reuse and collaboration. Function-based entry controls (RBAC), OAuth2 and different insurance policies could be utilized to Kafka subjects utilizing present API administration instruments. By virtualizing subjects and permitting secure cluster sharing, groups can cut back pointless duplication of techniques whereas sustaining entry management. Encryption and visitors shaping make it simpler to maneuver occasion workloads into cloud-based Kafka providers.

This additionally offers builders a single, unified management pane throughout REST, occasion and AI-based APIs. This simplifies growth and improves operational visibility.

This opens the door to a variety of real-time enterprise purposes. For instance, a telecommunications supplier may use occasion gateways to reveal streaming community telemetry to each inner instruments and third-party builders constructing analytics apps. These APIs could possibly be versioned, rate-limited and secured, similar to any REST API, however powered by dwell Kafka streams. This method permits new income streams with out duplicating knowledge pipelines or rebuilding core techniques.

A Extra Strategic Function for Kafka

When occasion streams are discoverable, safe and simple to eat, they’re extra prone to grow to be strategic property. For instance, a Kafka subject monitoring fee occasions could possibly be uncovered as a self-service API for inner analytics groups, customer-facing dashboards or third-party companions.

This unlocks sooner time to worth for brand new purposes, permits higher reuse of present knowledge infrastructure, boosts developer productiveness and helps organizations meet compliance necessities extra simply.

Kafka is already doing the heavy lifting for real-time knowledge throughout the enterprise. However to get full ROI, organizations should transfer past merely deploying Kafka and make it accessible, governable and aligned with the broader developer and enterprise ecosystem.

Occasion gateways supply a sensible and highly effective technique to shut the hole between infrastructure and innovation. They make it attainable for builders and enterprise groups alike to construct on high of real-time knowledge, securely, effectively and at scale. As extra organizations transfer towards AI-driven and event-based architectures, turning Kafka into an accessible and governable a part of your API technique could also be one of many highest-leverage steps you’ll be able to take, not only for IT, however for all the enterprise.

Concerning the writer: Saju Pillai is the senior vp of engineering at Kong. A seasoned engineering government skilled in constructing groups and merchandise from the bottom up at each startups and world companies, Pillai labored as a principal engineer at Oracle Corp programming HTTP servers and Fusion Middleware tech. He then went on to construct and efficiently exit a startup within the RBA area. Earlier than becoming a member of Kong, Saju most lately constructed Concur’s core platform as the corporate’s VP of engineering and later ran Concur’s R&D and Infrastructure Operations as CTO and SVP of engineering.