Shutterstock
Information streaming platform Confluent has launched new enhancements to Confluent Cloud, its totally managed service constructed on Apache Kafka. A key improve is the introduction of Snapshot queries that permit customers to question each batch and streaming knowledge utilizing the identical SQL interface. The corporate additionally launched new non-public networking and safety features that make stream processing safer and enterprise-ready.
Agentic AI, which refers to autonomous techniques able to reasoning, planning, and taking motion on behalf of customers, is swiftly transitioning from an rising idea to a foundational aspect in enterprise know-how. Nevertheless, for AI brokers to make the fitting selections and ship most worth, they want streaming and historic datasets. This is usually a drawback if knowledge is housed in disparate techniques, reminiscent of legacy databases and separate cloud storage platforms. Advanced ETL jobs could also be wanted to merge knowledge, however that will add complexity, prices, and inefficiency to the general workflow.
Confluent goals to unravel this situation by Snapshot queries by offering AI brokers with unified entry. As a substitute of counting on brittle pipelines that shuffle knowledge between batch and stream techniques, Snapshot queries are designed to deliver the whole lot collectively in a single place. This implies groups can work with a single question language and interface to investigate previous developments and react to dwell occasions, with out spinning up separate workloads or syncing throughout instruments.
“Agentic AI is shifting from hype to enterprise adoption as organizations look to realize a aggressive edge and win in as we speak’s market,” mentioned Shaun Clowes, Chief Product Officer at Confluent. “However with out high-quality knowledge, even essentially the most superior techniques can’t ship actual worth. The brand new Confluent Cloud for Apache Flink options make it potential to mix real-time and batch knowledge in order that enterprises can belief their agentic AI to drive actual change.”
In accordance with Confluent, Snapshot queries could possibly be notably helpful for producing studies that replicate knowledge’s state at a selected time, analyzing historic knowledge for auditing for compliance functions, and debugging points by inspecting previous knowledge states. Snapshot queries would even be helpful for builders constructing agentic AI techniques and occasion processing workflows that require historic knowledge enrichment.
Out there in early entry by Confluent Cloud for Apache Flink, Snapshot queries depend on the platform’s superior SQL question optimizer that determines whether or not knowledge ought to be fetched from Kafka matters or from open desk codecs like Apache Iceberg or Delta Lake. The characteristic makes use of Tableflow to materialize Kafka occasion streams into these tables, enabling environment friendly historic entry alongside real-time processing.
This interprets to much less complexity for customers. For instance, a developer constructing a fraud detection system not has to manually orchestrate pipelines to drag historic transaction patterns from one system and dwell exercise from one other. As a substitute, they’ll write a single SQL question, and the brand new snapshot question engine robotically determines the place the related knowledge lives and retrieves it effectively.
Agentic AI wants greater than pace, it wants context. As IDC’s Stewart Bond places it, the purpose is to “unify disparate knowledge sorts, together with structured, unstructured, real-time, and historic data, in a single surroundings.” That’s precisely what Confluent is aiming to ship with its newest Flink-powered snapshot queries.
Many organizations hesitate to deploy real-time techniques at scale resulting from safety and compliance dangers, particularly in hybrid cloud environments. Confluent’s newest updates deliver a brand new stage of safety and effectivity to Flink workloads, notably by CCN (Confluent Cloud Community) routing and IP filtering.
(SkillUp/Shutterstock)
The CCN routing works by permitting groups to reuse their current non-public networking configurations already in place for Kafka. This implies they don’t should create new community setups from scratch to run Flink workloads, saving time and decreasing complexity. By extending the identical safe connections to Flink, groups can keep constant safety insurance policies throughout each knowledge techniques. CCN routing is now usually obtainable on Amazon Net Providers (AWS) in all areas the place Flink is supported.
Many organizations operating in hybrid environments want tighter management over which knowledge may be accessed publicly. IP filtering for Flink helps by limiting entry to accepted IP addresses and making it simpler to trace any unauthorized makes an attempt. When paired with CCN routing, it offers groups extra management over their Flink workloads and helps meet safety and compliance wants in real-time settings. IP Filtering is now usually obtainable for all Confluent Cloud customers.
The updates to the platform present that Confluent is eager to maneuver past Kafka and grow to be a full streaming knowledge platform constructed for contemporary AI wants. It isn’t simply including new capabilities, but in addition signaling a broader shift in technique.
Whereas Snowflake constructed its basis on batch processing and Databricks promotes the lakehouse mannequin, Confluent is concentrated extra on delivering real-time intelligence. Apache Flink is on the middle of that push. For agentic AI and different fast-moving workloads, Flink offers techniques the recent context they should make good selections on the fly.
Associated Objects
5 Drivers Behind the Fast Rise of Apache Flink
Confluent Goes On Prem with Apache Flink Stream Processing
Sure, Actual-Time Streaming Information Is Nonetheless Rising
