Cisco IT’s community observability transformation

From knowledge overload to enhanced digital resilience. Cisco IT unified telemetry knowledge throughout its huge community, enabling automation to deal with 99.998% of alerts and attaining zero main incidents – empowering engineers to proactively handle community well being at scale.  

 The information downside: overload, restricted perception, and silos

Cisco IT manages an unlimited, complicated setting with lots of of 1000’s of property – together with computer systems, switches, entry factors, house gadgets, and a wide selection of functions and companies – in addition to exterior programs like web service and cloud suppliers. Every of those property generates telemetry, presenting a problem to successfully monitor and make sense of excessive volumes of numerous knowledge throughout our surroundings. 

In our earlier community operations mannequin, we outsourced a perform chargeable for community observability monitoring, second-level help for triage, and technical experience. This outsourced perform relied on conventional monitoring strategies involving guide processes and siloed dashboards.  

Consequently, we lacked management to tailor how telemetry was processed, routed, and actioned – resulting in generic metrics and restricted perception into essential areas like consumer expertise and utility efficiency. For instance, whereas we might see that the community was operational, we had restricted visibility into essential areas like consumer expertise and utility efficiency. 

Recognizing this knowledge downside, we determined to convey the outsourced community operations perform in-house. This gave us full management to design and implement a modernized community observability technique, enabling us to raised leverage our wealth of telemetry and finally strengthen Cisco’s digital resilience. 

Nevertheless, this shift wasn’t nearly altering crew tasks. It additionally meant dropping our current community observability system and requiring our smaller crew to handle the huge quantity of telemetry knowledge. 

So as to add to the stress, as a result of contractual obligations, we got simply 40 days to make this transition and construct a totally new community observability system. 

Contained in the blueprint: Constructing a contemporary observability system

The duty at hand wasn’t simply to interchange and mirror the outsourced community operations and legacy observability system, however to construct one thing higher. We needed to construct a system that would deal with huge volumes of knowledge, ship deeper, actionable, and proactive insights, and allow a leaner crew to be extra productive. 

To realize this, we designed a community observability mannequin centered on three key areas:   

1. Accumulate: Gathers telemetry and metrics from 1000’s of gadgets, functions, and platforms – each inside owned and unowned, exterior environments
2. Monitor:  Makes use of instruments and algorithms to course of and analyze the collected knowledge, serving to to determine patterns, anomalies, and potential points throughout the community
3. Act: Initiates human or automated responses when recognized issues meet predefined rule standards, enabling well timed remediation.  

Determine 1: Cisco IT’s community observability mannequin

Whereas this method is run by a centralized networking crew, knowledge and rule creation are democratized – permitting engineers and repair house owners throughout IT to outline and customise their very own alert guidelines through GitOps. This ensures the system adapts to distinctive and evolving enterprise wants.

To function this community administration technique, we use a mix of Cisco options:

• Cisco’s community administration options, together with Catalyst Middle, SD-WAN Supervisor, Meraki Dashboard, and Nexus Dashboard, accumulate and monitor detailed telemetry, efficiency metrics, and safety standing knowledge on their respective property. This gives complete visibility and assurance, along with their different core capabilities for managing community gadgets.
• ThousandEyes gives real-time, end-to-end visibility into community and utility efficiency. It additionally extends this visibility into exterior, unowned environments comparable to public web and cloud companies. These granular insights feed into the observability system, giving us an entire view of consumer expertise and connectivity – regardless of the place workers are working.
• Splunk Cloud Platform acts as a unified operations dashboard – aggregating and visualizing telemetry knowledge from the above options that have been beforehand siloed. It allows real-time monitoring, enabling engineers to rapidly concentrate on essentially the most essential alerts.

Collectively, Splunk and ThousandEyes permit us in Cisco IT to proactively monitor, analyze, and act on hundreds of thousands of occasions every day.

Determine 2: Cisco IT’s observability system instruments and integrations

Automation is a essential part of our community observability technique. By feeding telemetry knowledge and incident outcomes into our Giant Language Fashions (LLMs) and automation programs, we will effectively course of and prioritize hundreds of thousands of every day alerts to cut back engineer workload and pace up response instances, enhancing end-user expertise.

The payoff: Enhanced resilience, effectivity, and past

From the start, we acknowledged that this initiative would contain important upfront work. Nevertheless, the outcomes have far exceeded our preliminary expectations.

Since deploying this new observability technique and system:

• 0 main incidents have occurred, down from 3-4 per quarter beforehand.
• 10x extra telemetry knowledge is being monitored, enabling broader and deeper insights into community well being, utility efficiency and consumer expertise at a subsequent stage of element.
• 4x higher visibility, with every day alert quantity rising from lots of of 1000’s to 4 million, leading to earlier detection and proactive decision of potential points earlier than they escalate.
• Automation now handles 99.998% of 4 million every day alerts generated, minimizing the necessity for guide intervention, and enabling sooner identification and backbone of points by way of real-time, automated triage and response workflows.

Maybe most significantly, this effort laid a basis that permits us to repeatedly scale our AI-driven automation and lengthen AIOps capabilities throughout the broader Cisco IT setting.

Classes realized: Methods that made the distinction

Modernizing our observability technique and system was a fast-paced journey, stuffed with precious classes. Listed here are some key takeaways and methods to assist different groups seeking to do the identical:

• Collaborative possession: Usher in subject material consultants from throughout the group, share data extensively, and construct a democratized tradition the place everybody has a stake in observability and operational success.
• Accumulate telemetry from all over the place: Complete monitoring begins with capturing knowledge throughout your total setting.
• Knowledge normalization and enrichment: Unifying numerous knowledge sources is essential for holistic visibility. Spend money on a high-quality, well-maintained CMDB to maintain your stock and knowledge correct. Use your CMDB to complement alerts with enterprise context, possession, and criticality.
• Rule experimentation: Encourage democratized groups to develop and refine alerting and automation guidelines to maintain alert volumes manageable and related.
• AI-driven automation: Feed enriched knowledge into automation and LLMs to streamline remediation and take steps towards self-healing operations.

We’re thrilled and happy with the work and outcomes that our groups have achieved, however our journey doesn’t finish right here. We are going to proceed to iterate, enhance, and advance our AI-driven automation capabilities.