End-to-End Kubernetes Observability Project: AI-Powered Troubleshooting with Honeycomb
Implementing an End-to-End AI Observability Project
Introduction to the Project
- The video focuses on implementing an end-to-end AI observability project using a live microservice application.
- OpenTelemetry libraries will be injected into the application, with data routed to Honeycomb for monitoring and observability.
- The presenter expresses enthusiasm about Honeycomb's capabilities in simplifying cloud-native debugging, highlighting its importance for the audience.
Project Overview
- The project is deployed on an Elastic Kubernetes Service (EKS) cluster, but it can also work with Azure Kubernetes Service (AKS) or Google Kubernetes Engine (GKE).
- A URL shortener app consisting of multiple microservices will be used, including link service, redirect service, and database.
- The presenter assures viewers that all steps will be shared via a GitHub repository for easy access.
Understanding OpenTelemetry
- OpenTelemetry is described as a standard protocol for sending event traces and metrics to third-party monitoring systems like Honeycomb.
- It provides a universal standard that remains consistent across different vendors, unlike previous dedicated agents for each product.
Setting Up the Environment
- Viewers are advised to ensure they have prerequisites installed if they are new to AWS or EKS clusters.
- Instructions include logging into AWS using CLI commands and creating an EKS cluster either through command line or UI.
Configuring Honeycomb and Slack Integration
Initial Setup Steps
- After provisioning the EKS cluster, viewers should check their Nginx ingress controller setup before deploying the application.
Creating a Honeycomb Account
- Users are guided through signing up for a free account on Honeycomb which allows 20 million events per month under its free tier.
API Key Configuration
- An API key is generated in Honeycomb which needs to be set as an environment variable in the terminal for further configuration.
Integrating Slack Webhooks
Configuring Incoming Webhook in Slack
- Viewers learn how to add an incoming webhook integration in Slack to send real-time alerts from Honeycomb.
Finalizing Integration Settings
- The webhook URL from Slack must be added back into Honeycomb settings under team integrations to complete the setup process.
Setting Up Monitoring Triggers
Adding Alerts in Honeycomb MCP
- Instructions are provided on adding triggers within Honeycomb’s MCP (Monitoring Control Plane), including verifying connections with commands.
Testing Alert Functionality
- A script is introduced that creates load on the system; this helps test whether alerts function correctly when latency spikes occur.
Simulating Latency Spikes
Running Load Test Scripts
- A separate script is run that injectively simulates latency into the system allowing users to observe alert behavior during high-load scenarios.
Observing Trigger Responses
- After introducing chaos by injecting latency, viewers see how alerts fire based on predefined conditions set earlier.
Analyzing Results and Insights
Using Visual Tools for Analysis
- Viewers learn how to use visual tools within Honeycomb’s interface such as heat maps and bubble-up features to analyze performance issues effectively.
Conclusion of Demonstration
- The presenter encourages viewers to implement what they've learned while providing links for additional resources related to setting up their own observability projects.