Full Course (Lessons 1-10) AI Agents for Beginners

What are AI Agents?

Introduction to AI Agents

• The course aims to explain what AI agents are, their best use cases, and the requirements for building them.

• It will cover fundamentals from concept to code, including a written lesson with translations and code samples.

Components of an AI Agent

• The first component is a large language model (LLM), which enables reasoning by identifying user tasks and creating plans.

• Memory is crucial; it can be short-term (contextual conversation data) or long-term (data that helps improve task completion).

• Tools accessed via APIs perform actions or provide data necessary for decision-making in task execution.

Real-world Example

• A simple analogy used is brushing teeth, where planning involves timing and tools like toothbrushes and toothpaste.

• Contextual memory includes preferences such as toothpaste flavor, illustrating how agents utilize both short-term and long-term memory.

Building AI Agents with Code

Introduction to Semantic Kernel

• The course introduces using Microsoft's Semantic Kernel framework for building AI agents with access to LLMs.

• A chapter dedicated to setting up Jupyter notebooks will help beginners run the provided code samples effectively.

Implementing Functions in Code

• The class destination plugin lists functions/tools available for the agent, allowing it to suggest random destinations from a predefined list.

• Users can experiment with different LLM models; GPT4 Mini is suggested as an example for this lesson.

User Interaction Simulation

Day Trip Planning Scenario

• A user interaction simulates requesting a day trip, showcasing how LLM interprets natural language requests beyond basic commands.

• If the user dislikes the initial suggestion (e.g., Sydney), the agent utilizes its memory context not to repeat suggestions when asked again.

Conclusion of Interaction

• The agent successfully suggests alternative destinations based on previous interactions while maintaining contextual awareness throughout the conversation.

Next Steps in Learning

Upcoming Lessons on Agentic Frameworks

• The next lesson will explore various agentic frameworks, their purposes, and criteria for selection.

Building AI Agents for Task Management

Understanding Agentic Frameworks

• The focus is on building AI agents that enhance control over task management, particularly in scenarios requiring multiple agents to collaborate on tasks.

• Contextual understanding is crucial; agents must be aware of their environment and the context in which they operate to perform effectively.

• For example, an agent booking a hotel room needs to know about available options. Agentic frameworks help manage this contextual information.

• These frameworks facilitate agent collaboration by defining how agents work together and communicate, enhancing overall task completion efficiency.

• Various agentic frameworks exist; this course focuses on three: Azure AI Agent Service, Semantic Kernel, and Autogen.

Overview of Specific Frameworks

Azure AI Agent Service

• Currently designed for single-agent use through both code and UI, integrating well with existing Azure services.

Semantic Kernel

• An enterprise-focused framework aimed at improving developer experience for teams creating production-level AI agents. Supports C, Java, and Python.

Autogen

• Developed by Microsoft Research, it emphasizes enabling researchers and developers to experiment with cutting-edge ideas in code.

Best Practices for Using Agentic Frameworks

• Start small by working with one agent using the Azure AI Agent Service before scaling up to multi-agent setups with frameworks like Semantic Kernel or Autogen based on your goals (production vs. exploration).

Practical Application in Code Editor

Sample Implementations

• The session transitions into practical coding examples using the three discussed frameworks: Semantic Kernel, Autogen, and Azure AI Agent Service.

Semantic Kernel Example

• Demonstrates defining plugins such as a destinations plugin that randomly selects a destination when planning a trip using chat completion capabilities.

Autogen Example

• Similar setup where a client is created for model interaction. Custom tools can be defined alongside system messages guiding the agent's responses.

Azure AI Agent Service Example

What Makes a Good AI Agent?

Introduction to AI Agents

• The discussion begins with generating Python code for creating a bar chart that visualizes the interactions of an agent running different tools, showcasing its run status and generated images.

• The course will explore various samples, allowing learners to experiment with different settings across services and frameworks.

Key Principles of AI Agents

• The third lesson focuses on defining what constitutes a good AI agent, emphasizing principles for their creation and real-world applications.

• Three key parts are identified: Space, Time, and Core. These elements are crucial in designing effective agents.

Space

• This refers to the environment where the agent operates. It should connect events, people, and knowledge while being easily discoverable by users.

• User interface (UI) design should include clear instructions about the agent's capabilities and limitations.

Time

• This principle highlights how agents can improve over time through user interaction.

• Implementing reflection design patterns allows users to see past prompts and interactions, enhancing long-term context awareness.

Core

• Embracing uncertainty is vital as it builds trust between users and agents.

• Providing visible controls (like play/pause options in videos) empowers users with feedback tools for better interaction management.

Applying Design Principles in Code

• Transitioning into coding, the speaker references using Semantic Kernel with GitHub models to create an interactive travel planning assistant.

• The code sample involves a destinations plugin that returns random locations for trip planning while clarifying user expectations regarding the agent's capabilities.

Enhancing User Interaction

• Clear communication about what the agent can do is essential; suggestions help guide user interactions effectively.

• Users receive specific instructions on planning vacations or day trips based on preferences like location features (e.g., mountains or beaches).

Example Interaction Flow

• An example illustrates how a user interacts with the travel assistant, starting from greetings to specifying trip details such as desired locations.

• The session concludes with an itinerary generation based on user input, demonstrating practical application of discussed principles in action.

Next Steps in Learning About AI Agents

Upcoming Topics

Tool Use Design Pattern in AI Agents

Overview of Tool Use Design Pattern

• The tool use design pattern enables LLMs (Large Language Models) to interact with external tools, enhancing their ability to complete tasks that require additional resources.

• Examples include using APIs for real-time data like flight status or currency exchange, showcasing the versatility of AI agents in various applications.

Use Cases for AI Agents

• An AI agent can generate database queries to access customer loyalty points and travel preferences without human intervention.

• Combining multiple tools allows agents to automate workflows, such as analyzing emails and forwarding relevant information to customer service representatives.

Considerations for Tool Calling

• Important factors include security measures ensuring agents have appropriate access levels and error handling due to potential service outages.

• Future lessons will delve deeper into trustworthy AI agents and their production environments.

Building a Tool Calling AI Agent

Code Implementation

• The demonstration uses a semantic kernel tool example featuring two plugins: one listing available destinations and another checking booking availability.

• Function behavior can be set to auto, allowing the agent to decide when it is appropriate to call specific functions based on user input.

User Interaction Scenarios

• Users can inquire about available destinations, prompting the agent to call the get destinations function for a list of options.

• When asking if a specific destination like Barcelona is available, the agent calls the get availability function instead.

Natural Language Processing Capabilities

• The power of LLMs lies in their ability to interpret natural language queries without needing explicit programming for geographical classifications (e.g., Europe).

• For instance, when asked about vacation spots not in Europe, the model autonomously identifies locations like Tokyo and New York based on its training data.

Conclusion on Tool Calling

Summary of Insights

Understanding Agentic RAG: Enhancing LLMs with Retrieval-Augmented Generation

Introduction to Agentic RAG

• The concept of Agentic RAG (Retrieval-Augmented Generation) is introduced, which enhances Large Language Models (LLMs) by allowing them to retrieve relevant information from a database based on user queries.

• Unlike traditional frameworks, Agentic RAG adds capabilities for analyzing queries and creating plans for smaller tasks using necessary data sources and tools.

Query Analysis and Information Retrieval

• The agent analyzes complex questions that may require multiple steps to answer, verifying if the retrieved information suffices before repeating the retrieval process if needed.

• A key feature is the agent's ability to maintain long-term memory of previous attempts, improving future responses without redundant steps.

Code Implementation Overview

• Transitioning to code implementation, the speaker uses a semantic kernel Azure search example to demonstrate how to set up an index using Azure AI search.

• The prompt plugin class allows augmentation of user prompts post-request, enhancing direction for the agent in retrieving information effectively.

Combining Tools for Enhanced Responses

• The augmented prompt directs the agent first to review retrieved context; if it doesn't answer the query, it can utilize available plug-in functions for additional data retrieval.

• An example plugin called weather info plugin provides average temperatures from various locations when direct data isn't available in Azure search.

User Interaction and Contextual Responses

• Documents related to travel insurance are added into the Azure search index. These documents contain essential information about destinations and their benefits.

• In a practical scenario, a user queries about Kontoso's travel insurance coverage. The system retrieves relevant documents while also providing contextual support through dropdown options.

Handling Complex Queries with Agentic RAG

• When asked about average temperatures at specific destinations like Maldes, if initial context does not suffice, function calls are made to gather necessary temperature data.

Building Effective AI Agents

Overview of Destinations and Functionality

• The discussion begins with examples of destinations like the Swiss Alps and African safaris, showcasing how function calls can identify cold destinations.

• Encouragement to experiment with code on personal setups, emphasizing the power of retrieval-augmented generation (RAG).

Security and Privacy in AI Development

• Introduction to key topics for Lesson 6: building effective AI agents while considering security and user privacy.

• Importance of system messages in controlling AI agent behavior when working with large language models (LLMs).

System Message Framework

• Explanation of a scalable approach to creating prompts using a system message framework, which helps define clear instructions for AI agents.

• A basic system message example is provided, illustrating how it can guide an LLM in generating better prompts.

Iterative Prompt Building Process

• Emphasis on the iterative nature of prompt creation; achieving the perfect prompt often requires multiple attempts.

• Discussion on security threats related to developing agents and introduction to "human in the loop" architecture for scenarios requiring human intervention.

Practical Application of System Messages

• Transitioning to practical coding examples that illustrate the previously discussed concepts about system messages.

• Description of variables used in a travel agent context, highlighting roles and responsibilities within an AI agent framework.

Enhancing Agent Capabilities

• The importance of defining roles clearly allows for more sophisticated prompts that enhance agent functionality.

• Example provided where responsibilities are outlined for an AI assistant role, focusing on customer satisfaction and minimizing errors.

Conclusion and Next Steps

• Encouragement to explore further details in written chapters regarding system message frameworks.

Planning Design Pattern in AI Agents

Introduction to the Planning Design Pattern

• The planning design pattern is introduced as a method for AI agents to take tasks and create structured plans for completion.

• It emphasizes breaking down complex tasks into manageable subtasks, enhancing clarity and organization.

Example of Task Breakdown

• A practical example is provided: generating a travel plan for a 3-day vacation, which includes subtasks like booking flights, hotels, transportation, and activities.

• The effectiveness of using multiple agents is highlighted; each subtask can be assigned to different agents or processes for efficient execution.

Structuring Outputs and Validation

• The output from the agent can be structured using features of large language models (LLMs), allowing other systems to process the information effectively.

• Validation mechanisms are discussed to ensure that all necessary information is gathered before proceeding with subsequent steps in the agentic system.

Code Implementation Overview

• An introduction to code implementation using the Pyantic package for data validation and structure is presented.

• The model structure includes defining subtasks with assigned agents and task details, ensuring organized responses from the agent.

Application of AI Agents in Travel Planning

• The main task involves creating a travel request that encompasses various subtasks handled by specific agents.

• The advantages of AI agents over traditional booking systems are noted, particularly their ability to understand natural language and provide tailored recommendations based on user needs.

Multi-Agent Design Pattern

Understanding Multi-Agent Collaboration

• The multi-agent design pattern involves multiple AI agents collaborating towards a common goal or task.

• Different design patterns apply depending on the goals of these agents; one example given is group chat functionality where messages are broadcasted among agents.

Practical Applications

AI Agent Collaboration and Workflow Patterns

Understanding AI Agent Handoffs

• A customer can input messages into a chat, triggering task requests routed to the appropriate agent.

• In scenarios requiring information from multiple AI agents, a "handoff pattern" is utilized where each agent completes specific steps before passing tasks along.

Collaborative Filtering in AI Agents

• Collaborative filtering involves multiple AI agents acting as specialists, each responding uniquely to their designated tasks.

• This design pattern is particularly effective for analysis that benefits from diverse inputs or perspectives on data.

Practical Application of Multiple Agents

• The code example demonstrates two agents: one generating content and another reviewing it, showcasing the "reviewer checker pattern."

• The concierge agent focuses on recommending non-touristy experiences while providing insights without creating new examples.

Interaction Dynamics Between Agents

• The front desk agent is instructed to provide one recommendation per response, allowing the concierge to review and suggest improvements without making new recommendations.

• A termination function is defined to end interactions when the concierge approves the front desk's suggestion based on authenticity criteria.

Conversation Flow and Example Output

• The conversation structure allows alternating turns between the front desk and concierge agents based on user input.

• For instance, if a user asks about Paris, the front desk might recommend popular sites like the Louvre but will be refined by the concierge for more authentic suggestions.

Metacognition in AI Agents

Defining Metacognition in AI Context

• Metacognition refers to "thinking about thinking," which applies to how AI agents can analyze their performance over time.

Enhancing AI Through Self-Evaluation

Understanding Metacognition in AI Systems

The Role of Metacognition in Decision-Making

• Metacognition enhances the transparency of agentic systems, allowing them to adapt to changing environments and improve data interpretation accuracy.

• It enables agents to reflect on their decision-making processes, incorporating customer preferences into future decisions for a more personalized experience.

• Effective metacognitive systems must be designed to gather, store, and retrieve feedback from past interactions to adapt as user needs evolve.

Practical Application: Flight Booking Example

• A code sample demonstrates how metacognition can be implemented in an AI agent tasked with booking flights based on user preferences.

• The system maintains a customer preferences object that tracks user choices over time, enhancing the interaction quality by recalling previous inputs.

• Before suggesting flight options, the agent reflects on stored preferences (e.g., preferred flight times), streamlining the booking process compared to traditional UI methods.

User Interaction Scenarios

• In a simulated conversation, when a user requests a flight to Barcelona and specifies "later flights," the agent retrieves relevant options efficiently.

• If the user changes their mind about timing (e.g., preferring earlier flights), the agent quickly adapts without complex calculations due to its understanding of context.

• The agent acknowledges contextual cues (like stress about timing), which informs its suggestions for scheduling activities around available flight times.

Leveraging Past Interactions

• When initiating a new request (e.g., "Book me a flight to Paris"), the agent utilizes historical data from previous interactions to suggest suitable options based on established preferences.

How to Effectively Deploy AI Agents in Production?

Overview of Chapter 10

• This chapter focuses on deploying AI agents effectively, addressing common mistakes and cost management strategies.

• The course aims to guide learners from concept to code, providing written lessons and code samples for practical understanding.

Evaluating AI Agents

• Proper evaluation of AI agents requires a comprehensive look at the entire system, establishing evaluation points at each step.

• Key aspects include assessing the agent's ability to identify user intent and select appropriate tools for task completion.

• Consideration of tool responses is crucial; errors or downtime can significantly impact performance.

• User feedback mechanisms (e.g., thumbs up/down) are essential for evaluating agent responses and improving performance over time.

• Continuous evaluation allows for adjustments in the agent's system, such as changing models or services based on feedback.

Handling Errors in Production

• The lesson includes a code sample demonstrating how to handle HTTP errors when launching AI agents into production.

• A scenario is presented where an external service may fail (e.g., flight times not available), necessitating backup functions.

• Implementing a backup function ensures that the agent can continue operating even if primary services are down.

Example Scenario: Booking Flights

• An example illustrates user input ("book me a flight to Barcelona") leading to an initial failed request due to an HTTP error.

• The agent then successfully retrieves flight times using a backup function, showcasing effective error handling in action.