Building AI Agents that actually work (Full Course)

Understanding AI Agents

Introduction to AI Confusion

• The speaker expresses confusion about AI terminology, including terms like MCPs and agent harnesses, indicating a need for simplification.

• Remy Gasill is introduced as a guest who will explain AI concepts in beginner-friendly terms.

Setting Up AI Agents

• The goal of the episode is to teach beginners how to set up their own virtual assistants for various roles within a company.

• Remy assures that by the end of the episode, listeners will understand how to build agents that can manage different departments using various platforms.

Transition from Chat Models to Agents

• The discussion highlights a shift in the AI landscape from simple chat models to more complex agents, emphasizing productivity gains (10-20 times more).

• The importance of understanding the difference between chat models (question-answer format) and agents (goal-result format) is stressed.

Defining Agents vs. Chat Models

• An agent is defined as moving from asking questions to achieving specific goals through task execution.

• The analogy of chat being like ping-pong while agents focus on achieving results over time is presented.

Understanding the Agent Loop

• When given a task, an agent follows an "observe, think, act" loop until it completes its assigned task.

• A practical example illustrates how an agent would research information before executing tasks such as building a website based on user prompts.

Components of an Agent

• An effective agent consists of four components:

• LLM (Large Language Model), which serves as its brain.

• A continuous loop that allows it to keep working until tasks are completed without needing constant user input.

Understanding AI Agent Harnesses

What is an Agent Harness?

• An agent harness is a platform that connects various tools and contexts to facilitate a loop process in AI applications.

• Popular AI agent platforms are essentially different types of agent harnesses, designed to streamline interactions and processes.

Demonstration of the Loop Process

• The speaker prepares to demonstrate the loop process using three demo folders in Claude Code, showcasing how these systems operate.

• The project feature in chat models like Claude and ChatGPT allows users to manage all chats and sources in one place, including custom instructions known as system prompts.

Local vs Cloud Projects

• Unlike cloud-based projects, the demonstration focuses on local projects stored on the user's computer, enhancing control over the development environment.

• The goal is to build a minimalist portfolio site for Greg Eisenberg while utilizing features from Claude Code and other apps like Codeex and Anti-gravity.

Security Considerations with AI Agents

Managing Security Risks

• Security should focus on scoping what access agents have; major companies behind these tools prioritize security due to their reputations at stake.

• Users can control permissions granted to agents, ensuring limited access reduces risks if an agent were compromised. This includes providing read-only access where necessary.

Comparison with Open Claw

• Open Claw serves as another example of an agent harness but lacks structured security measures, making it less reliable compared to more established platforms.

Learning How to Use Agent Harnesses

Driving Analogy for Understanding Tools

• Learning about agent harnesses is likened to learning how to drive; once you understand key concepts (like controls), you can navigate any tool effectively regardless of its specific features or interface.

Building a Website Using Agent Loops

• During the demonstration, Claude Code initiates research on Greg Eisenberg by connecting with Plexity as part of its operational loop process. This showcases real-time functionality within the platform's capabilities.

Observing Results from Different Platforms

• Each platform (Claude Code, Anti-gravity, Codeex) operates through similar loops but may display results differently; Claude Code excels at visualizing this process effectively compared to others like Anti-gravity and Codeex.

Overview of Website Creation and Automation

Demonstrating the Agent Loop

• The discussion begins with a demonstration of an agent's ability to create websites, comparing different tools like Anti-Gravity and Codeex.

• The process involves researching a subject (Greg Eisenberg), generating HTML code, and deploying it on a local server.

• The final step includes verifying the website's completion by taking screenshots for review.

Potential Applications of Automated Website Creation

• A conversation arises about the potential market for clean websites, suggesting that many businesses could benefit from automated solutions.

• An idea is proposed to send cold emails offering pre-made websites for $250, highlighting a business model based on automation.

Building an Executive Assistant Agent

Structuring Workspaces for Efficiency

• The speaker outlines their workspace organization, emphasizing folders for each client and department heads to streamline tasks.

• Focus shifts to creating an "executive assistant" folder aimed at automating daily tasks to save time.

Onboarding Agents Like Employees

• Building agents is likened to onboarding real employees; context about the business must be provided for effective task execution.

• Emphasis is placed on explaining business operations, clients, and tools necessary for the agent’s success.

Utilizing Co-work as an Agent Tool

Features of Co-work Interface

• Co-work is introduced as another tool similar to others but noted for its user-friendly interface that aids understanding of processes.

Initial Setup Challenges

• When attempting to generate a cold email, the agent lacks context due to no prior information being fed into it.

Understanding Memory in Agents vs. Chat Models

Differences in Memory Functionality

• A critical distinction is made between chat models (like ChatGPT), which have automatic memory storage versus agents that require manual setup of memory parameters.

Benefits of Controlled Memory in Agents

• Controlled memory allows users to dictate what information is retained by agents, preventing irrelevant context from affecting outputs.

Understanding Agents and Context Files

The Importance of Context in Agent Functionality

• Agents require a context file to function effectively; without it, they lack the necessary information to perform tasks like writing a cold email.

• An agents.md file serves as a system prompt, similar to custom instructions for GPTs, providing essential context about the user and their preferences.

• Each session loads this context before responding to queries, ensuring that agents have relevant background information at their disposal.

Creating and Utilizing an Agents File

• Different platforms (Claude Code, Gemini, Codeex/OpenClaw) use variations of the agents file but maintain the same core concept of contextual loading.

• Users can create an agents.mmd file by utilizing tools like Claude Chat or Co-work to extract personal business details through guided questions.

• Properly naming files allows agents to automatically load context when performing tasks, enhancing efficiency.

Transitioning from Prompt Engineering to Context Engineering

• The shift from prompt engineering (creating specific prompts for tasks) to context engineering emphasizes loading comprehensive business information into agents.

• With sufficient context loaded, users can issue simple prompts (e.g., "write me a cold email") while still receiving high-quality outputs tailored to their needs.

Managing Contextual Information Across Sessions

• To enhance memory retention across sessions, users may create folders containing various contextual files related to their business and preferences.

• By instructing the agent's main file (claude.md) to reference these additional context files, users can ensure that important details are consistently available during interactions.

Addressing Memory Limitations in Agents

• Current limitations mean that agents do not retain intricate details shared across sessions unless manually updated in the main configuration file.

• This poses challenges for maintaining consistency in communication styles or preferences if updates are not regularly made.

Understanding Memory Management in AI Agents

Importance of Context Files

• The speaker discusses the limitations of AI agents when they do not retain user preferences, emphasizing that without manual updates to context files, the agent fails to remember specific instructions.

• A practical example is provided where a snippet is added to an agents.mnd file to enhance memory retention and ensure preferences are saved.

Structuring Memory Files

• The speaker suggests adding important commands at the top of the memory file for better visibility and effectiveness in guiding the agent's behavior.

• Instructions include keeping memory files current by updating them with new information or corrections as they arise.

Building Effective Memory Systems

• The concept of memory.md is introduced as a repository for retaining user preferences over time, akin to how a good employee remembers details about their work.

• The discussion highlights that effective memory management leads to improved performance and reduced errors in AI tasks.

Compounding Knowledge Over Time

• As users interact with AI tools like Co-work, having structured memory files (clone MD and memory MD) becomes crucial for achieving desired outcomes.

• Some advanced agent systems have built-in memory features, but understanding manual setups remains essential for users.

Managing Memory File Size

• Users can test if their AI retains new information (e.g., favorite color), demonstrating how accumulated data builds a comprehensive memory file over time.

• Examples illustrate how different roles (like executive assistants or marketing heads) require tailored preferences stored within these memory systems.

Best Practices for Memory Management

• A recommendation is made regarding maintaining manageable sizes for claw.md files—ideally no more than 200 lines—to prevent inefficiencies from excessive rules overlapping.

• Users are encouraged to focus on saving substantial corrections rather than trivial details, allowing greater control over what gets recorded in memory.

Connecting Tools for Enhanced Productivity

• Once foundational elements are established, connecting various tools (like Gmail and Calendar via MCP - Multi-channel Protocol) becomes necessary for maximizing productivity gains.

Connecting Tools with MCP

The Need for a Universal Translator

• The integration of various tools requires understanding their unique languages; for instance, Claude speaks English while Notion speaks Spanish.

• Anthropic developed the Model Context Protocol (MCP) to act as a translator between different tools, allowing seamless communication without extensive custom development.

Simplifying Tool Connections

• MCP provides an easy and standardized method to connect multiple applications, making it user-friendly to integrate popular apps like Gmail and Google Calendar.

• Users can connect their preferred tools through a straightforward process in platforms like Claude Code and Manis, all utilizing the MCP framework.

Future of AI Operating Systems

• The speaker envisions a future where individuals will have personal AI operating systems (AIOS), managing various tasks across departments without needing to switch between different applications.

• By integrating tools such as Gmail, Google Drive, and Notion into one central platform (Claude Code), users can streamline their workflow significantly.

Automating Tasks with AI

High Value Tasks

• The ability to summarize emails or manage meeting notes is highlighted as an essential task that showcases the value of connected tools.

• Questions arise about the importance of these tasks; however, having all tools integrated reduces context-switching and enhances productivity.

Building Skills for Automation

• Automating repetitive processes by creating skills allows users to enhance efficiency over time; even small automations can lead to significant time savings.

• Regularly turning manual processes into automated skills can eventually lead to comprehensive life automation through agents.

Compounding Efficiency Gains

• Connecting various tools enables more complex workflows; for example, summarizing meetings leads directly into drafting proposals and setting up projects in Notion.

• Even simple tasks become exponentially faster when using integrated systems, leading to substantial productivity gains over time.

The Future of Work: AI Operating Systems

Integration of AI in Daily Tasks

• The speaker demonstrates an email draft generated from meeting notes, showcasing how AI can streamline communication by integrating insights directly into emails.

• Cody Schneider's vision is shared, suggesting that future employees will utilize personal AI operating systems to enhance productivity and automate manual tasks.

Understanding Skills as SOPs for AI

• Skills are defined as Standard Operating Procedures (SOPs) for AI, allowing users to automate repetitive tasks without needing to explain processes repeatedly.

• Without skills, creating proposals involves extensive back-and-forth adjustments; skills package these preferences into a reusable format.

Creating and Utilizing Skills

• Skills function like memory files but are specifically designed for job-related tasks. They encapsulate processes into markdown files for easy reference.

• Most agent harnesses now include skills as a feature, enabling users to create and manage their own skill sets effectively.

Practical Examples of Skill Creation

• An example skill for writing viral hooks is discussed, illustrating how it includes both the process and references necessary for execution.

• Two methods for creating skills are outlined: using existing content (like course transcripts) or manually documenting a process after completing a task.

Organizing Skills and References

• The speaker explains that when creating skills, additional reference folders may be automatically generated by the system based on the input provided.

Building Skills Live

Demonstrating Skill Creation

• The speaker proposes a live demonstration of building a skill, suggesting the creation of a "daily brief skill" that summarizes calendar events, inbox messages, and project statuses to prepare for the day.

• This skill can be scheduled to run automatically at 9:00 AM each morning, enhancing daily productivity by providing an organized overview.

Automating Email Drafting

• An example is given where the speaker wants to draft an email referring someone named Moltoshi to a friend, Sebastian, who runs an AI automation agency.

• By using skills like "Sebastian refer skill," the process of drafting referral emails can be automated, saving time on repetitive tasks.

Compounding Time Savings

• The speaker emphasizes that even small time savings from automating processes can compound significantly over time when applied consistently across various tasks.

• They encourage identifying all repetitive processes in daily life and setting up skills to streamline these tasks for greater efficiency.

Skill Organization and Structure

• A folder structure is introduced as part of organizing skills. The speaker mentions having a workspace dedicated to AI with sub-agents for specific tasks.

• An example of an "ads analyst skill" is provided, which automates the analysis of competitors' ads and landing pages—previously a manual task taking several hours.

Creating Comprehensive Reports

• The ads analyst skill scrapes data from ad libraries and generates detailed reports on visual and copy analysis along with suggestions for improvement.

• After initially running through the process manually with Claude (the assistant), the speaker packages this into a reusable skill for future use.

Chaining Skills Together

• The concept of chaining multiple skills together is discussed; for instance, combining meeting preparation with podcast research skills to create efficient workflows.

• These interconnected skills can automate notifications or reminders about upcoming meetings or podcasts while preparing relevant information seamlessly.

Automating Daily Tasks with Skills

Introduction to Automation

• The speaker discusses using co-work tools to automate daily tasks, such as running a morning briefing skill at 9:00 AM, creating an automated workflow that simplifies routine activities.

Personal Use Case for Automation

• The speaker shares their experience of searching for a unique car color and feature set, utilizing automation to scrape car marketplaces every three hours for new listings.

• They emphasize the time-saving aspect of this automation, noting that without it, they would spend significant time refreshing multiple car websites.

Skills Development for Team Efficiency

• The speaker mentions developing various skills tailored for different teams within their organization, including a weekly research skill that scrapes Twitter and Reddit for AI updates.

• They highlight the importance of combining these skills with tools like MCP (Multi-Channel Processing), allowing agents to manage business processes effectively.

Building Advanced Agents

Overview of OpenClaw Functionality

• The speaker explains how OpenClaw operates similarly to other systems but focuses on managing meta ads through an agents.mmd file.

• They describe adding personality and identity attributes to the agent, enhancing its functionality by connecting context files with necessary tools.

Skill Creation Process

• A detailed process is shared about building specific skills such as ad creative generation and copywriting within the OpenClaw framework.

• The integration of scheduled tasks (cron jobs) with these skills allows for efficient management of advertising processes.

Choosing the Right Tools

Recommendations on Tool Usage

• The speaker advises beginners on tool selection, suggesting that while OpenClaw is powerful, it may be challenging to learn compared to co-work or simpler platforms like Perplexity Computer or Manis.

Building Processes Before Migration

• It’s recommended to develop all processes in simpler environments before migrating them into more complex systems like OpenClaw. This ensures smoother transitions and better functionality.

Global vs Project-Level Skills

Understanding Skill Application Levels

• The discussion includes differentiating between global skills applicable across all projects versus project-specific skills tailored for individual needs.

Examples of Skill Implementation

• An example is given regarding a truncate skill designed to shorten text without losing meaning; this is implemented globally due to its frequent use across various contexts.

Conclusion on Agent Management

• The session concludes by reiterating the flexibility in structuring agent functionalities based on personal or organizational needs while emphasizing effective task completion through connected tools.

Building Powerful AI Agents for Business

Developing Context Files and Skills

• To create effective AI agents, start by determining the roles you want to build out. Utilize a chat model like Claw to assist in developing context files through an interview-style process.

• Engage with the chat model by prompting it to "Ask me questions to build this out," which helps in gathering necessary information for context files.

• Connect all required tools and begin building skills through daily use, leading to the development of powerful AI agents tailored for various aspects of your business.

• The goal is to have specialized AI agents that can support every department within your organization effectively.

Conclusion and Acknowledgments

• The speaker expresses gratitude towards Remy for sharing insights on building AI agents and mentions including links in the show notes for further exploration of Remy's work.