How To Build, Deploy And Manage Agents At Scale

How to Build, Deploy, and Manage AI Agents at Scale

Introduction to AI Agents

• Joe Mora introduces the topic of building, deploying, and managing AI agents at scale, emphasizing the challenges of moving from concept to production.

• Joe Mora shares his background as CEO and founder of Crew AI and expresses excitement about collaborating with Snowflake for this discussion.

Understanding AI Agents

• The concept of AI agents is becoming more accessible; however, there remains a significant gap between understanding them and successfully implementing them in production.

• Joe explains that while LLMs (Large Language Models) can generate content like emails, their true potential lies in making decisions based on multiple inputs.

Agency in AI

• The term "agency" is introduced as a key characteristic of AI agents; they control application flow and make decisions autonomously rather than following simple if-then rules.

• Joe discusses how these agents can choose actions based on goals rather than just performing predefined tasks.

Benefits of Agentic Automation

• Companies are interested in agentic automation because it enables complex automations that were previously impossible due to limitations in traditional systems.

• An example is provided where an agent can navigate through various steps (A to B to C), demonstrating flexibility in achieving goals without strict sequential operations.

Cognition and Adaptability

• Key capabilities of these agents include cognition for decision-making and real-time adaptability when faced with unexpected data or situations.

• The concepts of self-healing (working around runtime issues) and self-improving (learning optimal pathways over time) are highlighted as essential features for effective agent performance.

Reliability and Governance

• Building reliable systems is crucial; it's not just about guardrails but also governance structures that ensure responsible use of AI agents.

• Different use cases may require varying levels of agency; some might benefit from collaborative groups (like Crew), while others need more deterministic approaches.

Spectrum of Agency

• Joe emphasizes the importance of balancing agency within systems—having a structured backbone while allowing pockets of agency depending on specific needs.

Understanding Agentic Systems in Conversational AI

Introduction to Agentic Systems

• The discussion begins with an example of a user interacting with a conversational agent, emphasizing the importance of understanding the initial message.

• A single language model (LM) call is sufficient for processing simple messages without needing an agent; complex queries may require deeper analysis.

Handling Complex Queries

• Complex inquiries, such as financial data requests from a CFO or technical questions about fine-tuning LLMs, can trigger extensive research processes involving multiple agents.

• These agents collaborate behind the scenes to query databases and compile reports before delivering comprehensive answers back to the user.

Differences in Software Development Approaches

• Traditional software development follows a clear lifecycle with defined inputs and outputs, contrasting sharply with the unpredictability of agent systems where data input and output are less deterministic.

• Testing methodologies like Test Driven Development (TDD) or Behavior Driven Development (BDD) face challenges in non-deterministic environments typical of agent systems.

Optimizing Agent Performance

• The optimization process for agents involves context engineering, which focuses on enhancing every aspect of API requests to maximize value extraction from each interaction.

• Context engineering includes optimizing prompts, system instructions, chat history, and tools used during interactions to improve overall performance.

Key Features of Crew AI

• Crew AI facilitates rapid deployment by allowing users to modify various parameters such as system prompts and instructions while providing essential frameworks for effective operation.

• Important components within Crew AI include memory management, guard rails for safety, training protocols, and advanced prompting techniques that enhance output quality.

Agents and Tasks Framework

• The core concepts within Crew AI revolve around 'agents'—which have roles and goals—and 'tasks,' which consist of descriptions and expected outcomes that guide their operations.

• Defining roles for agents along with clear task descriptions plays a crucial role in context engineering by shaping prompts and instructions effectively.

Competitive Landscape in AI Agents

• There is significant competition among platforms aiming to simplify agent creation; however, transitioning from prototyping to production reveals diminishing returns on value across different solutions.

Understanding the Challenges in Deploying AI Agents

Key Blockers in Deployment

• Customers often face blockers when trying to deploy AI agents, raising questions about privacy, dedicated IPs, and existing server resources.

• Many users quickly reach the prototype stage but struggle to transition into production, highlighting a common bottleneck in deployment processes.

The Role of DevOps and DataOps

• DevOps facilitates software deployment by materializing its value; similarly, DataOps ensures data is clean and formatted for use.

• There is a need for "agentic ops" to make AI agents adoptable, reliable, and deployable—an area where Core AI focuses its efforts.

Core AI's Offerings

• Core AI provides an open-source platform for learning and prototyping alongside an Agent Management (AM) platform that supports deployment features like monitoring and training.

• A new stack is emerging for building agents that integrates data management with agentic applications, emphasizing interoperability as a key concern for customers.

Customer Priorities in Agent Development

• Customers prioritize interoperability, governance (data protection), evaluation of performance, and guardrails to ensure safe development environments.

Building Agents: Pro Code vs. No Code

• Core AI offers both pro code and no code options for building agents; no code allows rapid development while pro code enables deeper customization.

• Users can start building with no code on the Crew platform before exporting their projects into traditional coding formats.

Utilizing Crew Studio for Agent Creation

Features of Crew Studio

• Users can create accounts on app.creati.com to access Crew Studio, which allows visualization of agents through drag-and-drop functionality or chat-based commands.

Example Use Case: Deep Research Automation

• An example use case involves creating an agent that conducts deep research on a specific topic and generates a comprehensive report based on various perspectives.

Agent Collaboration in Task Execution

• The system utilizes multiple agents working collaboratively to break down tasks based on user prompts while leveraging private tools available within user accounts.

Task Breakdown Process

• The process includes academic research analysis from different angles leading up to the synthesis of information into a final cohesive report.

How to Build and Execute Tasks with Agents

Overview of Agent Independence and Execution

• The agents can operate independently, allowing for asynchronous execution where multiple tasks run in parallel while the final task waits for their completion.

• Users have the flexibility to modify various parameters such as agents, goals, backstories, roles, tools, and settings to enhance functionality.

Task Customization and Execution Process

• Users can initiate a task by entering a specific topic; in this case, "fine-tuning small language models," followed by executing the command.

• During execution, users are encouraged to focus on logs rather than the execution screen itself. Logs provide detailed traces of LLM calls, messages, prompts, and raw data for tracking purposes.

Insights from Execution Logs

• The logs reveal extensive activities including Google searches, website scraping, content reading across three parallel tasks like downloading and analyzing papers.

Final Report Generation

• Upon completion of execution, users can access a comprehensive report that includes sections like introduction and market analysis. This report is customizable for further improvements.

• The platform allows automation enhancements by reviewing final outputs and updating them accordingly. Users can download code generated using CI open source.

Technical Aspects of Project Configuration

• Downloaded projects contain structured folders with configuration files (YAML format), detailing agent setups and tasks alongside actual code implementations.

• Users can delve into technical coding aspects within the project structure to create or modify agents effectively.

Future Directions in Development

• While time constraints limit deep dives into coding during this session, there is an expressed eagerness to explore more technical elements in future discussions about building agents.

• Emphasis on diverse methods available for constructing agents highlights the platform's versatility and potential for advanced development.