AutoGen, AG2, Agents, Frameworks, Open-source, and Best Practices

Interview with AutoGen Founders: Dr. Xi Wang and Dr. Shing Yun Wu

Origin Story of AutoGen

• The interview begins with a discussion about the origins of AutoGen, highlighting its collaboration between Penn State University and Microsoft.

• Both founders previously worked on an open-source project called FL, which focuses on automated machine learning and hyperparameter tuning, initiated in 2019 during Dr. Wang's internship at Microsoft Research.

• The FL library gained significant traction, especially among enterprise users in the fintech industry, leading to millions of downloads due to its efficiency in handling tabular data.

Transition to Generative Models

• Recognizing the potential of generative models like GPT-3 prompted the founders to explore how automated techniques could optimize inference parameters for these models.

• They discovered that selecting optimal inference parameters significantly improved model performance across various tasks such as math problem-solving and text generation.

• The realization emerged that effective use of language models involves navigating a complex design space beyond simple inference processes.

Development of a Unified Framework

• To facilitate this navigation, they aimed to create a solid foundational framework akin to existing machine learning frameworks (e.g., Scikit-learn, PyTorch).

• At the time they began building AutoGen, there was no unified framework available for language models, motivating them to develop one themselves.

Aha Moment: Collaboration Between Agents

• The conversation shifts towards the concept of agents collaborating; Dr. Wu reflects on his "aha moment" while solving challenging math problems using iterative coding and debugging processes.

• He emphasizes the importance of involving tools and human input in problem-solving scenarios where AI may struggle alone.

User-Centric Approach

• The need for an executor agent capable of executing code alongside human tutors became apparent during their exploration into math tutoring applications.

Understanding the Evolution of AI Interaction

The Motivation Behind AI Agents

• The original motivation for developing AI agents was to mimic human interaction with systems like ChatGPT, which could generate code but not execute it. Users had to manually run the generated code and provide feedback.

• This led to a two-agent chat system: one agent (the assistant) is powered by a large language model, while the other (the user proxy) simulates human behavior in executing tasks.

Autonomous Interaction Between Agents

• The process of these agents interacting is fully autonomous, allowing them to debug and communicate effectively without constant human intervention.

• ChatGPT's success can be attributed to its chat interface, which allows natural feedback loops between humans and AI, enhancing learning and problem-solving capabilities.

Dialogue as a Tool for Knowledge Creation

• Drawing from personal experiences in quantum computing seminars, the speaker emphasizes that dialogue is crucial for generating new knowledge and understanding complex concepts.

• The success of ChatGPT exemplifies this theory by showcasing how advanced models can facilitate meaningful conversations beyond just human-AI interactions.

Expanding Communication Capabilities

• As technology evolves, there's potential for multiple AI models to converse with each other, expanding beyond simple one-on-one interactions with humans.

• Newer models like GPT-4 demonstrate enhanced capabilities such as reasoning and debugging, allowing them to take on various roles within collaborative workflows.

Automating Complex Processes

• By automating more steps in workflows—like code generation, testing, and review—the complexity of tasks that can be achieved increases significantly.

• One application discussed involves using AI agents for data generation processes where an AI replaces the user role to automate training data creation based on diverse parameters.

Generalizing Agent Interactions

• The conversation pattern established between two agents has broader implications for modeling complex processes across different domains.

• Initial studies focused on math problem-solving have shown promise in refining agent interactions further by accommodating more roles and communication styles among multiple agents.

Development Timeline of Autogen Framework

• The development of the autogen framework began around late 2022 alongside tuning parameters for models.

Development and Impact of Autogen

Initial Development Timeline

• The team focused on building the agent framework (FW) and gathering initial user feedback, which led to iterative improvements.

• The first research paper was written in August, detailing a timeline of about three to four months for developing the initial version of Autogen.

Open Source Approach

• The project has been public since its inception in April, with all code written as part of the open-source Flamel project.

• Early access to Autogen was provided to a smaller community before it gained mainstream attention; reactions were overwhelmingly positive.

Community Response and Growth

• The developers were surprised by the popularity of their work, which exceeded their expectations and motivated them to improve further.

• A goal was set for Autogen to achieve a specific number of stars on GitHub within a year; however, this target had to be adjusted after just days due to rapid growth.

Open Source in Academia vs. Enterprise

• There is a notable difference in how open source is adopted between academia (e.g., Microsoft Research) and enterprise sectors, particularly within AI communities.

• The speaker reflects on their previous lack of knowledge regarding open source before diving into artificial intelligence.

Journey into Open Source

• The speaker shares that their entry into open source was not straightforward but rather incidental during their years at Microsoft Research.

• Initially focused on writing research papers without engaging in open-source projects until they began working on the Flamel project.

Open Source Development and User Adoption

The Journey of Open Source Libraries

• Previous user requirements led to the development of open-source libraries, which opened new avenues for learning and improvement.

• Addressing user concerns enhanced library usability, driving follow-up research and increasing adoption from small to large-scale companies, including major players like Microsoft.

• Lessons learned from earlier projects influenced the decision to build Autogen as an open-source initiative, ensuring a different approach compared to previous efforts.

• Starting with fully open-source code allowed for rapid user feedback and accelerated enterprise adoption, leading to significant insights into enterprise use cases.

Personal Reflections on Open Source Contributions

• The speaker shares their journey through three stages: starting as a PhD student focused on sharing research for reproducibility and impact.

• Early experiences in academia emphasized making research accessible due to high standards set by advisors, fostering a culture of transparency.

• Transitioning from personal research to broader impacts through the FLO project resulted in millions of downloads and real-world applications, providing personal fulfillment as a researcher.

The Impact of Autogen

• With wider adoption during the Autogen stage, there was increased visibility and appreciation for user contributions, prompting deeper thoughts about benefits for contributors and users alike.

• Open source fosters a healthy ecosystem where contributors can share their work while benefiting from others' contributions; this collaborative spirit enhances overall quality.

Benefits of Open Source for Enterprises

• Transparency is a key advantage of open-source software; all code is available for scrutiny, building trust among users—especially enterprises seeking reliable solutions.

• A thriving community around open source facilitates easier onboarding for new employees who can leverage widely-used software rather than internal-only tools.

The Unique Nature of AI in Open Source

• There’s something special about the AI industry that makes open source feel like the default mode; many contributors come from academic backgrounds that value openness.

• Users appreciate being able to access code freely, allowing them to tinker with it personally or enhance it before sharing improvements back with the community.

Practical Insights on Using Agents

Best Practices for Multi-Agent Systems

Key Considerations in Agent Models

• The discussion emphasizes the importance of selecting appropriate models and configurations for agents, including considerations on verbosity in prompts and system messages.

Starting with Prototyping

• A recommendation is made to begin with prototyping, focusing on a single instance of the task. This approach allows for experimentation with various methods.

Initial Setup Guidelines

• It is suggested to start with a simple setup involving two agents: one assistant agent to handle instructions and another user agent equipped with necessary capabilities like code execution.

Handling Complexity in Tasks

• For complex tasks, it may be beneficial to break them down into smaller components, utilizing specialized agents that can focus on simpler tasks for improved quality.

Communication Topology Among Agents

• As the number of agents increases beyond two, communication patterns become more complex. Various conversation patterns are introduced to facilitate effective collaboration among multiple agents.

Optimizing Agent Configurations

Freedom in Model Selection

• With multiple agents, there is greater flexibility in choosing different models and tools tailored to specific roles or tasks, allowing optimization across various metrics such as quality and cost.

New Features in AG2

• Recent updates introduce features like Captain Agent and Swarm Agent. These represent new paradigms for automating agent creation tailored to specific tasks.

Functionality of Captain Agent

• The Captain Agent automates the process by analyzing tasks, decomposing them, and creating teams of agents dynamically based on user input without requiring manual intervention.

Limitations of Automation

• While automation through the Captain Agent can streamline processes, it may struggle with overly complex tasks or when existing tools are insufficient. Users may need to supplement generated solutions manually.

Integration of Swarm Patterns

Best Practices for AI and Human Collaboration

Transitioning Between Agents

• Discusses the importance of defining transitions or instructions for handing off tasks to different agents, emphasizing structured information sharing.

• Suggests combining various methods in development processes as best practices, with an acknowledgment that more guidelines may emerge over time.

Mimicking Human Organization

• Proposes treating language models like humans with medium intelligence, suggesting that human organizational strategies can inform problem-solving approaches.

• Highlights the value of mimicking HR structures within companies as a starting point for developing AI systems.

Human Involvement in AI Processes

• Raises questions about when humans should be involved in AI processes, particularly regarding complex enterprise-level projects.

• Notes that the level of human involvement may vary based on technological advancements and the capabilities of AI agents.

Minimal Human Engagement Requirements

• Outlines minimal human engagement needed when AI reaches high capability levels, focusing on intent origination from humans.

• Emphasizes the necessity for clarity in initial requests to avoid ambiguity during task execution by AI.

Feedback and Decision-Making

• Stresses the iterative nature of refining intents between humans and AI until clarity is achieved.

• Identifies key moments where human feedback is essential, especially if results deviate from expectations or require verification.

Teaching and Learning Dynamics

• Discusses scenarios where humans must override AI decisions when necessary, highlighting a dynamic relationship between human oversight and machine autonomy.

Understanding Human-AI Interaction

The Motivation Behind Learning from AI

• Individuals are driven by curiosity and the desire to improve their skills, prompting them to engage with AI agents.

• There is a natural incentive for humans to learn useful skills from AI, reflecting a symbiotic relationship between human intelligence and artificial intelligence.

Human in the Loop: Identifying Gaps

• A key approach to integrating human input involves identifying gaps between medium-level AI intelligence and actual human capabilities.

• Highly intelligent humans can outperform AI models on complex tasks, necessitating human involvement for verification and confirmation.

• Ethical considerations, particularly in high-stakes applications, highlight the need for human oversight to ensure solutions are accurate and secure.

Transitioning from Autogen to AG2

Reasons for Forking the Code

• The decision to fork the original autogen code stemmed from collaborative efforts among contributors who sought more flexibility in project direction.

• As the project grew under Microsoft’s umbrella, complexities arose that slowed progress due to corporate constraints affecting decision-making.

Enhancing Project Agility

• The transition to AG2 allows for faster development cycles and greater community engagement without being hindered by corporate bureaucracy.

• Establishing AG2 as a neutral platform encourages contributions from various organizations, enhancing collaboration across different sectors.

Future Plans for AG2

Community Engagement and Development Goals

Use Case Driven Development and Future of Autonomous Agents

Current Development Principles

• The focus is on use case driven development, aiming to support various needs derived from specific use cases.

• Recent releases include the Captain agent, swarm agent, and Knowledge Graph, indicating ongoing updates in technology.

• Plans are underway to expand autogen capabilities to real-time applications, enhancing support for real-time voice agents.

Future of Proactive Agents

• Discussion on the future potential of agents becoming more autonomous and proactive in completing tasks without explicit user commands.

• Emphasis on the importance of a well-designed human interaction interface (UI/UX) for effective task completion by agents.

Challenges in Autonomy

• Concerns about authorization mechanisms; users want control over important communications sent by agents.

• The need for a "human-in-the-loop" mechanism to ensure that critical tasks are reviewed before execution.

Technological Viability and Responsibility

• Technology is close to enabling fully autonomous systems; however, quality assurance remains a concern when agents act independently.

• Users currently provide initial requests; automating this step could enhance efficiency but raises questions about accountability.

Outlook on Agent Capabilities

• In 5 to 10 years, it is anticipated that agents will handle most tasks with minimal human supervision while allowing users the option to intervene as needed.

Future of Agent Technology

The Role of Agents in Personal Task Management

• The speaker expresses a strong belief in the potential of agents, emphasizing their desire for agents to connect with personal accounts to perform tasks efficiently.

• There is a specific mention of the need for agents not just to send emails but to draft them for review, highlighting the importance of collaboration between human and AI.

• The speaker conveys excitement about future developments in agent technology that could significantly save time and enhance productivity.

• Gratitude is expressed towards collaborators for their contributions to open-source projects and the development of frameworks that support agent technology.