How JP Morgan Built An AI Agent for Investment Research with LangGraph | LangChain Interrupt

Introduction to Ask David

Overview of the Conference and Team

• David expresses excitement about participating in the Interrupt Conference and thanks the organizers for the opportunity.

• He introduces himself and Jane from JP Morgan Private Bank, highlighting their role in managing extensive lists of investment products.

Challenges in Investment Research

• The research team faces challenges due to the manual and time-consuming process of answering client inquiries, which limits scalability.

• They aim to automate this process to provide precise answers quickly, introducing "Ask David," an AI-powered solution designed for investment questions.

The Vision Behind Ask David

Enhancing Efficiency with Automation

• David reassures that automation will not replace jobs but enhance efficiency, emphasizing the importance of meeting stakeholder expectations.

• The potential benefits of "Ask David" are discussed, focusing on its ability to transform how investment questions are answered.

Technical Insights into Ask David

Functionality and Data Management

• Jane explains that "Ask David" is a domain-specific QA agent utilizing decades of structured data for efficient information retrieval.

• She highlights the challenge of managing unstructured data (emails, notes, recordings), noting advancements that can leverage this information effectively.

Scaling Insights Generation

• Proprietary models and analytics previously required human expertise; now they can be scaled through automation for broader client access.

Real-Time Decision Making with Ask David

Client Interaction Scenarios

• A scenario is presented where financial advisors can instantly access insights during client meetings using "Ask David," enhancing real-time decision-making capabilities.

System Architecture Overview

• The architecture includes a supervisor agent orchestrating tasks among sub-agents while maintaining user experience customization through memory access.

• Different agents handle structured queries via SQL or API calls while unstructured data requires preprocessing before effective information retrieval.

Conclusion: Future Implications

Workflow Integration

• An end-to-end workflow graph illustrates how planning notes integrate with general QA flows, showcasing the comprehensive approach taken by "Ask David."

How Does the Multi-Agent System Handle User Inquiries?

Overview of Inquiry Handling

• The system categorizes user inquiries into general questions (e.g., investing in gold) and specific queries about funds, directing them to appropriate subgraphs with specialized agents.

• Each flow includes a personalization node for tailoring answers and a reflection check to ensure accuracy before finalizing responses.

Example of Fund Termination Inquiry

• An example inquiry regarding fund termination is processed, revealing that it was due to performance issues, with links provided for further details on fund performance.

• The inquiry is identified as related to a specific fund, prompting the supervisor agent to extract relevant information from the document search agent.

Personalization and Reflection Mechanisms

• Information retrieved is personalized based on user roles; for instance, a due diligence specialist may require detailed answers while an advisor might need general insights.

• A reflection node uses language models to validate generated answers. If an answer fails this check, the system attempts to generate a new response.

Summarization Process

• The entire process concludes with summarization, which involves summarizing conversations, updating memory systems, and providing final answers.

What Lessons Were Learned During Development?

Iterative Development Approach

• Emphasis on starting simple and refactoring often; initial development focused on basic functionality before evolving into more complex specialized agents.

• As comfort with specialized agents grew, integration into multi-agent flows occurred progressively.

Evaluation Driven Development

• Importance of evaluation-driven development highlighted; shorter development phases in GNI projects necessitate longer evaluation periods for accuracy assurance.

• Early consideration of metrics is crucial; accuracy remains paramount in financial applications. Continuous evaluation fosters confidence in improvements over time.

Tips for Effective Evaluation

• Independent evaluation of sub-agents is essential for identifying weak links that can enhance overall accuracy.

• Selecting appropriate metrics based on agent design is critical; e.g., conciseness for summarization tasks or trajectory evaluations for call processes.

How Can Human Expertise Enhance AI Performance?

Role of Human Review

• Ground truth isn't always necessary for effective evaluation; various metrics provide insights without requiring exhaustive data sets.

Improving Accuracy Through Engineering

• Initial application of general models yields low accuracy (<50%); quick improvements can be made through strategies like chunking and algorithm adjustments.

Achieving Higher Accuracy Levels

• Transitioning from 80% to 90% accuracy involves workflow chains and creating subgraphs tailored to specific question types. Reaching above 90% often requires human involvement in the loop.