Introduction to AI Evaluation

Overview of the Session

• Kannaw Mishra introduces himself as a product manager focused on integrating AI into consumer and enterprise products.

• Emphasizes the importance of delivering reliable product features rather than treating them as experimental projects, highlighting the distinction between product management and machine learning research.

• Promises attendees a practical checklist and framework for evaluating AI systems that can be applied immediately without requiring advanced degrees.

Agenda for Discussion

• Outlines the session's agenda, which includes high-level discussions on product evaluation, real-life case studies, failure modes of large language models (LLMs), and evaluation techniques.

• Plans to cover specific metrics necessary for model evaluation and provide a comprehensive framework for building AI systems.

Personal Journey in AI Evaluation

Background Experience

• Shares his initial frustration as a business consultant regarding decision-making effectiveness despite having outputs but lacking feedback loops.

• Transitioned into product management with a focus on understanding user needs, problem identification, and measurement of solutions' success.

Achievements in Product Management

• Highlights significant contributions at Amazon leading to substantial cost savings through optimized systems.

• Discusses recent projects aimed at making AI trustworthy by ensuring human actions align with model predictions.

Lessons Learned from Mistakes

Importance of Problem Definition

• Reflects on early mistakes made by jumping to solutions without properly defining problems, emphasizing that framing is crucial in AI contexts.

Strategic Nature of Evaluation

• Stresses that effective AI evaluation is not merely procedural but strategic; it serves as a foundation for quality assurance in products.

The Reality of AI Evaluation

Challenges in Current Evaluations

• Discusses findings from McKenzie’s 2024 report indicating inflated scores due to benchmark contamination where test data leaks into training datasets.

Real-world Implications

• Cites examples like Air Canada's chatbot incident illustrating severe consequences stemming from poor evaluations leading to legal liabilities.

Understanding Differences in Software Models vs. ML Models

Deterministic vs. Predictive Systems

• Compares traditional software models (deterministic like vending machines with predictable outcomes) against machine learning models (predictive nature with continuous error rates).

Debugging Challenges

Large Language Models: Understanding Their Challenges and Risks

Introduction to Large Language Models

• Large language models (LLMs) are fundamentally different from traditional models; they generate diverse outputs rather than fixed labels.

• Analyzing LLM behavior requires inspecting prompts, embeddings, and various parameters like temperature, as their interpretability is low.

Risks of Large Language Models

• Without constant monitoring, LLMs can quickly drift into unsafe territories, posing significant risks.

• The success metrics for LLM applications differ from classic ML models; conversational capabilities change how we evaluate performance.

Common Failure Modes in LLMs

• Six major failure modes threaten user trust and brand integrity:

• Hallucination

• Bias and fairness issues

• Robustness concerns

• Toxicity in responses

• Prompt injection vulnerabilities

• Context loss during interactions

Hallucination

• Hallucinations occur when an LLM fabricates facts confidently; a notable case involved lawyers relying on a GPT-generated summary that led to legal repercussions.

Bias and Fairness

• Historical biases can be perpetuated by AI systems, exemplified by Amazon's hiring algorithm penalizing candidates based on biased data.

Robustness

• AI models may struggle with common user errors; research indicates minor typos can significantly reduce accuracy.

Toxicity

• Instances like Microsoft's chatbot learning inappropriate language highlight the potential for PR crises and safety threats.

Prompt Injection

• Malicious users may exploit vulnerabilities in LLM prompts to extract sensitive information or generate harmful content.

Context Loss

• Forgetting critical user information mid-conversation poses unacceptable safety risks with serious legal implications.

Evaluation Strategies for Mitigating Risks

• Two evaluation approaches exist: offline (like testing ingredients before cooking) and online (serving the dish to customers).

Offline Evaluation

• This method allows quick adjustments without real-world consequences, enabling rapid refinement of model outputs.

Online Evaluation

• Real-time assessment involves tracking user engagement metrics post-release to validate product effectiveness against business KPIs.

Case Study: Discover Weekly Feature Analysis

• In evaluating features like Spotify's Discover Weekly:

• Offline tests focus on prediction power based on historical data.

Evaluation Strategies for Product Success

Offline Evaluations

• Both offline and online evaluations are essential for product success, aiming to identify issues before they reach users.

• The toolkit includes "golden data sets," which consist of expert-written answer keys (20-50 per use case) to measure model outputs against standards like factuality and conciseness.

• For specific applications, such as financial assistants, accuracy in calculations is crucial; marketing bots are evaluated on the effectiveness of ad copy.

• Adversarial suits are employed to test models by exposing vulnerabilities through challenging prompts designed to elicit toxic responses or break the model's functionality.

• Automated scoring moves beyond traditional metrics, utilizing benchmarks tailored for large language models (LLMs), focusing on fact accuracy and robustness under pressure.

Online Evaluations

• While automated scores cover 80% of evaluation needs, human feedback is necessary for capturing nuances like brand tone and cultural context that machines may overlook.

• A reinforcement learning from human feedback (RLHF) loop involves domain experts scoring answers based on helpfulness, honesty, and humbleness using a lightweight reward model.

• Crowd workers provide cost-effective feedback but require regular calibration to prevent biases from affecting results; initial costs can be around $100/hour versus $10/hour for crowd workers.

• AI can serve as a judge after proper calibration of crowd workers, significantly reducing costs while maintaining quality in evaluations.

• High-stakes testing should prioritize human involvement while allowing AI to handle routine assessments efficiently.

Monitoring in Production

• Continuous monitoring is vital; dashboards track key metrics like hallucinations per thousand calls—anything over 1% necessitates immediate investigation.

• Policy violations must remain below 0.05%, with tools scanning outputs for safety compliance and misinformation prevention in sensitive areas like healthcare.

• Prompt injection success rates should also stay below 0.05%, requiring constant testing with canary prompts to detect new attack vectors effectively.

• Features should drive at least a 5% increase in conversion or engagement; if not achieving this metric, reconsideration of deployment is warranted.

Best Practices in AI Evaluation and Management

Importance of Monitoring Metrics

• If any critical metric breaches a threshold for 5 consecutive minutes, revert to a previous model version. Quick fixes should be communicated through Slack channels.

Case Study: Netflix's Recommendation Engine

• Netflix employs a private evaluation set for its recommendation engine, ensuring secrecy and rigor. This set is rotated monthly to prevent benchmark contamination.

• The result includes a significant reduction in browse-to-play time by 30 seconds, enhancing user engagement. Unbiased evaluations linked to core business APIs drive tangible value.

Case Study: OpenAI's Safety Measures

• OpenAI proactively tested their models with 2000 adversarial inputs, reducing toxicity by 95% using constitutional AI principles. A white paper is available for further reading.

• They released a public system card detailing capabilities and limitations, fostering transparency about risks associated with the product.

Case Study: Alexa's Multi-modal Testing Challenges

• Evaluating multi-modal AI like Alexa involves tracking various metrics such as speech recognition accuracy and intent F1 scores to ensure effective understanding of user input.

Product Manager’s Evaluation Framework

Defining Success Before Development

• Establish success criteria across four categories before coding begins: business success (impact on revenue/cost), user success (NPS score improvement), technical success (accuracy/latency targets), and risk mitigation (hallucination/toxicity thresholds).

Following an Evaluation Pipeline

• Implement an evaluation pipeline that includes human feedback loops, shadow deployments at low traffic percentages, and A/B testing prior to full launches.

Continuous Improvement Cycle

• Treat evaluation as an ongoing process with weekly KPI dashboards, monthly red team drills, and quarterly benchmark refreshes to adapt to model drift and changing user behavior.

Enterprise-Level Evaluation Considerations

Benchmark Contamination Awareness

• Regularly rotate benchmarks monthly to avoid contamination; monitor perplexity scores as indicators of performance quality.

Multi-modal Changes Complexity

• For text-to-image evaluations, assess both image quality and relevance alongside human aesthetic grades. Voice bots require composite error rates for transcription accuracy.

Common Pitfalls in AI Development

Avoiding the Demo Trap

• Ensure thorough offline testing that includes adversarial scenarios before launching or demonstrating models; real-world inputs can often lead to unexpected failures.

Preventing Metric Tunnel Vision

AI Evaluation Strategies and Best Practices

Importance of Balancing Metrics

• Optimizing for metrics like Rouge can lead to robotic user experiences; it's crucial to find a balance between different evaluation metrics.

• Always consider the trade-offs from an ROI perspective when evaluating AI performance, especially in diverse applications (e.g., chatbots vs. medical AIs).

Risk Management in AI Evaluation

• Understanding your threshold of risk is essential; high-stakes scenarios require human feedback checks, while lower-risk situations may allow for AI judgment.

• Neglecting human handoff can result in users feeling abandoned if AI fails without a fallback option.

Graceful Degradation and User Experience

• Plan for graceful degradation by providing clear alternatives or human agents when AI responses are uncertain.

• Avoid generating false information ("hallucination") when the AI is unsure about its response.

Maintaining Benchmark Relevance

• Benchmarks can become outdated, leading to inflated results that do not reflect real-world quality; regularly rotate static benchmarks.

• Keep your golden data set updated and ensure continuous learning with new datasets to maintain trustworthiness and impact.

Continuous Improvement and Human Involvement

• Evaluation should be viewed as strategic risk management; humans must coexist with AI during evaluations to capture nuances missed by automated systems.

• Models will drift over time, necessitating iterative improvements in evaluation processes rather than treating them as one-time tasks.

Final Thoughts on Shaping the Future of AI