The Most Brutal AI Benchmark Ever Made #benchmark #ai #perplexity
Perplexity's Open Source Benchmark: A Challenge for Research Agents
Overview of the Benchmark
- Perplexity has released a benchmark that exposes the limitations of research agents, with its best score still being inadequate.
- The benchmark consists of 500 real-world tasks derived from actual research jobs, including due diligence and market analysis.
Task Complexity
- Each task requires agents to find information on 50 companies, detailing every employee and providing evidence for each fact.
- The median task demands 50 entries backed by 245 records, totaling approximately 170,000 records across all tasks.
Evaluation Criteria
- The benchmark does not have a fixed answer key; it evaluates claims based on the accuracy of cited pages.
- Agents struggle primarily in two areas: discovery (finding all required information) and evidence (supporting claims with accurate citations).
Performance Insights
- Perplexity's search agent leads with a soft score of 0.36, while Anthropic follows at 0.25; other systems score below 0.13.
- Even leading systems incur costs exceeding $5 per task, indicating widespread challenges in achieving satisfactory performance.
Scoring Challenges
- In terms of hard scores requiring perfect accuracy, even top performers succeed only about one in seven times.
- Perplexity notes that while partial progress is common, complete coverage remains elusive.
Future Implications
- The grading pipeline also serves as a training data source for improving agent performance based on identified weaknesses.
- The open-source nature of this benchmark raises questions about its integrity—whether it's an honest assessment or merely a showcase by its creators.