The MIT Paper Everyone Building Agents Should Read Right Now

Name: The MIT Paper Everyone Building Agents Should Read Right Now
Uploaded: 2026-01-14T23:04:05.000Z
Duration: 2 h 10 min 46 s

Scalable Context Windows for AI Agents

Introduction to Recursive Language Models

The paper discusses a scalable approach to extending the effective context window of AI agents, potentially increasing it by one or two orders of magnitude.

This advancement could significantly enhance due diligence across numerous documents and allow local models to handle extensive code bases with millions of lines.

The recursive language models (RLMs) proposed in the paper enable large language models (LLMs) to process arbitrarily long prompts effectively.

Key Features of Recursive Language Models

RLMs treat long prompts as part of an external environment, allowing LLMs to examine, decompose, and recursively call themselves over prompt snippets.

RLMs can manage inputs up to two orders of magnitude beyond standard model context windows while maintaining comparable or lower costs per query.

For example, if a model has a million-token context, RLM could extend this capability up to 100 million tokens without significant performance degradation.

Performance Insights

The cost-effectiveness of RLM is notable; it processes only necessary information rather than overwhelming the context window with excessive data.

Current models like Claude Code can handle large code bases well; however, RLM's approach may offer additional advantages in processing efficiency and effectiveness.

Comparative Analysis with Existing Models

A chart comparing GPT5's raw performance against its performance using the RLM scaffold shows that traditional methods suffer from significant performance drops as context length increases.

The introduction of RLM techniques allows for better maintenance of performance even at extended token lengths (up to 1 million tokens), which is crucial for complex tasks.

Implications for Practical Applications

This research holds substantial implications for fields requiring heavy contextual understanding, such as legal document review or policy analysis where accuracy is paramount.

Users have already begun leveraging existing tools like Claude Code for similar tasks; however, the advancements presented in this paper suggest further enhancements are possible.

Limitations and Challenges

Despite progress in reasoning capabilities within modern language models, they still face limitations regarding context lengths and exhibit issues like "context rot," where performance deteriorates over lengthy interactions.

Understanding Inference Time Compute and Recursive Language Models

Context Degradation in Frontier Models

The quality of frontier models like GPT-5 degrades quickly as context length increases, leading to "context rot." This indicates that performance improves with Recursive Language Models (RLMs).

Inference Time Compute Discussion

There is confusion around the term "inference time compute," which differs from previously discussed "test time compute." Clarification on terminology is sought.

Data Processing Insights

RLMs draw inspiration from out-of-core algorithms, allowing data processing systems with limited fast memory to handle larger datasets by managing data fetching effectively.

Long Context Management Techniques

Common methods for addressing long context problems include context condensation or compaction, where prompts are summarized once they exceed a certain length. However, this method often lacks expressiveness for tasks requiring detailed access to prompt parts.

Challenges of Summarization

Compaction techniques are lossy; deciding what information to retain or discard can be task-dependent. Intelligent compaction attempts have been made but remain fragile.

Introduction of Recursive Language Models (RLM)

RLMs offer a new paradigm for scaling input and output lengths in LLMs by treating long prompts as part of the environment rather than directly feeding them into the neural network.

Programmatic Interaction with Long Context Data

Instead of extending context windows, RLM allows interaction with much larger contexts programmatically, potentially reducing the need for extremely large token limits.

Functionality of RLM

An RLM initializes a programming environment where it can interact with prompts symbolically. It enables the LLM to write code that examines and decomposes prompts iteratively.

Recursive Task Decomposition

RLM encourages LLM-generated code to create subtasks recursively, allowing focused attention on specific sections within extensive documents through systematic narrowing down.

Addressing Limitations in Prior Approaches

The design of RLM tackles foundational limitations seen in previous models that could not scale inputs beyond their context window. Questions arise about how this approach compares to existing recursive decomposition methods.

Evaluation of RLM Performance

The effectiveness of RLM is evaluated using both a closed model (GPT-5) and an open model (Quen Coda), highlighting advancements in handling complex tasks within large language models.

The Evolution of Language Models and Context Handling

Advancements in Model Parameters

The landscape of language models has shifted dramatically, with the capability to deploy 1 trillion parameter models on consumer hardware, a significant increase from previous standards.

Potential for Open-Source Solutions

There is potential for real open-source agentic products that can operate entirely on local machines without relying on cloud services.

Diverse Applications and Tasks

The discussion highlights various complex tasks such as deep research, information aggregation, software engineering, legal analysis, and long document extraction that these advanced models can address effectively.

Comparison of Approaches

A comparison is made between ILM (Incremental Learning Models), direct LLM calls, and context compaction retrieval tools to evaluate their effectiveness in handling complex tasks.

Understanding the Prompt Object

The concept of a "prompt object" is introduced; it serves as an environment where the language model executes Python code to achieve specific goals through iterative processing.

Challenges with Retrieval-Augmented Generation (RAG)

Limitations of RAG Approach

RAG has fallen out of favor due to its brittleness; it was initially designed to overcome context window limitations but struggles with semantic similarity matching.

Historical Context Window Issues

Early large language models had limited context windows (e.g., 8k tokens), which restricted their ability to provide accurate responses based solely on training data.

Ineffectiveness in Complex Queries

RAG's reliance on semantic similarity often leads to inaccurate results when faced with complex prompts or lengthy documents due to its rudimentary retrieval mechanisms.

Scaling Challenges in Long Context Tasks

Effective Context Window Insights

Recent findings suggest that the effective context window for LLMs may be shorter than their maximum token capacity, particularly for more complex tasks.

Task-Specific Performance Degradation

It’s hypothesized that more complex problems will show performance degradation at shorter lengths compared to simpler ones, emphasizing the need for tailored approaches based on task complexity.

This structured summary captures key insights from the transcript while providing timestamps for easy reference.

Understanding Task Complexity in Language Models

The Challenge of Complex Documentation

Traditional approaches to handling policies and contracts are failing due to their dense and complex nature, which requires heavy logical reasoning.

Tasks are characterized by how their complexity scales with prompt length; for instance, "needle in a haystack" problems maintain constant complexity as prompt length increases.

Advanced models can solve these "needle in a haystack" tasks effectively even with large token settings, but struggle with tasks that require detailed context from shorter prompts.

Logical Coherence and Its Impact on Performance

The NIA task illustrates that as context grows, the complexity remains stable; traditional semantic retrieval methods may suffice for simpler tasks.

However, models face challenges when dealing with logically coherent documents where different parts depend on each other, leading to performance breakdown.

Logical coherence means that understanding one part of the document often requires knowledge of previous sections, complicating processing.

Independence vs. Dependency in Q&A Documents

In independent Q&A documents (e.g., insurance policies), language models perform well since questions do not rely on each other.

Conversely, if questions are interconnected, model performance declines significantly due to the need for contextual understanding across multiple entries.

Scaling Problem Complexity with Prompt Length

The evaluation design focuses on verifying both prompt lengths and scaling patterns for problem complexity based on information density required for task completion.

For independent Q&A scenarios, only one question-answer pair might be needed; however, complex legal documents may require extensive reading to grasp interrelated clauses.

Benchmarking Model Performance Across Tasks

A set of 50 single "needle in a haystack" tasks demonstrates consistent processing costs regardless of input size due to the fixed number of needles being searched.

Another benchmark mentioned is Browse Comp Plus—a multihop question answering test requiring reasoning over multiple documents—highlighting the need for advanced capabilities in deep research contexts.

Multihop Question Answering and Synthesis in Document Retrieval

Evaluation Methodology

The evaluation set consists of 150 randomly sampled tasks, with 1,000 documents guaranteed to contain gold and evidence documents for each task.

Each task requires synthesizing information from multiple documents, making it more complex than single-document tasks like SNI.

This approach models scenarios similar to software engineers reading documentation, where understanding interactions across different references is crucial.

The study focuses on multihop retrieval across a curated corpus of 100k documents but evaluates performance based on a subset of 1,000 documents.

A long reasoning benchmark is introduced that necessitates examining and transforming input chunks semantically before aggregating them into a final answer.

Scoring Metrics and Performance

The scoring system includes numeric answers scored as y = 0.7, while other answers are evaluated based on exact matches; the specifics of this metric remain unclear.

Focus is placed on the trek course split involving 50 tasks that require nearly all entries in the dataset, leading to linear processing costs relative to input length.

The emphasis is on synthesis performance—how well can one identify and aggregate relevant information from various sources?

Aggregation Techniques

New queries were added to the existing tree trek core split to specifically require aggregating pairs of chunks for constructing final answers.

Appendix one provides explicit queries used in this benchmark; F1 scores are reported over these answers as a method for evaluating recall versus precision.

Understanding Precision and Recall

Precision measures how often the model's predictions are correct (e.g., predicting sick patients), while recall assesses how many actual cases were identified correctly within a population.

High precision indicates fewer false positives (correctly identifying sick individuals), whereas high recall reflects capturing most true cases even if some incorrect predictions occur.

Understanding F1 Score and Its Implications

Precision, Recall, and F1 Score

The F1 score is a crucial metric that balances precision and recall; a higher F1 indicates better performance in both areas.

High precision with low recall is ineffective, as is high recall with low precision; the goal is to achieve both simultaneously.

Reporting F1 scores becomes essential when tasks require processing all pairs of dataset entries, leading to increased computational costs.

Computational Complexity

As input length increases, compute time scales quadratically, significantly impacting processing efficiency.

Longbench v2 code QA presents challenges for modern models by requiring reasoning over fixed files in a codebase to derive correct answers.

Comparative Analysis of RLMs and Other Methods

Model Comparisons

The study compares Recursive Language Models (RLMs) against task-agnostic methods using contemporary large language models like GPT5 and Quen Coder 480B.

Cost evaluations are based on specific sampling parameters chosen for commercial versus open frontier models.

Implementation Details

The implementation details include an RLM that loads context as a string without sub-calls, affecting flexibility and effectiveness.

A trade-off exists between the capabilities of recursive models (RLM with GT5 mini for specialization) versus the root model (GBT5).

Exploring Recursive Approaches

Ablation Studies

An ablation study examines RLM's performance without sub-calls, indicating reduced flexibility compared to recursive approaches.

The absence of recursion in one method suggests it will be less effective due to limited interaction with context.

Iterative Summarization Techniques

An iterative agent summarizes context documents progressively; if input exceeds model window size, it chunks data accordingly.

Hybrid Retrieval Mechanisms

Codeact Plus BM25 Methodology

The comparison includes Codeact plus BM25 retrieval which combines keyword matching with semantic search techniques for improved results.

React Loop Framework

Unlike RLM, Codeact directly provides prompts to the language model rather than offloading them into the code environment.

Results Discussion

Performance Evaluation

Main experiments focus on how performance degrades as input context grows; visual aids such as charts may enhance understanding of these trends.

Performance Comparison of RLMs and Base Models in Long Context Tasks

Overview of Code QA and Task Complexity

The discussion begins with a focus on code QA, emphasizing the performance comparison across various methods within long context benchmarks.

The task length range for CoQa is highlighted, indicating it spans from 23K to 4.2 million tokens, showcasing the varying complexity involved.

Notably, Ulong pairs exhibit lower token lengths but are assessed based on document complexity scores, raising questions about reporting these scores.

Performance Insights of RLM vs. Base Models

It is observed that RLM without sub calls outperforms code QA tasks compared to the Quen 3 coder model; GPT5 shows superior performance across all metrics.

A significant difference in performance scores is noted between browser comp and Quen coder models, suggesting instability in model performance until a certain intelligence threshold is reached.

The need for intelligent orchestration in recursive models is emphasized as critical for achieving stable performance across diverse tasks.

Key Observations on RLM Capabilities

RLM can effectively scale beyond 10 million tokens while outperforming base LMS and existing task agnostic agents by two times in long context tasks.

Cost efficiency is highlighted: GBT5 incurs higher costs than RLM GBT5 while still delivering better performance metrics, suggesting a trend towards cheaper yet effective solutions.

On specific tasks like Ulong and Ulong pairs, RLM demonstrates substantial improvements over base models with F1 scores significantly higher than those achieved by other models.

Emerging Behaviors and Recursive Sub Calling

The discussion notes that even without specialized training for scaffolding, the experimental approach yields promising results at 58% effectiveness.

The necessity of a ripple environment for managing long inputs becomes apparent; recursive sub calling enhances handling of information-dense inputs effectively.

Engineering Trade-offs in Model Design

A key characteristic of RLM involves offloading context management to an external environment (epsilon), allowing scalability beyond traditional limits.

An ablation study indicates that even without recursion capabilities, the presence of a Python interpreter allows for improved handling of long input contexts.

The conversation concludes with insights into engineering trade-offs: recursion benefits information density while ripple environments are more suited for lengthy but less dense inputs.

Recursive Language Models and Their Performance

Recursive Subcalling in RLMs

Recursive language models (RLMs) are essential for information-dense tasks, such as ulong or long pairs, where they perform semantic transformations through recursive subcalling.

RLMs demonstrate a performance improvement of 10% to 59% over ablation models that do not utilize subcalling, indicating the effectiveness of this approach.

Layers of Subcalling

The potential exists for multiple layers of subcalls; if sub-agents reach intelligence levels comparable to GBT5 mini, further recursion could enhance task-solving capabilities.

Cost analysis shows that while naive context handling is cheaper, it underperforms compared to RLM approaches which provide better returns on investment.

Cost Distribution and Efficiency

At the median cost level, RLM approaches are more cost-effective than raw models. Even at higher percentiles, RLM maintains lower costs while delivering superior performance.

Token efficiency has improved significantly with RLM methods compared to previous cursor-based systems that incurred high token costs.

Performance Scaling with Input Length and Complexity

As input length increases from 2^13 to 2^18, the performance degradation of base GPT5 is much steeper than that of RLM, which scales better with complexity.

In benchmarks comparing RLM using GBT5 against base GPT5 across various tasks, RLM consistently outperforms GPT5 in longer contexts.

Trade-offs Between Base LM and RLM

For small input contexts, base LMs outperform RLM due to their inherent representation capacity advantages; thus, a one-size-fits-all approach is not viable.

The choice between using a base LM versus an RLM should consider context size and problem complexity for optimal results.

Inference Costs and Variability

While inference costs for RLM remain comparable to those of base models, they exhibit high variance due to differing trajectory lengths based on task complexity.

Iterative interactions within the context lead to significant variations in iteration lengths across different tasks and runs.

RLMs and Model Performance Insights

Understanding RLMs and Subcalling

RLMs (Recursive Language Models) are model-agnostic inference strategies, but their performance varies based on context management and subcalling capabilities of different models.

The performance of models like Quenfree Coder can suffer when using subcalling compared to RLM, indicating that more intelligent models tend to perform better in these scenarios.

Task Dependency in Model Performance

Both GBT5 and Quenfree exhibit strong performance with RLM relative to their base models; however, they show varied behavior across tasks, emphasizing the task-dependent nature of model effectiveness.

In specific tasks like browse comp plus, RLM nearly solves all tasks while Quenfree struggles significantly, suggesting a need for tailored prompts to enhance performance.

Prompt Design and Model Capability

The fixed system prompt used across experiments may limit the potential of certain models; minor adjustments could lead to improved outcomes for Quenfree compared to GBT5.

An example illustrates how RLM performs semantic transformations differently than GPT5, highlighting differences in handling subqueries.

Emerging Patterns in RLM Trajectories

The architecture of RLM allows agents greater flexibility in processing data economically and following complex reasoning paths by delegating computations to sub-agents.

While the academic implications are intriguing, industry applications seem ahead in utilizing similar concepts already present in existing code frameworks.

Input Filtering and Context Reasoning

A key insight is that the LM's ability to filter input context without explicit visibility contributes significantly to maintaining strong performance on large inputs.

This capability mirrors human-like reasoning where not every detail needs reading; instead, selective focus can yield effective results as demonstrated through practical experiments with legal documents.

Chunking Strategies and Task Performance

The use of regex queries by RLM demonstrates its ability to search efficiently within long contexts without needing complete visibility over all tokens.

Decomposing reasoning chains into unbounded lengths via recursive subcalls enhances task performance but requires careful consideration of chunking strategies.

Observations on Decomposition Choices

Simple partitioning methods such as uniform chunking or keyword searches were primarily observed during experiments; more complex strategies were not utilized effectively.

The combination of recursion with basic decomposition techniques creates a flexible framework for processing information-dense problems effectively.

Understanding Recursive Language Models and Their Applications

The Power of Recursion in Code and Document Processing

The ability to segment code by line or match keywords in legal documents enhances the power of recursion, allowing for the processing of complex documents effectively.

Combining simplicity with architecture leads to emergent behavior, particularly in answer verification through subLM calls with small context.

Answer Verification Strategies

Instances of answer verification by RLM (Recursive Language Model) using subLM calls are observed, which can avoid context rot while verifying answers.

Some verification strategies may be redundant, increasing task costs; an example shows a model reproducing a correct answer multiple times before ultimately choosing incorrectly.

Recursive Outputs and Token Management

RLMs can produce unbounded tokens beyond base limits by recursively calling each other, leading to increased capacity for processing outputs.

This recursive capability allows models to break down outputs without absorbing everything into context, enabling efficient understanding of information.

Complex Decomposition and Synthesis

Through iterative construction via programmatic and subRLM output calls, RLM demonstrates complex decomposition and synthesis over long contexts.

Evidence from ulong pairs trajectories shows how RLM stores outputs from subLM calls and reconstructs them into final answers.

Limitations and Future Directions for RLM Implementation

While RLM shows strong performance beyond existing LMS limitations at reasonable inference costs, optimal implementation mechanisms remain underexplored.

Current experiments focus on synchronous sub-calls within a Python ripple environment; asynchronous strategies could significantly reduce runtime and inference costs.

Exploring Asynchronous Capabilities

The limitation of synchronous calls restricts flexibility; asynchronous capabilities could allow root agents to call multiple sub-agents simultaneously for enhanced efficiency.

Investigating smaller models' performance when called asynchronously raises questions about their effectiveness during recursive processes.

Max Recursion Depth Considerations

A maximum recursion depth is set at one; while this showed strong performance, it parallels concerns seen in traditional modeling regarding overfitting on certain tasks.

Exploring Recursive Layered Models (RLMs)

The Future of RLMs

Discussion on the potential for deeper layers of recursion in future work, emphasizing the importance of this exploration.

Mention of existing frontier models and the idea that explicitly training models as RLMs could enhance performance; current implementations are already being discussed in coding communities.

Agent Swarms and Data Collection

Reference to agent swarms and asynchronous methods, indicating that many are already experimenting with these concepts using tools like claw code and codeex.

Introduction of RLMs as a significant abstraction for engineers, suggesting it can transform problem-solving approaches.

Practical Applications of RLMs

Example provided on utilizing RLM in claim decomposition tasks, highlighting its efficiency compared to traditional methods.

Emphasis on the utility of RLM for complex documents such as legal contracts, which have historically posed challenges for large language models.

Implications and Community Engagement

Acknowledgment of the limitations faced by large language models when dealing with extensive documents like 200-page merger agreements.

Encouragement for audience feedback regarding this content style, indicating a willingness to produce more informal yet insightful discussions.