Unleashing the Power of Large Language Models for Deep Research

As large language models (LLMs) continue to advance, their role as research assistants is increasingly profound. These models are transcending simple factual inquiries and delving into “deep research” tasks, which demand multi-step reasoning, the evaluation of conflicting information, data sourcing from various web resources, and synthesizing this information into coherent outputs.

This emerging capability is marketed under various brand names by leading labs—OpenAI terms it “Deep Research,” Anthropic refers to it as “Extended Thinking,” Google’s Gemini offers “Search + Pro” features, and Perplexity calls theirs “Pro Search” or “Deep Research.” But how effective are these models in real-world applications? A recent report from FutureSearch, titled Deep Research Bench (DRB): Evaluating Web Research Agents, delivers a comprehensive evaluation, showcasing both remarkable abilities and notable shortcomings.

What Is Deep Research Bench?

Developed by the FutureSearch team, Deep Research Bench is a meticulously designed benchmark that assesses AI agents on multi-step, web-based research tasks. These are not simple inquiries but reflect the complex, open-ended challenges faced by analysts, policymakers, and researchers in real-world situations.

The benchmark comprises 89 distinct tasks across eight categories, including:

• Find Number: e.g., “How many FDA Class II medical device recalls occurred?”
• Validate Claim: e.g., “Is ChatGPT 10x more energy-intensive than Google Search?”
• Compile Dataset: e.g., “Job trends for US software developers from 2019–2023.”

Each task is carefully crafted with human-verified answers, utilizing a frozen dataset of scraped web pages termed RetroSearch. This approach ensures consistency across model evaluations, eliminating the variable nature of the live web.

The Agent Architecture: ReAct and RetroSearch

Central to Deep Research Bench is the ReAct architecture, which stands for “Reason + Act.” This model mirrors how human researchers approach problems by contemplating the task, executing relevant searches, observing outcomes, and deciding whether to refine their approach or conclude.

While earlier models explicitly followed this loop, newer “thinking” models often embed reasoning more fluidly into their actions. To ensure evaluation consistency, DRB introduces RetroSearch—a static version of the web. Agents utilize a curated archive of web pages gathered through tools like Serper, Playwright, and ScraperAPI. For complex tasks like “Gather Evidence,” RetroSearch can offer access to over 189,000 pages, all time-stamped to ensure a reliable testing environment.

Top Performing AI Agents

In the competitive landscape, OpenAI’s model o3 stood out, achieving a score of 0.51 out of 1.0 on the Deep Research Bench. Although this may seem modest, interpreting the benchmark’s difficulty is crucial: due to task ambiguity and scoring nuances, even an exemplary model likely caps around 0.8—referred to as the “noise ceiling.” Thus, even the leading models today still trail well-informed, methodical human researchers.

The evaluation’s insights are illuminating. o3 not only led the results but also demonstrated efficiency and consistency across nearly all task types. Anthropic’s Claude 3.7 Sonnet followed closely, showcasing adaptability in both its “thinking” and “non-thinking” modes. Google’s Gemini 2.5 Pro excelled in structured planning and step-by-step reasoning tasks. Interestingly, the open-weight model DeepSeek-R1 kept pace with GPT-4 Turbo, illustrating a narrowing performance gap between open and closed models.

A discernible trend emerged: newer “thinking-enabled” models consistently outperformed older iterations, while closed-source models held a marked advantage over open-weight alternatives.

Challenges Faced by AI Agents

The failure patterns identified in the Deep Research Bench report felt alarmingly familiar. I’ve often experienced the frustration of an AI agent losing context during extensive research or content creation sessions. As the context window expands, the model may struggle to maintain coherence—key details might fade, objectives become unclear, and responses may appear disjointed or aimless. In such cases, it often proves more efficient to reset the process entirely, disregarding previous outputs.

This kind of forgetfulness isn’t merely anecdotal; it was identified as the primary predictor of failure in the evaluations. Additional recurring issues include repetitive tool use—agents running the same search in a loop, poor query formulation, and too often reaching premature conclusions—delivering only partially formed answers that lack substantive insight.

Notably, among the top models, differences were pronounced. For instance, GPT-4 Turbo exhibited a tendency to forget previous steps, while DeepSeek-R1 was prone to hallucinate or fabricate plausible yet inaccurate information. Across the board, models frequently neglect to cross-validate sources or substantiate findings before finalizing their outputs. For those relying on AI for critical tasks, these shortcomings resonate all too well, underscoring the distance we still need to cover to build agents that truly mimic human-like thinking and research abilities.

Memory-Based Performance Insights

Intriguingly, the Deep Research Bench also assessed “toolless” agents—language models that function without access to external resources, such as the web or document retrieval. These models rely exclusively on their internal information, generating responses based solely on their training data. This means they can’t verify facts or conduct online searches; instead, they form answers based purely on recollections.

Surprisingly, some toolless agents performed nearly as well as their fully equipped counterparts on specific tasks. For instance, in the Validate Claim task—measuring the plausibility of a statement—they scored 0.61, just shy of the 0.62 average achieved by tool-augmented agents. This suggests that models like o3 and Claude possess strong internal knowledge, often able to discern the validity of common assertions without needing to perform web searches.

However, on more challenging tasks like Derive Number—requiring the aggregation of multiple values from diverse sources—or Gather Evidence, which necessitates locating and evaluating various facts, these toolless models struggled significantly. Without current information or real-time lookup capabilities, they fell short in generating accurate or comprehensive answers.

This contrast reveals a vital nuance: while today’s LLMs can simulate “knowledge,” deep research does not rely solely on memory but also on reasoning with up-to-date and verifiable information—something that only tool-enabled agents can genuinely provide.

Concluding Thoughts

The DRB report underscores a crucial reality: the finest AI agents can outperform average humans on narrowly defined tasks, yet they still lag behind adept generalist researchers—particularly in strategic planning, adaptive processes, and nuanced reasoning.

This gap is especially evident during protracted or intricate sessions—something I have experienced, where an agent gradually loses sight of the overarching objective, resulting in frustrating disjointedness and utility breakdown.

The value of Deep Research Bench lies not only in its assessment of surface-level knowledge but in its investigation into the interplay of tool usage, memory, reasoning, and adaptability, providing a more realistic mirroring of actual research than benchmarks like MMLU or GSM8k.

As LLMs increasingly integrate into significant knowledge work, tools like FutureSearch‘s DRB will be crucial for evaluating not just the knowledge of these systems, but also their operational effectiveness.