As Large Language Models (LLMs) transition from simple chatbots to autonomous agents, the methods we use to feed them data must evolve. While Retrieval-Augmented Generation (RAG) remains the industry standard for grounding models in external data, its “vanilla” implementation—converting text chunks into vectors for semantic lookup—often falters when faced with interconnected documents, technical jargon, or multi-hop queries.
For Machine Learning Engineers (MLEs) and Product Managers (PMs), understanding the shift toward Agentic Search is critical. This approach moves away from static lookups toward dynamic, iterative, and hierarchical strategies that mirror how a human expert navigates a complex knowledge base.
1. The Limitations of Traditional RAG
In a standard RAG pipeline, documents are pre-processed into fixed-size chunks, embedded into a vector space, and retrieved based on cosine similarity. While efficient, this architecture faces several hurdles in production environments:
- Loss of Global Context: Chunking often breaks the “connective tissue” of a document, leading to misinterpretations of headers or page-spanning tables.
- Multi-hop Reasoning: If a query requires synthesizing information from Page 5 of Document A and Page 100 of Document B, a simple similarity search may not surface both simultaneously.
- Retrieval Noise: Irrelevant “top-k” results can pollute the LLM’s context window, leading to “Lost in the Middle” phenomena or hallucinations.
To solve these, we are seeing the rise of Hierarchical and Agentic retrieval patterns.
2. Hierarchical Retrieval: RAPTOR and PageIndex
Instead of treating a corpus as a flat list of chunks, hierarchical approaches build a structured map of the data.
RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval)
RAPTOR constructs a tree of document summaries. It recursively clusters chunks and generates high-level summaries for those clusters, allowing the retriever to look at the “forest” (summaries) before diving into the “trees” (specific chunks). This is particularly effective for queries that require a thematic overview of a large dataset.
PageIndex: Reasoning-Based Tree Search
Similar to RAPTOR, PageIndex transforms documents into a “Table-of-Contents” style JSON structure. It enables an LLM to navigate a document’s hierarchy—moving from section summaries down to granular paragraphs.
Key Advantages:
- Preserves Structure: Maintains the relationship between chapters, sections, and subsections.
- High Accuracy: PageIndex previously achieved 98.7% accuracy on FinanceBench, showcasing its strength in professional document analysis (e.g., legal filings and technical manuals).
3. Agentic RAG: The Iterative Loop
Agentic RAG transforms retrieval from a single-shot event into a multi-step conversation between the agent and the data. Instead of just searching once, the agent analyzes the results and decides its next move.
The Agentic Search Loop:
Query → Search → Analyze Results → Adjust Strategy → Search Again → Precise Results
Corrective RAG (CRAG) and Self-RAG
These frameworks introduce an “Evaluator” gate. If the retrieved documents are deemed irrelevant or of low quality, the agent can:
- Ignore the results to prevent hallucinations.
- Rewrite the query for a better vector match.
- Escalate to a secondary tool (like a web search) to fill the knowledge gap.
4. Lessons from Coding Agents: Parallelized Sub-Agents
One of the most sophisticated examples of agentic search is found in tools like Claude Code. When tasked with a broad objective—such as documenting every template context in a repository—it eschews the “document dump” method.
Instead, it utilizes Context Isolation through Sub-Agents:
- Parallelization: The supervisor agent spawns sub-agents focused on narrow domains (e.g., one for Database templates, one for UI templates).
- Reduced Pollution: By giving each sub-agent only the relevant file context, the system avoids “context poisoning” and stays within token limits.
- Exact Symbol Search: Unlike semantic RAG, these agents often use
grepor AST (Abstract Syntax Tree) parsing to find exact function names or variables, combining the speed of traditional search with the reasoning of an LLM.
5. Implementing “Deep Agents”: The Four Pillars
For teams looking to move beyond naive tool-calling, four factors distinguish a true “Deep Agentic” search architecture:
| Feature | Description |
|---|---|
| Strategic Planning | Using tools like TodoWrite and TodoRead to manage a multi-step search plan. |
| Persistent Memory | Writing “playbooks” (e.g., an Agents.md file) that compress findings and recurring pitfalls for long-running tasks. |
| Dynamic Delegation | The ability to automatically spawn and monitor sub-agents for parallel task execution. |
| Advanced System Prompts | Detailed instructions including few-shot examples for handling “no-result” scenarios or conflicting data. |
Summary
For Product Managers, the trade-off is clear: Agentic search increases latency and token cost but significantly improves accuracy and reliability for complex domains. For MLEs, the frontier is no longer just better embeddings, but better orchestration—building systems that know when to search, when to reflect, and when to ask for more context.
References
https://blog.langchain.com/deep-agents/
https://machinelearningmastery.com/beyond-vector-search-5-next-gen-rag-retrieval-strategies/