Loading…
Stack Guide · LlamaIndex
LlamaIndex occupies a specific and important slot in the AI stack: it's the framework you reach for when your documents are complex, your data sources are many, or your indexing requirements go beyond 'split text, embed chunks, store in vector database.'
LlamaParse handles PDFs the way they actually look, tables intact, columns correct. LlamaHub connects to the data sources your documents actually live in. And the indexing layer gives you the metadata filtering and hierarchical structure that plain similarity search can't provide.
These are our notes from production builds: when LlamaIndex earns its complexity cost, when it doesn't, and the patterns we've settled on.
Building a RAG pipeline with LlamaIndex?
The value is specific: complex document ingestion, structured indexing, and accurate extraction from formats that other tools handle poorly.
LlamaIndex was purpose-built for the problem of turning a messy document collection into something an LLM can reason over accurately. When you have hundreds or thousands of documents with different formats, sections, and metadata, LlamaIndex's indexing primitives (document nodes, metadata filters, hierarchical indexes) give you finer control than LangChain's retrieval chains.
Some queries can't be answered from a single retrieved chunk. They require synthesizing information across multiple documents, comparing positions, aggregating data, or tracing a concept through a corpus. LlamaIndex has first-class support for this through its query engines, sub-question decomposition, and multi-document summarization patterns.
Most PDF parsing libraries treat a PDF as a flat stream of text. LlamaParse understands layout: it preserves table structure, handles multi-column layouts, and correctly associates captions with figures. For financial documents, technical manuals, and research papers where table data matters, LlamaParse produces dramatically cleaner input than PyMuPDF or unstructured alone.
LlamaIndex makes it easy to attach structured metadata (source, date, author, section, document type) to every node in an index, then filter on that metadata at query time. This matters when you need retrieval that respects document structure: 'find relevant chunks from Q3 financial reports' rather than 'find relevant chunks from all documents'.
LlamaHub is the connector ecosystem, over 100 data source integrations maintained alongside the core framework. It solves the problem of getting documents out of the tools where they actually live.
100+ maintained data source connectors
LlamaHub provides connectors for Slack, Notion, Google Drive, Confluence, GitHub, Jira, Salesforce, and dozens more, all maintained by the LlamaIndex team and the community. Rather than writing a custom loader that reads from the Notion API and maps it to document nodes, you install the connector and configure credentials.
Consistent document node format
Every LlamaHub connector outputs the same document node format with consistent metadata fields. This means a pipeline that reads from Confluence and one that reads from Google Drive can feed into the same index without format translation in the middle. The uniformity compounds as you add more sources.
Multi-source ingestion pipelines
For enterprise RAG applications that need to ingest from five or six internal data sources simultaneously, LlamaHub removes the scaffolding work. The ingestion pipeline handles batching, chunking, embedding, and upserting to your vector store. You configure it, you don't write it.
LlamaIndex has real setup and conceptual overhead. When your use case doesn't require its strengths, that overhead is cost without benefit.
Simple single-source RAG
If you're embedding one type of document from one source and running standard similarity search, LlamaIndex is overkill. LangChain's retrieval chain or even the raw vector database SDK with direct embedding calls is less code, easier to understand, and easier to debug. Only reach for LlamaIndex when the document structure complexity justifies it.
Agent orchestration workflows
LlamaIndex has agent abstractions, but they're not its strength. If your primary need is multi-step agent behavior with conditional routing, human-in-the-loop checkpoints, or parallel agent execution, use LangGraph. LlamaIndex is the data layer; LangGraph is the control flow layer. Using LlamaIndex agents for complex orchestration is using the wrong tool.
Clean structured text documents
If your documents are already well-structured (clean HTML, JSON records, or Markdown with consistent formatting) the value-add from LlamaIndex's parsing and indexing machinery is minimal. The overhead of setting up a LlamaIndex pipeline is only worth it when the input data is messy enough to require it.
LlamaIndex earns its place in our stack for a specific class of project. Here is what that looks like in practice.
For any document that contains tables, LlamaParse is our default parser. The quality difference compared to raw PDF text extraction is significant for financial statements, technical specifications, and research papers. We pay for the API calls. It's worth it for data accuracy.
When a document collection has known section structure (chapters with subsections, or policies with numbered clauses), we build hierarchical indexes that mirror that structure. This allows retrieval at the right granularity: a query about a specific clause retrieves the clause, not a random chunk that happens to contain relevant words.
For applications that need to ingest from multiple internal tools, we use LlamaHub connectors to standardize the ingestion pipeline. The Notion, Confluence, and Google Drive connectors have saved meaningful development time on projects where the client has documents spread across multiple platforms.
LlamaIndex is powerful enough that misuse costs real money and time. These are the patterns we encounter repeatedly.
Using LlamaIndex for agent orchestration
LlamaIndex's agent abstractions work, but they're thin wrappers over ReAct-style loops and lack the state management and conditional branching that complex production agents need. Teams that build their agent logic inside LlamaIndex agents eventually hit walls that require rewriting in LangGraph anyway. Start with LangGraph for agents; use LlamaIndex for data.
Ignoring chunking strategy defaults
LlamaIndex ships with default chunk sizes and overlap settings. Those defaults are reasonable for generic text but rarely optimal for your specific content. Financial documents need different chunk sizes than support tickets. Code documentation needs different overlap than narrative prose. Benchmarking retrieval quality against your actual data before settling on chunk settings is always worth the time.
Not batching LlamaParse API calls
LlamaParse is a paid API and costs add up quickly when ingesting large document collections. Calling it document-by-document in a loop during development is fine, but in production you should batch submissions, cache results, and re-parse only when documents change. Not doing this is how a modest document collection generates a surprisingly large monthly API bill.
Tell us about your document sources and what the retrieval needs to do. We can give you an honest assessment of whether LlamaIndex is the right data layer, and how to set it up so it actually works in production.