"Do we need a vector database?" is one of the most common questions in early AI architecture conversations. It's also one where the answer depends almost entirely on the size and nature of your data — not on how sophisticated you want to sound.
Here's the honest decision framework.
What a vector database actually does
When you send text to an embedding model (OpenAI's text-embedding-3-small, Cohere's embed-v3, etc.), you get back a list of numbers — a vector — that represents the semantic meaning of that text. Two pieces of text that mean similar things produce vectors that are close together in this high-dimensional space. Two unrelated pieces of text produce vectors that are far apart.
A vector database stores these embeddings and lets you query by semantic similarity rather than exact keyword match. You embed the user's question, then find the stored documents whose embeddings are closest to the question embedding, and pass those documents as context to the LLM.
This is the foundation of retrieval-augmented generation (RAG). The vector database is the retrieval layer.
What it is not: a replacement for a relational database, a search engine for exact-match queries, or a way to make a small knowledge base smarter than it would be in a system prompt. It's a tool for one specific job.
When you need one
Your corpus is larger than roughly 50 pages. The key question is whether your entire knowledge base fits comfortably in a context window. Modern models support 128K tokens (GPT-5) or even 1M tokens (Gemini 2.5 Pro). At 128K tokens, you can fit roughly 90,000–100,000 words of text — or about 350–400 pages of a standard business document.
If your documentation fits inside a single large context window with room to spare for the conversation itself, you might not need a vector database. But consider what happens when your docs grow, or when you're paying per token for every query. At scale, stuffing 300 pages into every request is expensive and slow.
Your knowledge base changes frequently. Vector databases make it practical to add, remove, and update chunks of your knowledge base without rebuilding everything. If you're managing product documentation that changes monthly, a vector database with a proper ingestion pipeline is much easier to maintain than a monolithic system prompt you rewrite by hand.
You need semantic search across a large corpus. If users ask open-ended questions — "what's our refund policy for digital products?" — and the answer lives somewhere in 1,000 pages of policy documentation, exact keyword search will fail them frequently. Semantic search via embeddings handles paraphrase and indirect phrasing.
You have multiple types of content. Product docs, support transcripts, contract terms, pricing sheets — a vector database lets you manage all of these as separate collections and control which ones are searched for a given query type.
You're building a RAG system with more than a handful of users. At production traffic, loading large amounts of text into every context window is slow and expensive. A vector database query typically takes 10–100ms. Stuffing 50,000 tokens into every request adds latency and cost with every single call.
When you don't need one
Everything fits in 32K tokens or less. If your entire knowledge base — all your FAQ content, product documentation, policies — fits in a single context window with room for the conversation history, just put it in the system prompt. It's simpler, cheaper to build, and easier to debug. The LLM will use it reliably.
Your chatbot has a fully structured FAQ. If users ask from a known set of ~50–100 questions and the answers are short, a structured lookup (even a simple JSON file the agent searches) works fine. The semantic search capability of a vector database is unnecessary when the question space is small and well-defined.
You're prototyping or building a proof of concept. Don't spend time setting up a vector database for a two-week prototype. Use in-memory storage or just put the data in the system prompt. Validate the concept first.
Your queries are always structured. If every question maps to a database lookup — "what's the status of order #12345?" — you don't need semantic search. You need a function call to your orders API.
Alternatives for smaller use cases
pgvector
If you're already using PostgreSQL, the pgvector extension adds vector storage and similarity search directly to your existing database. You get a familiar query interface, no new infrastructure, and transactional consistency with your other data.
Works well for: up to a few hundred thousand vectors, teams who already know Postgres, projects where adding a new managed service isn't worth it.
Limitations: similarity search performance degrades without careful indexing above ~1M vectors. Not designed for the specific query patterns (approximate nearest neighbor at scale) that dedicated vector databases optimize for.
In-memory for prototypes
Libraries like hnswlib or LangChain's in-memory vector store let you run vector search entirely in process memory. Fine for development, fine for a prototype with a fixed small dataset. Not appropriate for production systems that need to persist data across restarts.
Simple semantic similarity with an API
For very small corpora, you can skip vector storage entirely: embed all your documents at query time, compute cosine similarity in Python/TypeScript, and return the top results. This works if you have 20–30 short documents and aren't worried about latency.
Managed vector database options
Once you've decided you need a dedicated vector database, the main options:
Pinecone. Fully managed, strong performance at scale, simple API. Pricing starts free for small indexes and moves to $70+/month for production-grade serverless usage. Easiest to get started with; least operational overhead.
Qdrant. Open source with a managed cloud offering. More flexible filtering and payload storage than Pinecone. Free tier available; cloud starts at around $25/month. Can self-host if data residency is a requirement.
Weaviate. Open source with cloud hosting. Strong multi-tenancy support, which matters if you're building a SaaS product where each customer has their own knowledge base. Cloud pricing starts at around $25/month.
Chroma. Lightweight, open source, designed for development and smaller production use cases. Good fit for projects that want to self-host without complex infrastructure.
Cost comparison at production scale (approximate, varies by vector count and query volume):
| Option | 1M vectors | 10M vectors | |---|---|---| | Pinecone (serverless) | ~$70–120/mo | ~$400–600/mo | | Qdrant Cloud | ~$60–100/mo | ~$300–500/mo | | Weaviate Cloud | ~$75–120/mo | ~$350–550/mo | | pgvector (self-managed RDS) | ~$50–80/mo | ~$150–300/mo | | Self-hosted Qdrant (EC2) | ~$30–60/mo | ~$80–150/mo |
Self-hosted options are cheaper but require operational maintenance. For most early-stage projects, a managed service with a free or low-cost tier is the right call — you can always migrate later.
The decision in one rule
Under 50 pages of documentation that changes rarely: put it in the system prompt, skip the vector database.
Over 50 pages, growing knowledge base, or multi-source content: use a vector database. Start with pgvector if you're already on Postgres; use Pinecone or Qdrant if you want managed and production-ready from day one.
The bottom line
A vector database is the right tool for semantic search over large, changing corpora. It's overkill for a chatbot backed by a small, stable knowledge base. The line is roughly 50 pages — below that, put the content in the system prompt. Above it, or when the data is dynamic and multi-source, reach for a vector database.
Don't add infrastructure complexity because it sounds like the right thing to do. Add it when the alternative genuinely breaks down.