What Is a Vector Database? Why RAG Applications Need One

Short answer: A vector database stores embeddings, which are lists of numbers that represent the meaning of text, images or other data, and quickly finds the vectors most similar to a query vector. RAG (Retrieval-Augmented Generation) applications need one because they must search thousands or millions of document chunks by meaning, not just by keywords, and return the most relevant passages to the LLM in milliseconds.

Key takeaways

• Embeddings turn text into vectors where similar meanings are close together.
• A vector database performs fast similarity search over those vectors.
• Most use approximate nearest neighbour (ANN) indexes such as HNSW for speed at scale.
• Options range from libraries (FAISS) to embedded databases (ChromaDB), managed services (Pinecone) and extensions of familiar databases (pgvector for PostgreSQL).
• For learning, start simple. Production needs metadata filtering, scalability and access control.

What are embeddings?

An embedding model converts a piece of text into a fixed-length list of numbers (a vector). Texts with similar meanings produce vectors that are close together. For example, “How many leave days do I get?” and “annual holiday entitlement” share few words but end up near each other in vector space. That’s what lets a RAG system find the right passage even when the user’s wording differs from the document.

How does a vector database work?

1. Ingest: split documents into chunks and create an embedding for each chunk.
2. Store: save each vector with its original text and metadata (source, page, date, department).
3. Index: build an index that makes similarity search fast.
4. Query: embed the user’s question and find the k most similar vectors.
5. Filter (optional): restrict results by metadata, for example only HR documents or only documents the user may access.
6. Return: pass the matching chunks to the LLM as context.

How is similarity measured?

Common measures include cosine similarity (the angle between vectors), dot product and Euclidean distance. Use the measure recommended for your embedding model.

What is approximate nearest neighbour (ANN) search?

Comparing a query against every stored vector (exact search) becomes slow with millions of vectors. ANN indexes trade a tiny amount of accuracy for very large speed gains. A widely used method is HNSW (Hierarchical Navigable Small World graphs), described by Malkov and Yashunin. Others include IVF (inverted file) indexes and product quantization.

Why does RAG need a vector database?

• Semantic search: it finds relevant passages by meaning, not exact keywords
• Speed: millisecond retrieval across large document collections
• Metadata: it keeps the source and page for citations
• Filtering: it enforces which documents each user can retrieve
• Updates: you can add, update or delete chunks without retraining any model

New to RAG? Start with What Is RAG? Retrieval-Augmented Generation Explained.

Vector database vs traditional database

Aspect	Traditional (relational) database	Vector database
Data	Rows and columns	High-dimensional vectors + metadata
Typical query	Exact match, filters, joins (SQL)	“Find items most similar to this vector”
Search type	Keyword / exact	Semantic / similarity
Index	B-tree, hash	ANN indexes such as HNSW, IVF
Example use	Orders, customers, transactions	RAG, semantic search, recommendations

The line is blurring: PostgreSQL with the pgvector extension can store and search vectors alongside normal relational data. Strong SQL skills remain valuable. See SQL for Data Science.

Popular vector databases and libraries

Option	Type	Good for
FAISS	Open-source similarity-search library from Meta	Fast local experiments, research, custom pipelines
ChromaDB	Open-source, developer-friendly vector store	Learning, prototypes, small-to-medium apps
Pinecone	Managed cloud vector database	Production without managing infrastructure
pgvector	PostgreSQL extension	Teams already using PostgreSQL
Weaviate, Qdrant, Milvus	Open-source vector databases (self-hosted or cloud)	Scalable production systems with filtering

Features and pricing change quickly, so check each project’s official documentation before choosing for production.

FAISS vs ChromaDB: which should beginners learn?

ChromaDB is often the easiest starting point. It stores text, metadata and vectors together with a simple Python API. FAISS is a lower-level library: extremely fast and flexible, but you manage metadata and persistence yourself. Learning both gives you a good understanding of what managed databases do for you.

How do you choose a vector database?

• Scale: thousands vs hundreds of millions of vectors
• Hosting: local, self-hosted or fully managed
• Filtering: how well it combines metadata filters with similarity search
• Hybrid search: support for combining keyword and vector search
• Existing stack: already on PostgreSQL? pgvector may be enough
• Security: access control, encryption, data residency
• Cost: storage, queries and operational effort

Common mistakes

• Mixing embeddings from different models in one index
• Poor chunking: chunks that are too big, too small or cut mid-sentence
• Not storing metadata, which makes citations and filtering impossible
• Skipping evaluation of retrieval quality, the step that decides answer quality
• Choosing an enterprise database for a 500-document prototype

A beginner project to practise

1. Collect 20–50 PDFs or web pages on one topic
2. Chunk them and create embeddings
3. Store them in ChromaDB with source and page metadata
4. Write a search function that returns the top 5 chunks for a question
5. Measure retrieval: for 20 test questions, does the right chunk appear in the top 5?
6. Connect an LLM to generate answers with citations

Frequently asked questions

Is a vector database required for RAG?

Not strictly. Small projects can use an in-memory index or a library like FAISS. As data grows, a vector database adds persistence, filtering and scalability.

What is the difference between embeddings and a vector database?

Embeddings are the vectors. The vector database stores, indexes and searches them.

Can a vector database replace SQL databases?

No. They solve different problems and are often used together.

What is hybrid search?

Hybrid search combines keyword search (good for exact terms, codes and names) with vector search (good for meaning). It often improves RAG retrieval.

Which vector database is best?

There’s no single best one. ChromaDB or FAISS suits learning, pgvector suits PostgreSQL users, and managed or open-source vector databases suit larger production workloads.

Conclusion

A vector database is the retrieval engine behind RAG: it stores embeddings and finds the most relevant content by meaning, fast. Learn embeddings, similarity, ANN indexing and metadata filtering, and build one small project. Next, compare RAG vs Fine-Tuning and follow the LLM Engineer Roadmap.

Build RAG applications hands-on in our Generative AI App Developer course or see our Enterprise RAG Chatbot project.