Hybrid RAG: Combine Keyword and Vector Search Effectively

Hybrid retrieval combines more than one way of finding evidence—often keyword search and vector search—before sending candidate passages to a language model. It can help when one route finds exact terms and another finds paraphrases. It does not automatically make answers more accurate: a poor retriever can add noise, a fusion rule can rank the wrong passages highly, and the model can still misunderstand good evidence.

Treat hybrid RAG as a testable design choice. Keep the simplest approach that meets your retrieval and answer-quality goals, and measure whether each additional search leg earns its latency, cost and maintenance.

What each retrieval method contributes

Keyword or full-text search

Lexical search is useful when the query contains an exact identifier, product code, error message, name or phrase that must appear in the source. PostgreSQL full-text search, for example, builds and searches indexed terms with language-aware processing. Tokenization, stemming, synonyms and punctuation rules still affect what it finds.

Vector or semantic search

Vector search can retrieve passages whose meaning is related to a query even when they use different words. Its behavior depends on the embedding model, chunking, index settings and corpus. A similarity score is a ranking signal for that search—not a calibrated probability that the passage is relevant or that an answer is true.

Structured or graph retrieval

A database filter or graph query can help when the task depends on explicit attributes or relationships, such as a product’s current region or a dependency between services. Add this leg only when the data is structured and maintained well enough to support the question. A knowledge graph is not a universal upgrade for ordinary document search.

Build a small, permission-aware pipeline

1. Define the query and access scope. Normalize the request, identify useful exact terms or filters, and derive the caller’s permissions from trusted application context.
2. Retrieve candidates. Run keyword and vector searches in parallel when they can be queried independently. Apply tenant and document permissions in every relevant retrieval path; do not rely on ranking or the model to hide unauthorized records.
3. Deduplicate. Merge repeated chunks by stable document/chunk identity, keeping source version and access metadata with each candidate.
4. Fuse rankings. Start with Reciprocal Rank Fusion (RRF) when the retrievers return ranked lists with differently scaled scores.
5. Optionally rerank. If evaluation shows that ordering remains a problem, test a reranker on a bounded candidate set and measure its extra latency and cost.
6. Generate from evidence. Pass a concise, diverse set of authorized passages with source identifiers. Instruct the model to distinguish evidence from user instructions, acknowledge gaps and avoid unsupported claims.
7. Verify access again before display or action. Permissions should be enforced by the application and data-owning services, not by a prompt.

Why rank fusion is a useful baseline

Keyword systems and vector systems often produce scores with different meanings and ranges. Adding their raw values—or min-max normalizing each tiny result set—can give a misleading sense that the scores are directly comparable. RRF uses positions instead: for a document d, add 1 / (k + ranki(d)) for each result list that contains it. The constant k controls how strongly high ranks dominate. A passage near the top of several lists can rise in the fused ranking without comparing the retrievers’ raw scores.

RRF is a practical baseline, not a relevance guarantee. It cannot surface a passage that none of the candidate searches returned, and it does not turn its fused value into a probability. Candidate-window size, duplicate handling, ties and source imbalance all affect the result. Tune these choices on judged examples rather than assuming one default fits every corpus.

Weighted RRF or a learned ranking model may help when one retriever is consistently more useful for a query class, but they add knobs and data requirements. Keep a simple RRF baseline so you can tell whether complexity actually improves results.

Choose parallel search or routing deliberately

Parallel retrieval is easy to reason about: ask each selected retriever for candidates, then merge. It can improve coverage but also adds work and may make the slowest dependency determine response time. Use explicit timeouts and bounded result windows, and define what the system does when one leg fails.

Routing selects retrievers based on query features or a classifier. It can reduce unnecessary calls, but a routing mistake can prevent the right evidence from ever being considered. Evaluate the router’s missed-route rate alongside answer quality. A low vector score alone is not a universal signal that keyword search will succeed; score distributions change with the query and index. Treat fallback thresholds as a measured product rule, not as a confidence guarantee.

Evaluate retrieval separately from the answer

Create a representative set of real questions with reviewed relevant sources. Include paraphrases, exact identifiers, rare terms, short queries, ambiguous requests, stale documents, no-answer cases and queries across customer or language boundaries. Protect or remove personal information from the evaluation data.

• Compare keyword-only, vector-only, hybrid and (if used) reranked results on the same test set.
• Measure retrieval with metrics such as Recall@k, mean reciprocal rank and nDCG, and inspect failures by query type.
• Check permission correctness and source freshness separately; a relevant but unauthorized or obsolete passage is still a failure.
• Evaluate generated answers for factual support, citation correctness, abstention when evidence is missing, and harmful reliance on retrieved instructions.
• Track p50/p95 latency, error rate, token/context use and operating cost for each pipeline stage.
• Use a holdout set and review difficult examples manually; do not tune repeatedly on the same small set and call it general improvement.

Run ablations: remove one retrieval leg and see what changes. If hybrid search does not improve the target queries enough to justify its cost, simplify the system or improve the underlying content and indexing.

Common failure patterns

Exact product codes disappear

Inspect tokenization, punctuation, field boosts and metadata filters in the keyword leg. Ensure the identifier is indexed as users enter it, not just embedded semantically.

Top results look related but do not answer the question

Review chunk boundaries, embedding coverage, candidate depth and the relevance judgments. Test a reranker only after you know candidate recall is adequate.

The hybrid list is noisy

Check duplicate chunks, stale indexes, overly broad lexical matching and whether both retrievers are returning candidates within the user’s authorized scope. Rank fusion cannot compensate for bad candidates.

Latency spikes after adding retrievers

Measure each leg, cap parallel work and set clear deadlines. Remove low-value queries or choose a simpler route when offline evaluation supports that trade-off.

Roll out incrementally

Start with one concrete retrieval failure that matters to users. Build a small gold set, add the second retrieval method, fuse with a transparent baseline, then compare retrieval, answer quality, access correctness, latency and cost against what you already run. Keep a rollback path and monitor index freshness and regressions after updates.

Hybrid RAG is useful when complementary retrieval methods recover evidence that a single method misses. The win comes from better candidates and measured ranking—not from combining more components by default.

Further reading

• PostgreSQL: Introduction to Full Text Search
• pgvector: Hybrid Search with PostgreSQL full-text search
• Cormack, Clarke & Büttcher: Reciprocal Rank Fusion (SIGIR 2009)
• NIST AI 600-1: Generative AI Profile

Retrieval quality is specific to the corpus, users and task. Benchmark the systems and versions you plan to deploy.