Memory Management in Large-Scale RAG Deployments

“Memory” in a retrieval-augmented generation (RAG) system can mean several different things: stored embeddings, vector-index structures, database buffers, application caches, or the GPU memory used by a self-hosted language model. These are separate budgets with different fixes. A useful plan measures each one before compressing data, dropping documents or adding infrastructure.

Map the memory budget first

When a hosted model API handles generation, its model weights and GPU memory are not part of your application server’s RAM budget. You still need to manage the prompt and retrieved context sent to the API, but do not combine those costs with the vector database’s memory estimate.

Estimate raw vector storage

For a simple first estimate, multiply the number of vectors by their dimensions and bytes per value. For example, a 1,536-dimensional float32 vector contains 6,144 bytes of values. In pgvector, a vector uses approximately 4 × dimensions + 8 bytes, so one million such vectors are about 5.7 GiB for vector values and that type’s per-value overhead alone.

That is not a server-sizing number. It excludes table and tuple overhead, IDs and metadata, indexes, replicas, temporary build space, database buffers, operating-system cache and concurrent query memory. Measure the actual table and index sizes in the database you run. pgvector notes that an index need not all fit in RAM, although keeping frequently used pages in memory can improve performance.

Chunking changes this estimate because every chunk becomes a vector. More, smaller chunks can improve retrieval granularity but add rows, identifiers, metadata and index entries; larger chunks reduce row count but may return irrelevant text. Tune chunk size and overlap against retrieval quality and storage, not memory in isolation.

Choose an index for the workload

Start with an exact-search baseline on a representative evaluation set. Approximate indexes trade some search quality for speed or memory, and their behavior depends on the data and settings. HNSW commonly uses additional graph structure; IVF partitions the search space; product or scalar quantization compresses stored representations. These approaches are not interchangeable, and none guarantees a particular recall or latency on your corpus.

For PostgreSQL with pgvector, check the current extension documentation for supported dimensions, operators, index options and filtered-search behavior. Measure the actual index with PostgreSQL’s relation-size functions. For Faiss, its index-selection guidance compares RAM use and precision/speed trade-offs and describes options that store or compress vectors differently. Choose based on the index you actually deploy rather than a generic claim that every vector must stay in RAM.

Reduce data carefully—not by popularity alone

• Remove true duplicates: deduplicate repeated source material where doing so will not remove meaningful context or provenance.
• Keep an authoritative source: retain document IDs, version, permissions and source location so retrieved passages can be traced and refreshed.
• Retire obsolete content deliberately: delete or archive material according to its owner’s retention policy. Low query frequency alone is not evidence that a policy or record is safe to discard.
• Review chunking: reduce unnecessary overlap and avoid creating embeddings for empty boilerplate, but rerun retrieval tests after changing the segmentation.
• Separate hot and cold data only when supported: tiering can lower the memory footprint of hot data, but disk access, page faults and promotion behavior may add latency.

Compression is a measured quality trade-off

Half-precision storage uses fewer bytes per vector value than float32. Lower-bit scalar or product quantization can reduce the vector payload further; binary quantization can reduce it substantially in supported systems. The total index shrinks by less than the raw-vector ratio when metadata and index structures remain, and retrieval quality may change. A theoretical 75% reduction in the vector-value payload from 32-bit to 8-bit values is not a promise of 75% lower system memory or unchanged recall.

Test compressed search against the original vectors using a labelled query set. Record recall@k or another retrieval metric alongside answer grounding and latency. Where supported, one pattern is to use a compressed representation to find candidates and rerank those candidates using higher-precision vectors. This costs extra computation and should be compared with the simpler alternatives.

Do not apply PCA, truncate dimensions or replace each cluster with a “representative” vector as a drop-in storage tweak. Such transformations change the retrieval representation; they require a compatible query transformation, suitable training data where applicable, and evaluation on the target corpus. Re-embedding with a model that emits fewer dimensions is a different migration and may require rebuilding the index.

Bound ingestion and query concurrency

During ingestion, process a bounded number of documents at a time. Large batches may improve throughput but can increase peak memory, temporary storage and retry cost. Use back-pressure so the embedding or database stage cannot accumulate an unbounded queue, and make ingestion safe to resume after a failure.

Batching independent searches can improve throughput in some libraries, but waiting to fill a batch can increase an individual visitor’s latency. Set a maximum batch size and wait time, then load-test with the concurrency and traffic pattern you expect. Measure peak process memory as well as the average; averages hide the brief spikes that cause out-of-memory failures.

If you host the language model yourself

Model memory is a separate capacity problem. The weights consume memory according to parameter count and representation precision; inference also needs runtime workspace and KV cache for active requests. KV-cache use grows with the number of concurrent sequences and their context lengths, subject to the model architecture and serving implementation. Quantizing weights can make a model fit smaller hardware, but it can affect output quality and does not remove the other memory costs.

Measure the serving stack’s actual VRAM and throughput at the intended context length and concurrency. If a provider hosts the model, focus instead on controlling retrieved context, prompt size, request limits and the provider’s documented context constraints.

Measure the outcomes that matter

• Capacity: table, index and cache size; process RSS; peak RAM and VRAM; temporary space during index builds.
• Performance: retrieval and end-to-end p50/p95 latency, throughput, disk reads, queue time and timeout rate.
• Quality: recall@k or another retrieval measure, source-grounded answer quality, and performance on permission-filtered queries.
• Change safety: results after document updates and deletions, re-embedding, index rebuilds and permission revocation.
• Cost: hardware and storage required for the measured workload, including replicas and ingestion—not only the raw vector payload.

Run the same test set before and after a change. If memory falls but relevant evidence is missed, or p95 latency worsens because the index pages are repeatedly read from disk, the optimization may not be a win.

A practical optimization order

1. Count chunks, dimensions, data types, replicas and index types; measure the real table and index sizes.
2. Separate database, application and model-serving budgets so work targets the constrained resource.
3. Remove confirmed duplicates and tune chunking while checking retrieval quality.
4. Compare exact search and candidate approximate indexes on representative queries and filters.
5. Test lower-precision storage or quantization only if measurements show vector/index memory is a bottleneck.
6. Bound ingestion and concurrency, then repeat tests at realistic peak load.
7. Write down the quality, freshness and latency thresholds that would trigger a rollback.

The takeaway

Large RAG deployments do not have one “memory problem”: they have vector storage, index, database, application and sometimes model-serving budgets. Estimate and measure those separately. Optimize the actual bottleneck, validate retrieval quality after each change, and never trade away source traceability or access controls for a smaller footprint.

Further reading

• pgvector documentation: storage, index sizing and search options
• Faiss: Guidelines to choose an index
• Faiss: Index types and memory/precision trade-offs

Database, index and serving capabilities vary by product and version. Check the documentation for your deployed versions and benchmark against your own corpus.