Retrieval Performance and Scalability Evaluation

Key Takeaways

• Accuracy: The system retrieves the correct document chunk within the top 200 results 97.6–98.7% of the time across all tested datasets (Hybrid search, baseline KB).
• Token Budget: A 50k token context window captures 97%+ of relevant chunks for well-structured documents. Complex financial documents benefit from a 75k token budget.
• Hybrid Search: Combining vector and keyword search consistently improves ranking quality by +5 to +9 Mean Reciprocal Rank (MRR) points over vector-only, with no meaningful latency penalty. Recommended as default.
• Scalability: Retrieval quality remains stable as the knowledge base grows from approximately 136k chunks (representing ~8,000–15,000 financial documents) to 400k chunks (representing ~15,000–25,000 financial documents).
• Latency: A single search query completes in ~ 0.9–1.2 seconds end-to-end (including embedding generation and network round-trip).

Introduction

Retrieval is the foundation of Retrieval-Augmented Generation (RAG). Before the language model can generate an answer, the retrieval layer must find the right document chunks from a potentially large knowledge base. The quality of the AI's response depends directly on the quality of this retrieval step — if the relevant information is not retrieved, no amount of language model sophistication can compensate.

We conducted a systematic evaluation of our retrieval layer to answer five practical questions:

1. Does the system find the right documents? — How accurately does retrieval identify relevant content?
2. How much context window do we need? — What token budget is sufficient to capture relevant information?
3. Should we enable Hybrid search? — Does combining keyword and vector search improve results?
4. Can we grow the knowledge base without losing accuracy? — How does retrieval quality scale?
5. How fast is it? — What latency can users expect for a single search query?

We benchmarked across three datasets representing different document types commonly found in financial services, using four search modes and four knowledge base sizes ranging from ~136,000 to ~400,000 chunks.

Methodology

Datasets

We benchmarked across three datasets representing different document types and complexity levels relevant to the financial services industry:

Independent Variables

• Hybrid: Combines dense vector search with keyword search (BM25), merging results using Distribution-Based Score Fusion (DBSF). Both search components run in parallel, so hybrid latency is driven by the slower of the two (typically the vector component).
• Vector Only: Pure semantic search using dense vector embeddings and approximate nearest neighbor (HNSW) graph traversal.
• Elasticsearch Only: Keyword search (BM25).

Measures

Results

Effect of Search Mode

Fixed: Baseline KB (~136k chunks) - Dataset type | Varies: Search mode (Hybrid vs Vector Only vs Elastic only vs Postgres FTS)

Averaged across all three datasets, Hybrid (DBSF) achieves the highest recall from k=5 onwards, reaching 98.0% by k=200. Vector Only follows at 95.7%. Elasticsearch BM25 plateaus lower at 93.2%.

Effect of Dataset Type

Fixed: Baseline KB (~136k chunks) | Varies: Dataset (HR, Legal, Due Diligence)

All three datasets reach high recall (>97%) by k=200, but differ in how quickly they climb. HR is fastest, reaching 96.3% by k=50 due to its clear policy structure. Legal follows closely, jumping from 84.4% at k=10 to 96.1% at k=50. Due Diligence starts lower but converges to 97.9% by k=200.

Effect of Knowledge Base Size

Fixed: Hybrid (DBSF) search | Varies: KB size (Baseline ~136k to ~200k to ~300k to ~400k)

Recall remains stable as the knowledge base grows. Across all three datasets, Recall@200 holds between 97.4% and 98.7% from the baseline (136k chunks) through to the largest tier (400k chunks) — a difference of less than 0.5 percentage points when tripling the collection size.

Conclusion

We return to the five questions that motivated this evaluation:

Does the system find the right documents?

Yes. With Hybrid search, the system retrieves the correct document chunk within the top 200 results 97–99% of the time across all three datasets.

How much context window do we need?

50k tokens for standard documents, 75k for complex ones. HR and Legal plateau at 50k tokens (97.6–98.7% recall).

Should we enable Hybrid search?

Yes — recommended as the default.

Can we grow the knowledge base without losing accuracy?

Yes — recall remains stable across all KB sizes.

How fast is it?

~0.9–1.2 seconds per search query and limit of 200 chunks, end-to-end.

Practical Recommendations