# Retrieval Performance and Scalability Evaluation

## Key Takeaways

- **Accuracy:** The system retrieves the correct document chunk within the top 200 results **97.6–98.7% of the time** across all tested datasets (Hybrid search, baseline KB).
- **Token Budget:** A **50k token** context window captures 97%+ of relevant chunks for well-structured documents. Complex financial documents benefit from a **75k token** budget.
- **Hybrid Search:** Combining vector and keyword search consistently improves ranking quality by **+5 to +9 Mean Reciprocal Rank (MRR) points** over vector-only, with no meaningful latency penalty. Recommended as default.
- **Scalability:** Retrieval quality remains stable as the knowledge base grows from approximately **136k chunks** (representing ~8,000–15,000 financial documents) to **400k chunks** (representing ~15,000–25,000 financial documents).
- **Latency:** A single search query completes in ~ **0.9–1.2 seconds** end-to-end (including embedding generation and network round-trip).

---

## Introduction

Retrieval is the foundation of Retrieval-Augmented Generation (RAG). Before the language model can generate an answer, the retrieval layer must find the right document chunks from a potentially large knowledge base. The quality of the AI's response depends directly on the quality of this retrieval step — if the relevant information is not retrieved, no amount of language model sophistication can compensate.

We conducted a systematic evaluation of our retrieval layer to answer five practical questions:

1. **Does the system find the right documents?** — How accurately does retrieval identify relevant content?
2. **How much context window do we need?** — What token budget is sufficient to capture relevant information?
3. **Should we enable Hybrid search?** — Does combining keyword and vector search improve results?
4. **Can we grow the knowledge base without losing accuracy?** — How does retrieval quality scale?
5. **How fast is it?** — What latency can users expect for a single search query?

We benchmarked across **three datasets** representing different document types commonly found in financial services, using **four search modes** and **four knowledge base sizes** ranging from ~136,000 to ~400,000 chunks.

---

## Methodology

### Datasets

We benchmarked across three datasets representing different document types and complexity levels relevant to the financial services industry:

| Dataset | Description | Questions | Documents | Pages | Chunks |
| --- | --- | --- | --- | --- | --- |
| **HR** | Human resources policies and employment guidelines. Semi-structured corporate documents typical of internal knowledge bases. | 82 | 16 | 370 | 588 |
| **Legal** | EU regulatory documents — directives, regulations, and compliance texts. Highly structured with formal language, standard in regulatory and compliance workflows. | 77 | 56 | 1,108 | 2,484 |
| **Due Diligence** | Financial reports, investment documents, and mixed-format content including tables and charts. Representative of research and due diligence processes. | 94 | 9 | 299 | 393 |

### Independent Variables

| Variable | Levels |
| --- | --- |
| **Dataset type** | HR, Legal, Due Diligence |
| **Search mode** | Hybrid (DBSF), Vector Only, Elastic search Only, Postgres FTS |
| **Knowledge base size** | ~136k chunks, ~200k, ~300k, ~400k chunks |

- **Hybrid:** Combines dense vector search with keyword search (BM25), merging results using Distribution-Based Score Fusion (DBSF). Both search components run in parallel, so hybrid latency is driven by the slower of the two (typically the vector component).  
- **Vector Only:** Pure semantic search using dense vector embeddings and approximate nearest neighbor (HNSW) graph traversal.
- **Elasticsearch Only:** Keyword search (BM25).

### Measures

| Measure | Definition |
| --- | --- |
| **Recall@k** | The fraction of relevant (expected) chunks found within the top _k_ retrieved results. Recall@200 = 0.95 means 95% of relevant chunks appear in the top 200. |
| **MRR (Mean Reciprocal Rank)** | Average of 1/rank, where rank is the position of the first relevant chunk. MRR = 0.50 means the first relevant chunk appears, on average, at position 2. Higher is better. |
| **Latency** | End-to-end wall-clock time for a single search query, including embedding generation (via cloud API), vector search, keyword search (for Hybrid mode), result merging, and full HTTP round-trip. |

### Results

#### Effect of Search Mode

> **Fixed:** Baseline KB (~136k chunks) - Dataset type | **Varies:** Search mode (Hybrid vs Vector Only vs Elastic only vs Postgres FTS)

**Averaged across all three datasets**, **Hybrid (DBSF)** achieves the highest recall from k=5 onwards, reaching 98.0% by k=200. **Vector Only** follows at 95.7%. **Elasticsearch BM25** plateaus lower at 93.2%.

#### Effect of Dataset Type

> **Fixed:** Baseline KB (~136k chunks) | **Varies:** Dataset (HR, Legal, Due Diligence)

All three datasets reach high recall (>97%) by k=200, but differ in how quickly they climb. **HR** is fastest, reaching 96.3% by k=50 due to its clear policy structure. **Legal** follows closely, jumping from 84.4% at k=10 to 96.1% at k=50. **Due Diligence** starts lower but converges to 97.9% by k=200.

#### Effect of Knowledge Base Size

> **Fixed:** Hybrid (DBSF) search | **Varies:** KB size (Baseline ~136k to ~200k to ~300k to ~400k)

**Recall remains stable as the knowledge base grows.** Across all three datasets, Recall@200 holds between 97.4% and 98.7% from the baseline (~136k chunks) through to the largest tier (~400k chunks) — a difference of less than 0.5 percentage points when tripling the collection size.

---

## Conclusion

We return to the five questions that motivated this evaluation:

### Does the system find the right documents?
**Yes.** With Hybrid search, the system retrieves the correct document chunk within the top 200 results **97–99% of the time** across all three datasets.

### How much context window do we need?
**50k tokens for standard documents, 75k for complex ones.** HR and Legal plateau at 50k tokens (97.6–98.7% recall).

### Should we enable Hybrid search?
**Yes — recommended as the default.**

### Can we grow the knowledge base without losing accuracy?
**Yes — recall remains stable across all KB sizes.**

### How fast is it?
**~0.9–1.2 seconds per search query and limit of 200 chunks**, end-to-end.

### Practical Recommendations
| Recommendation | Detail |
| --- | --- |
| **Default search mode** | Enable Hybrid (DBSF) — better quality, negligible latency cost |
| **Token budget** | 50k tokens for standard documents, 75k for complex/diverse content |
| **Scalability** | KB can grow to 400k+ chunks with stable recall on well-structured documents |