# Elasticsearch

## Overview

Elasticsearch is a powerful keyword search engine integrated into our RAG (Retrieval-Augmented Generation) system, specifically designed to serve the unique needs of Financial Services Industry (FSI) clients. It uses the industry-standard BM25 (Best Matching 25) scoring algorithm to provide highly relevant keyword-based search results for your financial documents and content.

When you enable the **COMBINED** search type in the InternalSearch tool of the Unique AI assistant, our platform intelligently combines:

- **Qdrant** for vector-based semantic search
  - excels at concept queries such as "What are our policies on managing customer credit risk?"
- **Elasticsearch** for keyword-based search using BM25
  - excels at keyword queries such as "Basel III Tier 1 capital ratio"

This hybrid approach ensures you get the best of both worlds: semantic understanding through vectors and precise keyword matching through Elasticsearch's advanced text analysis.

This service is currently in **BETA**. We may continue to refine the indexing and retrieval methods as we improve the system. If you encounter any issues while using the service, we’d appreciate your feedback.

## Who it’s for

- Admins who configure Spaces to optimize the experience for AI chat users relying on internal document searches, particularly when their queries contain specific keywords or technical terms

**Can this feature be enabled on non-azure or self-hosted tenants?**

## Benefits

Elasticsearch provides superior relevance scoring for keyword-based queries and improves the search performance overall. Our platform previously used PostgreSQL's built-in full-text search with n-gram-based similarity matching (`pg_trgm` extension). While functional, this approach had several limitations for FSI requirements.

Key benefits of Elasticsearch over PostgreSQL FTS:

**Superior Relevance Scoring**
- BM25 Algorithm: Industry-standard relevance scoring vs. basic term frequency
- Document Length Normalization: Better handling of varying document sizes common in financial documents
- Term Frequency Saturation: Prevents over-weighting of frequently repeated term

**Enhanced Performance**
- Dedicated Search Engine: Purpose-built for search vs. general database operations
- Advanced Indexing: Optimized inverted indices vs. simple GIN indices
- Horizontal Scaling: Can scale independently from your database

## Example queries

**Regulatory Compliance**
- Regulation References: "Section 225 of Dodd-Frank", "Basel III capital requirements"
- Compliance Codes: "CCAR stress testing", "GDPR Article 17", "SOX Section 404"
- Policy Numbers: "Policy AML-2023-001", "Procedure RISK-001-2024"

**Investment Research**
- Financial Instruments: "10-year Treasury bonds", "S&P 500 futures", "EUR/USD options"
- Financial Metrics: "price-to-earnings ratio", "debt-to-equity", "return on equity"
- Market Data: "Q3 2024 earnings", "dividend yield 3.5%", "beta coefficient"

## Step-by-Step Guide

- **Open the Unique AI settings by clicking on the edit icon**

- **Verify that the “InternalSearch” tool has “searchType” set to “COMBINED”**

## Limitations