# Seed Parameter Benchmarking Report

### **What is a Seed Parameter?**

A seed parameter is an initial value used to initialize the random number generator in computational processes, ensuring reproducibility of results by generating the same sequence of random numbers each time. In the context of Large Language Models (LLMs), a seed parameter can be used to control the randomness in tasks such as text generation, allowing for consistent outputs across multiple runs with the same input and seed. By setting a seed, developers can experiment with different model behaviors while maintaining the ability to replicate specific outcomes for analysis or demonstration purposes.

This will be set automatically; customers do not need to do anything to activate it.

### **Purpose**

This benchmarking report evaluates the impact of using a seed parameter in GPT-4o responses. Both versions use the same configuration, except for the seed:

- **Version A**: GPT-4o using a 30,000-token context window with **no seed**  
- **Version B**: GPT-4o using a 30,000-token context window with **a fixed seed**

The goal is to assess how the seed affects consistency when generating responses to the user prompts. Each version was evaluated across identical input and retrieval conditions.

### **Performance Overview – Model Consistency Check**

| Metric (rounded) | No Seed (A) | With Seed (B) |
| --- | --- | --- |
| One or more answers differ in actual meaning | 0.7% | 0.00% |
| Obvious differences, but same meaning | 69% | 62% |
| Very slight difference (e.g., word choice) | 9% | 11% |
| Identical answers (LLM choice of source can vary) | 20% | 27% |

This evaluation measures how consistent each model version is across multiple runs.

While Version B (with a fixed seed) significantly reduced major inconsistencies, it still showed some variation in structure and content.

Version A (no seed) displayed broader differences in phrasing and layout.

### **Example**

**Question: “What regulatory body supervises the fund manager?”**

- _No Seed (A) – Run 1:_ Lists regulatory framework and includes citation.
- _No Seed (A) – Run 2:_ Omits citation, uses different sentence structure.
- _With Seed (B):_ Maintains consistent structure, but varied emphasis on secondary authorities.

### **Conclusion**

Using a seed reduces randomness but does not guarantee deterministic output. While it helps remove major inconsistencies (e.g., contradictions), variations in phrasing, formatting, and detail still appear — especially in long-form or document-based queries.

### **RAG Configuration Details**

**Space used:**

Key parameters:

- **Context window**: 30,000 tokens
- **Chunk relevancy sorting**:
- **Search method**:
- **LLM seed**: Enabled only for version B
- **Temperature**: