# Re-embedding a Knowledge Base

## Re-embedding a Knowledge Base (Switching the Embedding Model)

Audience: Unique Solution Engineering (running on customer-managed tenants) and platform administrators with `chat.admin.all`. This is an **advanced maintenance operation** that re-processes a tenant's entire vector store and can run for **hours** on large tenants. Coordinate with Unique before running on a production tenant.

## Overview

**Re-embedding** regenerates the vector embeddings for **all** of a company's ingested content using a different embedding model, then atomically switches the company's live vector collection to the new model. Use it when you need to:

- Upgrade to a higher-quality / higher-dimensional embedding model (e.g. `text-embedding-ada-002` → `text-embedding-3-large`).
- Change the embedding dimension of a tenant's vector store.
- Re-baseline embeddings after a model or provider change.

**What it does under the hood** (operation type `SWITCH_EMBEDDING_MODEL`):

1. **preparing** — creates a new, temporary vector collection sized for the new model's dimension.
2. **reembedding** — streams every chunk, re-embeds it with the new model, and writes the vectors into the new collection (in batches).
3. **swap** — once the new collection is complete, atomically points the company's live collection at it and updates the relational store.
4. **cleanup** — removes the old collection.

The job is **checkpointed** (a worker restart resumes from the last position) and **non-destructive by default**: the old collection and live search keep working throughout the run, and the swap only happens at the very end.

## Who can run it

- **Permission:**`chat.admin.all` (Zitadel) **and**`CONTENT:MANAGE`.
- **Scope:** the `companyId` is taken from the **authenticated admin user** — you run it as an admin _of the company being re-embedded_.

## Before you start (prerequisites)

| # | Prerequisite | Why it matters |
| --- | --- | --- |
| 1 | **Target model is registered & deployed for the tenant** | The new embedding model must be available in the tenant's embedding configuration. Confirm with Unique which model keys are available. |
| 2 | **Sufficient embedding quota (TPM/RPM)** | Re-embedding pushes the _entire_ corpus through the model. Low quotas make large tenants impractical (jobs spend their time rate-limited). See **Sizing**. |
| 3 | **Per-request token limit respected** | Embedding providers cap tokens per request (commonly **300,000 tokens/request**). `batch size × average chunk tokens` must stay under that cap or requests fail with a `400`. See **Configuration**. |
| 4 | **Run a dry run first** | Always preview with `dryRun: true` to validate the plan and get a chunk-count estimate before committing. |
| 5 | **Plan for duration** | Large tenants take **hours**. Plan an overnight/weekend window and monitor. |

The operation is resilient to concurrent ingestion and to transient vector-DB / rate-limit errors (they are retried automatically). For the cleanest, fastest run you may still choose to pause heavy ingestion for the company during the switch.

## Step 1 — Dry run (preview, makes no changes)

```graphql
mutation {
  createMaintenanceJob(input: {
    operationType: "SWITCH_EMBEDDING_MODEL"
    newEmbeddingModel: "text-embedding-3-large"
    newEmbeddingDimension: 3072
    dryRun: true
  }) { id status }
}
```

A dry run validates the request, estimates the number of chunks to process, and completes **without changing any data**. Read the job back (Step 3) — its `result` contains the plan, including `estimatedChunks`, the current collection, and the target model/dimension. Use the estimate to size the run (see **Sizing**).

## Step 2 — Run the switch

```graphql
mutation {
  createMaintenanceJob(input: {
    operationType: "SWITCH_EMBEDDING_MODEL"
    newEmbeddingModel: "text-embedding-3-large"
    newEmbeddingDimension: 3072
    destructive: false
    dryRun: false
  }) { id status createdAt }
}
```

**Input fields**

| Field | Required | Description |
| --- | --- | --- |
| `operationType` | ✅ | "SWITCH_EMBEDDING_MODEL" |
| `newEmbeddingModel` | ✅ | The model key as registered for the tenant (e.g. `text-embedding-3-large`). |
| `newEmbeddingDimension` | ✅ | The vector dimension of the new model (e.g. `3072` for `text-embedding-3-large`). **Must match the model's actual output dimension.** |
| `destructive` | ➖ (default `false`) | `false` = keep the old collection until the new one is durable ( **recommended**, allows rollback). `true` = delete the old collection up front (no rollback). |
| `dryRun` | ➖ (default `false`) | `true` = preview only (Step 1). |

**Invoking it (cURL)** — authenticate with a `chat.admin.all` bearer token; `companyId` is derived from the token:

```bash
curl -X POST "$INGESTION_GRAPHQL_URL" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"query":"mutation { createMaintenanceJob(input:{operationType:\"SWITCH_EMBEDDING_MODEL\",newEmbeddingModel:\"text-embedding-3-large\",newEmbeddingDimension:3072,destructive:false,dryRun:false}){ id status } }"}'
```

`$INGESTION_GRAPHQL_URL` is the tenant's ingestion GraphQL endpoint and `$ADMIN_TOKEN` is a Zitadel token for a chat.admin.all user — ask Unique for the endpoint if you don't have it. The mutation returns the new job's id.

## Step 3 — Monitor

A background worker claims the job within ~60 seconds of creation. Track it with:

```graphql
query {
  maintenanceJob(jobId: "mjob_xxx") {
    id status phase processed total failed errorMessage startedAt completedAt result
  }
}
```

List all jobs for the company with `maintenanceJobs { id status phase processed total operationType createdAt }`.

| Field | Meaning |
| --- | --- |
| `status` | `PENDING` → `RUNNING` → `COMPLETED` (or `FAILED` / `CANCELLED`). |
| `phase` | `preparing` → `reembedding` → `swap` → `cleanup`. |
| `processed` / `total` | Progress (chunks re-embedded vs. estimated total). |
| `failed` | Per-item failures (non-fatal). |
| `errorMessage` | Populated when `status = FAILED`. |

**Cancel** a running job with `cancelMaintenanceJob(jobId: "mjob_xxx")`.

## Sizing & duration

Throughput is bounded by the embedding model's **tokens-per-minute (TPM)** quota:

```none
estimated minutes  ≈  (total_chunks × avg_tokens_per_chunk) ÷ TPM_quota
chunks per minute  ≈  TPM_quota ÷ avg_tokens_per_chunk
```

**Worked example:** ~750,000 TPM quota, ~400 tokens/chunk → ~1,800 chunks/min → a **1.9M-chunk** tenant ≈ **~17 hours**. A few-thousand-TPM quota would make the same tenant take many days — raise the quota for large tenants.

> Re-embedding is processed sequentially (one batch at a time). The single biggest lever on duration is the embedding deployment's TPM quota.

## Safety, atomicity & rollback

- **Non-destructive by default:** users keep searching on the existing model for the entire run; the switch is atomic at the very end. If the job fails **before** the swap, nothing changes for users.
- **Checkpoint / resume:** progress is recorded continuously; a worker restart resumes from the last checkpoint rather than starting over.
- `destructive: true` deletes the old collection _up front_ — only use it when you cannot keep both collections and you accept that there is no rollback.
- **Rollback after a completed switch:** run the switch again, targeting the previous model and dimension (this is a full re-embed back to the old model).

## Troubleshooting

| Symptom (in `errorMessage` / logs) | Cause | Resolution |
| --- | --- | --- |
| `429 ... exceeded call rate limit ... S0 pricing tier` | Embedding deployment quota too low | Raise the model deployment's TPM/RPM quota, then re-run. Transient 429s are retried automatically; a _persistently_ throttled deployment will still exhaust retries. |
| `400 Invalid 'input': maximum request size is 300000 tokens per request` | Batch is larger than the provider's per-request token cap | Reduce the batch size (`MAINTENANCE_OPS_CHUNK_BATCH_SIZE`), e.g. `512 → 128`. Re-run. |
| Transient vector-DB connection drops (`EPIPE`, socket errors, "Insert into qdrant did not work") | Network / keep-alive blips to the vector DB | Retried automatically in current versions. If it persists, check vector-DB health/capacity. |
| Dimension mismatch on upsert | `newEmbeddingDimension` ≠ the model's real output dimension | Set `newEmbeddingDimension` to the model's actual dimension (e.g. 3072 for `text-embedding-3-large`). |
| `A maintenance job for this company is still holding the lock` | A job is active or recently finished (one job per company at a time) | Wait for it to finish, or `cancelMaintenanceJob`, then retry. |
| Job stays `PENDING` | No worker picked it up | Ensure the ingestion / maintenance worker is running. |

## Configuration reference

| Setting | Default | Purpose |
| --- | --- | --- |
| `MAINTENANCE_OPS_CHUNK_BATCH_SIZE` | `512` | Number of chunks per embedding request. Lower it if `batch × avg-chunk-tokens` approaches the provider's per-request token cap (≈300k). |
| Per-company embedding model registration | — | The new model must be registered/available for the tenant before switching. |

**Note on multi-endpoint embedding distribution:** in environments where embedding requests are distributed across multiple endpoints, confirm that **per-company model overrides are honored** for the target company before running — otherwise the re-embed may use the default model. Check with Unique if your tenant uses multi-endpoint distribution.

## Known limitations

- **One maintenance job per company at a time** (enforced by a per-company lock).
- **A**`FAILED` **job is not auto-resumed** — re-trigger it to restart (it re-embeds into a fresh collection; the partial collection from the failed run is garbage-collected).
- **Runtime scales with corpus size and embedding quota** — large tenants take hours; plan accordingly and monitor.