# RAG & Knowledge Base Operations

**Audience**: Platform administrators with `chat.admin.all`. These are **advanced maintenance operations** on a tenant's RAG stores (vector database + search index). They can run for **hours** on large tenants. Coordinate with Unique before running on a production tenant.

## What these are

Maintenance operations let an administrator **reshape or rebuild a company's Retrieval-Augmented Generation (RAG) stores without re-ingesting source files**. They run as **asynchronous, checkpointed background jobs** and are **non-destructive by default** — the live collection keeps serving search until an atomic swap at the very end.

All operations share the same framework (below); each has its own runbook for the specifics.

## Available operations

| Operation | `operationType` | What it does | When to use |
| --- | --- | --- | --- |
| [**Re-embedding**](https://docs.unique.ai/administrators/knowledge-base-for-admins/rag-knowledge-base-operations/re-embedding-a-knowledge-base) | `SWITCH_EMBEDDING_MODEL` | Re-embeds all content with a **different embedding model** and switches the live vector collection to it | Upgrading or changing the embedding model / dimension |
| [**Re-sharding a Collection**](https://docs.unique.ai/administrators/knowledge-base-for-admins/rag-knowledge-base-operations/re-sharding-a-collection) | `RESHARD_COLLECTION` | Recreates the company's vector collection with a **different shard count** and swaps to it (copies vectors, no re-embed) | Scaling a vector collection for size / query performance |
| [**Rebuild Index**](https://docs.unique.ai/administrators/knowledge-base-for-admins/rag-knowledge-base-operations/rebuild-index-vector-elasticsearch) | `REBUILD_INDEX` | Rebuilds the **vector index and/or the Elasticsearch full-text index** from existing data | Recovering from index issues, or applying new index settings |

* * *

## How maintenance jobs work (shared framework)

- **Permission:**`chat.admin.all` (Zitadel) **and**`CONTENT:MANAGE`. The `companyId` is taken from the authenticated admin user — you run it as an admin _of the target company_.

- **One job per company at a time** — enforced by a per-company lock. Start a new one only after the previous job is terminal (or cancel it).

- **Trigger:** the `createMaintenanceJob(input)` mutation returns a job `id` with status `PENDING`.

- **Execution:** a background worker claims the job within ~60s and runs it through operation-specific phases. `status`: `PENDING → RUNNING → COMPLETED` (or `FAILED` / `CANCELLED`).

- **Dry run first:** every operation accepts `dryRun: true`, which validates the request and estimates the work **without changing any data**. Always preview before committing.

- **Checkpoint / resume:** progress is persisted continuously; a worker restart resumes from the last checkpoint rather than starting over.

- **Resilience:** transient vector-DB connection drops and embedding rate-limit (`429`) errors are retried automatically.

- **Non-destructive by default:** with `destructive: false`, the old collection is kept until the new one is durable and the swap is atomic — users are unaffected if a job fails before the swap. `destructive: true` deletes the source up front (no rollback) — use only when you cannot keep both copies.

### Monitoring (all operations)

```graphql
query {
  maintenanceJob(jobId: "mjob_xxx") {
    id status phase processed total failed errorMessage startedAt completedAt result
  }
}
```
List all jobs for the company with `maintenanceJobs { id status phase operationType processed total createdAt }`. Cancel a running job with `cancelMaintenanceJob(jobId: "mjob_xxx")`.

| Field | Meaning |
| --- | --- |
| `status` | `PENDING → RUNNING → COMPLETED` (or `FAILED` / `CANCELLED`) |
| `phase` | Operation-specific (see each runbook) |
| `processed` / `total` | Progress |
| `failed` | Per-item failures (non-fatal) |
| `errorMessage` | Populated on `FAILED` |

### Common prerequisites & sizing

- **Vector-DB / index capacity:** non-destructive operations briefly hold **two collections** (source + new) — ensure the vector DB has headroom for a second copy.

- **Embedding quota** _(re-embedding only):_ bounded by the embedding model's tokens-per-minute. Roughly `minutes ≈ (total_chunks × avg_tokens_per_chunk) ÷ TPM`.

- **Per-request limits** _(re-embedding only):_ embedding providers cap tokens per request (~300,000). See the Re-embedding runbook.

- **Duration:** all operations scale with the company's chunk count; large tenants take hours. Plan a window and monitor.

### Common troubleshooting

| Symptom | Cause | Resolution |
| --- | --- | --- |
| `A maintenance job for this company is still holding the lock` | A job is active or recently finished | Wait, or `cancelMaintenanceJob`, then retry (one job per company) |
| Job stays `PENDING` | No worker picked it up | Ensure the ingestion / maintenance worker is running |
| Transient vector-DB errors (`EPIPE`, socket) | Network / keep-alive blips | Retried automatically; if persistent, check vector-DB health/capacity |
| Job `FAILED` partway | Operation-specific error | Read `errorMessage`; see the operation's runbook. Re-trigger to restart (jobs do not auto-resume after `FAILED`) |

* * *

## Runbooks

- [**Re-embedding (Switching the Embedding Model)**](https://docs.unique.ai/administrators/knowledge-base-for-admins/rag-knowledge-base-operations/re-embedding-a-knowledge-base) — `SWITCH_EMBEDDING_MODEL`

- [**Re-sharding a Collection**](https://docs.unique.ai/administrators/knowledge-base-for-admins/rag-knowledge-base-operations/re-sharding-a-collection) — `RESHARD_COLLECTION`

- [**Rebuild Index (Vector / Elasticsearch)**](https://docs.unique.ai/administrators/knowledge-base-for-admins/rag-knowledge-base-operations/rebuild-index-vector-elasticsearch) — `REBUILD_INDEX`

_Each runbook covers the operation's specific parameters, phases, prerequisites, and troubleshooting. Shared mechanics (permissions, monitoring, dry run, safety) are described above.