RAG & Knowledge Base Operations

Audience: Platform administrators with chat.admin.all. These are advanced maintenance operations on a tenant's RAG stores (vector database + search index). They can run for hours on large tenants. Coordinate with Unique before running on a production tenant.

What these are

Maintenance operations let an administrator reshape or rebuild a company's Retrieval-Augmented Generation (RAG) stores without re-ingesting source files. They run as asynchronous, checkpointed background jobs and are non-destructive by default — the live collection keeps serving search until an atomic swap at the very end.

All operations share the same framework (below); each has its own runbook for the specifics.

Available operations

How maintenance jobs work (shared framework)

• Permission:chat.admin.all (Zitadel) andCONTENT:MANAGE. The companyId is taken from the authenticated admin user — you run it as an admin of the target company.

• One job per company at a time — enforced by a per-company lock. Start a new one only after the previous job is terminal (or cancel it).

• Trigger: the createMaintenanceJob(input) mutation returns a job id with status PENDING.

• Execution: a background worker claims the job within ~60s and runs it through operation-specific phases. status: PENDING → RUNNING → COMPLETED (or FAILED / CANCELLED).

• Dry run first: every operation accepts dryRun: true, which validates the request and estimates the work without changing any data. Always preview before committing.

• Checkpoint / resume: progress is persisted continuously; a worker restart resumes from the last checkpoint rather than starting over.

• Resilience: transient vector-DB connection drops and embedding rate-limit (429) errors are retried automatically.

• Non-destructive by default: with destructive: false, the old collection is kept until the new one is durable and the swap is atomic — users are unaffected if a job fails before the swap. destructive: true deletes the source up front (no rollback) — use only when you cannot keep both copies.

Monitoring (all operations)

query {
  maintenanceJob(jobId: "mjob_xxx") {
    id status phase processed total failed errorMessage startedAt completedAt result
  }
}

List all jobs for the company with maintenanceJobs { id status phase operationType processed total createdAt }. Cancel a running job with cancelMaintenanceJob(jobId: "mjob_xxx").

Common prerequisites & sizing

• Vector-DB / index capacity: non-destructive operations briefly hold two collections (source + new) — ensure the vector DB has headroom for a second copy.

• Embedding quota (re-embedding only): bounded by the embedding model's tokens-per-minute. Roughly minutes ≈ (total_chunks × avg_tokens_per_chunk) ÷ TPM.

• Per-request limits (re-embedding only): embedding providers cap tokens per request (~300,000). See the Re-embedding runbook.

• Duration: all operations scale with the company's chunk count; large tenants take hours. Plan a window and monitor.

Common troubleshooting

Runbooks

• Re-embedding (Switching the Embedding Model) — SWITCH_EMBEDDING_MODEL

• Re-sharding a Collection — RESHARD_COLLECTION

• Rebuild Index (Vector / Elasticsearch) — REBUILD_INDEX

_Each runbook covers the operation's specific parameters, phases, prerequisites, and troubleshooting. Shared mechanics (permissions, monitoring, dry run, safety) are described above.