# SharePoint Connector - Configuration

The SharePoint Connector uses a **YAML-based tenant configuration file** for all settings. The configuration file path is specified via the `TENANT_CONFIG_PATH_PATTERN` environment variable.

## Environment Variables

The following environment variables control application-level behavior. They are set outside the tenant configuration YAML (typically in Helm `connector.env`).

| Variable | Default | Description |
| --- | --- | --- |
| `NODE_ENV` | `production` | Environment mode (`development`, `production`, `test`) |
| `PORT` | `9542` | HTTP port the application binds to |
| `LOG_LEVEL` | `info` | Log verbosity: `fatal`, `error`, `warn`, `info`, `debug`, `trace`, `silent` |
| `LOGS_DIAGNOSTICS_DATA_POLICY` | `conceal` | Controls whether sensitive data (site names, file names) is logged in full (`disclose`) or redacted (`conceal`) |
| `LOGS_DIAGNOSTICS_CONFIG_EMIT_POLICY` | `{"emit":"on","events":["on_startup","on_sync"]}` | JSON object controlling when configuration is logged. Set `emit` to `off` to disable |
| `TENANT_CONFIG_PATH_PATTERN` | — (required) | Glob pattern to tenant configuration YAML files (e.g., `/app/tenant-configs/*-tenant-config.yaml`) |
| `OTEL_METRICS_EXPORTER` | — | OpenTelemetry metrics exporter (e.g., `prometheus`) |
| `OTEL_EXPORTER_PROMETHEUS_HOST` | — | Prometheus exporter bind host |
| `OTEL_EXPORTER_PROMETHEUS_PORT` | — | Prometheus exporter bind port |
| `NODE_EXTRA_CA_CERTS` | — | Path to a PEM file containing additional CA certificates for TLS verification if pod's trust store doesn't have them |
| `HEALTH_SYNC_HISTORY_SIZE` | `5` | Number of recent sync runs kept in the sliding window for health evaluation |
| `HEALTH_SYNC_SITE_FAILURE_THRESHOLD` | `0.5` | Per-site failure ratio (0–1) across the window that marks the service unhealthy when exceeded |
| `HEALTH_CONNECTIVITY_TIMEOUT_MS` | `3000` | Timeout in milliseconds for each reachability ping used by the health endpoint |

The following environment variables are loaded from Kubernetes secrets:

| Variable | Description |
| --- | --- |
| `SHAREPOINT_AUTH_PRIVATE_KEY_PASSWORD` | Password for an encrypted certificate private key (optional, only if key is password-protected) |
| `ZITADEL_CLIENT_SECRET` | Zitadel client secret (required when `unique.serviceAuthMode` is `external`) |
| `PROXY_PASSWORD` | Proxy password (required when proxy `authMode` is `username_password`) |

## Configuration Sources

Sites can be configured in two ways:

| Source | Description | Use Case |
| --- | --- | --- |
| `config_file` | Static YAML configuration | Simple deployments, fixed site list |
| `sharepoint_list` | Dynamic configuration from SharePoint list | Self-service, frequent changes |

## Tenant Configuration File

### Static Sites Configuration (config_file)

```yaml
sharepoint:
  # ... auth and base configuration ...

# Deployment-wide defaults applied to every site below; per-site values
  # win when set. See "Site Defaults" further down for full semantics.
  siteDefaults:
    syncColumnName: FinanceGPTKnowledge
    storeInternally: enabled
    syncStatus: active
    syncMode: content_only
    permissionsInheritanceMode: inherit_scopes_and_files

sitesSource: config_file
  sites:
    # Overrides syncMode for this site; everything else inherits from siteDefaults.
    - siteId: 12345678-1234-1234-1234-123456789abc
      ingestionMode: recursive
      scopeId: scope_bu4gokr0atzj0kfiuaaaaaaa
      maxFilesToIngest: 1000
      syncMode: content_and_permissions
    # Overrides syncColumnName for this site; everything else inherits from siteDefaults.
    - siteId: 87654321-4321-4321-4321-cba987654321
      syncColumnName: HRKnowledge
      ingestionMode: flat
      scopeId: scope_bu4gokr0atzj0kfiubbbbbb
```

### Dynamic Sites Configuration (sharepoint_list)

Configure sites dynamically via a SharePoint list:

```yaml
sharepoint:
  # ... auth and base configuration ...

sitesSource: sharepoint_list
  sharepointList:
    siteId: your-config-site-id-here
    listId: 00000000-0000-0000-0000-000000000000
```

You can use the CSV import template when populating the SharePoint list for `sharepoint_list`-based configuration.

## SharePoint Base Configuration

The `sharepoint` section of the tenant YAML contains authentication and base settings that apply to all sites:

```yaml
sharepoint:
  tenantId: 12345678-1234-1234-1234-123456789012
  baseUrl: https://acme.sharepoint.com
  graphApiRateLimitPerMinuteThousands: 780
  auth:
    mode: certificate
    clientId: 00000000-0000-0000-0000-000000000000
    privateKeyPath: /app/key.pem
    thumbprintSha1: AB12CD34EF56...
```

| Option | Required | Default | Description |
| --- | --- | --- | --- |
| `tenantId` | Yes | — | Azure AD tenant ID |
| `baseUrl` | Yes | — | Company SharePoint URL (e.g., `https://acme.sharepoint.com`). Must not end with a trailing slash |
| `graphApiRateLimitPerMinuteThousands` | No | `780` | Microsoft Graph API rate limit in thousands of requests per minute |
| `auth` | Yes | — | Authentication configuration |
| `siteDefaults` | No | `{}` | Deployment-level fallbacks applied to every per-site config |

### Authentication

The connector uses certificate-based authentication (`auth.mode: certificate`):

| Option | Required | Description |
| --- | --- | --- |
| `auth.mode` | Yes | `certificate` |
| `auth.clientId` | Yes | Azure AD application client ID |
| `auth.privateKeyPath` | Yes | Path to the private key file in PEM format |
| `auth.thumbprintSha1` | One of SHA1/SHA256 required | SHA-1 thumbprint of the certificate |
| `auth.thumbprintSha256` | One of SHA1/SHA256 required | SHA-256 thumbprint of the certificate |
| `auth.privateKeyPassword` | No | Injected from `SHAREPOINT_AUTH_PRIVATE_KEY_PASSWORD` env var if the key is encrypted |

## Unique Platform Configuration

The `unique` section configures how the connector communicates with the Unique platform:

```yaml
unique:
  serviceAuthMode: cluster_local
  ingestionServiceBaseUrl: http://node-ingestion.finance-gpt:8091
  scopeManagementServiceBaseUrl: http://node-scope-management.finance-gpt:8094
  apiRateLimitPerMinute: 100
  serviceExtraHeaders:
    x-company-id: "company-id"
    x-user-id: "service-user-id"
```

| Option | Required | Default | Description |
| --- | --- | --- | --- |
| `serviceAuthMode` | Yes | — | `cluster_local` or `external` |
| `ingestionServiceBaseUrl` | Yes | — | Base URL for the Unique ingestion service |
| `scopeManagementServiceBaseUrl` | Yes | — | Base URL for the Unique scope management service |
| `apiRateLimitPerMinute` | No | `100` | Rate limit for Unique API requests per minute |
| `ingestionConfig` | No | — | Optional object passed when submitting files for ingestion |

## Proxy Configuration

The connector supports HTTP/HTTPS proxy for environments where internet access is only available through a proxy. Proxy settings are configured via environment variables (managed by the Helm chart's `proxyConfig` section).

| Mode | Description |
| --- | --- |
| `none` | Proxy disabled (default) |
| `no_auth` | Proxy enabled without authentication |
| `username_password` | Basic authentication proxy |
| `ssl_tls` | TLS client certificate proxy |

**Common options** (required for `no_auth`, `username_password`, and `ssl_tls` modes):

| Variable | Description |
| --- | --- |
| `PROXY_HOST` | Proxy server hostname |
| `PROXY_PORT` | Proxy server port |
| `PROXY_PROTOCOL` | `http` or `https` |
| `PROXY_SSL_CA_BUNDLE_PATH` | (Optional) Path to CA bundle for verifying proxy TLS certificate |
| `PROXY_HEADERS` | (Optional) JSON string of custom headers for CONNECT request |

**`username_password` mode** adds:

| Variable | Description |
| --- | --- |
| `PROXY_USERNAME` | Proxy username |
| `PROXY_PASSWORD` | Proxy password (loaded from secret) |

**`ssl_tls` mode** adds:

| Variable | Description |
| --- | --- |
| `PROXY_SSL_CERT_PATH` | Path to TLS client certificate |
| `PROXY_SSL_KEY_PATH` | Path to TLS client key |

## SharePoint List Configuration

When using `sharepoint_list` as the sites source, create a SharePoint list with the following columns. Only `siteId` is strictly required as a column on the list — any other column whose value is set via Site Defaults can be omitted from the list entirely, and rows will inherit the deployment-wide value.

| Column Display Name | Type | Description |
| --- | --- | --- |
| `siteId` | Single line text | SharePoint site ID (UUID or compound format: `hostname,siteCollectionId,webId` for subsites) |
| `syncColumnName` | Single line text | Column that marks files for sync |
| `ingestionMode` | Choice | `flat` or `recursive` |
| `uniqueScopeId` | Single line text | Unique scope ID. Either `scope_<id>` (existing root) or `in_parent:scope_<parentId>` (auto-resolve under parent). |
| `maxFilesToIngest` | Number | Maximum new + updated files per sync cycle; sync fails for the site if exceeded |
| `storeInternally` | Choice | `enabled` or `disabled` |
| `syncStatus` | Choice | `active`, `inactive`, or `deleted` |
| `syncMode` | Choice | `content_only` or `content_and_permissions` |
| `permissionsInheritanceMode` | Choice | Optional inheritance mode |
| `subsitesScan` | Choice | `enabled` or `disabled` (default: `disabled`) |

### Benefits of SharePoint List Configuration

- **Self-service**: Site owners can request sync without IT involvement
- **No redeployment**: Add/modify sites without restarting the connector
- **Audit trail**: SharePoint tracks changes to the configuration list
- **Approval workflows**: Use SharePoint approval flows for governance

## Per-Site Configuration Options

**Important:** The connector is a singleton — each SharePoint site must be configured in at most one connector process per Unique instance. Configuring the same site in multiple processes leads to conflicting state and unexpected behavior of the connector.

| Option | Values | Default | Description |
| --- | --- | --- | --- |
| `siteId` | UUID or compound ID | — (required) | SharePoint site ID. Subsites use compound format: `hostname,siteCollectionId,webId` |
| `syncColumnName` | String | `FinanceGPTKnowledge` | Display name or internal name of the sync flag column (display name takes priority) |
| `ingestionMode` | `flat`, `recursive` | — (required) | Flat ingests all to one scope; recursive maintains hierarchy |
| `scopeId` | `scope_<id>` or `in_parent:scope_<parentId>` | — (required) | Where to mount this site's content |
| `maxFilesToIngest` | Number | — (unlimited) | Maximum new + updated files per sync cycle; sync fails for the site if exceeded |
| `storeInternally` | `enabled`, `disabled` | `enabled` | Whether to store content in Unique |
| `syncStatus` | `active`, `inactive`, `deleted` | `active` | Control sync behavior |
| `syncMode` | `content_only`, `content_and_permissions` | — (required) | What to sync |
| `permissionsInheritanceMode` | `none`, `inherit_files`, `inherit_scopes`, `inherit_scopes_and_files` | `inherit_scopes_and_files` | Inheritance settings for content_only |
| `subsitesScan` | `enabled`, `disabled` | `disabled` | Recursively discover and sync content from subsites |

### Choosing between fixed scope and `in_parent:` auto-resolve

**Fixed (`scope_<id>`)** — each site needs its own scope, pre-created by the operator before the site is configured. Best when scopes are managed centrally and named or permissioned individually, since the operator stays in full control of the scope's identity, ACLs, and lifecycle.

**Auto (`in_parent:scope_<parentId>`)** — the connector finds-or-creates a child scope under the parent on every sync, named after the SharePoint site's URL slug. Removing a site (via `syncStatus: deleted`) removes the auto-created scope. If a sibling scope under the parent already has the same site name and isn't claimed by us, the connector aborts the sync with a typed error rather than guessing to stay on the safe side and not sync a site into a user folder.

### Permissions Inheritance Modes

Only used when `syncMode` is `content_only`. It controls whether newly created scopes / files inherit permissions from their parent. If scopes / files are configured to not inherit permissions, any newly created scopes / files will not be visible to platform users, only to service user. To grant access to these new scopes / files, admin has to use API on behalf of the service user.

| Mode | Scopes Inherit | Files Inherit |
| --- | --- | --- |
| `inherit_scopes_and_files` | Yes | Yes |
| `inherit_scopes` | Yes | No |
| `inherit_files` | No | Yes |
| `none` | No | No |

## Site Defaults

`sharepoint.siteDefaults` lets you set deployment-level fallbacks for any per-site option except `siteId`. Each site (whether sourced from `config_file` or from a `sharepoint_list` row) is merged with the defaults: if the per-site value is set, it wins; otherwise the default is used. This keeps individual site entries terse and makes it easy to change a policy across an entire deployment in one place.

### With `config_file`

```yaml
sharepoint:
  # ... auth and base configuration ...

siteDefaults:
    syncColumnName: FinanceGPTKnowledge
    ingestionMode: recursive
    storeInternally: enabled
    syncStatus: active
    syncMode: content_only
    permissionsInheritanceMode: inherit_scopes_and_files
    subsitesScan: disabled

sitesSource: config_file
  sites:
    # Inherits everything from siteDefaults except scopeId / maxFilesToIngest
    - siteId: 12345678-1234-1234-1234-123456789abc
      scopeId: scope_bu4gokr0atzj0kfiuaaaaaaa
      maxFilesToIngest: 1000
    # Overrides syncColumnName and ingestionMode for this site
    - siteId: 87654321-4321-4321-4321-cba987654321
      syncColumnName: HRKnowledge
      ingestionMode: flat
      scopeId: scope_bu4gokr0atzj0kfiubbbbbb
```

### With `sharepoint_list`

`siteDefaults` works identically for `sharepoint_list`: any column whose value is set on a row wins; any column that is blank falls back to the default. This means **columns covered by `siteDefaults` can be omitted from the SharePoint list entirely** — only the columns you want to vary per row need to exist. At minimum, the list must carry `siteId`; everything else can live in `siteDefaults`.

```yaml
sharepoint:
  # ... auth and base configuration ...

siteDefaults:
    syncColumnName: FinanceGPTKnowledge
    ingestionMode: recursive
    storeInternally: enabled
    syncStatus: active
    syncMode: content_only
    permissionsInheritanceMode: inherit_scopes_and_files
    subsitesScan: disabled
    # Common pattern: every site auto-creates a child under one shared parent scope,
    # so the list does not need a `uniqueScopeId` column at all.
    scopeId: in_parent:scope_bu4gokr0atzj0kfiucccccc

sitesSource: sharepoint_list
  sharepointList:
    siteId: your-config-site-id-here
    listId: 00000000-0000-0000-0000-000000000000
```

With the example above, the SharePoint list can be reduced to a single `siteId` column — every other per-site field is supplied by `siteDefaults`. Add columns back to the list only when you need per-site overrides for those fields.

### Merge Rules

- **Per-site value wins when "set".** For string-typed fields (including `siteId`, `scopeId`, `syncColumnName`), "set" means non-`undefined` and non-empty after trim — so a blank cell in a SharePoint list row falls back to the default. For numeric/enum fields, any non-`undefined` value counts as set.

- **Required-after-merge.**`ingestionMode`, `scopeId`, and `syncMode` are required on the final merged config. If a per-site entry omits them and `siteDefaults` does not supply them either, the merger throws — and because sites are merged eagerly at the start of every sync cycle, a single unmergeable row aborts the **entire sync cycle**.

- **`siteId` cannot be defaulted.** It must always be set per site.

### SharePoint Site Configuration

### Finding SharePoint Site IDs

Site IDs are required to configure which SharePoint sites the connector scans. The connector supports both `/sites/` and `/teams/` managed paths.

**Via Browser:**

Navigate to: `https://{tenant}.sharepoint.com/sites/your-site/_api/site/id` (or `/teams/your-team/_api/site/id` for team sites)

**Via Microsoft Graph Explorer:**

```none
GET https://graph.microsoft.com/v1.0/sites/{tenant}.sharepoint.com:/sites/{site}
```

**For nested subsites**, extend the path:

```none
GET https://graph.microsoft.com/v1.0/sites/{tenant}.sharepoint.com:/sites/{parentSite}/{subsite}/{nestedSubsite}
```

## Subsites Scanning

### Overview

When `subsitesScan` is set to `enabled` for a site, the connector recursively discovers all subsites under that site and syncs their content alongside the parent site's content. This means you only need to configure the top-level site — all nested subsites are discovered and included automatically.

### How It Works

1. **Discovery** — During each sync cycle, the connector calls the Graph API (`GET /sites/{siteId}/sites`) to list direct child subsites.

2. **Content fetching** — For each discovered subsite, the connector fetches document libraries and site pages using the same `syncColumnName` as the parent site.

3. **Scope hierarchy** — Subsite content is ingested under the parent site's scope tree.

4. **File diff** — Subsite items are keyed under the parent site's ID in the file-diff mechanism.

### Configuring Document Libraries for Sync

Users mark documents for sync by setting a specific column in the SharePoint document library. The connector picks up flagged files on the next scan cycle.

## Processing Configuration

The `processing` section of the tenant configuration file controls file processing behavior:

```yaml
processing:
  stepTimeoutSeconds: 30
  concurrency: 1
  maxFileSizeToIngestBytes: 209715200
  allowedMimeTypes:
    - application/pdf
    - application/vnd.openxmlformats-officedocument.wordprocessingml.document
    - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
    - application/vnd.openxmlformats-officedocument.presentationml.presentation
    - application/x-asp
    - text/plain
    - text/html
    - text/csv
  mimeTypeOverridesByExtension:
    .csv: text/csv
  scanIntervalCron: "*/15 * * * *"
```

## Supported File Types

Configure allowed types via the `allowedMimeTypes` processing option.

### Logging

The connector produces structured JSON logs:

```json
{
  "timestamp": "2024-01-15T10:30:00.000Z",
  "level": "info",
  "message": "Sync cycle started",
  "traceId": "abc123",
  "siteId": "xxx-xxx-xxx"
}
```

### Health Endpoint

The connector exposes a `GET /health` endpoint that reports operational health. It is separate from the existing `GET /probe` endpoint used for K8s liveness/readiness probes.

### Response Examples

**Healthy (200):**

```json
{
  "status": "ok",
  "info": {
    "sync": {
      "status": "up",
      "lastSyncAt": "2026-03-18T10:15:00.000Z",
      "recentSyncs": 5,
      "sites": {
        "site-aaa": { "failures": 0, "total": 5 },
        "site-bbb": { "failures": 1, "total": 5 }
      }
    },
    "connectivity": {
      "status": "up",
      "graph": "reachable",
      "sharepoint": [
        { "tenant": "default", "status": "reachable" }
      ]
    },
    "uniqueApi": {
      "status": "up",
      "ingestion": "reachable",
      "scopeManagement": "reachable"
    }
  },
  "error": {},
  "details": { "...same as info when healthy..." }
}
```

**Unhealthy (503):**

```json
{
  "status": "error",
  "info": {
    "connectivity": { "status": "up", "..." : "..." },
    "uniqueApi": { "status": "up", "..." : "..." }
  },
  "error": {
    "sync": {
      "status": "down",
      "lastSyncAt": "2026-03-18T10:15:00.000Z",
      "threshold": 0.5,
      "failingSites": ["site-bbb"],
      "sites": {
        "site-aaa": { "failures": 0, "total": 5 },
        "site-bbb": { "failures": 4, "total": 5 }
      }
    }
  },
  "details": { "...all checks combined..." }
}
```

### Complete Re-ingestion

To perform a complete re-ingestion of all synced SharePoint content, follow the outlined steps to pause the connector, delete content, and re-enable it.