Logging Architecture Overview — Unique AI Documentation

Logging Architecture Overview

Overview

Unique AI and its deployment models generate a variety of logs to support effective monitoring, security, and operational visibility. Understanding these logs is critical to maintaining system integrity and achieving compliance with industry standards. This document provides a detailed overview of the different types of logs produced, their sources, and how Unique AI, or in some deployment scenarios, the client, utilizes these logs for auditing and analysis.

Layer Descriptions

Before diving into the specifics of each layer in the logging architecture, it is important to understand the structure of the table used to describe the logs at every layer. Each table provides a consistent framework that outlines the key aspects of the logs generated within that layer. The table format is reused across all layers for clarity and consistency, and it includes the following columns:

Type: This column defines the type of log or the category of the activity being captured. It helps to quickly identify the nature of the log, such as security-related events, system modifications, or application-specific actions. Each layer may contain various types of logs depending on its scope and function within the system.
Description: The description provides a brief but informative explanation of what each log type entails. This includes details on the kind of events that the log records, how the log supports auditing, compliance, or troubleshooting, and its relevance to overall system monitoring. This column helps the user understand the purpose and context behind each type of log.
Data Class: Data classification is crucial for understanding the sensitivity and regulatory requirements related to the logged data. This column categorizes the data to help users manage it according to legal, security, or operational policies. For example, some logs may contain sensitive information subject to retention and access controls, while others may be less critical.
Examples: To further illustrate each log type, this column provides concrete examples of scenarios or events that would trigger the generation of a log. These examples help clarify how each log type functions in real-world situations. For instance, an example may detail how a user interaction or a system event creates a specific log entry.
Log Destination: This column specifies where the logs are stored based on the deployment model. Different deployment models – such as Unique Multi-Tenant, Unique Single-Tenant, and Customer Managed Tenant – may handle log storage and management differently. This column outlines whether logs are retained in shared environments, customer-managed infrastructure, or single-tenant setups. It also provides guidance on log retention periods and access controls.

Layers

Now that we have a clear understanding of the structure and categories used to describe logs within each layer, we can dive deeper into the specific layers of the logging architecture. Each layer plays a distinct role in the overall system and provides valuable insights through its logs. Below is an overview of the different layers and the types of logs they generate.

1. Provider Layer

This is the foundation that consists of cloud infrastructure components such as Azure, AWS, or on-premises equivalents. It manages infrastructure-level resources like identity management, authentication, and privilege activities.

Key Logs:

Type	Description	Data Class	Examples	Log destination
IDP Audit Logs	The range of Identity Providers (IDPs) is extensive, covering solutions such as Azure Entra ID, Active Directory Federation Services (ADFS), and custom implementations. In more complex environments, multiple IDPs may be deployed, including scenarios involving local Active Directory (AD) or LDAP.	C2	- Who created the user account (user@domain.com), when it was created, under whose approval, and which ticket was associated with the request? - When was the user assigned to the 'AI Engineers' team, under whose approval, and in response to which access request?	Logs are captured by Azure Activity Logs and retained for 90 days.
IDP Privilege Escalation Logs	Security best practices, such as the Principle of Least Privilege, are integral to responsible system design. These strategies often involve Just-In-Time (JIT) access or Privilege Escalation mechanisms. It is essential to log all actions associated with these mitigations for auditing and compliance purposes.	C2	- Who activated the 'Break Glass Administrator' role, when it was activated, and which linked request authorized this action? - When did user@domain.com escalate their permissions for network zone NZ1, and what was the reason for the escalation?

2. Resource Layer

This layer consists of the essential compute, network, and storage resources that support the application. It handles the infrastructure necessary to manage workload communications and operations. The Resource Layer is responsible for logging both Control Plane and Data Plane activities.

Key Logs:

Type	Description	Data Class	Examples	Log destination
Control Plane Activity Logs	All resources provide Control Plane actions. These actions are recorded in the Activity Log, detailing which user performed the action and when it occurred	C2	- Who increased the database server size from 4 CPUs to 8 CPUs? - Who modified the size of the Kubernetes Node Pool?	Logs are captured by Azure Activity Logs and retained for 90 days.
Data Plane Activity Logs	Not all resources support Data Plane actions. For those that do, the Activity Log captures which user performed the action and when it occurred.	C2	- Who, if possessing escalated privileges, accessed the key 'some-key' in the last n weeks? - Who rotated the 'rabbitmq-password' secret yesterday?

3. Kubernetes Layer

Kubernetes orchestrates the deployment and management of containerized applications. This layer captures logs that show how resources within the Kubernetes cluster are used and modified.

Key Logs:

Type	Description	Data Class	Examples	Log destination
Audit Logs	Kubernetes, including but not limited to Azure AKS, operates as its own ecosystem with distinct Control Plane and Data Plane components that generate dedicated audit logs.	C2	- Who scaled the 'node-chat' service from 2 to 4 pods, and when did this action occur? - Who deleted the Pod Disruption Budget for a specific service, and when was it done?

4. Workload Layer

The Workload Layer contains the actual applications running on Kubernetes. Logs from this layer are critical for tracking user interactions, service requests, and application performance.

Key Logs:

Type	Description	Data Class	Examples
Service Audit Logs	Unique AI provides a range of APIs, both public and internal. Audit logs meticulously document every request to ensure compliance with legal regulations and requirements.	C3	- Who listed all ingested files in a failed state, when did it happen, how was it done, and from where was the request made? - Who listed all users, including their last names, when did it occur, how was it done, and from where was the request made?
Service Application Logs	Unique AI generates standard application logs (such as debug, info, warn, and error logs, when enabled). These logs are valuable for investigating incidents and verifying application functionality.	C1	- How many database connections were created by the 'node-chat' service? - How many results were returned from the Vector Database?
IDP Audit Events	Unique utilizes an existing Identity Provider (IDP) solution, which generates its own audit events to track all actions related to authentication and authorization.	C2	- Who assigned the 'chat.admin' role to user@domain.com last week? - Who issued themselves a token, when did it happen, and potentially from which device?

5. DevOps Tooling Layer

This layer contains the tools and processes used during the development lifecycle, including version control, build systems, and deployment pipelines. Logs from this layer ensure that the development process is traceable and secure.

Key Logs:

Type	Description	Data Class	Examples
Source Control Audit Logs	Nearly all organizations, including clients and Unique, manage as much as possible through code. This includes the product, its configuration, the infrastructure configuration, and even the provider setup. As a result, access to these codebases must be properly audited.	C2	- When was a specific user invited, under whose approval, and which access request was associated with the invitation? - How frequently does a specific user log in, and from which geographical region?
Commit History	All changes to the codebase are automatically logged by the version control system.	C1	- Who modified the product to function as 'B' instead of 'A'? - Who changed the configuration for the Vector Database?
Test Automation Audit Logs	All changes to version control undergo testing in accordance with Unique's testing procedures. Logs from these tests are maintained for traceability and auditing.	C2	- When change 'X' occurred, which tests were executed and what were the results? - If tests failed but the change was still approved, who granted the exception and why?
Build Audit Logs	When a change is built (including its artifacts), the actions taken are logged for traceability and auditing purposes.	C2	- When was change 'C' built, and who triggered the build? - Which artifacts were generated to proceed further along the pipeline?
Deployment Audit Logs	Whenever a system is updated, the relevant actions are logged for traceability and auditing. This applies to both deployments and releases, even in environments where these steps are separated.	C2	- When and by whom was the deployment approved or executed, and under which change request (who triggered the automation)? - Deployments require segregated duties and sign-off, so the person who signed off is also logged.

Connection Between the Layers

Each of the layers builds on the one below it, creating a hierarchy that supports full operational visibility:

Provider Layer manages the cloud infrastructure and supports the Resource Layer.
The Resource Layer provides compute, network, and storage resources for the Kubernetes Layer to orchestrate containerized applications.
Kubernetes Layer manages the lifecycle of workloads, providing infrastructure for the Workload Layer to function.
Finally, the DevOps Tooling Layer ensures traceability and control over the entire development and deployment pipeline, connecting all other layers for comprehensive monitoring.