File Ingestion — Unique AI Documentation
File Ingestion
Overview
The ingestion service powers Unique's knowledge base by converting uploaded files into vector embeddings for use by the platform's LLMs. It supports various file types, including PDF, Word, Excel, and others with flexible ingestion methods to meet diverse needs. Administrators can use different ingestion methods and customize ingestion settings and chunking strategies for each knowledge base folder, optimizing processing for specific document sets.
Who is it for
This documentation focuses specifically on administrators who need to manage and configure advanced ingestion capabilities for their organizations. Administrators who require granular control over document processing, cost management, and optimization of search results will benefit most from understanding the comprehensive ingestion options, customization features available within the platform.
Permission Requirements: Users must have the Can manage permission for a folder to access its ingestions configuration. Only users with the role knowledge.write can be permissioned the can manage role.
Benefits and Use Cases
The ingestion service processes diverse company documents through a unified platform, offering flexible customization to optimize results. Folder-level settings allow organizations to tailor processing methods to specific document types, improving search accuracy and relevance.
By combining the default service, specialized MDI tools, and emerging vision-enhanced features, the platform delivers scalable, high-quality solutions that adapt to evolving document management needs.
The Unique platform supports multiple services for ingesting PDF documents:
- Unique Ingestion
- Docling
- Microsoft Document Intelligence (MDI)
- MDI with Image Content Extraction
Ingestion Overview, Default and Recommendations
Each service can parse structured PDFs with a single-column layout and extract simple tables. However, their capabilities vary when handling more complex documents:
- Image-based PDFs: Scanned or printed PDFs lack structured content, requiring OCR techniques for extraction.
- Multi-Column Layout: PDFs with multiple columns, charts, tables, and text need pre-trained layout detection models to identify page elements and preserve logical content flow.
- Complex Tables Detection: Extracting tables with merged cells, missing borders, or checkmarks requires specialized AI models to recognize different table components.
- Image Content Extraction: Many PDFs contain unstructured visual elements like charts, logos, or photos. AI models with image-to-text capabilities are needed to extract this content in a searchable form.
- On-Prem Deployment: The service can operate in a closed environment without internet access.
| Ingestion service | Capabilities | Available Regions | Performance |
| Image-based PDFs | Multi-Column Layouts | Complex Tables Detection | Image Content Extraction | On-Prem Deployment |
| Base Unique Ingestion | ❌ | ❌ | ❌ | ❌ | ✅ | All regions | 10-15s per page |
| Docling | 🟡 | ✅ | 🟡 | ❌ | ✅ | All regions | 10-20s per page |
| MDI | ✅ | ✅ | ✅ | ❌ | ❌ | Check here for Azure AI Document Intelligence | 10-20s per page |
| MDI with Image Content ExtractionEXPERIMENTAL | ✅ | ✅ | ✅ | ✅ | ✅
(MDI on-prem) | Check here for Azure AI Document Intelligence | 20-30s per page |
✅ - fully supported 🟡 - partially supported ❌ - not supported
Why we recommend MDI or Docling?
Both MDI and Docling offer meaningful improvements over the Base Unique Ingestion service, which lacks support for image-based PDFs, multi-column layouts, complex tables, and image content extraction. The choice between the two depends on your deployment context and document complexity:
MDI (Microsoft Document Intelligence) is the recommended default for cloud deployments. It is built on Azure AI's document processing models and delivers superior accuracy — particularly for complex tables without grid lines, irregular layouts, and image-based PDFs requiring OCR. Its layout detection is more robust and better suited to enterprise-grade document diversity. For use cases where document fidelity and extraction precision are critical, MDI consistently outperforms Docling. MDI also supports image content extraction when used in its extended configuration (MDI with Image Content Extraction), albeit at a higher processing cost (20–30s per page).
Docling is a strong alternative, especially for on-premises deployments. It supports multi-column layouts and offers partial support for image-based PDFs and complex tables. While its accuracy on borderless tables and scanned content is not as high as MDI, it is a capable, self-hostable solution that runs fully within your infrastructure — with no dependency on external cloud services. This makes it the preferred choice when data residency, air-gapped environments, or strict compliance requirements apply.
Recommendation
On-Prem Customers: Both Docling and MDI are now viable options for on-premises deployment. Docling can be self-hosted directly within your infrastructure with no external dependencies. MDI also supports on-premises deployment via Docker containers, which can be run on your own hardware, Azure Kubernetes Service (AKS), Azure Container Instances, or a Kubernetes cluster deployed to Azure Stack. If your documents include complex tables or image-based PDFs and your infrastructure supports container workloads, MDI on-prem is the higher-accuracy choice. For simpler setups or fully air-gapped environments, Docling remains a reliable fallback.
Cloud Customers: Use MDI as the default ingestion service. It provides higher accuracy than Docling, particularly for tables without grid lines, scanned documents, and complex multi-column layouts. For documents containing charts, photos, or other visual elements that need to be made searchable, use MDI with Image Content Extraction (note: this is currently experimental and carries a higher per-page processing time of 20–30s).
Step-by-Step Guide
Step 1: Open the Ingestion Configuration
Navigate to the folder in the knowledge base that you want to review or set up for ingestion. Then, click the ‘ Configure Files Ingestion’ button located on the right.
Step 2: Change the Ingestion Configuration
PDFs on Unique are ingested page by page.
There are 2 modes of ingesting documents as it is implemented:
PDFTODOCX_ONLY: Use our default library: PDFs are converted using pdf2docx (default)DOC_INTELLIGENCE_DEFAULT: Use MDI on all pages of the document
Word
The default process directly extracts the content of a Word file, including text and tables with their underlying formatting. However, it does not extract content from images (e.g., if a table is embedded as an image in the Word file). There is an option to use the MDI service for Word files, which can also extract text from images. This process first converts the Word file to a PDF to utilize the full capabilities of the MDI service:
WORD_DEFAULT_INGESTION: Use the default Word ingestion mechanism (without MDI)INGEST_WORD_AS_PDF: Convert the Word document to PDF and use the PDF ingestion service on the Word document
Enable MDI
The default pipeline currently in place may not adequately process certain PDF and Word documents, particularly when encountering improperly formatted data (e.g., tables in financial documents, images with text).
Microsoft Document Intelligence (MDI ff.) can enhance Unique's capability to accurately ingest documents that contain complex tables and graphics. The latest GA version 2024-11-30 of Microsoft’s Document Intelligence is used to ingest documents.
Enable Agentic Document Ingestion
This service is deprecated and no longer supported. The configuration should be removed as early as possible.
Tips & Tricks
Improving Document Ingestion Quality
Converting a document from PDF to a format that can be read by large language models (LLMs), such as Markdown, is a critical step for ensuring the high quality of outputs from Unique AI. Maintaining the integrity of text structures, especially those within tables and complex structures, is a particularly challenging task that is currently the focus of extensive research.
Layout
Structured Data: Text should have clear headings, subheadings, bullet points or numbers. Clear and distinct paragraphs focused on a single topic with consistent spacing between lines and paragraphs.
Footnotes: The link between footnote and usage in paragraph can get lost. Include information in paragraphs if possible.
Links: Avoid links to other paragraphs as this connection might not be understood
Multi Columns: Avoid multi-column layouts if possible.
Remove Noise: Footnotes, page numbers, headers, and footers should be eliminated if they do not contribute to the content’s meaning.
Ingestion Error Codes
Please always check the ingestion state in the knowledge center to see if the file ingestion has been successful or not.
| Display Name | Error State | Description |
|---|---|---|
| Ingestion failed (General Error) | FAILED | Generic failure state when the ingestion process encounters an unspecified error |
| Ingestion failed (Images are not supported) | FAILED_IMAGE | File ingestion failed because the uploaded file is an image format, which is not supported for text processing |
| Ingestion failed (While creating chunks) | FAILED_CREATING_CHUNKS | Error occurred during the chunking phase where the document content is split into smaller segments |
| Ingestion failed (While creating embedding) | FAILED_EMBEDDING | Failure during the embedding generation process where text is converted to vector representations |
| Ingestion failed (While fetching the file) | FAILED_GETTING_FILE | Error retrieving the file from its source location during the ingestion process |
| Ingestion failed (While parsing the text from the document) | FAILED_PARSING | Document parsing failed, unable to extract readable text content from the file |
| Ingestion failed (General error or time limit exceeded) | FAILED_REDELIVERED | Ingestion failed after retry attempts, either due to persistent errors or timeout |
| Ingestion failed (Could not parse a lot of text. Document might have no meaning) | FAILED_TOO_LESS_CONTENT | Document contains insufficient meaningful text content for successful ingestion |
| Rejected by malware scanner | FAILED_MALWARE_FOUND | File was rejected during security scanning due to detected malware or suspicious content |
| Metadata validation failed | FAILED_METADATA_VALIDATION | File metadata does not meet validation requirements or contains invalid data, e.g., it did not contain a required sensitivity label. |
Limitations
Take into account price variations across different ingestion methods.