Custom API Search

2 min read

Overview

Custom API Search enables direct integration with proprietary or internally managed search APIs. It acts as a lightweight adapter over any HTTP-based search service that can return results in the required response format.

This provider is intended for teams that already have their own search infrastructure, such as internal knowledge systems, licensed datasets, or domain-specific indexes and want to expose those results through a unified search interface.

It supports both GET and POST request methods, custom headers, authentication schemes, and flexible parameter placement (query string or request body). This makes it suitable for APIs with non-standard authentication or request structures.

Custom API Search does not impose crawling, indexing, or ranking behavior. It relies entirely on the upstream API for search logic, making it appropriate for domain-specific or vertical search engines. Scraping can be enabled or disabled depending on the capabilities of the underlying service.

Features

• Support for any compatible search API
• GET and POST request methods
• Flexible parameter configuration
• Custom headers and authentication
• Configurable request timeout
• Supports additional web scraping

API Response Specification

Your custom search API MUST return responses in the following JSON format:

{
  "results": [
    {
      "url": "https://example.com/page",
      "title": "Page Title",
      "snippet": "Brief description",
      "content": "Full page content"
    }
  ]
}

Field Definitions

Environment Variables

The following environment variables define parameters used to construct the request payload for the Custom API Search provider.

These parameters can be configured either through the Space configuration (UI) or via environment variables.

If a parameter is defined using an environment variable, the corresponding option in the UI will be hidden and the environment variable value will take precedence.

CUSTOM_WEB_SEARCH_API_ENDPOINT=https://your-api.example.com/search
CUSTOM_WEB_SEARCH_API_METHOD=POST  # GET or POST
CUSTOM_WEB_SEARCH_API_HEADERS='{"Authorization": "Bearer YOUR_KEY"}'
CUSTOM_WEB_SEARCH_API_ADDITIONAL_QUERY_PARAMS='{"api_version": "v2", "format": "json"}'
CUSTOM_WEB_SEARCH_API_ADDITIONAL_BODY_PARAMS='{"lang": "en", "safe": true}'

Variable Descriptions

| Variable | Description | | CUSTOM_WEB_SEARCH_API_ENDPOINT | Base URL of the search API | | CUSTOM_WEB_SEARCH_API_METHOD | HTTP method (GET or POST) | | CUSTOM_WEB_SEARCH_API_HEADERS | JSON-encoded HTTP headers | | CUSTOM_WEB_SEARCH_API_ADDITIONAL_QUERY_PARAMS | Extra query parameters appended to the URL | | CUSTOM_WEB_SEARCH_API_ADDITIONAL_BODY_PARAMS | Extra parameters included in the request body (POST only) |

Scraping

Admins can configure whether the search engine requires an additional scraping step.

If the provider’s endpoint returns the full page content, no extra scraping is needed as the agent will work with the content.

If the endpoint returns only partial data (e.g. only snippets or abstracts), an additional scraping step can be enabled.

In that case, the system retrieves the URLs returned by the search engine and fetches the full webpage content directly to enrich the results.

Scraping can be configured under Space > Sources & Tools > Web Search > Search Engine Configuration > Custom API > Requires Scraping.

Web Search Space Configuration

The Custom API search engine can be configured from the spaces configuration page under Space > Sources & Tools > Web Search > Search Engine Configuration.

Configuration Examples

GET Request Configuration

from unique_web_search.services.search_engine.custom_api import CustomAPIConfig, CustomAPI
from unique_web_search.settings import CUSTOM_API_REQUEST_METHOD

config = CustomAPIConfig(
    search_engine_name=SearchEngineType.CUSTOM_API,
    api_endpoint="https://your-api.example.com/search",
    api_request_method=CUSTOM_API_REQUEST_METHOD.GET,
    api_headers={"X-API-Key": "your_api_key"},
    api_additional_query_params={
        "limit": 10,
        "format": "json",
        "lang": "en"
    },
    timeout=60
)

Resulting Request to the Server

GET https://your-api.example.com/search?query=test&limit=10&format=json&lang=en
Headers:
  X-API-Key: your_api_key

POST Request Configuration

config = CustomAPIConfig(
    search_engine_name=SearchEngineType.CUSTOM_API,
    api_endpoint="https://your-api.example.com/search",
    api_request_method=CUSTOM_API_REQUEST_METHOD.POST,
    api_headers={
        "Authorization": "Bearer YOUR_TOKEN",
        "Content-Type": "application/json"
    },
    api_additional_query_params={
        "api_version": "v2"
    },
    api_additional_body_params={
        "lang": "en",
        "safe_search": True,
        "max_results": 10
    },
    timeout=120,
    requires_scraping=False
)

search = CustomAPI(config)
results = await search.search("custom search query")

Resulting Request to the server

POST https://your-api.example.com/search?api_version=v2
Headers:
  Authorization: Bearer YOUR_TOKEN
  Content-Type: application/json

Body (json):
{
  "query": "custom search query",
  "lang": "en",
  "safe_search": true,
  "max_results": 10
}