Cloudflare changelogs | AI Search

AI Search - AI Search makes it easier to build a search engine for your data

Thu, 06 Aug 2026 00:00:00 GMT

AI Search gets you from a data source to a working search endpoint quickly. This release adds what you need to put that endpoint in front of real users: your own domain, authentication, and one endpoint across several instances. It also adds crawling for sites without a complete sitemap, so your index covers everything you want it to find.

Each of the following is a new option. The previous behavior is still the default, so nothing changes until you change it.

Serve search from your own domain

A public endpoint is a URL that a site or app can query directly, with no authentication in front of it. By default that URL is a generated hostname on search.ai.cloudflare.com. You can now serve the same endpoint from a custom domain, a hostname in a zone that you own:

https://search.example.com/search

Restrict who can query your content

Once your endpoint is on your own domain, you can put Cloudflare Access in front of it. For example, you usually want to give /mcp to specific agents rather than to anyone who finds the URL. Agents authenticate with an Access service token, and people who open the endpoint in a browser sign in through your identity provider.

Search several instances from one URL

A namespace can expose its own public endpoint with /search, /chat/completions, and /mcp paths that fan out across the instances you choose:

curl https://ns-<NAMESPACE_ENDPOINT_ID>.search.ai.cloudflare.com/search \
  --header "Content-Type: application/json" \
  --data '{
    "messages": [{ "content": "How do I configure AI Search?", "role": "user" }],
    "ai_search_options": { "instance_ids": ["docs", "support"] }
  }'

Index your sites without a sitemap

Website data sources support a new discover parse type. It starts at the source URL and collects pages from both your sitemaps and the links it finds while crawling:

curl -X POST "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/ai-search/instances" \
  -H "Authorization: Bearer <API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "my-ai-search",
    "type": "web-crawler",
    "source": "example.com",
    "source_params": {
      "web_crawler": {
        "parse_type": "discover",
        "discover_options": { "source": "links", "limit": 5000, "depth": 3 }
      }
    }
  }'

To learn more, refer to the AI Search documentation.

AI Search - Use AI Search with the Agents SDK, AI SDK, and LangChain

Thu, 30 Jul 2026 00:00:00 GMT

You can now use AI Search directly from popular agent frameworks, adding grounded retrieval to an existing app instead of calling the REST API by hand. The new Agents section has guides for the Vercel AI SDK, LangChain, and the Cloudflare Agents SDK. The AI SDK integration is a new package, and the LangChain integration is a new retriever in the existing langchain-cloudflare package.

Vercel AI SDK

The ai-search-provider ↗ package connects AI Search to the AI SDK, and targets AI SDK v6 (ai@^6). Pass instance.chat() to generateText or streamText to generate a response grounded in your indexed content, with the retrieved chunks returned as sources. You can also expose instance.search() as a tool for agent loops.

import { createAISearchNamespace } from "ai-search-provider";
import { generateText } from "ai";

const aiSearch = createAISearchNamespace({ binding: env.AI_SEARCH });

const { text, sources } = await generateText({
	model: aiSearch.get("knowledge-base").chat(),
	messages: [{ role: "user", content: "How does caching work?" }],
});

import { createAISearchNamespace } from "ai-search-provider";
import { generateText } from "ai";

const aiSearch = createAISearchNamespace({ binding: env.AI_SEARCH });

const { text, sources } = await generateText({
	model: aiSearch.get("knowledge-base").chat(),
	messages: [{ role: "user", content: "How does caching work?" }],
});

LangChain

The langchain-cloudflare package (PyPI ↗, GitHub ↗) provides CloudflareAISearchRetriever, a standard LangChain retriever backed by AI Search. Use it on its own, wrap it with create_retriever_tool to give an agent a search tool, or drop it into a RAG chain. It works with REST credentials or a Worker binding inside a Python Worker.

from langchain_cloudflare import CloudflareAISearchRetriever

retriever = CloudflareAISearchRetriever(
    account_id=ACCOUNT_ID,
    api_token=API_TOKEN,
    instance_name="knowledge-base",
    retrieval_type="hybrid",
)

docs = retriever.invoke("How do I configure Workers AI?")

Cloudflare Agents SDK

The Cloudflare Agents SDK could already reach AI Search through the Workers binding. The new guide walks through building a stateful chat agent that provisions its own instance, indexes content, and searches it from a tool.

import { tool } from "ai";
import { z } from "zod";

const instance = env.AI_SEARCH.get("knowledge-base");

// Expose AI Search to the agent's model as a tool it can call.
const searchKnowledgeBase = tool({
	description: "Search the knowledge base for relevant content.",
	inputSchema: z.object({ query: z.string() }),
	execute: ({ query }) => instance.search({ query }),
});

import { tool } from "ai";
import { z } from "zod";

const instance = env.AI_SEARCH.get("knowledge-base");

// Expose AI Search to the agent's model as a tool it can call.
const searchKnowledgeBase = tool({
	description: "Search the knowledge base for relevant content.",
	inputSchema: z.object({ query: z.string() }),
	execute: ({ query }) => instance.search({ query }),
});

For the full walkthroughs, including creating an instance and indexing content, refer to the Agents guides.

AI Search - Filter AI Search list items by exact object key

Wed, 08 Jul 2026 00:00:00 GMT

In AI Search, you can upload files to an instance, or connect a data source such as an R2 bucket, to make your content searchable with natural language. Each file becomes an item identified by an object key (its filename or path). The list items endpoint returns the items in an instance.

That endpoint now accepts a key query parameter, so you can look up a single item by its exact object key without paging through the full list. This complements the existing item_id filter for when you know the key but not the ID.

curl "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/ai-search/instances/<INSTANCE_NAME>/items?key=docs/readme.md" \
  -H "Authorization: Bearer <API_TOKEN>"

Keys are unique per data source, so combine key with source (for example, source=builtin) to disambiguate when the same key exists across multiple sources.

For more information, refer to managing items.

Workers AI, AI Search - Workers AI toMarkdown and AI Search now supports GIF and BMP image conversion

Wed, 08 Jul 2026 00:00:00 GMT

Workers AI Markdown conversion (toMarkdown) now supports .gif and .bmp image files, in addition to the JPEG, PNG, WebP, and SVG formats already supported.

GIF and BMP files run through the same image pipeline as other formats. Each image is resized if needed (and for animated GIFs, only the first frame is used), then passed to an object-detection model to identify what it contains. Those detected objects prompt a vision model that writes a natural-language description of the image, which becomes searchable, machine-readable Markdown.

AI Search uses toMarkdown automatically to process the files it ingests, so any .gif and .bmp files are included the next time your index syncs, with no configuration changes required. This helps when your content mixes formats, for example a support knowledge base full of screenshots or an archive of BMP scans.

Learn more about Markdown conversion and the full list of AI Search's supported file types.

AI Search - Manage AI Search sync jobs with Wrangler CLI

Thu, 02 Jul 2026 00:00:00 GMT

When you connect a data source to your AI Search instance, AI Search runs sync jobs to keep your index up to date with your content. You can now manage those jobs directly from Wrangler.

For example, you can trigger a sync job from your CI/CD or automated pipelines with the jobs create command so your index refreshes when you push a change:

wrangler ai-search jobs create my-instance

This creates an asynchronous sync job that checks for changes in your data source, and sends new, modified, or deleted files to be indexed. The following commands are available:

Command	Description
`wrangler ai-search jobs create`	Trigger a new sync job
`wrangler ai-search jobs list`	List sync jobs for an instance
`wrangler ai-search jobs get`	Get details for a job
`wrangler ai-search jobs cancel`	Cancel a running job
`wrangler ai-search jobs logs`	View log entries for a job

All commands accept --namespace/-n (defaults to default) and --json for structured output that automation and AI agents can parse directly. The list and logs commands also support --page and --per-page for pagination, and cancel prompts for confirmation unless you pass -y/--force.

For full usage details, refer to the AI Search Wrangler commands documentation.

AI Search - Control AI Search similarity cache freshness

Wed, 24 Jun 2026 00:00:00 GMT

AI Search now gives you more control over similarity cache freshness. Similarity cache helps reduce latency and inference cost by reusing responses for semantically similar queries.

With these updates, you can choose how long responses are eligible for reuse and clear cached responses when they may be stale.

Cache duration now defaults to 48 hours

Previously, AI Search cached responses for a fixed duration of 30 days. Cached responses now use the instance's cache_ttl setting, and the default is 48 hours.

You can set cache_ttl when creating or updating an instance to choose a cache duration from 10 minutes to 6 days.

Use a shorter TTL when your source content changes frequently and freshness is more important. Use a longer TTL when your content is stable and you want more cache reuse.

For example, set cache_ttl to 518400 to retain cached responses for 6 days:

{
	"cache_ttl": 518400
}

Purge cached responses

You can also purge all cached responses for an instance on demand. Purging cached responses does not delete indexed content or source files.

It prevents AI Search from reusing previous cached responses, so subsequent similar queries generate fresh answers and repopulate the cache.

curl -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-search/instances/$INSTANCE_NAME/purge_cache" \
  -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN"

You can also purge cached responses from the instance settings page in the Cloudflare dashboard.

Refer to similarity cache for the full list of supported cache_ttl values and more details about cache behavior.

AI Search - Manage AI Search namespaces with Wrangler CLI

Wed, 10 Jun 2026 00:00:00 GMT

AI Search now supports namespace-level Wrangler commands, making it easier to manage namespaces from your terminal, scripts, and agent workflows.

The following commands are available:

Command	Description
`wrangler ai-search namespace list`	List AI Search namespaces
`wrangler ai-search namespace create`	Create a new AI Search namespace
`wrangler ai-search namespace get`	Get details for a namespace
`wrangler ai-search namespace update`	Update a namespace description
`wrangler ai-search namespace delete`	Delete an AI Search namespace

Create a namespace for a new application or tenant directly from the CLI:

wrangler ai-search namespace create docs-production --description "Production documentation search"

List namespaces with pagination or filter by name or description:

wrangler ai-search namespace list --search docs --page 1 --per-page 10

Use --json with list, create, get, and update to return structured output that automation and AI agents can parse directly.

Instance-level commands also now support a --namespace flag, so you can interact with instances inside a specific namespace from the CLI:

wrangler ai-search list --namespace docs-production

For full usage details, refer to the AI Search Wrangler commands documentation.

AI Search - AI Search instances now include built-in storage and namespace Workers Bindings

Thu, 16 Apr 2026 12:00:00 GMT

New AI Search instances created after today will work differently. New instances come with built-in storage and a vector index, so you can upload a file, have it indexed immediately, and search it right away.

Additionally new Workers Bindings are now available to use with AI Search. The new namespace binding lets you create and manage instances at runtime, and cross-instance search API lets you query across multiple instances in one call.

Built-in storage and vector index

All new instances now comes with built-in storage which allows you to upload files directly to it using the Items API or the dashboard. No R2 buckets to set up, no external data sources to connect first.

const instance = env.AI_SEARCH.get("my-instance");

// upload and wait for indexing to complete
const item = await instance.items.uploadAndPoll("faq.md", content);

// search immediately after indexing
const results = await instance.search({
	messages: [{ role: "user", content: "onboarding guide" }],
});

Namespace binding

The new ai_search_namespaces binding replaces the previous env.AI.autorag() API provided through the AI binding. It gives your Worker access to all instances within a namespace and lets you create, update, and delete instances at runtime without redeploying.

// wrangler.jsonc
{
	"ai_search_namespaces": [
		{
			"binding": "AI_SEARCH",
			"namespace": "default",
		},
	],
}

// create an instance at runtime
const instance = await env.AI_SEARCH.create({
	id: "my-instance",
});

For migration details, refer to Workers binding migration. For more on namespaces, refer to Namespaces.

Cross-instance search

Within the new AI Search binding, you now have access to a Search and Chat API on the namespace level. Pass an array of instance IDs and get one ranked list of results back.

const results = await env.AI_SEARCH.search({
	messages: [{ role: "user", content: "What is Cloudflare?" }],
	ai_search_options: {
		instance_ids: ["product-docs", "customer-abc123"],
	},
});

Refer to Namespace-level search for details.

AI Search - AI Search now has hybrid search and relevance boosting

Thu, 16 Apr 2026 12:00:00 GMT

AI Search now supports hybrid search and relevance boosting, giving you more control over how results are found and ranked.

Hybrid search

Hybrid search combines vector (semantic) search with BM25 keyword search in a single query. Vector search finds chunks with similar meaning, even when the exact words differ. Keyword search matches chunks that contain your query terms exactly. When you enable hybrid search, both run in parallel and the results are fused into a single ranked list.

You can configure the tokenizer (porter for natural language, trigram for code), keyword match mode (and for precision, or for recall), and fusion method (rrf or max) per instance:

const instance = await env.AI_SEARCH.create({
	id: "my-instance",
	index_method: { vector: true, keyword: true },
	fusion_method: "rrf",
	indexing_options: { keyword_tokenizer: "porter" },
	retrieval_options: { keyword_match_mode: "and" },
});

Refer to Search modes for an overview and Hybrid search for configuration details.

Relevance boosting

Relevance boosting lets you nudge search rankings based on document metadata. For example, you can prioritize recent documents by boosting on timestamp, or surface high-priority content by boosting on a custom metadata field like priority.

Configure up to 3 boost fields per instance or override them per request:

const results = await env.AI_SEARCH.get("my-instance").search({
	messages: [{ role: "user", content: "deployment guide" }],
	ai_search_options: {
		retrieval: {
			boost_by: [
				{ field: "timestamp", direction: "desc" },
				{ field: "priority", direction: "desc" },
			],
		},
	},
});

Refer to Relevance boosting for configuration details.

AI Search - Website Source CSS content selectors for precise content extraction in AI Search

Wed, 08 Apr 2026 00:00:00 GMT

AI Search now supports CSS content selectors for website data sources. You can now define which parts of a crawled page are extracted and indexed by specifying CSS selectors paired with URL glob patterns.

Content selectors solve the problem of indexing only relevant content while ignoring navigation, sidebars, footers, and other boilerplate. When a page URL matches a glob pattern, only elements matching the corresponding CSS selector are extracted and converted to Markdown for indexing.

Configure content selectors via the dashboard or API:

curl "https://api.cloudflare.com/client/v4/accounts/{account_id}/ai-search/instances" \
  -H "Authorization: Bearer {api_token}" \
  -H "Content-Type: application/json" \
  -d '{
    "id": "my-ai-search",
    "source": "https://example.com",
    "type": "web-crawler",
    "source_params": {
      "web_crawler": {
        "parse_options": {
          "content_selector": [
            {
              "path": "**/blog/**",
              "selector": "article .post-body"
            }
          ]
        }
      }
    }
  }'

Selectors are evaluated in order, and the first matching pattern wins. You can define up to 10 content selector entries per instance.

For configuration details and examples, refer to the content selectors documentation.

AI Search - New Workers AI models for text generation and embedding in AI Search

Wed, 08 Apr 2026 00:00:00 GMT

AI Search now supports four additional Workers AI models across text generation and embedding.

Text generation

Model	Context window (tokens)
`@cf/zai-org/glm-4.7-flash`	131,072
`@cf/qwen/qwen3-30b-a3b-fp8`	32,000

GLM-4.7-Flash is a lightweight model from Zhipu AI with a 131,072 token context window, suitable for long-document summarization and retrieval tasks. Qwen3-30B-A3B is a mixture-of-experts model from Alibaba that activates only 3 billion parameters per forward pass, keeping inference fast while maintaining strong response quality.

Embedding

Model	Vector dims	Input tokens	Metric
`@cf/qwen/qwen3-embedding-0.6b`	1,024	4,096	cosine
`@cf/google/embeddinggemma-300m`	768	512	cosine

Qwen3-Embedding-0.6B supports up to 4,096 input tokens, making it a good fit for indexing longer text chunks. EmbeddingGemma-300M from Google produces 768-dimension vectors and is optimized for low-latency embedding workloads.

All four models are available without additional provider keys since they run on Workers AI. Select them when creating or updating an AI Search instance in the dashboard or through the API.

For the full list of supported models, refer to Supported models.

AI Search - Create, manage, search AI Search instances with Wrangler CLI

Wed, 01 Apr 2026 00:00:00 GMT

AI Search supports a wrangler ai-search command namespace. Use it to manage instances from the command line.

The following commands are available:

Command	Description
`wrangler ai-search create`	Create a new instance with an interactive wizard
`wrangler ai-search list`	List all instances in your account
`wrangler ai-search get`	Get details of a specific instance
`wrangler ai-search update`	Update the configuration of an instance
`wrangler ai-search delete`	Delete an instance
`wrangler ai-search search`	Run a search query against an instance
`wrangler ai-search stats`	Get usage statistics for an instance

The create command guides you through setup, choosing a name, source type (r2 or web), and data source. You can also pass all options as flags for non-interactive use:

wrangler ai-search create my-instance --type r2 --source my-bucket

Use wrangler ai-search search to query an instance directly from the CLI:

wrangler ai-search search my-instance --query "how do I configure caching?"

All commands support --json for structured output that scripts and AI agents can parse directly.

For full usage details, refer to the Wrangler commands documentation.

AI Search - New AI Search REST API endpoints for /search and /chat/completions

Mon, 23 Mar 2026 00:00:00 GMT

AI Search now offers new REST API endpoints for search and chat that use an OpenAI compatible format. This means you can use the familiar messages array structure that works with existing OpenAI SDKs and tools. The messages array also lets you pass previous messages within a session, so the model can maintain context across multiple turns.

Endpoint	Path
Chat Completions	`POST /accounts/{account_id}/ai-search/instances/{name}/chat/completions`
Search	`POST /accounts/{account_id}/ai-search/instances/{name}/search`

Here is an example request to the Chat Completions endpoint using the new messages array format:

curl https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai-search/instances/{NAME}/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {API_TOKEN}" \
  -d '{
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful documentation assistant."
      },
      {
        "role": "user",
        "content": "How do I get started?"
      }
    ]
  }'

For more details, refer to the AI Search REST API guide.

Migration from existing AutoRAG API (recommended)

If you are using the previous AutoRAG API endpoints (/autorag/rags/), we recommend migrating to the new endpoints. The previous AutoRAG API endpoints will continue to be fully supported.

Refer to the migration guide for step-by-step instructions.

AI Search - AI Search UI snippets and MCP support

Mon, 23 Mar 2026 00:00:00 GMT

AI Search now supports public endpoints, UI snippets, and MCP, making it easy to add search to your website or connect AI agents.

Public endpoints allow you to expose AI Search capabilities without requiring API authentication. To enable public endpoints:

Go to AI Search in the Cloudflare dashboard. Go to AI Search ↗
Select your instance, and turn on Public Endpoint in Settings. For more details, refer to Public endpoint configuration.

UI snippets

UI snippets are pre-built search and chat components you can embed in your website. Visit search.ai.cloudflare.com ↗ to configure and preview components for your AI Search instance.

To add a search modal to your page:

<script
	type="module"
	src="https://<PUBLIC_ENDPOINT_ID>.search.ai.cloudflare.com/assets/v0.0.25/search-snippet.es.js"
></script>

<search-modal-snippet
	api-url="https://<PUBLIC_ENDPOINT_ID>.search.ai.cloudflare.com/"
	placeholder="Search..."
>
</search-modal-snippet>

For more details, refer to the UI snippets documentation.

MCP

The MCP endpoint allows AI agents to search your content via the Model Context Protocol. Connect your MCP client to:

https://<PUBLIC_ENDPOINT_ID>.search.ai.cloudflare.com/mcp

For more details, refer to the MCP documentation.

AI Search - Custom metadata filtering for AI Search

Mon, 23 Mar 2026 00:00:00 GMT

AI Search now supports custom metadata filtering, allowing you to define your own metadata fields and filter search results based on attributes like category, version, or any custom field you define.

Define a custom metadata schema

You can define up to 5 custom metadata fields per AI Search instance. Each field has a name and data type (text, number, or boolean):

curl -X POST https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai-search/instances \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {API_TOKEN}" \
  -d '{
    "id": "my-instance",
    "type": "r2",
    "source": "my-bucket",
    "custom_metadata": [
      { "field_name": "category", "data_type": "text" },
      { "field_name": "version", "data_type": "number" },
      { "field_name": "is_public", "data_type": "boolean" }
    ]
  }'

Add metadata to your documents

How you attach metadata depends on your data source:

R2 bucket: Set metadata using S3-compatible custom headers (x-amz-meta-*) when uploading objects. Refer to R2 custom metadata for examples.
Website: Add <meta> tags to your HTML pages. Refer to Website custom metadata for details.

Filter search results

Use custom metadata fields in your search queries alongside built-in attributes like folder and timestamp:

curl https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai-search/instances/{NAME}/search \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {API_TOKEN}" \
  -d '{
    "messages": [
      {
        "content": "How do I configure authentication?",
        "role": "user"
      }
    ],
    "ai_search_options": {
      "retrieval": {
        "filters": {
          "category": "documentation",
          "version": { "$gte": 2.0 }
        }
      }
    }
  }'

Learn more in the metadata filtering documentation.

AI Search - AI Search now with more granular controls over indexing

Mon, 09 Feb 2026 00:00:00 GMT

Get your content updates into AI Search faster and avoid a full rescan when you do not need it.

Reindex individual files without a full sync

Updated a file or need to retry one that errored? When you know exactly which file changed, you can now reindex it directly instead of rescanning your entire data source.

Go to Overview > Indexed Items and select the sync icon next to any file to reindex it immediately.

Crawl only the sitemap you need

By default, AI Search crawls all sitemaps listed in your robots.txt, up to the maximum files per index limit. If your site has multiple sitemaps but you only want to index a specific set, you can now specify a single sitemap URL to limit what the crawler visits.

For example, if your robots.txt lists both blog-sitemap.xml and docs-sitemap.xml, you can specify just https://example.com/docs-sitemap.xml to index only your documentation.

Configure your selection anytime in Settings > Parsing options > Specific sitemaps, then trigger a sync to apply the changes.

Learn more about indexing controls and website crawling configuration.

AI Search - AI Search path filtering for website and R2 data sources

Tue, 20 Jan 2026 00:00:00 GMT

AI Search now includes path filtering for both website and R2 data sources. You can now control which content gets indexed by defining include and exclude rules for paths.

By controlling what gets indexed, you can improve the relevance and quality of your search results. You can also use path filtering to split a single data source across multiple AI Search instances for specialized search experiences.

Path filtering uses micromatch ↗ patterns, so you can use * to match within a directory and ** to match across directories.

Use case	Include	Exclude
Index docs but skip drafts	`/docs/`	`/docs/drafts/`
Keep admin pages out of results	—	`/admin/`
Index only English content	`/en/`	—

Configure path filters when creating a new instance or update them anytime from Settings. Check out path filtering to learn more.

AI Search - Create AI Search instances programmatically via REST API

Tue, 20 Jan 2026 00:00:00 GMT

You can now create AI Search instances programmatically using the API. For example, use the API to create instances for each customer in a multi-tenant application or manage AI Search alongside your other infrastructure.

If you have created an AI Search instance via the dashboard before, you already have a service API token registered and can start creating instances programmatically right away. If not, follow the API guide to set up your first instance.

For example, you can now create separate search instances for each language on your website:

for lang in en fr es de; do
  curl -X POST "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai-search/instances" \
    -H "Authorization: Bearer $API_TOKEN" \
    -H "Content-Type: application/json" \
    --data '{
      "id": "docs-'"$lang"'",
      "type": "web-crawler",
      "source": "example.com",
      "source_params": {
        "path_include": ["**/'"$lang"'/**"]
      }
    }'
done

Refer to the REST API reference for additional configuration options.

AI Search - AI Search support for crawling login protected website content

Wed, 19 Nov 2025 00:00:00 GMT

AI Search now supports custom HTTP headers for website crawling, solving a common problem where valuable content behind authentication or access controls could not be indexed.

Previously, AI Search could only crawl publicly accessible pages, leaving knowledge bases, documentation, and other protected content out of your search results. With custom headers support, you can now include authentication credentials that allow the crawler to access this protected content.

This is particularly useful for indexing content like:

Internal documentation behind corporate login systems
Premium content that requires users to provide access to unlock
Sites protected by Cloudflare Access using service tokens

To add custom headers when creating an AI Search instance, select Parse options. In the Extra headers section, you can add up to five custom headers per Website data source.

For example, to crawl a site protected by Cloudflare Access, you can add service token credentials as custom headers:

CF-Access-Client-Id: your-token-id.access
CF-Access-Client-Secret: your-token-secret

The crawler will automatically include these headers in all requests, allowing it to access protected pages that would otherwise be blocked.

Learn more about configuring custom headers for website crawling in AI Search.

AI Search - Reranking and API-based system prompt configuration in AI Search

Tue, 28 Oct 2025 00:00:00 GMT

AI Search now supports reranking for improved retrieval quality and allows you to set the system prompt directly in your API requests.

Rerank for more relevant results

You can now enable reranking to reorder retrieved documents based on their semantic relevance to the user’s query. Reranking helps improve accuracy, especially for large or noisy datasets where vector similarity alone may not produce the optimal ordering.

You can enable and configure reranking in the dashboard or directly in your API requests:

const answer = await env.AI.autorag("my-autorag").aiSearch({
	query: "How do I train a llama to deliver coffee?",
	model: "@cf/meta/llama-3.3-70b-instruct-fp8-fast",
	reranking: {
		enabled: true,
		model: "@cf/baai/bge-reranker-base",
	},
});

Set system prompts in API

Previously, system prompts could only be configured in the dashboard. You can now define them directly in your API requests, giving you per-query control over behavior. For example:

// Dynamically set query and system prompt in AI Search
async function getAnswer(query, tone) {
	const systemPrompt = `You are a ${tone} assistant.`;

	const response = await env.AI.autorag("my-autorag").aiSearch({
		query: query,
		system_prompt: systemPrompt,
	});

	return response;
}

// Example usage
const query = "What is Cloudflare?";
const tone = "friendly";

const answer = await getAnswer(query, tone);
console.log(answer);

Learn more about Reranking and System Prompt in AI Search.

AI Search - AI Search (formerly AutoRAG) now with More Models To Choose From

Thu, 25 Sep 2025 00:00:00 GMT

AutoRAG is now AI Search! The new name marks a new and bigger mission: to make world-class search infrastructure available to every developer and business.

With AI Search you can now use models from different providers like OpenAI and Anthropic. By attaching your provider keys to the AI Gateway linked to your AI Search instance, you can use many more models for both embedding and inference.

To use AI Search with other model providers:

Add provider keys to AI Gateway
1. Go to AI > AI Gateway in the dashboard.
2. Select or create an AI gateway.
3. In Provider Keys, choose your provider, click Add, and enter the key.
Connect a gateway to AI Search: When creating a new AI Search, select the AI Gateway with your provider keys. For an existing AI Search, go to Settings and switch to a gateway that has your keys under Resources.
Select models: Embedding models are only available to be changed when creating a new AI Search. Generation model can be selected when creating a new AI Search and can be changed at any time in Settings.

Once configured, your AI Search instance will be able to reference models available through your AI Gateway when making a /ai-search request:

export default {
  async fetch(request, env) {
    
    // Query your AI Search instance with a natural language question to an OpenAI model
    const result = await env.AI.autorag("my-ai-search").aiSearch({
      query: "What's new for Cloudflare Birthday Week?",
      model: "openai/gpt-5"
    });

    // Return only the generated answer as plain text
    return new Response(result.response, {
      headers: { "Content-Type": "text/plain" },
    });
  },
};

In the coming weeks we will also roll out updates to align the APIs with the new name. The existing APIs will continue to be supported for the time being. Stay tuned to the AI Search Changelog and Discord ↗ for more updates!

AI Search - New Metrics View in AutoRAG

Fri, 19 Sep 2025 00:00:00 GMT

AutoRAG now includes a Metrics tab that shows how your data is indexed and searched. Get a clear view of the health of your indexing pipeline, compare usage between ai-search and search, and see which files are retrieved most often.

You can find these metrics within each AutoRAG instance:

Indexing: Track how files are ingested and see status changes over time.
Search breakdown: Compare usage between ai-search and search endpoints.
Top file retrievals: Identify which files are most frequently retrieved in a given period.

Try it today in AutoRAG.

AI Search - Faster indexing and new Jobs view in AutoRAG

Tue, 08 Jul 2025 00:00:00 GMT

You can now expect 3-5× faster indexing in AutoRAG, and with it, a brand new Jobs view to help you monitor indexing progress.

With each AutoRAG, indexing jobs are automatically triggered to sync your data source (i.e. R2 bucket) with your Vectorize index, ensuring new or updated files are reflected in your query results. You can also trigger jobs manually via the Sync API or by clicking “Sync index” in the dashboard.

With the new jobs observability, you can now:

View the status, job ID, source, start time, duration and last sync time for each indexing job
Inspect real-time logs of job events (e.g. Starting indexing data source...)
See a history of past indexing jobs under the Jobs tab of your AutoRAG

This makes it easier to understand what’s happening behind the scenes.

Coming soon: We’re adding APIs to programmatically check indexing status, making it even easier to integrate AutoRAG into your workflows.

Try it out today on the Cloudflare dashboard ↗.

AI Search - View custom metadata in responses and guide AI-search with context in AutoRAG

Thu, 19 Jun 2025 00:00:00 GMT

In AutoRAG, you can now view your object's custom metadata in the response from /search and /ai-search, and optionally add a context field in the custom metadata of an object to provide additional guidance for AI-generated answers.

You can add custom metadata to an object when uploading it to your R2 bucket.

Object's custom metadata in search responses

When you run a search, AutoRAG now returns any custom metadata associated with the object. This metadata appears in the response inside attributes then file , and can be used for downstream processing.

For example, the attributes section of your search response may look like:

{
	"attributes": {
		"timestamp": 1750001460000,
		"folder": "docs/",
		"filename": "launch-checklist.md",
		"file": {
			"url": "https://wiki.company.com/docs/launch-checklist",
			"context": "A checklist for internal launch readiness, including legal, engineering, and marketing steps."
		}
	}
}

Add a `context` field to guide LLM answers

When you include a custom metadata field named context, AutoRAG attaches that value to each chunk of the file. When you run an /ai-search query, this context is passed to the LLM and can be used as additional input when generating an answer.

We recommend using the context field to describe supplemental information you want the LLM to consider, such as a summary of the document or a source URL. If you have several different metadata attributes, you can join them together however you choose within the context string.

For example:

{
	"context": "summary: 'Checklist for internal product launch readiness, including legal, engineering, and marketing steps.'; url: 'https://wiki.company.com/docs/launch-checklist'"
}

This gives you more control over how your content is interpreted, without requiring you to modify the original contents of the file.

Learn more in AutoRAG's metadata filtering documentation.

AI Search - Filter your AutoRAG search by file name

Thu, 19 Jun 2025 00:00:00 GMT

In AutoRAG, you can now filter by an object's file name using the filename attribute, giving you more control over which files are searched for a given query.

This is useful when your application has already determined which files should be searched. For example, you might query a PostgreSQL database to get a list of files a user has access to based on their permissions, and then use that list to limit what AutoRAG retrieves.

For example, your search query may look like:

const response = await env.AI.autorag("my-autorag").search({
	query: "what is the project deadline?",
	filters: {
		type: "eq",
		key: "filename",
		value: "project-alpha-roadmap.md",
	},
});

This allows you to connect your application logic with AutoRAG's retrieval process, making it easy to control what gets searched without needing to reindex or modify your data.

Learn more in AutoRAG's metadata filtering documentation.

AI Search - Metadata filtering and multitenancy support in AutoRAG

Wed, 23 Apr 2025 00:00:00 GMT

You can now filter AutoRAG search results by folder and timestamp using metadata filtering to narrow down the scope of your query.

This makes it easy to build multitenant experiences where each user can only access their own data. By organizing your content into per-tenant folders and applying a folder filter at query time, you ensure that each tenant retrieves only their own documents.

Example folder structure:

customer-a/logs/
customer-a/contracts/
customer-b/contracts/

Example query:

const response = await env.AI.autorag("my-autorag").search({
	query: "When did I sign my agreement contract?",
	filters: {
		type: "eq",
		key: "folder",
		value: "customer-a/contracts/",
	},
});

You can use metadata filtering by creating a new AutoRAG or reindexing existing data. To reindex all content in an existing AutoRAG, update any chunking setting and select Sync index. Metadata filtering is available for all data indexed on or after April 21, 2025.

If you are new to AutoRAG, get started with the Get started AutoRAG guide.

AI Search, Vectorize - Create fully-managed RAG pipelines for your AI applications with AutoRAG

Mon, 07 Apr 2025 00:00:00 GMT

AutoRAG is now in open beta, making it easy for you to build fully-managed retrieval-augmented generation (RAG) pipelines without managing infrastructure. Just upload your docs to R2, and AutoRAG handles the rest: embeddings, indexing, retrieval, and response generation via API.

With AutoRAG, you can:

Customize your pipeline: Choose from Workers AI models, configure chunking strategies, edit system prompts, and more.
Instant setup: AutoRAG provisions everything you need from Vectorize, AI gateway, to pipeline logic for you, so you can go from zero to a working RAG pipeline in seconds.
Keep your index fresh: AutoRAG continuously syncs your index with your data source to ensure responses stay accurate and up to date.
Ask questions: Query your data and receive grounded responses via a Workers binding or API.

Whether you're building internal tools, AI-powered search, or a support assistant, AutoRAG gets you from idea to deployment in minutes.

Get started in the Cloudflare dashboard ↗ or check out the guide for instructions on how to build your RAG pipeline today.

Cloudflare changelogs | AI Search

AI Search - AI Search makes it easier to build a search engine for your data

Serve search from your own domain

Restrict who can query your content

Search several instances from one URL

Index your sites without a sitemap

AI Search - Use AI Search with the Agents SDK, AI SDK, and LangChain

Vercel AI SDK

LangChain

Cloudflare Agents SDK

AI Search - Filter AI Search list items by exact object key

Workers AI, AI Search - Workers AI toMarkdown and AI Search now supports GIF and BMP image conversion

AI Search - Manage AI Search sync jobs with Wrangler CLI

AI Search - Control AI Search similarity cache freshness

Cache duration now defaults to 48 hours

Purge cached responses

AI Search - Manage AI Search namespaces with Wrangler CLI

AI Search - AI Search instances now include built-in storage and namespace Workers Bindings

Built-in storage and vector index

Namespace binding

Cross-instance search

AI Search - AI Search now has hybrid search and relevance boosting

Hybrid search

Relevance boosting

AI Search - Website Source CSS content selectors for precise content extraction in AI Search

AI Search - New Workers AI models for text generation and embedding in AI Search

Text generation

Embedding

AI Search - Create, manage, search AI Search instances with Wrangler CLI

AI Search - New AI Search REST API endpoints for /search and /chat/completions

Migration from existing AutoRAG API (recommended)

AI Search - AI Search UI snippets and MCP support

UI snippets

MCP

AI Search - Custom metadata filtering for AI Search

Define a custom metadata schema

Add metadata to your documents

Filter search results

AI Search - AI Search now with more granular controls over indexing

Reindex individual files without a full sync

Crawl only the sitemap you need

AI Search - AI Search path filtering for website and R2 data sources

AI Search - Create AI Search instances programmatically via REST API

AI Search - AI Search support for crawling login protected website content

AI Search - Reranking and API-based system prompt configuration in AI Search

Rerank for more relevant results

Set system prompts in API

AI Search - AI Search (formerly AutoRAG) now with More Models To Choose From

AI Search - New Metrics View in AutoRAG

AI Search - Faster indexing and new Jobs view in AutoRAG

AI Search - View custom metadata in responses and guide AI-search with context in AutoRAG

Object's custom metadata in search responses

Add a context field to guide LLM answers

AI Search - Filter your AutoRAG search by file name

AI Search - Metadata filtering and multitenancy support in AutoRAG

AI Search, Vectorize - Create fully-managed RAG pipelines for your AI applications with AutoRAG

Add a `context` field to guide LLM answers