Advanced configuration

You may want to offer free access to certain pages while charging for others:

• Allow free access to homepages, category pages, or navigation to help crawlers discover paid content.
• Exclude functional pages like login, search, or API endpoints that don't contain chargeable content.
• Start with Pay Per Crawl on a small section of your site before expanding.
• Offer free access to promotional or archived content while charging for premium articles.

To get started, use Configuration Rules to exclude specific URI patterns from charging.

1. Go to Rules > Overview in the Cloudflare dashboard.

   Go to Overview ↗

2. Select Create rule > Configuration Rule.

3. When incoming requests match: Set your URI pattern.

   • Field: URI Full
   • Operator: wildcard
   • Value: https://*example.com/public/*

4. Select Disable Pay Per Crawl > Add

5. Select Deploy.

Example patterns:

• Free homepage: URI Full equals https://example.com/
• Free directory: URI Full wildcard https://*example.com/public/*

The price you specify in Pay Per Crawl settings applies to the entire zone by default, but you can implement a differentiated pricing policy by selecting Enable dynamic pricing and having your origin HTTP responses include a crawler-price header. For example:

crawler-price: USD 3.14

When the crawler-price header is present in a response, the price it specifies will be used instead of the default price specified in the Pay Per Crawl settings for the zone.

Pay Per Crawl adds a cf-pay-per-crawl header to every origin request. This header indicates the pricing mode in effect, and can be used by the origin to decide whether or not to include a crawler-price header with the response.

cf-pay-per-crawl: protocol=cloudflare, pricing=in-band

Currently, the only possible value for the protocol indicator is cloudflare. For the pricing indicator the value can be one of the following:

• zone-default: When the zone does not have in-band pricing enabled.
• in-band: When the zone has dynamic pricing enabled.
• bypass: When the request is not subject to payment (for example, not a bot).

If you prefer to maintain your origin as is, you can use a Worker to include the crawler-price header in responses. From a Worker you can, for example, select the price based on the incoming request's properties (including information added by the Cloudflare global network) or the content itself.

The following Worker script implements a simple example policy that selects the price based on the requested URL path, while still taking advantage of Cloudflare Cache:

function getContentPriceUSD(request, response) {
	const requestPath = new URL(request.url).pathname;

	if (requestPath.startsWith("/premium-content/")) {
		return 3.14;
	}

	if (requestPath.startsWith("/free-content/")) {
		return 0.0;
	}

	return null; // Use the default price set in the zone configuration.
}

export default {
	async fetch(request, env, ctx) {
		// Obtain the response first (and allow it to be cached if possible).
		let response = await fetch(request, { cf: { cacheEverything: true } });

		// Indicates the pricing mode in effect ("bypass", "zone-default", "in-band").
		const cfPayPerCrawl = request.headers.get("CF-Pay-Per-Crawl") || "";

		// If in-band pricing is enabled, use the request/response to select a price.
		if (cfPayPerCrawl.match(/\bpricing=in-band\b/)) {
			const contentPrice = getContentPriceUSD(request, response);

			if (contentPrice !== null) {
				// Make the response mutable, to allow setting the price header.
				response = new Response(response.body, response);
				response.headers.set("Crawler-Price", `USD ${contentPrice.toFixed(2)}`);
			}
		}
		return response;
	}
};

• Configuration Rules documentation
• Workers documentation