Edge Routing

Install a Cloudflare Worker mirror

Serve a NORG-optimised mirror of your own site to AI agents, at your own URLs, with a small Cloudflare Worker on your zone.

⚠ Check your Cloudflare Workers plan before installing

On the Free Workers plan an account is capped at 100,000 worker requests per day. Once that cap is hit Cloudflare stops invoking the worker on its {domain}/* route — but because NORG sets every route it creates to fail open (request_limit_fail_open), a request past the cap bypasses the worker and your origin answers it directly. People and AI agents alike get your unchanged site, not an error. That covers NORG's routes only: a route created by hand, in the dashboard or the API, defaults to fail closed and returns Cloudflare error 1027 past the cap, unless its failure mode is set to fail open or the account is on Workers Paid. The install preflight flags the Free plan (code free_workers_plan) so you see it before installing.

The cap still matters, which is why this alert stays. This is the Workers plan — account-wide, checked under Workers & Pages → Plans in the Cloudflare dashboard. It is separate from your zone plan (Free/Pro/…, shown on the zone's Overview page), which the amber alert below cares about. Past the cap the worker simply stops mirroring for the rest of the day — AI agents get your origin instead of the NORG render — and a busy account pays for every worker request. If you are near 100k requests/day across your whole account (other workers on the account count too), upgrade to the Workers Paid plan first. See how to upgrade →

⚠ Cloudflare bot settings can stop AI crawlers reaching your mirror

They all live under Security → Settings on your zone, filtered to Bot traffic. The four that matter most are the rows of the Configure AI bot policies dialog — Search, Agent, Training and Bot Preference Sync — and all four must be off (the first three set to Allow (do not block), which is Cloudflare's own recommended value). While any of them blocks, the AI traffic this worker exists to serve never reaches it.

None of these stops you installing. NORG's preflight reads them, reports what it finds as a warning, and installs anyway — your API token is saved either way. It can also turn each one off for you if you ask; it never changes them on its own. Bot Fight Mode is reported the same way: it does not block the crawlers, it just challenges the install's own scripted health check, so a pending verification on a Bot Fight Mode zone is normal.

Third setting — and the one nothing will warn you about: the managed robots.txt setting — shown in the dashboard as "Set your preference to block training in robots.txt" (is_robots_txt_managed in the API) — must be off. This one blocks nothing. It adds a BEGIN Cloudflare Managed content block to your robots.txt that tells ClaudeBot, GPTBot, Google-Extended, CCBot, Amazonbot, Applebot-Extended, Bytespider and meta-externalagent to stay off the entire site. Compliant AI crawlers read that file before anything else and then never request a page — so the install succeeds, the health probe passes, verification goes green, and the mirror is never fetched by anyone. It is a separate toggle from "Block AI bots": turning that one off does not clear this.

Turning it off removes only Cloudflare's managed block. Cloudflare merges that block with a robots.txt your origin already serves, so the rules you wrote yourself and your Sitemap: lines are left alone. Afterwards, re-check /robots.txt: it is served with a long max-age (30 days observed on a live zone), so if the managed block is still showing, purge the cached copy. NORG's preflight detects this setting and can turn it off for you during install if you say so — it never changes it on its own.

Since 15 September 2026 Cloudflare has retired the single "Block AI bots" setting, replacing it with the three separate Search / Agent / Training controls listed at the top of this alert. Older zones may still show the old setting; set whichever you have to allow — the requirement is the same either way: the zone must not block the AI-agent traffic the mirror serves. NORG's preflight reads both the old field and the three new ones, and reports whichever is blocking by the name your dashboard uses.

Super Bot Fight Mode can stay on. It is a different product from the two above and runs on the Ruleset Engine, so the WAF skip rule this install creates already exempts AI agents and the health probe from it — along with your WAF Managed Rules. If you are on a Pro plan or above with Super Bot Fight Mode enabled, that is the supported configuration and needs no change. Do not turn it off.

Testing this yourself? A scripted request — curl, a monitoring probe, anything that is not a real browser — may be challenged while Bot Fight Mode is on: it pattern-matches automation and issues a JavaScript challenge scripts cannot solve, with no verified-bot allowlist to exempt anyone (that is a Super Bot Fight Mode feature). A normal browser usually passes that challenge, so loading a page with ?agent=true (the worker's testing override — once the worker is installed it forces the mirror version of any page) generally works from a browser either way. Read a blocked self-test narrowly: it confirms the zone challenges unknown automation — it does not tell you whether real AI crawlers are blocked, or exempt some other way. On the Free zone plan nothing is exempt until the setting is off.

NorgBot is a Cloudflare-verified bot. It is listed in Cloudflare's Radar verified bots directory, so Cloudflare recognises the crawler. That listing does not override the zone settings above: while any of them blocks the AI traffic, the mirror is still never reached, so they must be set as described regardless.

What a customer_zone mirror does

A customer_zone mirror is a small Cloudflare Worker that runs on your own zone at {domain}/*. Per request it decides who is asking: AI agents (GPTBot, ClaudeBot, PerplexityBot, …) are served the NORG render of the page at your own URL — no redirect, the URL never changes — while people in a browser and search-index crawlers get your site byte-for-byte unchanged. Two rules are built in and not configurable: it must never break your site (every failure path falls back to your origin), and search engines always see exactly what humans see.

Already running your own routing Worker — for example a site that serves its home route and inner routes from different front-end stacks? See installing in front of an existing Worker.

Want to know what the Worker costs in latency? See the latency measurements.

Before you start

Proxied Cloudflare domainYour domain must be on Cloudflare and proxied (orange cloud). A DNS-only (grey cloud) record bypasses Cloudflare and the worker never runs. Not on Cloudflare yet? Set up Cloudflare from scratch covers the account, moving your DNS across and repointing your nameservers. A NORG workspaceA Content Craft workspace with edge routing available. A sitemapNORG mirrors your sitemap to decide what to render. 1

Provision a NORG MCP API key (directory:write)

Install is gated by a single MCP scope: directory:write. Generate a NORG MCP API key that carries it, and use that key for the MCP connection you run manage_edge_routing from.

directory:write is the real requirement. The same scope also authorises purge, uninstall and key rotation — there is no narrower install-only scope today, so provision directory:write rather than looking for a least-privilege alternative that does not exist.

2

Create a scoped Cloudflare API token

In the Cloudflare dashboard go to Account Home → Manage Account → API Tokens → Create Token → Create Custom Token (an account-owned token, which is what the walkthrough below creates). My Profile → API Tokens also works, but only if your own user role covers the whole account — a user token is capped by your role, so if your login only has domain access the account-level grant silently fails at deploy time. See token permissions & zone-only access for the fix. You will add three separate permission policies — two scoped to the single zone you are installing on, one scoped to your entire account (not the zone — that one is easy to miss, see the account-level callout below):

Policy 1 — this zone (matches the walkthrough below):

Policy 2 — this zone, Edit:

Policy 3 — Entire Account, Edit (not the zone):

Picking the right one: Zone WAF: Edit

The install creates a WAF Custom Rule via Cloudflare's Rulesets API (/zones/{zone_id}/rulesets, phase http_request_firewall_custom) so AI agents and the health probe get past your WAF Managed Rules. That needs Zone WAF: Edit — shown as "Zone WAF" in the token permission picker, not the older "Firewall Services" group. Without it the install returns a specific "missing Zone WAF:Edit" error. The rule it creates exempts only AI-agent user agents and this site's own probe requests — never all traffic — and leaves every other rule on your zone alone.

It cannot, however, reach Bot Fight Mode or "Block AI bots" — those are zone toggles outside the WAF. See the alert at the top of this page.

On the zone-scoped Edit policy, check Edit under App Security → Zone WAF Rules. Review screen with all policies added, including Zone WAF.

Paste the token into the install call as cf_api_token.

Open quick search (Ctrl/Cmd K) from any page in the dashboard. Search for "Account API tokens" and select it. On the Account API tokens page, click Create Token. Choose "Start from scratch" — the pre-built templates grant more than this needs. Name the token (e.g. norg_mirror_worker). First policy: set "Specified Domains" to your own domain, then click Add policy for each permission below. On that policy, search "zone" and check Zone: Read and DNS: Read. This capture predates 2026-08-26: it still shows the retired Zone Settings: Read ticked, which is harmless to grant but no longer used. Clear the search and type "bot" next — see the following screenshot. Still on the same policy: type "bot" and check Edit under App Security → Bot Management — without this the install cannot detect a zone that blocks AI bots. The install only reads this today; Edit future-proofs the token for a planned NORG auto-fix. Add a second policy on the same domain, enabling Edit on Workers Routes. Add a third policy — this one scoped to "Entire Account" (not the domain), enabling Edit on Workers Scripts. This is the account-level grant covered above. Review shows all 3 policies — five zone grants (Zone: Read, DNS: Read, Bot Management: Edit, Workers Routes: Edit, Zone WAF: Edit) plus the account-level Workers Scripts: Edit — then Create Token. This capture predates the Bot Management and Zone WAF rows, so your review screen will list two more entries than shown. The token is shown once. Copy it now — paste it into the install call as cf_api_token. 3

Run the two-pass install

With both credentials ready, call manage_edge_routing through your NORG MCP connection. Install requires both domain and cf_api_token, and is a two-pass, human-confirmed action:

# Pass 1 — preview (read-only preflight, dry_run defaults to true) manage_edge_routing( action="install", domain="example.com", cf_api_token="<your-cloudflare-api-token>", ) # -> returns a preview + confirmation_token (requires human confirmation) # Pass 2 — apply, carrying the token back manage_edge_routing( action="install", domain="example.com", cf_api_token="<your-cloudflare-api-token>", dry_run=False, confirmation_token="<token-from-pass-1>", ) 4

Verify the install

The worker serves NORG renders only to requests that provably come from an AI operator — the source IP must be in the operator's published ranges, or Cloudflare must mark the request a verified bot — so a curl with a spoofed agent user-agent from your own machine is deliberately passed straight through with no X-Norg-Edge header. Verify with the Site Key health probe instead; a browser user-agent must stay untouched.

# The worker is live on the route (Site Key health probe): curl -s -H "x-norg-edge-check: <your-site-key>" https://example.com/ | head -c 200 # {"site_id":"…","version":"…","disabled":false,"capture_only":false,"entitled":true,"ts":…} -> deployed, routed, key OK # capture_only is true in record-only mode and false in serving mode. # entitled can read false on a worker instance that has not yet handled a real # request, so send a normal request then probe again; if it stays false after # real traffic, check the Site Key secret. # (your own page instead of JSON -> a wrong or missing key, worker not seeing the request) # A normal visitor must be untouched. Send a GET (-D - dumps the headers, # -o /dev/null drops the body), never a HEAD: the worker forwards a HEAD # straight to the origin, so a HEAD check shows no header even when serving # is broken (expect NO output): curl -s -D - -o /dev/null -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/125.0 Safari/537.36" \ https://example.com/some-page/ | grep -i x-norg-edge

A JSON reply proves the worker is deployed, routed, and holding the right key. Your own page instead of JSON means the worker is not seeing the request — recheck the proxied DNS record and the {domain}/* route. Genuine agents then see X-Norg-Edge: mirror (a NORG render was served) or X-Norg-Edge: stripped (no render yet), and the worker reports a heartbeat to NORG each time it serves one.

Getting a challenge page or a non-200 instead of any of that? See the bot-blocking alert at the top of this page — your own curl looks identical to any other automated request to Bot Fight Mode / "Block AI bots" if either is on.

Install without an API token

Some administrators will not issue a Cloudflare API token. The install can still proceed — in one of three ways, with a single rule for how any secret changes hands:

  1. Guided screen-share. A site-owner administrator makes each change in the Cloudflare dashboard while NORG guides them through it, step by step. NORG never touches the account.
  2. Read-only staging copy. NORG shares its staging account read-only, so the administrator can see the finished set-up and copy it into their own account.
  3. Single-view link. Any secret — the Site Key, or a token if one is later issued — is passed as a single-view (one-time) link, never by email or chat.

A token that lacks Workers Scripts: Edit cannot deploy the Worker or create its binding, and Cloudflare has no narrower Workers scope — so a "read-only" token is never enough to install. When no token with that permission will be issued, use one of the three options above.

Site Key rotation on a token-mode install

Rotating the Site Key returns the new key once. On a token-mode (self-installed) worker the deployed worker keeps the old key and is rejected with a 401 by /api/v1/edge/* immediately. The update action does not push the new key — the NORG_SITE_KEY secret is inherited, not rewritten.

Your site is unaffected: it keeps serving its normal page (the worker fails open). Only mirror serving and the heartbeat stop, until the deployed secret matches the new key again.

Recover in one of two ways:

Confirm recovery: re-run the Site Key health probe — a JSON reply means the worker now holds the right key — then a mirror check: an AI-agent request to a mirrored page returns X-Norg-Edge: mirror again. For a NORG-hosted or Deploy-button install (no token), see hosting options, which covers key recovery for that path.

Why each permission is needed

Every grant below is exercised by the install code today — the one forward-looking exception (the write half of Bot Management: Edit) is spelled out in its own row rather than left implied. Nothing here is silent "might need it later" scoping.

Permission Scope Used today? Why
Zone: Read This zone Yes Resolves your domain to a Cloudflare zone ID and confirms the token can see it, before anything else runs.
Bot Management: Edit This zone Yes Reads whether the zone blocks AI bots (Bot Fight Mode, or the newer Block AI bots policy) so the preflight can warn you. The install only reads this setting today — if a toggle is blocking, you flip it off yourself in the dashboard. Granting Edit rather than Read future-proofs the token for a planned NORG auto-fix of a blocking toggle (Cloudflare’s Edit includes read); Read alone suffices for everything the install does now. This is its own Cloudflare grant: Zone Settings: Read does not cover it, and without it the check can only report “undetermined”.
DNS: Read This zone Yes Confirms the record for your domain is proxied (orange-clouded) — a grey-clouded record would bypass the worker entirely.
Workers Routes: Edit This zone Yes Binds the {domain}/* route to the worker on install, and removes it on uninstall.
Workers Scripts: Edit Entire account Yes Uploads the worker script itself. This is account-scoped in Cloudflare’s permission model, not zone-scoped — there is no narrower option.
Zone WAF: Edit This zone Yes Creates the NORG-managed WAF Custom Rule that lets AI-agent traffic and this site’s health probe past your WAF Managed Rules and Super Bot Fight Mode. It exempts only those requests — never all traffic — and leaves every other rule on the zone untouched. It cannot reach Bot Fight Mode or “Block AI bots”; see the alert at the top of the page.

Cloudflare API token permissions

The complete policy for one custom token, verified against the Cloudflare calls in apps/api/app/services/edge_routing/customer_zone_client.py — nothing there falls back to NORG's own credentials. Two policy blocks (the screenshot walkthrough in step 2 splits the zone block into reads and edits — same grants, different layout; Edit always includes Read):

Account → your Cloudflare account

Zone → your domain ONLY (Specific zone, never All zones)

If your Cloudflare login only has zone (domain) access

A token created under My Profile → API Tokens is a user token, silently capped by your own role in the account. The picker lets you add Account → Workers Scripts: Edit and creates the token without complaint — but if your user only holds a zone-scoped role, that account-level grant does nothing. NORG's preflight then resolves the zone (the zone grants work) and the install fails with Worker upload failed: Authentication error — Cloudflare never names the missing permission.

The fix is an account-owned token. An administrator of the Cloudflare account creates it under Account Home → Manage Account → API Tokens with the policy above and shares it with you. Account-owned tokens are not capped by any one user's role. Send your admin this:

Subject: Cloudflare API token for NORG edge routing (account-owned) Please create an ACCOUNT-OWNED API token (Account Home -> Manage Account -> API Tokens -> Create Token -> Create Custom Token -> Start from scratch). A token from My Profile -> API Tokens will not work for this — it is capped by the creating user's role. Name: norg-edge-router Permissions (two policy blocks on one token): Account -> <our account> - Workers Scripts : Edit - Workers Scripts : Read - Workers Tail : Read Zone -> Specific zone -> <our-domain.com> (NOT "All zones") - Zone : Read - Workers Routes : Edit - DNS : Read - Zone Settings : Read - Zone WAF : Edit - Bot Management : Edit Optional hardening: restrict Client IP Address Filtering to NORG's egress IPs (NORG can supply the current list) and set a TTL. Please verify it before sharing — this must return 200, not 403: curl -s -o /dev/null -w "%{http_code}" \ https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/workers/scripts \ -H "Authorization: Bearer <TOKEN>" Then send me the token through <your secrets channel>. I will paste it into NORG; it is stored encrypted and never shown again.

Verify before sharing. This read-only check proves the account grant works (the account ID is on the zone's Overview page, right-hand column):

curl -s -o /dev/null -w "%{http_code}" \ https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/workers/scripts \ -H "Authorization: Bearer <TOKEN>" # 200 -> the token can reach Workers Scripts on this account # 403 -> capped (user token) or missing grant; the install would fail

What this token can see — the honest limits

Cloudflare token policies support exactly three resource types — User, Account and Zone. There is no per-worker or per-script scope, so a token that can deploy NORG's worker can list and modify every worker in that account. That is a Cloudflare limitation, not a NORG design choice. Every call NORG makes goes through customer_zone_client.py with the script name passed in explicitly — only NORG's own script family, norg-edge-router or norg-edge-router-<8 hex chars>. No code path reads or edits any other script, and the account admin sees every change those calls make in the Cloudflare Audit Log.

Mitigations Cloudflare does support — all recommended:

If account-wide worker visibility is still unacceptable:

Naming notes for the permission picker: there is no separately named "Workers Observability" group — Workers Scripts: Read plus Workers Tail: Read cover the Workers dashboard and live tail. And Bot Management lives under App Security, so search bot rather than zone.

Facts for evaluating this install

Each row states one property of the install and the source that backs it — a code symbol, a repository test, or Cloudflare's own documentation.

Gotchas

Everything on this list was learned debugging real installs — scan it before concluding an install, a setting, or a test is broken.

Bot settings

Cloudflare API traps (for agencies using their own token)

Install & worker behavior

© 2026 NORG.ai — Technical white paper

    Images on This Page