Bots & exclusions

Five things stop a hit from being counted as a visit. Four are server-side and apply to traffic that is already live; one is in the tracker and needs a redeploy.

Rule	Where	Applies to
Excluded IPs	Site settings	Every hit from those addresses
Excluded paths	Site settings	Any path matching a glob
Bot detection	Collector	Any user agent that names a crawler
Suspected automation	Collector	A client whose headers and behaviour give it away
data-exclude	The script tag	Any path matching a glob, before anything is sent

A hit caught by any of the four server-side rules is answered with an empty 202 — the browser is told everything is fine, because from the browser’s point of view it is. No event, no visit, and nothing towards your event allowance.

The checks run in that order, and the first match wins: an excluded IP is dropped before the user agent is even parsed.

The last two are the exception to “nothing is written”. A crawler is not a visitor, but it is worth knowing about, so it is recorded on Crawlers — a separate screen, on separate rows, outside the allowance.

Bot detection

The collector matches the User-Agent header against a directory of named crawlers: GPTBot, ChatGPT, ClaudeBot, Claude, Googlebot, Bingbot, Perplexity, Applebot, the SEO tools, the link-preview fetchers, the uptime monitors and the headless browsers. A match is recorded on Crawlers under one of four categories — AI answers (a page fetched to answer a question somebody just asked an assistant), Indexing (a search or assistant index), Training (pages collected as model training data) and Other (SEO tools, link previews, monitors, headless browsers and scripts).

Something that announces itself as a bot without a name the directory knows — bot, crawler, spider in the user agent — is filed as Unknown bot under Other, with the user agent kept so it can be named later.

Suspected automation

A scraper that copies a real Chrome user agent has nothing in its User-Agent to catch. The collector weighs what is left:

• Automation flags the browser sets about itself — navigator.webdriver, or a Selenium, Playwright, PhantomJS or Nightmare global left on the page.
• An impossible window — a screen or a window with no size at all, or a viewport of 800×600, which is what a headless browser opens at when nobody sets one.
• Missing browser headers — no Accept-Language, no client hints on a modern Chrome over HTTPS, no Sec-Fetch-* metadata, or neither an Origin nor a Referer on the request, which a real fetch always carries.
• An OS that contradicts the browser — a Sec-CH-UA-Platform of Linux under a Windows user agent. The platform comes from the operating system; the user agent is a string anyone can type.
• A deep landing with no referrer — straight to /blog/posts/something with no referrer and no campaign parameter, the way a crawler works through a sitemap.
• No interaction — ten seconds on a desktop page without a single pointer move, scroll, tap or keypress.

Each signal carries a weight, and only the sum decides. One weak signal is nothing — plenty of people read a page without touching it, and privacy browsers strip headers on purpose. A client that trips enough of them is filed under Suspected automation on Crawlers, with the list of signals that caught it, and is never counted as a visit or towards your allowance.

Headless browsers in end-to-end tests

Playwright, Puppeteer, headless Chrome and curl normally identify themselves, and now land on Crawlers under Other rather than in your reports. That is what you want in production and exactly what you do not want in an end-to-end test asserting on the dashboard. Test against a real browser, or assert on the 202 response instead.

Reporting crawlers from your server

Almost no AI crawler runs JavaScript. GPTBot, ClaudeBot, Google-Extended and the rest fetch the HTML and leave, so the tracker never loads and the visit that a site owner most wants to see is the one nothing can see.

Your own server did see it. One line of middleware forwards what it served, and the hit lands on Crawlers next to the ones the tracker caught:

POST https://ingest.tracing.tools/crawl
content-type: application/json

{
  "s": "pk_live_xxxxxxxx",
  "u": "https://acme.com/blog/how-it-works",
  "ua": "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
  "ip": "203.0.113.9",
  "r": "https://chatgpt.com/"
}

Field	Required	Meaning
s	yes	Site public key, the same one in your script tag
u	yes	The full URL the crawler asked for
ua	yes	The crawler’s User-Agent, exactly as you received it
ip	no	The crawler’s address. Only used to resolve a country; never stored
r	no	The Referer the crawler sent, if any

The answer is a 202 either way:

Body	Meaning
{"recorded":true,"crawler":"gptbot"}	Recorded under that crawler
{"recorded":false}	The user agent is a browser, or the path is excluded. Nothing written

A browser is never turned into a pageview by this endpoint — that is what the tracker is for. An unknown site key answers 404, and a body over 32 KiB answers 413. Excluded paths apply exactly as they do to the tracker.

Send it and forget it. The report must never delay the page, and a collector that is slow or unreachable must never break a request.

Next.js

middleware.ts — or proxy.ts on the newest Next:

import { NextResponse, type NextRequest } from "next/server";

export function middleware(request: NextRequest) {
  // fire and forget: keepalive lets it outlive the request
  fetch("https://ingest.tracing.tools/crawl", {
    method: "POST",
    keepalive: true,
    headers: { "content-type": "application/json" },
    body: JSON.stringify({
      s: process.env.NEXT_PUBLIC_TRACING_SITE_KEY,
      u: request.url,
      ua: request.headers.get("user-agent"),
      ip: request.headers.get("x-forwarded-for")?.split(",")[0]?.trim(),
      r: request.headers.get("referer"),
    }),
  }).catch(() => {});

  return NextResponse.next();
}

// pages only — never assets, and never the API
export const config = {
  matcher: ["/((?!_next/|api/|favicon.ico|.*\\.[a-z0-9]+$).*)"],
};

Express

app.use((req, res, next) => {
  fetch("https://ingest.tracing.tools/crawl", {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({
      s: process.env.TRACING_SITE_KEY,
      u: `${req.protocol}://${req.get("host")}${req.originalUrl}`,
      ua: req.get("user-agent"),
      ip: req.ip,
      r: req.get("referer"),
    }),
  }).catch(() => {});

  next();
});

Cloudflare Workers

If your site runs on a Worker, ctx.waitUntil keeps the report alive after the response has gone out:

export default {
  async fetch(request, env, ctx) {
    ctx.waitUntil(
      fetch("https://ingest.tracing.tools/crawl", {
        method: "POST",
        headers: { "content-type": "application/json" },
        body: JSON.stringify({
          s: env.TRACING_SITE_KEY,
          u: request.url,
          ua: request.headers.get("user-agent"),
          ip: request.headers.get("cf-connecting-ip"),
          r: request.headers.get("referer"),
        }),
      }).catch(() => {}),
    );

    return fetch(request);
  },
};

Report every page request, not just the ones you think are bots: the endpoint decides, and a user agent it does not recognise as a crawler is dropped rather than counted.

Excluded IPs

Settings → General → Excluded IPs. One address per line, or comma-separated.

203.0.113.9
198.51.100.42

The match is exact string equality against the client address the collector resolved. Consequences:

• No CIDR ranges. 203.0.113.0/24 matches nothing.
• No wildcards.
• IPv4 and IPv6 are different strings. A visitor whose browser prefers IPv6 needs the IPv6 address listed too — 2001:db8::1 and 203.0.113.9 are two entries.
• A dynamic home address changes, and the entry silently stops working.

Find your own address the reliable way: load your site, open the newest visit in Sessions — but the IP is not stored, so instead use any “what is my IP” service, and list both the v4 and v6 answers.

For a browser you control, localStorage.tracing_disabled = "1" is more reliable than an IP rule. See Opt-out & debugging.

Excluded paths

Settings → General → Excluded paths. Globs, one per line.

/admin/**
/wp-admin/**
/preview/*
/health

Matched against the pathname only — never the query string or the hash — and anchored at both ends.

Pattern	Matches	Does not match
/admin	/admin	/admin/users
/admin/*	/admin/users	/admin/users/7, /admin
/admin/**	/admin/users, /admin/users/7	/admin
*.json	/data.json	/api/data.json
**/draft	/blog/draft, /a/b/draft	/draft

* matches anything except /. ** matches across segments. Every other regular-expression metacharacter — ., +, (, [, $ — is escaped, so /v1.2/* matches only a literal /v1.2/….

To exclude a section and its index, list both:

/admin
/admin/**

Changes take effect within one minute: the collector caches the site row for 60 seconds.

data-exclude on the script tag

<script defer
  data-site="pk_live_xxxxxxxx"
  data-exclude="/admin/**,/preview/*"
  src="https://ingest.tracing.tools/t.js"></script>

Comma-separated, same two wildcards — but with one difference that matters: the tracker does not escape regex metacharacters. A . in a client-side pattern matches any character. Prefer the server-side setting unless you specifically want to avoid the request being made at all.

What exclusions do not do

• They do not remove data already collected. To start clean, use Settings → Danger zone → Reset site data, which deletes every event, session and trait for the site.
• They do not apply retroactively to the billing month. Events that were written before you added the rule have already been counted.
• Excluded paths do not exclude the visits those paths belong to. If a visitor sees /pricing and then /admin/x, the visit exists and its exit path is /pricing.

The bot device type still exists as a filter value, and the device dimension can return it. It is reachable only for rows written before crawlers moved to their own screen, or through data imported another way.

Related

• Crawlers
• Opt-out & debugging
• Settings
• Usage & overage