Skip to content
tracing.toolstracing.tools

Crawlers

/dashboard/<siteId>/crawlers — who is reading the site that is not a person.

An assistant that cites a page fetched it first. A model that knows the documentation read it. This screen is the record of that: one row per crawler, one line per crawler over the range, and the pages they went for.

Categories

Section titled “Categories”

The tabs across the top split every hit by what the fetch was for, not by who sent it. One company often runs three crawlers with three very different jobs, and a site owner feels differently about each.

Category What it means Examples
AI answers An assistant fetched the page while answering a question someone had just asked. Usually a handful of pages, in real time, often with a citation. ChatGPT-User, Claude-User, Perplexity-User
Indexing Crawled for a search engine, or for the search index an assistant queries. Googlebot, OAI-SearchBot, PerplexityBot
Training Collected pages as training data for a model. GPTBot, ClaudeBot, CCBot
Other Everything else with a user agent: SEO tools, link-preview fetchers, uptime monitors, headless browsers and scripts. AhrefsBot, Slackbot, HeadlessChrome

The count on each tab is the hits in that category over the current range. The screen opens on AI answers, or on the first category with any hits if that one is empty.

Chart and list

Section titled “Chart and list”

The card on the left plots the busiest crawlers in the open category, one line each, in the same buckets as every other screen: hours for today, yesterday and 24h; days for 7d, 30d, 90d, week and month; months for 12m.

The list on the right is the legend and the ranking at once. Each row carries the crawler, the company behind it, and its hits. Picking a row narrows the chart to that one line and the pages panel to what that crawler fetched, and adds a line saying what the crawler is doing on the site. All crawlers puts it back.

Two entries are not real names:

  • Unknown bot — announced itself as a bot, crawler or spider with a token the directory does not recognise. The raw user agent is kept, so it can be named later.
  • Suspected automation — looked like a browser but behaved like a script.

Pages crawled

Section titled “Pages crawled”

The paths behind the current category, or behind the selected crawler, ranked by hits. Rows are not clickable: a crawler hit carries no visitor, so there is nothing to filter the rest of the dashboard by.

Recent

Section titled “Recent”

The last twelve hits, newest first, regardless of the range. Each says how it reached us:

  • ran the script — the bot executed the tracking script like a browser would, and the collector recognised it.
  • from server — your server reported the request.

Most AI crawlers never execute JavaScript, so they only ever show up the second way. See reporting crawlers from your server.

Opting a crawler out

Section titled “Opting a crawler out”

Blocking is done on your side, in robots.txt — this product records what happened, it does not sit in front of your traffic. Every crawler that publishes a robots.txt token honours a Disallow under its own user agent, and ignores the rules written for anyone else.

The tokens for the AI crawlers, by category:

Crawler robots.txt user-agent Operator Category
GPTBot GPTBot OpenAI Training
ChatGPT-User ChatGPT-User OpenAI AI answers
OAI-SearchBot OAI-SearchBot OpenAI Indexing
ClaudeBot ClaudeBot Anthropic Training
Claude-User Claude-User Anthropic AI answers
Claude-SearchBot Claude-SearchBot Anthropic Indexing
Google-Extended Google-Extended Google Training
PerplexityBot PerplexityBot Perplexity Indexing
Perplexity-User Perplexity-User Perplexity AI answers
Meta-ExternalAgent Meta-ExternalAgent Meta Training
Applebot-Extended Applebot-Extended Apple Training
Bytespider Bytespider ByteDance Training
CCBot CCBot Common Crawl Training

So, to keep the site out of model training while staying in search and staying citable:

robots.txt
User-agent: GPTBot
Disallow: /


User-agent: ClaudeBot
Disallow: /


User-agent: Google-Extended
Disallow: /


User-agent: Applebot-Extended
Disallow: /


User-agent: Bytespider
Disallow: /


User-agent: CCBot
Disallow: /

A Disallow stops a well-behaved crawler from fetching. It does not stop it being recorded here if it fetches anyway — which is the point: the screen keeps showing you whether the rule is being honoured.

URL parameters

Section titled “URL parameters”
Parameter Values Default
range today, yesterday, 24h, 7d, 30d, 90d, 12m, week, month 30d
category answers, indexing, training, other answers
crawler a crawler slug from the list, e.g. gptbot none

Dashboard filters do not apply to this screen, so the filter bar is not shown.

Section titled “Related”