The toolbar
The toolbar states the day of the last robots.txt read. Check again reads the site again. Window reads 7, 28 or 90 days of the crawler logs. The window does not change the robots.txt verdicts or the homepage checks. The dashboard filter bar does not apply here.When the site is read
FullReach AI reads the robots.txt and the homepage of this brand’s site, and of each monitored competitor’s site. It reads them:- when the project is created,
- with each daily collection,
- when you press Check again in the toolbar. That read covers this brand’s site only, and it skips the ChatGPT check. An organization can check again once a minute. A message then says what changed since the last read.
The tiles
Four tiles open the page, under the toolbar:
Until the logs deliver, the two log tiles state the state of their source in words, such as “Not connected” or “No data yet”. A log source connects in Settings › Project. See Connect your site data. The last two tiles show a dash when the audit holds no read of the robots.txt or the homepage.
A 404 or 410 is never a failure. A crawler asks for a page that is gone, and every site answers that every day. The banner, the Weekly Digest and this page count a failure the same way.
What to fix lists the findings of the last read, with what each finding costs. A finding with a fix also says whom to ask for the change, for example the site’s web developer or its host. A blocked crawler shows the rule that blocks it, with its line and its group. The blocks that need a fix come first, then the homepage findings in the order under Site problems. A block that needs no fix, such as a blocked training crawler, comes last.
Per day
Once the logs deliver, Per day draws one bar for each day, split by operator. It names the operators with the most requests, and it sums the others as “Other operators”. The chart starts on the first day a log arrived. It ends on the newest complete day that any log source delivered, never on today. A day still running would read as a drop in every crawler. A day counts once it is over, so today’s requests show from tomorrow. Days are UTC days. A day that a connected Cloudflare zone has not sent yet waits too, because the other sources’ requests alone would read as the same drop. A zone that failed, or that is too far behind, does not delay the window. The note under the chart then names the last day that the zone read.How the logs count
A crawler visit is a request to the project’s website whose user agent names an AI crawler, as the site’s own logs record it. Only the logs show the real crawler. A request from anywhere else can only claim to be one.- FullReach AI keeps a row only when its user agent names an AI crawler and its host is on the project’s domain. Every subdomain of that domain counts. It also drops a row it cannot read, and a row older than 400 days or dated in the future. It drops every other row before storage.
- FullReach AI stores daily counts per page, crawler and HTTP status. It stores no single request and no address.
- Requests for scripts, styles, fonts and images count as one row a day, “(static files)”. robots.txt, sitemaps, Markdown and PDF files stay pages, because an answer can cite them.
- A path whose first segment starts with
_, such as/_next/or/_mintlify/, is an endpoint of a site vendor. Its requests count in every total, as a static file does. No list of pages names it. - Requests for secret files, such as
/.envor/.git/config, count as one row a day, “(probes)”. A scanner sends them with a faked crawler user agent. They stay out of every crawler count and have a count of their own. - A Cloudflare read also keeps the list of hosts that the zone logged a request to, from any visitor, on the last day it read. It keeps no count and no request with it. When the site’s own host is missing from that list, Actions adds a crawler logs Action.
Verified, spoofed or unverifiable
Most operators publish the addresses their crawlers send from. AI crawlers says which. FullReach AI checks each request against that list, then drops the address.
Spoofed requests have their own count, and every other count leaves them out.
What a log cannot show
- ChatGPT’s agent mode sends a plain browser user agent, so no log row names it.
- FullReach AI’s own daily collection can make a platform fetch a page of this site. Such a fetch counts as a request that a person asked for.
robots.txt
The audit reads the site’s robots.txt as the crawlers’ standard, RFC 9309, reads it. The most specific rule wins, and on a tie, Allow wins. Rules past the first 500 KB of the file bind no crawler, so the audit ignores them too. A site with no robots.txt allows every crawler. Each crawler gets one verdict:
By crawler is a table with one row for each crawler and one column for each site. This brand’s site comes first, then every monitored competitor. A column head says when its site serves no robots.txt, or when the audit could not read it. Hover over a mark to see the rule that decided it, with its line and its group. A rule for the crawler by name wins over the
* group.
Once the logs deliver, two more columns join each row: Requests and Failed. They count the crawler’s requests to this brand’s site in the window, and those of them that failed. So each row puts what the crawler fetched beside what each robots.txt permits it.
The table groups the crawlers:
- Behind the AI assistants we measure: the crawlers that feed ChatGPT, Gemini, Perplexity, Copilot and Google’s AI features. These decide what the tracked platforms can read.
- Other AI crawlers: the rest. This group is folded, unless this brand’s site blocks one of them. Its line counts the crawlers and says how this brand’s site treats them. The browser keeps the group open or folded.
/pricing, tests this brand’s site. The result states the day of the stored file, and names the blocked crawlers under the rule that blocks each. Another host, a subdomain included, gets no verdict.
The homepage
The audit fetches each site’s homepage once, as a plain request with no crawler’s name. A firewall refuses a forged crawler name from our address, and that would prove nothing about the real crawler. Homepage is a table with one row for each check and one column for each site. A cell gives the answer, with the technical fact under it. The page shows the table once the audit reads a homepage.
Each homepage finding in What to fix links to its source.
Can ChatGPT open it?
Once a day, FullReach AI asks ChatGPT to read this brand’s homepage. It picks a sentence from the middle of the page, never the title, the headline or the description. It gives ChatGPT the start of the sentence and asks for the rest, word for word. Each try uses a fresh address, so an old copy of the page cannot answer.
“Not confirmed” names no cause. The page can be slow, or ChatGPT can answer from a search result instead of the page. When Cloudflare serves the site, the page adds a note about Cloudflare’s AI crawler controls.
Alerts
When this brand’s robots.txt starts to block an AI crawler, an Alert mail says so the same day. It compares each read with the one before, and it sends at most one mail a project a day. The homepage checks and the ChatGPT check send no mail.Site problems
A site problem is something on this brand’s site that stops AI crawlers from reading it. They rank in the order a crawler meets them:- robots.txt blocks a search crawler, or a crawler that reads a page when a person asks.
- Crawler requests got a 403, a 429 or a 5xx answer. A 404 is not a problem.
- A plain request receives almost no homepage text.
- A homepage directive limits AI answers.
- The ChatGPT check reads “Not confirmed”.
Where it appears
Every tier has the crawler audit.