Can AI read your website?
Buyers now ask ChatGPT, Perplexity and Gemini before they Google. But can AI even read your website? We check live which AI crawlers are allowed in — and which ones you are (often unknowingly) locking out.
Checked live, in seconds.
Buyers ask AI before Google.
robots.txt live
We fetch your domain’s public robots.txt — no signup, no cache.
Per AI crawler
GPTBot, ClaudeBot, PerplexityBot, Google-Extended & co.: allowed, blocked, restricted or unlisted.
llms.txt
We check whether a curated table of contents for AI systems exists.
What it means
From total blackout to showcase setup — with the next sensible step.
This check shows the door. We show who gets cited.
This check shows the door. Whether AI assistants actually cite you behind it — and who gets cited instead — is what our full audit shows: competitor comparison, per-engine citation analysis and a prioritised action plan.
Sites that lock out AI crawlers do not exist in AI answers.
The AI access check verifies live whether the ten most important AI crawlers — GPTBot and OAI-SearchBot (ChatGPT), ClaudeBot (Claude), PerplexityBot, Google-Extended (Gemini), Bingbot (Copilot), Applebot, CCBot and Meta-ExternalAgent — are allowed to read your website according to robots.txt, and whether an llms.txt exists as curated context for AI systems.
Buyers increasingly ask ChatGPT, Perplexity or Google AI Overviews before they run a classic search. These systems do not rank ten blue links — they cite a handful of sources. A website whose robots.txt blocks AI crawlers (often unintentionally, through blanket bot rules) cannot appear in those answers. That is the new invisible visibility loss: no error, no warning, just absence.
The result shows one of four statuses per crawler: allowed, blocked, partial or unlisted. What follows from it — a per-vendor allowlist, llms.txt, machine-readable fact sheets — is part of our GEO work.
What is an llms.txt?
A text file in the website root (like robots.txt) that gives AI systems curated context: what the company does, for whom, with links to the most important pages. It does not replace SEO, but it makes it easy for AI assistants to describe and cite you correctly.
Should I allow all AI crawlers?
For visibility in AI answers: yes for the search and answer crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended). You can differentiate on training-only crawlers like CCBot — blocking those does not cost citability in the major answer engines.
Could my website be blocking AI crawlers without me knowing?
Frequently, yes. Many security plugins, CDN bot filters and old robots.txt templates block everything that is not called Googlebot. This check shows it in ten seconds — per crawler, with the source of the rule.
Is AI access enough to get cited?
No — access is the precondition, not the guarantee. What gets cited is extractable answers, clear structure, named entities and sourced numbers. That is exactly what we structure pages for in the GEO & SXO pillar.