LLM crawlers

LLM crawlers are the user agents with which AI providers retrieve web pages. There are two kinds: training crawlers such as GPTBot, ClaudeBot or CCBot collect content for the model knowledge of future models. Retrieval agents such as ChatGPT-User or PerplexityBot fetch a page live when a system performs a web search for an answer. Both read the delivered HTML — and neither of them executes JavaScript.

How do LLM crawlers work?

An LLM crawler retrieves a URL, reads the HTML, extracts the text and follows links if it is configured to do so. It respects robots.txt and can be blocked or allowed specifically through it. It does not wait for lazily loaded content, does not render scripts and does not repeat a request if the server responds too slowly — at the SISTRIX Meetup in September 2026, Chrissy Kunisch (ONE Beyond Search) cited a TTFB below 500 to 800 milliseconds as the benchmark above which requests are aborted.

A retrieval in the log file does not yet mean visibility: the agent validates a page as a candidate; it does not cite it for that reason (log files).

Why do LLM crawlers matter for AI visibility?

They are the only route by which your content reaches an AI system — into training or into a live answer. What a crawler cannot retrieve, cannot read or cannot load in time does not exist for the system. That is why the technical side is the entry ticket: it gets a page into the race, but it does not decide whether the page wins.

What types of AI bots are there?

Cloudflare groups AI bots by behaviour into three controllable categories (Cloudflare Radar, September 2026):

The lines are blurring: according to Cloudflare, more than 36% of crawler activity comes from mixed crawlers that combine search, agent use and training (July 2026). The distinction still matters, because each category can be controlled separately. Anyone who blocks OAI-SearchBot will not normally appear in ChatGPT’s search answers; training access via GPTBot is controlled independently of that (Kai Spriestersbach, AFAIK, August 2026). For agents that operate websites, see AI agents.

Google-Extended is not a bot but a control token in robots.txt. It governs whether Google may use content to train Gemini, and therefore does not appear in log files (Seokratie, August 2026). According to Google, Google-Extended does not affect inclusion in Google Search.

What does this mean for your website?

Check your robots.txt for blocked AI bots. Deliver important content server-side in the HTML, not via JavaScript. Keep the server response time low (TTFB) and the DOM lean. Avoid click walls and login walls in front of content that should be found. And analyse your log files for AI user agents to see which pages the systems actually retrieve — with the caveat that Google AI Overviews and AI Mode usually do not generate a live retrieval.