Common Crawl is a non-profit organisation that crawls billions of web pages every month and provides the raw data as an open archive. The archive comprises over 300 billion web pages from 19 years. It is one of the most important sources from which AI providers train the model knowledge of their language models.
How does Common Crawl work?
Common Crawl’s crawler, the CCBot, visits web pages like a search engine crawler and stores HTML, metadata and extracted text in standardised file formats. The data is published monthly as a new snapshot and is freely available to anyone. AI providers filter and clean this raw data before it goes into a training run; pages with little text, duplicates or poor quality frequently drop out at this stage.
Why does Common Crawl matter for AI visibility?
What is not in Common Crawl has a hard time in model knowledge. A website that blocks the CCBot, sits behind login walls or loads its content only via JavaScript appears in the training data not at all or only as an empty shell. Conversely, a page in Common Crawl is no assurance that a model has learned its content, but it is the prerequisite for it.
What does this mean for your website?
Check whether your important URLs appear in the Common Crawl index and whether the CCBot is allowed in your robots.txt. Deliver the text of your pages in the HTML, not only after rendering by JavaScript, and keep the source code lean; the AI providers’ cleaning step discards pages where markup outweighs content. How to handle LLM crawlers correctly overall is covered in its own entry.