URL Discovery is the first step of every indexing process: a crawler has to know a URL before it can fetch it. It learns URLs through links on pages it already knows and through the XML sitemap. Can crawlers find the relevant content at all? That is the question that comes before crawl budget, indexing and ranking.
How does URL Discovery work?
A crawler starts from known URLs — the homepage, the sitemap — and follows the HTML links it finds there. Every linked page becomes the next known URL. Pages without an incoming HTML link are orphans: they are found only if they are listed in the sitemap, and even then the crawler rates them as unimportant because nothing points to them. LLM crawlers do not execute JavaScript; links that only appear via script do not exist for them.
Click walls and login walls break the chain at a different point: the URL is known, but the content behind it cannot be reached without an interaction.
Why does URL Discovery matter for AI visibility?
Because it is the lowest link in the chain: URL Discovery, crawl management, index management, linking, content quality. A service page that is reachable only through a JavaScript menu is not discovered by AI crawlers, not taken into training and not fetched for a live answer — regardless of its quality.
What does this mean for your website?
All important pages must be reachable via HTML links — from the navigation, from related pages, from the footer. Keep the sitemap current. Do not put click or login walls in front of content that is meant to be found. And check the link graph: pages with only one incoming link are hard to find for humans too.