Crawling, explained
Crawlers mostly find pages through links and sitemaps. A new page with no links pointing to it can be hard to find, however good it is.
Crawling is not the same as ranking. It is only the first step. After a page is fetched, the search engine decides whether to index it, and only indexed pages can show in results.
You can guide crawlers with robots.txt, which asks them to skip certain paths. Well-behaved bots follow it, but it is not a security tool, and it does not reliably keep a page out of results.
Search Console's Crawl stats report shows how Google visits your site. AI companies run crawlers too, and you can decide in robots.txt which ones you allow.



