Crawling
01 Definition
Crawling meaning: The process search engines use to discover pages and follow links.
Crawling occurs when an automated program requests web resources and follows links to find new or updated content. A crawler must usually discover a page before the page can be indexed.
Also called
spideringweb crawling
Why it matters
How Crawling fits the work
Efficient crawling makes important content easier for search engines to find and refresh.
In context
The audit found that orphan pages were not being reached during crawling.
Practical note
Internal links sitemaps server performance and crawl directives all shape crawl access.