Crawler
01 Definition
Crawler meaning: An automated program that browses the web and fetches pages by following links.
A crawler is software that systematically visits web pages, reads their content, and follows links to find more pages. Search engines use crawlers to build their indexes, but many tools and services run crawlers too. Well-behaved crawlers identify themselves and follow robots.txt rules.
Also called
Why it matters
How Crawler fits the work
Crawlers are how search engines and many SEO tools discover content, so understanding them helps you control what gets fetched and how your site is represented.
In context
The SEO tool ran a crawler across the site and flagged 30 pages with missing title tags and broken internal links.
Practical note
Not all crawlers are search engines; some scrape content or check links, and some ignore robots.txt. Watch server logs to tell helpful crawlers from abusive or fake ones.