Robots.txt
01 Definition
Robots.txt meaning: A site-level file that gives web crawlers access instructions.
Robots.txt is a plain-text file at the root of a host that states which paths named crawlers may or may not request. It does not reliably keep a URL out of search results.
Also called
robots exclusion protocol
Why it matters
How Robots.txt fits the work
It helps manage crawler access to unhelpful or resource-intensive areas.
In context
The developer blocked crawler access to faceted parameter combinations in robots.txt.
Practical note
Never use robots.txt as a security control because the file is public and voluntary.