User Agent
01 Definition
User Agent meaning: A text string that identifies the browser or bot making a web request.
A user agent is a line of text a browser or crawler sends with each request to identify itself, including its name and sometimes version and platform. Servers and robots.txt can respond to specific user agents. Because it is self-reported, a user agent can be spoofed.
Also called
Why it matters
How User Agent fits the work
Robots.txt rules and server logic often target specific user agents, so knowing them lets you allow, block, or analyze particular crawlers and browsers.
In context
The robots.txt file allowed the Googlebot user agent to crawl the site while blocking a scraper that used a suspicious custom user agent string.
Practical note
User agent strings are easy to fake, so do not rely on them alone to confirm a real crawler. Use reverse DNS or official verification for bots like Googlebot.