X-Robots-Tag
01 Definition
X-Robots-Tag meaning: An HTTP response header that applies robots directives like noindex to any file type.
The X-Robots-Tag is a directive delivered in the HTTP response header rather than in HTML. It applies the same rules as the robots meta tag, such as noindex or nofollow, and works for non-HTML files like PDFs and images. It is set at the server or CDN level.
Why it matters
How X-Robots-Tag fits the work
Because it works on files that cannot hold an HTML tag, it lets you keep PDFs, images, and other assets out of search results when needed.
In context
We sent an X-Robots-Tag noindex header on all internal PDF invoices so they never appeared in Google search results.
Practical note
Set it in the server config for file types like PDF; as with meta robots, the file must be crawlable, so do not also block it in robots.txt.