Google’s John Mueller mentioned it might make sense to noindex your LLMs.txt file. This manner, the file would not get listed by engines like google, after which one way or the other a person lands on it and is confused.
John wrote on Bluesky, “That mentioned, utilizing noindex for it might make sense, as websites would possibly hyperlink to it and it might in any other case change into listed, which might be bizarre for customers.”
The query was:
Will Google view LLMs.txt recordsdata as duplicate content material? It appears stiff necked to take action, on condition that they know that it is not, and what it’s actually for.
Ought to I add a “noindex” header for llms.txt for Googlebot?
John Mueller replied in full:
It might solely be duplicate content material if the content material have been the identical as a HTML web page, which would not make sense (assuming the file itself have been helpful). That mentioned, utilizing noindex for it might make sense, as websites would possibly hyperlink to it and it might in any other case change into listed, which might be bizarre for customers.
John beforehand mentioned that so far as he is aware of no AI system at present makes use of llms.txt.
In any occasion, a variety of websites are adopting it, will you noindex it?
Discussion board dialogue at Bluesky.
