Just lately, Google stated that no AI system is at present utilizing the LLMS.txt file. However perhaps some are beginning to? OpenAI could also be beginning to uncover and crawl LLMS.txt information on web sites. Whereas Google’s Gary Illyes stated they will not be doing that.
Ray Martinez posted a screenshot of his log information displaying how OpenAI is crawling his LLMS.txt file each quarter-hour or so. He posted this screenshot on X and wrote, “Log file evaluation reveals that OpenAI crawls my LLMs.txt file on a number of websites. It is pinging our servers each quarter-hour searching for freshness.”
Right here is the screenshot:
Have you ever seen this throughout any of your websites?
In the meantime, Gary Illyes from Google, whereas on stage on the Google Search Central Stay Deep Dive occasion, stated Google would not help LLMs.txt and is not planning to. This was captured by Kenichi Suzuki who posted about this on LinkedIn.
Right here is a photograph:
Anthropic has docs on LLMS.txt, so does ElevenLabs and PineCone – as I’m certain some others do too. Right here is an effective weblog submit on Are LLMs.txt Recordsdata Being Carried out Throughout the Net?
Discussion board dialogue at X.
