A handful of dominant AI corporations have been quietly manipulating one of the vital influential public leaderboards for chatbot fashions, doubtlessly distorting perceptions of mannequin efficiency and undermining open competitors, in accordance with a brand new examine.
The analysis, titled “The Leaderboard Phantasm,” was revealed by a staff of consultants from Cohere Labs, Stanford College, Princeton College, and different establishments. It scrutinized the operations of Chatbot Enviornment, a extensively used public platform that enables customers to match generative AI fashions by pairwise voting on mannequin responses to consumer prompts.
The examine revealed that main tech corporations — together with Meta, Google, and OpenAI — got privileged entry to check a number of variations of their AI fashions privately on Chatbot Enviornment. By selectively publishing solely the highest-performing variations, these corporations have been capable of increase their rankings, the examine discovered.
