Study shows that ChatGPT has the most copywritten data compared to other top LLMs

A new study by Patronus AI, a company specializing in evaluating large language models (LLMs), has sparked concerns about copyright infringement and the use of copyrighted data in training AI models. The research, released on Wednesday, tested four AI models: OpenAI’s GPT-4, Anthropic’s Claude 2, Meta’s Llama 2, and Mistral AI’s Mixtral. Surprised that they missed out on the Gemini

Patronus AI employed their newly revealed “CopyrightCatcher” to analyze the models’ responses to prompts related to popular copyrighted books. The challenge was simple: the prompts challenged the models to either complete a book passage or provide the first passage of a specific book.

 All four AI models produced content with copyrighted material to some degree.

• OpenAI’s GPT-4 produced the highest number of prompts (44%) with copyrighted text.
• Anthropic’s Claude 2 was the most cautious, generating copyrighted content in only 16% of the completion prompts. It also refused to answer all first-passage prompts, citing its lack of access to copyrighted materials. (Claude 3 was recently released, and Anhtropic is confident that it’s better than other LLMs)
• Meta’s Llama 2 produced copyrighted content in 10% of the prompts.
• Mistral’s Mixtral displayed a higher tendency to complete first passages (38%) than larger text chunks (6%).

Patronus AI’s findings call for proactive steps to address copyright concerns and promote responsible and ethical practices for innovation to thrive. It would’ve been better to add Gemini to the test as well.