OpenAI Holds Back on Releasing Reliable Cheating Detection Tool for ChatGPT
San Francisco, August 5, 2024 – Despite widespreadconcerns about students using artificial intelligence (AI) to cheat, OpenAI, the company behind the popular chatbot ChatGPT, has decided not to release a reliable cheating detectiontool it has developed.
According to sources familiar with the matter and internal documents reviewed by The Wall Street Journal, the project has been mired in internaldebate for nearly two years, with a release planned about a year ago. It was just a button push away, one of the sources said.
The decision to hold back stems from a clash between OpenAI’s stated commitment totransparency and its desire to attract and retain users. A survey conducted by the company among its loyal ChatGPT users revealed that nearly one-third would abandon the platform if anti-cheating technology were implemented.
We are developing text watermarkingmethods that are technically promising, but there are also significant risks that we are weighing, and we are also exploring alternatives, said an OpenAI spokesperson. We believe that given the complexities involved and its potential impact on the broader ecosystem beyond OpenAI, a cautious approach is warranted.
Employees who support the release, includingthose involved in developing the tool, argue that these concerns pale in comparison to the potential benefits of the technology. Generative AI can produce entire essays or research papers in seconds for free based on a simple prompt. Teachers and professors say they desperately need help combating the misuse of generative AI.
It’s a hugeproblem, said Alexa Gutterman, a high school English and journalism teacher in New York City. Every teacher I work with is already talking about it.
A recent survey by the Center for Democracy & Technology, a non-profit organization focused on technology policy, found that 59% of middle and high schoolteachers believe some students have used AI to help with their schoolwork, an increase of 17 percentage points from the previous year.
OpenAI CEO Sam Altman and Chief Technology Officer Mira Murati were involved in discussions about the anti-cheating tool. Some sources say Altman was encouraging of the project but did notpush for its release.
The tool in question works by subtly altering the way ChatGPT selects words or word fragments, known as tokens, in a sentence. These changes leave a pattern, or watermark, that is invisible to the naked eye but can be detected by OpenAI’s technology. The detector provides a score indicatingthe likelihood that a document or part of a document was written by ChatGPT.
According to internal documents, these watermarks are 99.9% effective when enough new text is created by ChatGPT. The chances of that term paper being unwatermarked are less than the sun not rising tomorrow, said John Thickstun, a researcher at Stanford University who is part of a team developing a similar method for watermarking AI text.
However, OpenAI employees are concerned that the watermarks could be easily removed by simple techniques, such as translating the text into another language and back using Google, or having ChatGPT add emojis to the textand then manually removing them.
Another concern within the company is determining who would have access to the detector. If too few people have access, the tool would be of little use. If too many people have access, bad actors could potentially crack the company’s watermarking technology.
OpenAI employees have discussed providingthe detector directly to educators or to external companies that help schools identify AI-generated essays and plagiarized work.
Google has developed a watermarking tool that can detect text generated by its Gemini AI. The tool, called SynthID, is currently in testing and not yet widely available.
OpenAI has a toolthat can determine whether an image was created using its text-to-image generator DALL-E 3, which was released for testing this spring. According to informed employees, OpenAI has prioritized watermarking audio and visual materials over text because the risks associated with the former are greater, especially in this busy election yearin the United States.
In January 2023, OpenAI released an algorithm designed to detect text generated by multiple AI models, including its own. However, it had a success rate of only 26%, and OpenAI withdrew the algorithm seven months later.
External companies and researchers have also developedother tools to detect AI-generated text, many of which are used by teachers. However, these tools sometimes fail to detect text written by advanced large language models and can produce false positives.
Initially, students thought we had some kind of magic wand that could tell if they were using AI, said Mike Kentz, an educator who provides AI consulting to educators at a private high school in Georgia. By the end of the year…they’re like, ‘Wait, my teacher has no idea.’
Some teachers encourage students to use AI to assist with research or to get feedback on ideas. The problem arises when students useapplications like ChatGPT to complete all their assignments, without even knowing what they are submitting.
Last year, Josh McCrain, a political science professor at the University of Utah, assigned a writing assignment with instructions in small, difficult-to-read font that mentioned Batman. If students copied and pasted their assignments into AI,the instructions would be included.
Sure enough, a few students submitted papers that inexplicably mentioned Batman. Looking forward, McCrain is adjusting his writing assignments to focus more on current events that AI is less familiar with and is urging students not to outsource their work to AI. The point I try to hammer hometo students is: You need to learn this stuff, he said.
The debate over the watermarking tool began before OpenAI launched ChatGPT in November 2022 and has been a source of tension. The tool was developed by Scott Aaronson, a computer science professor at the University of Texas who has beenon leave for the past two years to conduct research on safety at OpenAI.
In early 2023, OpenAI co-founder John Schulman outlined the pros and cons of the tool in a shared Google document. OpenAI executives then decided to seek input from various stakeholders before taking further action.
Over the next year and a half, OpenAI executives repeatedly discussed the technology and sought new data to help them decide whether to release it. Internal documents show that a survey commissioned by OpenAI in April 2023 found that respondents globally supported the idea of using AI detection tools by a 4-to-1 margin.
That same month, a survey of ChatGPT users by OpenAI found that 69% of users believed that cheating detection technology would lead to false accusations against AI users. Nearly 30% of users said they would use ChatGPT less if it deployed watermarks while competitors did not.
One recurringinternal concern was that the anti-cheating tools could affect the writing quality of ChatGPT. Sources say OpenAI conducted a test earlier this year that found the watermarks did not impair ChatGPT’s performance.
According to these internal documents, employees involved in the test concluded: Given that we now know that the tooldoes not degrade output quality, our arguments against using text watermarking are less compelling.
In early June, senior OpenAI employees and researchers met again to discuss the project. They agreed that the watermarking technology worked well, but the results of the ChatGPT user survey from last year remained significant. According to those familiar withthe meeting, staff said that OpenAI should explore other, potentially less controversial but unproven methods.
Internal documents show that attendees also said that OpenAI needed to develop a plan by this fall to guide public opinion on AI transparency and potential new laws on the subject.
Views: 0