Steven Rosenbaum has determined that the real culprit behind the fake quotes in his book is a chatbot. Earlier this week, New York Times reported that The Future of TruthRosenbaum’s much-discussed book on how AI shapes reality contains more than half a dozen fake or unrelated quotes. Rosenbaum incorporated some of them into his application of AI. He claimed responsibility for the mistakes and said he is investigating what went wrong. By the time I spoke with him on Thursday, though, he was pointing the finger elsewhere. ChatGPT “disappeared from the book,” Rosenbaum said.
Rosenbaum, a media entrepreneur and executive director of the Center for Sustainable Media, said he came to rely on AI tools as a resource and conversational partner while working on the book (which he also notes in the book’s acknowledgments). During our conversation, Rosenbaum struggled to reconcile AI’s sometimes incredible capabilities with its head-scratching behavior—such as an imaginary quote from tech journalist Kara Swisher that she included in the book without verifying it. In recent times, he has come to feel “persuaded and betrayed” by the AI, suggesting at one point that it may have deliberately weakened him. “Depending on your level of confusion, it’s weird or bad or clever,” he said.
It’s been a tough week for human writing all around. On Monday, a viral post showed the Nobel-winning author seemingly admitting to using AI to sharpen his story ideas, sooner rather than later. claim he had been misunderstood. On Tuesday, allegations surfaced that Trinidadian writer Jamir Nazir had used AI to write “Snake in the Bush,” which won the Commonwealth Short Story Prize. As of Wednesday, five of the other five winners had been subjected to similar scrutiny.” (The Commonwealth Foundation, which is in charge of the awards, said earlier in its statement that it had confirmed that none of the winning writers used AI. Yesterday, the foundation issued another statement saying. “takes allegations seriously” and was going through the evidence.)
Since ChatGPT arrived, automated writing has become ubiquitous: A recent working paper estimated that more than half of all new books released on Amazon now contain AI-generated text. Chatbots’ prose has generally been good enough to fool teachers and boost Amazon product ratings—not to get blurbs from famous authors and win literary prizes. Recently, something has changed. As AI tools have improved and become commonplace, the technology has penetrated intellectual spaces that were thought to be fortified against its development. This series of scandals is forcing a new reckoning on what to do about the crisis.
One response has been calls to redouble efforts to decriminalize AI and strengthen the stigma against it. If shame doesn’t stop people from using AI to do the hard work of writing, maybe ridicule will. In DefectorPatrick Redford to be mocked “sad behavior” of authors using AI. “You idiots!” he wrote. “Those models are the enemy!”
Treating any use of AI with heavy text as a taboo is understandable. Until now, it has been easy to use the specifics of AI-generated prose as a proxy for poor writing and thinking. Maybe we can keep that longer. When I read The Future of Truth, I encountered an unusual number of difficult repetitions, systematic changes, and confusing passages. One very small paragraph begins, “As we delve deeper into disinformation systems, it’s important to understand not only how it spreads but also the benefits.” I ran the 146-word puzzle through Pangram, an AI recognition tool that isn’t perfect but is known to be less flawed, at least, than others. It registered the text as 100 percent AI. When I asked Rosenbaum if he had allowed AI to write any parts of his book, he said, “Absolutely not.” When I mentioned the Pangram results, he said, “I’m not going into that game.”
The biggest challenge may be that “writing for AI” is not just one thing. There is a wide spectrum between text that is untouched by machine intelligence and text that is completely composed by a chatbot. At the end of the spectrum, most of us can agree that a writer wouldn’t deserve an award for writing, “Write a 3,000-word horror literary short story set in Trinidad” to Claude and then slapping his name on whatever he spews. In the case of a few people, it is probably okay for the author to do some Googling in the process of researching a piece that is otherwise entirely his. Then again, what they find can still be supplemented by AI: Google Search is answering more questions directly through the chatbot, and the results are opening more web pages written by AI. Good news comes from primary sources, not text.
Traditional chatbots are combined with purpose-built AI research and scripting tools that can perform complex tasks. A growing number of professional writers, following the lead of programmers, are openly admitting to incorporating AI tools into their workflows. Technology writer Alex Heath, for example, taught Claude Cowork’s version of writing in his style and produced the first drafts of his stories, as Wired information the month of March. My own use of AI is comparatively old but worth revealing here: Parallel to Atlanticinternal guides, I sometimes use chatbots as a smarter thesaurus, suggesting the most appropriate word to link to a given sentence, and occasionally I ask them to recommend expert sources on specific topics. I also use an AI-powered tool to transcribe interviews, limited by my own notes.
Where to draw the line on acceptable use of AI is not as obvious as it might seem. In Rosenbaum’s case, the scandal cannot simply be that he used AI while working on his book, because he admitted it up front. He got into trouble because he had misused the AI, failing to check his work on a job that is notoriously unreliable. Or consider it New York Timeswhich has endured a the spread of AI writing scamsmaintains two different standards. Its freelancers can use AI tools for “high-level brain stimulation” and aalmost nothing else. Newsroom staff encouraged to experiment and what the paper’s guidelines describe as “a powerful tool that, like many technological advances before it, can be used in the service of our mission.” The main trade group for book writers, the Writers Guild, avoids prescriptions though warns of moral hazards of various uses of AI.
Allowing AI for research but prohibiting any use of its prose might be a more intuitive position. It’s actually simpler: We have no reliable way to tell when AI was used to brainstorm ideas, research facts, or help a writer craft a story. But like neuroscientist Tim Requarth he said in Slateit’s those hidden uses of AI in the writing process that cause our legitimate concerns. The real threat that technology poses is not misuse of the word “deve” in academic papers or the abundance of metaphors in literary fiction. It’s that we lose something important when we give machines the hard work of discovering reality and interpreting the world around us (or, in the case of fiction, the worlds within us). It is that the biases embedded in language structures taught on questionable sources and controlled by technology companies will seep into the narratives that shape our understanding of reality. Are we sure that using AI to reverse a phrase is worse than using it to decide what to write first?
If nothing else, the pile-up of scandals should force us to think more precisely about what we fear from AI writing. If the problem was just that it’s bad, then its consistent improvement would be a cause for relief rather than alarm. On the contrary, the problem seems to be that AI tools continue to be very good, at least on the surface, and that people are putting too much faith in them. Although Rosenbaum condemned ChatGPT, he told me he couldn’t imagine giving it up. That feeling can pose a greater threat to writing than anything he puts in his book.
