Every knowledge-based profession may one day reach a point where AI will outcompete human experts. In medicine, that day seemed to come in April. A group of Harvard and Stanford researchers announced the results of the study pitted ChatGPT against hundreds of doctors in a diagnostic obstacle course involving written medical puzzles and statements from real-world patients. The bot had won, and the humans weren’t exactly happy about it.
“I get a little concerned about how some of these findings might be applied,” Adam Rodman, lead author of the study, said at a press conference before publication in the journal. Science. The work was a professional exercise, he told reporters; As comprehensive as it was, it did not prove that ChatGPT or any other AI tool was ready to become a regular part of medical practice. His caution matched that of the others expertsbut as Rodman knew, most people will ignore the warning. AI has already entered the US healthcare system, evidence and protections be damned.
Even as I was watching Rodman’s press conference, I got a message on my phone from the management of the medical center where I work as a pathologist. They emailed me to say that an “AI powered clinical tool” is now available for me to use. This wasn’t the first time I got emails like this; it wasn’t the second or third time either. In fact, I’ve lost count of how many AI products have been released to us in recent years, there is none approved for medical use by the FDA.
This passion feels unparalleled. Healthcare is usually among the last fields to adopt new technologies; I still use the page, and I send faxes regularly. (Younger readers can ask Claude to explain what these things are.) The trend toward simple technology is partly a product of the safety-conscious culture of physicians: We know that any mistimed error can turn fatal. But these days, clinicians are allowed — encouraged, even — to run with the latest software, guided by the usual warning that “AI can make mistakes.”
Those mistakes can be the result. Although Rodman’s research shows that generative AI can help diagnose rare diseases or make sense of unusual symptoms, a randomized trial which was published in NEJM AI just a week before it found that deliberately erroneous results from an AI model could easily mislead doctors. Non-professionals can be similarly misled. Soon reading and Oxford scientists found that using AI did not significantly improve patients’ ability to recognize themselves or others. Another one oneled by researchers at Mount Sinai, suggested that chatbots may fail to alert users to potential medical emergencies.
Misdiagnosis is not the only concern. As AI permeates the healthcare system, mistakes are popping up in unexpected places. When I spoke with Rodman by phone after his press conference, he told me that he was surprised one day to find that his hospital, Beth Israel Deaconess Medical Center, had enlisted an AI to prepare messages for patients on his behalf—sometimes producing results for his audit that he described as “absolute nonsense.” (Sarah Finlaw, a spokeswoman for Beth Israel Lahey Health, told me that the use of AI tools is voluntary and depends on training and hospital support. She also said that any results from AI tools must be approved by a doctor.)
Part of the problem is that health-related AI products can be shipped without any review by FDA officials. If a software package intended for physicians is classified as “a clinical decision support tool,” and is not a medical device, usually escapes the attention of an agent. To count in this category, an AI-based program must generally rely on existing medical literature, refrain from analyzing medical examinations or images, explain the cause, and leave diagnosis and treatment to the physician. Most of the generative-AI products that doctors use today seem to meet these criteria.
Consumer wellness programs and devices can also pass FDA inspection as long as they are intended “Maintaining or promoting a healthy lifestyle” and not for diagnosing or treating specific conditions. With this in mind, Microsoft, OpenAI, Anthropicand xAI all warn users that their health-related chats are not intended to provide medical care or make recommendations for diagnosis and treatment. In practice, though, the distinction is not always clear. Elon Musk encourages people to use his Grok chatbot to develop second medical opinion and translation their X-ray and MRI images; a marketing video for Health ChatGPT shows a program that reassures people that their lab results are in good shape and encourages them to continue taking their cholesterol medications.
Many of these apps also invite users to connect their medical records with wearable health devices. AI companies won’t need to add all this data just to provide general health information. A new product from medical company Hims & Hers, called Labs AI, goes so far as to help interpret results from “130 biomarker tests” for its users and then provide “comprehensive, personalized, and actionable analysis of whole-body health, risk and patterns.” I, too, analyze a patient’s lab results and then provide personalized, actionable advice. What is the difference?
When I contacted the manufacturers of these products, they again confirmed that no real medical advice is given to consumers. Dominic King, vice president of health at Microsoft AI, told me in an emailed statement that its Copilot app provides “critical information and support for conversations with clinicians” and not “a single, consistent diagnosis.” Patrick Carroll, Hims & Hers’ chief medical officer, told me that AI Labs doesn’t diagnose or recommend treatments: “That role belongs to clinicians, and Labs is designed to bridge that boundary.” Anthropic and xAI did not answer my questions. OpenAI declined to comment for this article.
Perhaps that boundary – between doctor and algorithm – is artificial to begin with. One thought swirling around treatment literature is to stop treating AI products as if they were conventional medical devices. Given their human-like ability to learn new information and tailor responses to individual patients, medical AIs may function more like doctors than decongestants—so perhaps they should be evaluated in the same way as doctors. Instead of requiring FDA approval for every task it can perform, a chatbot can be required to pass a medical licensing exam and go through a period of supervision similar to a medical residency.
For now, though, that idea remains on the sidelines. Haider Warraich, a cardiologist and program manager at the Advanced Healthcare Research Projects Agency, a US government initiative to develop advanced health technology, is leading a major effort to get medical chatbots mainstreamed. His agent is provide funding for the development of an AI tool that is specifically designed for heart conditions, and then send it through the full FDA approval process. Warraich’s hope is that with such rigorous assessment, the chatbot will be able to safely assess and treat patients without the involvement of a doctor. Rodman praised this approach but warned that the process is ongoing take yearsby which time a plethora of new health AIs will have entered the market with little scrutiny.
In this way, the emergence of today’s AI health products reminds me of the rise, in the 2010s, of ride-sharing services like Uber and Lyft. The taxi industry is highly regulated, making it difficult for new players to enter the market. However by jumping and sometimes ignoring those rulesride-sharing companies were able to gain critical mass of users in a short period of time. Soon, governments had no choice but to act amend their laws to match what was just now. That model may end up playing out in medicine. Do regulations intended to ensure that medical products are safe and effective remain in use? Or will they instead be weakened or removed to clear the way for the tools everyone already uses?
We will find out soon. The health care system “isn’t going to slow down and wait for the evidence to mount,” Rodman told me. 80% of doctors are already using AI tools in their work, according to 2026 investigation and the American Medical Association. Patients they are not far behind. The benefits of AI may remain uncertain, but they are already overwhelmingly attractive.
