U.S. Hospitals Quietly Use Emotion AI to Flag “Difficult” Patients

As usual, the exam room is small, with two chairs, a table covered in paper, and a monitor that is angled slightly away from the patient. The doctor clicks his mouse halfway through a woman’s description of her dyspnea. As she continues to speak, a block of text silently blooms behind her on the screen. The entire time, the AI scribe has been paying attention. It is flagging, summarizing, and transcribing. The physician looks over at the monitor. The woman is still speaking and is unaware of the change.

In American hospitals, scenes like this are becoming commonplace. In 2024, artificial intelligence was being used in clinical practice in over 86% of U.S. health systems, which would have seemed unlikely only a few years ago. The tools include diagnostic algorithms that recommend potential conditions and billing codes before the doctor has finished asking questions, as well as AI scribes that transcribe appointments in real time. The majority of patients are unaware of what these systems are recording or when they are operating. That’s not a coincidence. It’s mostly due to the way these tools have been implemented: discreetly, as administrative enhancements, with minimal public explanation of what they truly capture.

The fact that some of these systems do more than just transcription is less talked about. A small but increasing number of hospitals are using emotion AI, which is software that can identify sentiment, behavioral cues, and what the industry refers to as “patient engagement levels” to profile patients in real time. Patients who express frustration, ask repeated questions, or communicate in ways that the algorithm interprets as non-compliant can be subtly flagged in their records before the clinical encounter is even complete, at least in certain implementations. The label that follows—sometimes “difficult patient,” sometimes variations with a more clinical sound—can influence how that individual is handled at each subsequent visit.

There is some validity to the case for these systems. There is a lot of pressure on hospitals. Burnout among doctors is a real problem. Anything that cuts down on the amount of time a doctor spends staring at a screen rather than looking at a patient is genuinely appealing, and documentation requirements are burdensome. AI will become as commonplace in medicine as the stethoscope, according to Dr. Robert Pearl, the former CEO of Permanente Medical Group. That might be accurate. However, a behavioral profile that endures in a patient’s file and shapes clinical judgments for years is not produced by a stethoscope.

What is lost when emotion is filtered by an algorithm is the deeper issue. The experience was vividly described by a doctor and anthropologist who went to a recent doctor’s appointment with a friend, a woman in her 70s with several chronic illnesses. The note produced by the AI scribe was precise, even remarkably fluid. However, when she mentioned stairs, it failed to pick up on the catch in her voice. It overlooked the hint of fear concealed by a casual comment about staying indoors. It overlooked the link between her present symptoms and a history she hadn’t yet mentioned aloud, which the doctor never bothered to elucidate. The synopsis was finished. The patient wasn’t.

When the AI is evaluating rather than merely summarizing, this disparity becomes more significant. There is strong evidence that emotion detection models struggle with linguistic and cultural variation because they are trained on datasets that represent the communication patterns of specific populations. An algorithm trained on a limited behavioral baseline may react very differently to a patient who speaks loudly because of hearing loss, who asks numerous clarifying questions out of anxiety rather than aggression, or who expresses distress through understatement rather than obvious distress. The flag is unaware of the distinction. It simply raises a flag.

This is comparable to what transpired with social media. Platforms developed engagement algorithms that were incredibly good at recognizing and reacting to specific behavioral cues. They then used these algorithms to maximize attention without giving much thought to the consequences that might follow. Nowadays, hospitals are using behavioral detection tools in a setting where the stakes are, if anything, higher, with much less regulatory oversight and virtually no public discussion regarding the use of the data. How patient behavioral profiles created by AI scribes are stored, shared among providers, or taken into consideration when making decisions about future care is not specifically regulated by the federal government. That disparity is substantial.

This is especially unsettling because the use of AI in healthcare isn’t intrinsically harmful. AI has been shown to improve outcomes in real, well-documented cases, such as early detection of sepsis, accurate radiological image reading that matches specialists, and early detection of medication errors. Recently, OpenEvidence’s AI system became the first to pass the U.S. Medical Licensing Exam with a perfect score. These are genuine achievements. The issue is that the same infrastructure that is used to monitor and classify patient behavior is also being used to enhance clinical decision-making. These two functions are being implemented simultaneously, within the same software, frequently with no distinction made between them.

It’s difficult to ignore the fact that patients who already experience the most conflict in the healthcare system are frequently the ones who are most likely to be flagged by emotion AI: people with chronic pain who have learned to speak up for themselves, people who have experienced rejection in the past and whose annoyance is interpreted as hostility, and people whose cultural backgrounds involve communication styles that don’t neatly fit onto the behavioral norms these algorithms were designed to recognize. In other words, the technology runs the risk of systematizing the very biases that medicine has been attempting to address for decades. Simply put, it’s leaving a record and working more quickly now.