Healthcare's High-Performance Frontier: Building Effective Human-AI Systems

And so what I really want to do is tell you a story in four parts. The first part is going to be that I believe medicine more than anything is an information processing discipline and that's why a meeting like this is so important. Second, I want to touch on why computers are so important especially in complex environments when we're trying to empower humans to make decisions. And then what I really want to do is talk about sort of the fact and fiction of AI in medicine. And I think there's a lot of lessons that can be extrapolated to other fields. And then lastly, I'm going to give you a little glimpse of the future.

So, let's start the story. Number one, I already said medicine is intrinsically a knowledge processing discipline. And this can be somewhat of a controversial comment. Some people think of medicine being anchored upon the basic science of human wellness and disease, while others think of it as an art in which we actually use that knowledge to diagnose and treat patients. And those two things are true. But I actually think most importantly what we are doing when we pursue the practice of medicine is we're managing information.

And that's because we have to think beyond biology. We have to think about human preferences. We have to think about emergent knowledge. We have to think about the ethics of the choices that we're making and put all of those pieces together. And that is something that we as humans do very well. And while computers are getting better at replicating some of those decisions, they don't always replicate those human strengths.

This is also why computation is so important in medicine. And I'll come back to that in a moment because the amount of information that we need to process vastly exceeds our human cognitive capacities. So, how do we augment those human cognitive capacities alongside our unique strengths in order to deliver better care to the patients that we serve?

And so and this is a really sort of seminal report by Zach Kohani. It is the case that medicine is not only a knowledge processing discipline in the future. I would argue it has always been a knowledge processing discipline. Now I did include a few examples in the slides. I cut them down for a little bit of sort of brevity. I could have gone all the way back to sort of the ancient Egyptians. But there is a history of medicine being a knowledge processing discipline.

So for example, if you go back to sort of the very initiation of modern medicine in the United States, you will see literal published books, registers from major hospitals that tell the stories, the narratives of individual patients. And that was used not only for recordkeeping, but also for teaching and for research. An interesting sidebar is the publisher of these reports from the Pennsylvania hospital was actually Thomas Jefferson. So one could argue Thomas Jefferson may be one of the very first informaticians.

If you look a little bit more contemporaneously, you could look at the work of someone like Larry Weed who really thought about how do we structure the electronic health record? How do we structure even paper records to be smarter and more usable? And this is where concepts like the problem oriented chart come into play where we start thinking about not only recording what happens to our patients but actually what choices we make and what are the outcomes so that we can be even more precise in learning from these records.

And then if you look most recently and this could be a controversial topic because many people have strong opinions about electronic health records. We have modern electronic health records that allow us to have records that are in more than one place at one time that are accessible to all of the care team that allow us to deploy modern AI.

But the arc across all of these examples, whether it be the Egyptian example, I didn't give you some of the earliest medical records produced by the Greeks or now what we see in modern electronic medical records are sort of five critical features. First that they comprise standardized observations about what happens to our patients. Second, in some form or another, they allow us to engage in case-based reasoning. Third, we can see what we do to our patients and then find out what the outcomes are so we can both teach and learn. In more modern examples, we see decision support. We start using that information to help support humans to make better evidence-based decisions. And importantly, they also allow for professional accountability.

And so I hope what you take away from this is medicine as a field has been engaged in information management since its very first origins.

So that leads to part two of my story. Where did computers come in? I've already given you a little bit of a preview when I started talking about medical records, but I really want to drill down on the topic of artificial intelligence, which is a theme for this meeting. And in particular, how does AI support humans in a collaborative manner so that we can make better decisions together?

So, first and foremost, if you look back again, and you can tell I'm a little bit of a student of history, you can see a similar arc of history around computation. I could have gone back to the very first mechanical computers that were used to prove basic statistical theories, but I'll use just a few examples. Some of you may be familiar with Enac, which was a mechanical electrical computer. It was actually used to do very complex mathematical operations that previously had not been possible, including the math that underpinned the Manhattan project.

An interesting side story if you read about it, they were not sure if their simulations were correct or not. So, the outcome was either that the first atomic test would be safe or they would lead to the end of the Earth as we know it. A little bit of a margin of error in those calculations. Nonetheless, this was an example of using computation to augment human computers, right? Because the very first computers were actually humans who did mathematical operations in order to scale the types of decisions that they could make.

A little bit more modern than this, you could look at the advent of the micro computer. I happen to use an example of an Apple 2 because that's what I grew up with. And I'm sure we could have a long conversation about which micro computer was the best, although we all know it was the Apple 2. But the point is all of a sudden these things that used to take a whole room of equipment could be compressed into a single device on the desktop. Imagine the power that we were giving to individual humans again to program these computers and to augment their decision-making capabilities.

And all of a sudden, computing went from being something that only a very few number of people could access to something that was widely available. And now in sort of the modern era, we have technologies like quantum computing. Computers that if you actually try to observe them while they compute, you'll perturb the process and they won't generate the results you're expecting. One could say that these are computers perhaps with performance anxiety. But the more important part of this is all of a sudden we have computers that can compute in multiple dimensions. So previously non-computable problems can be solved in near real time.

But again there's a common thread here. Whether it be mechanical computers, whether it be the first vacuum tube based computers, whether it be microcomputers or quantum computers, what we have been doing is building tools that allow us to manage information, reason upon it, and really augment human cognitive capacity. And that leads to the fundamentals of AI as we know it today. And as complex as AI may be, the basics are the same. Whether we're talking about rule-based systems, whether we're talking about statistical models, machine learning, or even generative AI and a future of agentic AI, we collect data, we label it, we train models that make sense of the world, often in ways that replicate humans, but not always. We deploy them, evaluate, and repeat.

And there's two really important lessons here. Number one is the concept of pattern recognition. We as humans make sense of the world around us through pattern recognition. If you walked into this room and you saw the tables in the front of the room, you were able to recognize that they were a table even though you may have never seen a table like this before because you use the patterns that you've learned through experience to know that was a table. In a similar way, we're training computers to be able to do the same thing. And now we're using those same patterns in order to be able to produce new content.

So where does that take us? That takes us to a future where we can actually use really advanced AI in order to reason about the world around us whether it be as I said machine learning or now generative models and the opportunities are tremendous and I know that's the theme of what you're all going to be talking about today. I think also important is this idea that we can use agentic AI to actually automate tasks. So that does lead to one fundamental question though that we need to be very thoughtful about. How do we assess intelligence? Because there really are two schools of thought about how we assess intelligence. One is that we think of it as a function of being able to do more faster. Two, we think about it as a function of being able to learn from prior experience. Unfortunately, those two things are reinforcing of each other. And so then the question becomes how do we understand those cycles? Is this about generating and understanding more data or is this about informing new experiences and learning over time?

So that comes back to my theme of healthcare being complex. How many people here think we can predict everything that happens when we're engaged in healthcare? Anyone? How many people think there are incredible emergent features when we are engaging in clinical decisionmaking? When we're thinking about how to diagnose patients, when we're thinking about how we are going to treat people and predict their outcomes. And how many people think that we don't have complete knowledge of what's going to happen? Anyone? That is why it's so hard for us as humans to answer every question in healthcare. And also why it's so important for us to use these tools like modern computation and AI to make better choices.

And I can't really sort of go into all the details, but we often use the term complex and complicated interchangeably, but they're not the same thing. Complicated environments are places where we can model and understand everything that's going to happen a priority. Complex environments like many that we all encounter are places as I describe where there are emergent features where we don't always know what's going to happen and where we have to be able to use fuzzy reasoning where we have to be able to generate new knowledge literally in real time in order to navigate those environments.

So what does that mean in an environment where humans have strengths and computers have strengths? It means that we have to build partnerships between humans and computers. There is a concept called the fundamental theorem of biomedical informatics my field that was actually coined by a good friend of mine Charles Freriedman and the postulate of this theorem is that it is the combination of computers and humans working together that is superior to either working alone now I think it's important to understand that this happens in that complex system there may be environments in which the computer will outperform humans and there will be cases where humans will outperform computers. And it is the aggregate of all of those scenarios in this system that really defines a complex environment in which AI and computers partner together.

So there is a long history of AI and medicine. This complex environment formed by this sort of fundamental theorem where humans and computers work together dating back to even before the 1990s actually going back in the 1970s with some of the very first rule-based systems that were used for diagnostic reasoning. More recently, we see the embedding of AI enabled clinical decision support in electronic health records. And in the future, we actually expect to see agentic solutions where computers become a partner to humans that can actually undertake certain tasks under the supervision of humans. We're already seeing that to some extent with the advent of ambient AI at the point of care where the computer is actually acting like a scribe in the room in order to generate records so that the computer gets out of the way, nobody's typing on a keyboard, and the provider and the patient can have a natural conversation. And if there's one thing I want to emphasize here about the history of AI in medicine, even though it seems like a very long history to those of us that have studied it, this is a very short time frame. It's not that long ago that the very first computers were being put into clinics. And today we have computers that are partnering with humans to make care better and more humanistic.

So what does that look like in terms of how we sort of evaluate back to the idea of intelligence, back to that idea of partnering humans and computers together? Well, it means we really have to think about this as a learning health system. And I'm sure that many people in this room have heard the concept of learning health systems before. But this is an environment where every time we interact with patients, it's an opportunity to improve the care they receive, their family receives, their community receives, where we collect every piece of data. We reason it on real time. We actually extract knowledge from that reasoning process, deliver it in the right time, place and format and then study how it impacts care. And that all happens in a virtuous cycle that repeats iteratively and continuously.

This is the antithesis of the unidirectional flow of research into practice that we often think about when we talk about evidence-based medicine where we do research, generate results, disseminate them, and then hope that they impact care. But instead, this is a constant cycle in which we're always improving. We're always learning. We're always capturing data. And that's again where this opportunity to apply modern data and computational technologies and certainly AI becomes so impactful.

And that leads to the third part of my story. So I've just painted a picture that we have a long history of medicine being an information processing discipline.

https://example.com/transcript

And similarly, we have a long history of using modern computation to make humans better in terms of their ability to reason in these complex environments. And now, how do we put those together with the sort of modern computational capabilities that we have at our fingertips today? Well, it's not all straightforward. That's the spoiler alert.

So, first, there are some big questions to be answered. And I love this tweet from my colleague Karen Deep Singh who's the chief AI officer at UC San Diego which is that research models are rarely implemented and implemented models are rarely researched. I'm going to repeat that research models are rarely implemented and implemented models are rarely researched. The point is that today most of the AI that we are deploying in healthcare has been minimally studied and most of the AI that we do the deepest research on where we understand the most about how it works we don't actually deploy into the production environment and that's largely because of technical barriers but more importantly cultural and policy barriers and we're going to have to tackle that if we're going to build an evidence-based learning health system that employs AI.

So what's the first major question? We do know that there are places where AI is improving the quality, safety and outcomes of care. But it is not consistent and it has to do with the tasks, the data, the way in which the AI has been deployed and importantly how we build those computer human relationships. And so while there's good reason to be optimistic, we still have to build the evidence base to understand how and where AI generates value. Where does it improve the quality and safety and outcomes of care? And certainly with modern generative AI that has both really powerful capabilities to produce new content, but also new types of error and bias that are introduced as a function of that generative AI, those questions get even more acute.

Number two, AI can absolutely improve clinical decision-making. Back to my prior point, when it has been adequately validated. Now, this may sound like me justifying my role as someone who primarily works in academic medicine, but there's incredible work to do to study AI just like we would study any other new diagnostic or therapeutic intervention in healthcare. We need to understand is it efficacious? Is it safe? How does it compare to the current standard of care? And what does it do in terms of outcomes? That means measuring things just like how much time we save or how much more we can bill or how many more patients are seen is not going to answer that question.

Number three, we also have critical questions concerning how humans and AI work together. So I'll give you an example. We have a decision support tool that we've deployed in our health system that looks at patients with high chronic or acute disease burden and it looks at whether or not they have endof life care plans so that when they present in the emergency department or other settings we don't automatically admit them to the ICU potentially where they may not survive without having that conversation around what their end of life care plans are. The decision support tool simply asks for a paliotative care consult to have that conversation before the patient makes a care decision. This is a simple behavioral nudge, but it's very complicated because in fact what we're talking about is a decision support tool that's going to trigger a conversation about end of life care planning. This is a deeply ethically complex domain. And so we have to think about how do humans with their ability to reason in those sort of fuzzy areas of ethics and patient preference and moral reasoning. How does that interact with AI in the complex environment that is healthcare?

Number four, there are cases and I want to emphasize this as much as I said we need to do more research. There are cases where AI is simply better than humans. Certainly we see this in the imaging domain. For example, we have an AI tool deployed in our health system that actually scans every chest image that's been captured for inatients and identifies whether or not they have an incidental pulmonary emblei, a clot in their lungs. These are often missed, but they can be life-threatening. We can find them more quickly and with higher accuracy using AI than we can with our human radiologists. So now the question doesn't become should we use AI but is it ethical to not use AI in those situations where we know that the tools are actually superior and like I said before in this learning health system concept where we're putting AI and humans together and we're constantly iterating we need to study and understand how these types of fully automated tools fit into that overall ecosystem.

Number five patient perception of AI use is actually quite positive. I know that there's often concerns are we not explaining to our patients how AI is being used are there sort of concerns around privacy or the use of patient data and these are all very important questions but I will also tell you when you disclose that AI is being used and you explain when and how it's being used patients are generally very positive about this and this has been studied time and time again in fact one of the greatest studies I've seen about this is a study in which in a sort of comparison between messages from a patient portal generated using AI versus those that were generated by human physicians. The patients found the messages generated by AI to be more timely and compassionate. I'll say it again. The AI message was more timely and compassionate. Why is that? Because the AI doesn't get tired. The AI doesn't have two or three or 400 messages to respond to at the end of the day. That's a great example where AI becomes an ideal partner with our human clinicians. But importantly, in that same study, the messages generated by AI had a disclaimer that said they had been produced with the help of AI and approved by the patients provider. So there was no mystery about how AI had been used in that setting.

Number six, providers are also seeing benefits. I talked about ambient AI in our system. We literally see an incredibly positive response to the use of generative AI for documentation purposes. We are reducing the amount of time our providers spend after hours, what we affectionately refer to as pajama time, by almost two to three hours a week. We're increasing their work RV use, so we're actually able to get them reimbursed for more of the care they deliver. They're closing their encounters on the same day that they see the patients rather than weeks later. And this is all because we're using an ambient scribe rather than having our providers type on a keyboard. We also see qualitatively improvements in burnout metrics and in patient perception because their providers are talking to them. This is the first technology in my career that our providers have actually asked for. Nobody asked for the electronic health record for better or for worse. And there are so many other technologies that we arrive and say this is going to be great. It's going to make your job better and then it doesn't. This is a technology demonstrabably is generating those outcomes is perceived as such and people want and so there are benefits here and I cannot emphasize enough we have a demographic cliff in terms of the patients that we're seeing but also in terms of our provider workforce we have to find ways to empower our providers and retain them in healthcare and this is a primary example of AI helping us to do that.

Seven, and this is an important charge for a group like this, we are still having a hard time taking systems built in a vendor ecosystem and scaling them across health systems.

# YouTube Transcript

There's no example more sort of well-known than the early warning systems for sepsis built into major commercial EHRs that basically produced results where a provider would have to review in some cases up to a hundred false positives to find one true positive where a patient was at risk for sepsis and we didn't know it. This created a lack of trust in AI as a result of that sort of incredibly poor performance. And part of that is that populations are different, workflows are different, environments are different.

And so the real question is, are we moving models between environments or are we moving the techniques between environments? And I think that's a real question. And then lastly, I want to emphasize we have to find ways to share this knowledge with other health systems. I'm in St. Louis, Missouri. We're a very large well-resourced delivery system anchored on one of the largest medical schools in the country. We have people and expertise that very few other organizations in our state have to evaluate AI and understand its impact.

How do we share that knowledge with our rural health partners, with our community partners, with our FQC's, with our critical access hospitals? Because we don't have enough capacity certainly in the state of Missouri and I would argue nationally to take care of every patient that needs care. And so the question is how do we find new ways to share this knowledge in a non-competitive way across healthcare delivery systems because right now we are competing on technical expertise to the detriment of our patients.

So let me try to bring all of those pieces together in sort of a final message. So I've told you that medicine is an information processing discipline and I've also told you that humans have historically and certainly in the present time used computers and now AI to augment human decision-making capabilities. And I've talked about some big questions that we have to answer if we're going to really sort of generate the type of value and impact that I think we all want to achieve.

But then the question is what are the risks versus the benefits? How do we get to that future in which we really realize all of the amazing opportunities I just described? So first of all, and I hope for those of you in the room that are a little bit of a nerd like me, you'll recognize this picture. People often talk about AI and medicine like it's the future. It's going to happen in the future. Spoiler alert, we are in the future right now. If you've seen the movie Back to the Future, you would actually know that we are presently five years into the future.

So, there are really, and we'll do this in a sort of game show type of format, two options. There's door number one. Door number one is that we make bad design decisions. That we don't address all of the questions that I just shared with you. That we don't think about that human AI partnership that we don't consider all of the factors that define the complex environment of medicine and we end up creating more data more noise in the system that we draw people's attention away from what matters that we create even more cognitive overload and people become even more fragmented that they are in terms of critical decisionmaking.

We can also create automation bias where we overrely on some of those tools that are very good. And we're having a lot of conversations in medicine now about how we train the next generation of providers. Do we have a problem with never skilling or deskilling because we don't have people learning how to do tasks that AI are doing for them? And we could exacerbate inequities through the digital divide. I talked about this earlier. What happens if rural hospitals or critical access hospitals or community hospitals serving underrepresented communities cannot get access to these technologies?

And probably most importantly, we create even more alert fatigue as a function of just overloading our human beings. Now, I would argue that there's examples of what this looks like already. There's a phenomenal study by my colleague Jonathan Chen at Stanford that shows that when you took common case studies from the New England Journal of Medicine with a known diagnostic endpoint and you asked expert clinicians to evaluate them, their accuracy ranged between 20 to 30% on their first attempt. The generative AI tool in this case ChatGPT was closer to 70 to 80%. And then importantly when you put the two of them together both got worse.

I want to repeat that putting the AI and the humans together generated worse outcomes than either acting alone. The opposite of Friedman's fundamental theorem. And why was that? Because this was the off-the-shelf version of ChatGPT. It wasn't built to support clinical decisionmaking. It wasn't an AI tool that had been tailored to this concept of human AI partnership in the complex environment that is healthcare. So, we know exactly what that looks like. I could talk about how the doctors argued with ChatGPT during that study which was fascinating but that's another talk.

There is an alternative more optimistic view. That's door number two. We make smart design choices and this leads to increased insights. We're able to filter all of the information that we have at our fingertips today so we can focus on what's important when it's important. We can create equity by design because the first step in understanding bias or inequity is actually measuring it in the data that we have and that's extremely hard to do today.

We can allow for contextual focus. For example, I have a very good friend who's a rheumatologist and he points out that when he sees his patients who have many other chronic disease states, they may have 20 or 30 different alerts that pop up in the electronic health record. They're all clinically appropriate, but only a handful are appropriate for his encounter when he's treating that patient for their vasculitis or their other rheumatologic diseases. He needs to see those alerts first, not last, not in a random order. That sounds like a simple task, but we do not do that today.

We can allow for better judgment support. I mentioned that example of a decision support tool that helps people to make end-of-life planning choices. This is a place where human ethics and moral judgment are equally important to the evidence. And so, how do we actually support those types of critical conversations? We can democratize access to data. I'd point out that this is an opportunity for us to truly empower our patients to be active members of the health care team by allowing them to access their data in ways that make sense given the decisions that they have to make in their daily lives.

And then finally, as the antithesis to alert fatigue, we can actually allow for adaptive notifications where we can actually give people, like I said, the right information in the right time, place, and format. And again, there are examples of this today. This is an incredibly impressive paper out of Microsoft research that shows when we take generative AI and we deploy it in a way that actually replicates how human experts navigate diagnosis and treatment planning where you have teams of AI working together with specialized expertise with a single AI orchestrating them. Imagine now you're attending physician and your residents or fellows for example in an academic setting. You can actually get better performance than anybody working alone.

So the opposite of the Chen paper that I talked about before because this is an example where the AI has been designed to replicate how humans operate in this complex environment. And so how do we get there? I just told you what I think we need to do but how do we get there?

So number one, we're going to have to embrace what's next, which is the agentic revolution, where these tools become not only tools that we direct in sort of a transactional way, but tools that we can set out to undertake a set of tasks, maybe to prioritize a work queue of a radiologist or to answer messages in a patient portal or to review the literature in support of our scientists. And that means every one of us is going to have to become a manager, a manager of a new type of workforce, which will be Agentic AI.

We're going to have to create what has been referred to as really logical traceability. Explainability might not be an achievable goal given the way that these tools work, even though people talk about it really at great length. Instead, we may need to talk about how do we track what data is used to build these tools, how they've been evaluated, and how they perform over time. Sort of a trust but verify model.

We need standardized benchmarking. Every single vendor that comes and talks to me tells me that their product is the bestin-class in the marketplace. That is logically impossible. The problem is there's no baseline for comparing these tools. There's no underwriters laboratory of data and metrics and especially metrics that focus on outcomes.

We're going to have to figure out how do we build better infrastructure. I didn't talk about it a lot today, but you are at the core of this. There is no AI strategy without a data strategy. And there's no data strategy in the complex environment that is healthcare without an interoperability strategy. And so therefore, we will have to tackle all three of those problems together.

Governance is going to have to change. I often refer to governance as the G-word in our organization because everyone's afraid of the word. That sounds like we're going to slow things down. But the reality is we do need to make smart decisions. We need to be responsible. We need to make sure we're addressing sort of the economies of scale that make AI sustainable. And we're have to do it in an environment where as soon as we deploy the technology, it changes in terms of its own performance and how the environment presents itself. These are dynamic technologies. Back to my example of the learning health system.

And then finally, we are going to have to reskill and I actually prefer the term upskill our workforce. Everyone is going to be impacted by these tools. And so, how do we give people the tools and capabilities to succeed in that AI enabled healthcare environment?

So, with that, I hope I've convinced you that AI is important in terms of the future of healthcare. That we have to acknowledge that we're in this complex environment. That we have a long history because of the fact that really medicine is an information processing discipline and that AI in computing is the way to empower our humans to thrive and be successful in that environment and improve the care that we deliver to our patients and to their families and the communities that we serve because this is intrinsically a human enterprise. It is not a technical enterprise. Thank you.