AI In Clinical Use: Lessons Learned - Nina Kottler, MD, MS (Radiology Partners)

Name: AI In Clinical Use: Lessons Learned - Nina Kottler, MD, MS (Radiology Partners)
Uploaded: 2022-01-20T02:13:49.000Z
Duration: 2 h 40 s

Introduction

In this section, the speaker introduces Nina Cutler and her work on bringing AI to clinical use.

Nina Cutler is a long-time friend of the Amy Center and has participated in many activities in the past.

She will be talking about her work on bringing AI to clinical use.

Nina's Background

In this section, the speaker provides an overview of Nina Cutler's background.

Nina got her bachelor's degree in math from Emory University.

She did graduate studies in applied math at USC and ultimately got a master's in applied math from NC State.

She went back to do her pre-med requirements at Wellesley while working as a system analyst at Raytheon on the Patriot missile system.

She then earned an MD at the University of Massachusetts and completed residency at UCSD.

She joined one of the first teleradiology practices, Nighthawk Teleradiology based in Sydney Australia, and served as a medical director there.

Currently, she serves as associate chief medical officer for AI for Rad Partners which is the largest radiology practice in the United States with over 2,500 radiologists reading 49 million cases a year.

Awards and Accomplishments

In this section, the speaker discusses some of Nina Cutler's awards and accomplishments.

Nina has been awarded the Rad Equals Trailblazer Award and an RSNA Research and Education Foundation Renkin Resident Research Award.

She is one of the most influential AI leaders in radiology today with practical experience implementing AI at scale.

Talk Title

In this section, the speaker introduces the title of Nina Cutler's talk.

The title of her talk is "AI and Clinical Use: Lessons Learned".

She will be sharing some of the lessons she has learned from her experience using artificial intelligence.

Importance of AI in Radiology

In this section, the speaker discusses the importance of AI in radiology.

The speaker strongly believes that AI is foundational within radiology.

It is still early on, but it's going to become an integral part of everything they do.

Disclosures

In this section, the speaker provides disclosures related to being a partner and equity owner at Radiology Partners.

All disclosures are related to being a partner and equity owner at Radiology Partners.

Some things that she does for them are just as a friendly physician and they do have minority interests but they're not selling anything today.

They're just talking about education and how everyone can improve themselves and learn more about artificial intelligence.

Evolution of Radiology

In this section, the speaker talks about how radiology has evolved over time.

The speaker shows an image of what radiology looked like when she was a medical student.

There were alternators everywhere, dictaphones, and no one had access to images. It would take days for anyone to get a report.

When PACS was introduced during residency, it quickly became a very quiet reading room which changed how they practice medicine and radiology.

Growth of Imaging Services and the Need for AI

In this section, the speaker discusses the growth of imaging services and how it has outpaced the growth of radiologists. The need for efficiency and speed in reading images has led to a loss of value in clinical consultation. The speaker believes that artificial intelligence (AI) can help bring back the value lost while also improving efficiency.

Radiology Staffing vs Imaging Services Growth

Radiology staffing only grew by about 25% while imaging services increased by 80%.

This increase has not stopped, leading to a need for more efficiency.

Loss of Value in Clinical Consultation

With PACS alone, robust clinical consultation was completely lost.

Consultation became an electronic piece of paper, losing the ability to communicate effectively.

Reading faster leads to a loss of value as well.

Need for AI

AI can bring back the value lost in clinical consultation while also improving efficiency.

It's not about replacing humans but collaborating with them.

Standards are not yet defined, so collaboration between vendors and academic centers is necessary.

Where to Deploy AI in Radiology

In this section, the speaker discusses where AI can be deployed in radiology. They break down radiology into different components and explain how each component can benefit from AI.

Components of Radiology

Radiology involves more than just interpretation, detection, and diagnosis.

Every moment from when an ordering clinician has a clinical question to following up with a patient can benefit from AI.

Different Kinds of AI Algorithms

Different kinds of AI algorithms are available on the market, and many have already been implemented.

AI can benefit scheduling, protocoling, acquiring images, work list optimization, patient information retrieval, detection, diagnosis, quantification, segmentation, reporting, communication and following up with patients.

The speaker's organization has either trialed or implemented or created all the yellow items in their diagram.

Best Practice Programs and CAD Triage Algorithm

In this section, the speaker discusses best practice programs to ensure appropriate follow-up and a follow-up program to ensure recommended information is received. They also discuss the CAD Triage Algorithm, which is an artificial intelligence product used for detection of critical findings.

Best Practice Programs

The speaker mentions that they have best practice programs in place to ensure appropriate follow-up.

They also have a follow-up program to make sure that recommended information is received by hospitals.

The speaker mentions that they have trialed a tool for automatic impression generation.

CAD Triage Algorithm

The CAD Triage Algorithm is an artificial intelligence product used for detection of critical findings.

When the algorithm detects a positive finding, it marks the study as reviewed and moves it to the top of the work list.

Initially, there was not much excitement about these algorithms because reading times were already fast for stat studies.

A pilot with two different CAD T algorithms showed increased efficiency in reading exams by about 10% overall.

The algorithm's negative predictive value being high enough increases confidence in detecting positive exams and decreases time spent looking at negative studies.

Benefits of Using CAD Triage Algorithm

In this section, the speaker discusses additional benefits found during their pilot study using two different CAD T algorithms.

Increased Sensitivity

Using the CAD T algorithm increased sensitivity in detecting pulmonary embolisms by about 4.4% and intracranial hemorrhages by 2.4%.

Increased Efficiency

Reading all non-contrast head CT exams was faster overall.

Increased confidence in detecting positive exams decreased time spent looking at negative studies, making the process more efficient.

Examples of Missed Findings

The images shown are examples of missed findings that were detected using the CAD T algorithm.

Importance of AI in Radiology

In this section, the speaker discusses the importance of AI in radiology and how it can help identify subtle findings that humans may miss. They also discuss the potential benefits and drawbacks of treating all cases identified by AI.

Importance of Identifying Subtle Findings

Humans may miss some subtle findings in radiology.

Treating all cases identified by AI may not necessarily improve patient care and could potentially increase costs or cause more harm than good.

The goal is to use AI to triage patients and determine which ones need to be monitored or treated.

Anecdotal Case

A 72-year-old female with altered mental status had a small acute subdural hematoma that was missed but later caught by AI.

The hemorrhage increased in size, causing mass effect and requiring a craniotomy.

Just because findings are subtle does not mean they should be ignored.

Radiologist Satisfaction with AI

It is important for radiologists to enjoy using the tool for it to be effective.

Net Promoter Score (NPS) is used to measure satisfaction, and a score of 64 is high for an NPS score.

Radiologists enjoyed using the tool so much that they asked for it to remain after the pilot program ended.

Facilitating Peer Learning with AI

Peer learning can help identify errors in interpretation, especially since chest x-ray error rates can be up to 40%.

Teaching radiologists what the AI is missing or overcalling can help improve the human-AI system.

Standard peer review statistically only reviews a small number of cases and may only find one error.

Overall, AI has the potential to greatly benefit radiology by identifying subtle findings that humans may miss and facilitating peer learning. However, it is important to use AI as a tool for triaging patients rather than treating all cases identified by AI. Additionally, radiologist satisfaction with the tool is crucial for its effectiveness.

The Importance of AI in Radiology

In this section, the speaker discusses the importance of using AI in radiology and how it can help improve patient outcomes.

Benefits of AI Review

Traditional peer review is limited in its ability to provide learning opportunities due to the sheer volume of cases.

AI algorithms are becoming more robust and can review a larger number of cases with greater accuracy.

With 95% sensitivity, AI could potentially provide 36 learning lessons from 38 chest x-ray errors.

Investing in Rad Eye Education

It's important to invest in rad eye education to become experts in using AI as a clinical tool.

Clinicians need to take an active role in driving where the technology goes by becoming local experts on AI.

Learning Opportunities

Radiology has a huge amount of information available on AI, providing ample opportunity for early learners and adopters to develop expertise.

Developing expertise will allow radiologists to drive where the technology goes in the future.

Conclusion

In this section, the speaker concludes by encouraging clinicians to take an active role in investing and developing expertise in using AI as a clinical tool.

Takeaways

Investing time and effort into learning about AI will benefit patients, clinicians, and radiologists alike.

Becoming experts on AI will allow radiologists to drive where the technology goes and ensure that it is used effectively for better patient outcomes.

AI and Radiology Interpretation

In this section, the speaker discusses how AI is used to interpret radiology results and how discrepancies are handled. The importance of educating radiologists on when to trust AI and when to rely on their own expertise is also emphasized.

Interpreting Results with AI

AI interprets computer vision and NLP results.

Discrepancies between the two interpretations are sent to a peer learning system.

Discrepancies could be due to missed findings by the radiologist or the AI.

Educating Radiologists on Trusting AI

Radiologists need education on when to trust AI and when to rely on their own expertise.

Continuous education is necessary as algorithms are frozen in time.

A massive database is used for teaching files and continuous education.

Lessons Learned in Change Management

In this section, the speaker shares three lessons learned in change management related to implementing clinical programs, including those involving AI.

Reverting to the Mean

Reverting back to the mean happens with education over time.

Continuous education is necessary as radiologists will revert back to their mean without it.

Bringing Data for Change

Data is necessary for managing clinical rollouts of best practice recommendations.

Scorecards were implemented retrospectively and increased scores after implementation.

It Takes an Army

Many people are needed for educating a large practice like theirs.

Educating Radiologists with AI

In this section, the speaker discusses how to effectively educate radiologists using AI and the importance of integrating AI into radiology workflows.

Importance of Education

The most effective way to disseminate information is through colleagues and peers.

People tend to look up to those they know closely rather than celebrities or public figures.

Integrating AI into Radiology Workflows

AI needs to be integrated into radiology workflows for maximum effectiveness.

However, current technology limitations make it difficult to fully integrate AI in PACS systems.

The speaker's practice created an NLP solution called Rekomd that listens to radiologist dictations and provides best practice recommendations based on patient metadata.

Rekomd Solution

Rekomd gathers patient metadata and combines it with radiologist dictations.

It provides best practice recommendations for specific findings such as thyroid nodules.

Rekomd is integrated because any changes made by the radiologist will cause the tool to reassess and provide a different recommendation if necessary.

Workflow Integration

Workflow integration is crucial in radiology, especially for triage, incidental findings, and critical results.

While some benefits can still be gained without full integration, oncology evaluation requires robust and feature-rich integration.

An example of workflow integration is comparing current studies with prior studies side-by-side.

Integrating AI with PACS

In this section, the speaker discusses how to integrate AI with PACS and presents five different options available for doing so.

Sending AI Results to PACS

One way to send AI results to PACS is through a DICOM secondary capture. This creates a hard copy of the image with the pathology frozen into it, which can be put as a separate series in your hanging protocol. The downside is that you cannot edit any of the annotations.

Another way is through a DICOM SR (structured report), which passes information from the AI system to your PACS and overlays it onto the actual image. The good thing about this method is that when it does the overlay, it's very well correlated with the underlying image. However, you still cannot edit annotations.

A third option is called a presentation state, which is an image overlay with information from the AI system. You can actually edit annotations on this one, but there are issues with interoperability between different pack systems.

Calling an AI Viewer or Embedding It Into PACS

Option number four involves having a separate AI viewer interface that opens up when you push a button on your PACS. This takes you out of your main viewing screens but allows for interactivity.

Finally, option number five involves embedding an AI viewer directly into your PACS. This would require significant development work and may not be feasible at present.

No timestamp provided

Viewer Inside of Your Viewer

In this section, the speaker discusses why a viewer inside of your viewer is necessary and how it can help in AI deployment.

Benefits of Having a Platform for AI Deployment

A platform is necessary to deploy artificial intelligence at scale.

Automating workflows is one benefit of having a platform for AI deployment.

Orchestration and Movement of Data

Orchestration is the purposeful and automated movement of data.

In AI, orchestration involves sending data to an AI system, receiving structured output from the system, and sending it downstream.

Accuracy in orchestrating data affects the accuracy of overall AI results.

Lessons Learned in Sending Data to an AI System

It's essential to send the right study, series, or image for the right patient to the right algorithm at the right time.

The accuracy of how well you can orchestrate your data affects the accuracy of your overall AI result.

Importance of Filtering Out Incorrect Data

In this section, the speaker emphasizes that filtering out incorrect data is crucial in achieving accurate results when using an AI system.

Examples of Incorrectly Sent Data

Sending data that an algorithm hasn't been trained on will result in poor interpretation.

If there's no way to filter out incorrect types of exams, inaccurate results are likely.

Choosing Which Algorithm to Send Data To

Knowing something about what's in your data helps you make better choices about which algorithm to send it to.

Looking only at DICOM tags may not be enough; knowing specific information about Hounsfield units can help make better choices.

Lessons Learned in Data Orchestration

In this section, the speaker summarizes five lessons learned about data orchestration.

Five Lessons Learned in Data Orchestration

Lesson 1: It's essential to send the right study, series, or image for the right patient to the right algorithm at the right time.

Lesson 2: Filtering out incorrect types of exams is crucial for accurate results.

Lesson 3: Knowing something about what's in your data helps you make better choices about which algorithm to send it to.

Lesson 4: The accuracy of how well you can orchestrate your data affects the accuracy of your overall AI result.

Lesson 5: Accuracy in orchestrating data requires purposeful and automated movement of data.

Using AI to Manage Unstructured Data

In this section, the speaker discusses how AI can be used to manage unstructured data and add structure to it.

Adding Structure to Unstructured Data

DICOM and HL7 standards provide information about what is in the image and report.

However, these standards are not always followed correctly, leading to incorrect or inconsistent data.

Normalizing data requires a lot of human effort.

AI can provide more robust information about what is in the data and correct some of the incorrect information provided by DICOM and HL7.

Sending AI Downstream

Generally, AI is sent only to radiologists or viewers.

Knowing more about the data can help make better choices on where to send it.

For example, if an algorithm has already identified missing reconstructions, it could be sent directly to a technician instead of wasting time waiting for a radiologist.

More data allows for more informed decisions on where to send the data.

Orchestration and Platforms

In this section, the speaker discusses orchestration and platforms in relation to AI systems.

Orchestration

Orchestration involves sending data downstream to multiple locations.

The AI system goes to an orchestrator before being sent downstream.

Data can be sent to multiple other places if you know what's in your data.

Platforms

A platform is necessary for normalizing and translating data.

A modern platform should be native and orchestrate data while doing all of these other things as well.

Structured information from imaging is either going downstream or disappearing entirely, which is a problem that a platform can solve.

Creating Structure from Unstructured Data

In this section, the speaker talks about creating structure from unstructured data using AI systems.

The AI system creates structure from unstructured data.

This structured data needs to go somewhere where it can be used effectively.

Capturing structured data with a platform allows it to be attached back to your data so you can decide where you're going to send it in a smart way.

Lessons Learned and Future of Radiology

In this section, the speaker shares some lessons learned and discusses the future of radiology in relation to technology-enabled land, AI, and data-enabled land.

Lessons Learned

Investing in radiology education is crucial when deploying AI systems.

Radiologists need to become information experts rather than just imaging experts.

Future of Radiology

Radiology will change significantly in tech-enabled land with more information available from various sources such as radiomics genomics molecular imaging ai.

A fully integrated system will allow changes made by radiologists to be re-evaluated by the system, providing robust consultation back to the referring clinician and patient.

Radiology practices that don't use AI will be replaced by those who do.

Final Thoughts

In this section, the speaker shares some final thoughts on embracing technology in radiology.

Radiologists need to embrace technology and learn about it themselves.

Technology is driving the future of radiology, and we need to change as a profession to provide the best care for our patients.

Radiology and AI: Implications for the Future

In this section, the speaker discusses the disconnect between available vendor products and what radiologists actually need. They also touch on peer learning activities and human-computer interaction.

Mismatch Between Vendor Products and Radiologist Needs

There is a disconnect between available vendor products and what radiologists actually need.

Early algorithms were related to image detection, which may not be the most useful thing for radiologists.

However, more AI vendors are working closely with radiologists to create more useful products.

Peer Learning Activities

Peer review can be punitive, but peer learning can be a positive experience.

It would be possible to use peer learning activities to find great calls as well as misses.

The subset of cases where AI detects something could be used to point out great learning lessons for good calls.

Human-Computer Interaction

As more algorithms operate on the same images, there is a risk of cacophony.

A platform is needed to orchestrate data and consolidate information back to the radiologist.

The platform can help vet different series from different AI algorithms.

Algorithm Development

External data scientists were hired initially to create a product that did not exist yet.

Now they are part of the team and work on natural language processing while external experts are brought in for computer vision tasks.

Changes have been made in how information is displayed and incorporated into workflow.

Supervised vs Unsupervised Algorithm Development

No content provided in this section.

Adversarial Attacks

No content provided in this section.

Tagging and Security Efforts

In this section, the speaker discusses tagging efforts and security measures taken by their team.

Tagging Efforts

The team uses clinicians, data scientists, and interns for tagging efforts.

Data scientists lead the effort after learning from the team.

Radiologists are not required for tagging efforts.

Clustering analysis can be used for tagging without initial labeling.

Security Measures

A whole division is dedicated to security efforts.

Education is a key component of security efforts, particularly around phishing attacks.

Phishing emails are sent to teammates to educate them on how to identify such attacks.

Software tools are also used in combination with education efforts to prevent attacks.

Conclusion

In this section, the speaker concludes the presentation and thanks attendees.

The presentation has ended.

The speaker thanks attendees for their questions and attendance.