Deep Learning for Medical Image Interpretation

Pranav Rajpurkar

PhD Candidate, Computer Science Stanford ML Group

Ophthalmologist-Level Performance Dec 2016

Gulshan et al, 2016

Dermatologist-Level Performance in Feb 2017

Esteva et al, 2017

Pathologist-Level Performance in Dec 2017

Bejnordi et al, 2017

Cardiologist-level Performance in July 2017

Rajpurkar et al, 2017

Detection of Breast Cancer with

Mammography

Rodríguez-Ruiz et al, 2018

View Classification of Echocardiograms

Madani et al, 2018

Detecting middle ear fluid using smartphones + AI

Chan et al, Science Translational Medicine 2019

Lung cancer screening with deep learning on low-dose chest computed tomography

Ardila  et al, 2019

Pathologist-level interpretable whole-slide cancer diagnosis with deep learning

Zhang et al, 2019

Similar image search for histopathology:

SMILY

Hedge et al, 2019

Weakly supervised deep learning on whole slide images

Work with Bora Uyumazturk, Amir Kiani, Dr. Jeanne Shen, Dr. Robyn Ball, Dr. Curt Langlotz, and others at Stanford Medical School

Ongoing work, Stanford ML Group

DL to predict microsatellite instability directly from histology in gastrointestinal cancer

Kather et al, 2019

Weakly supervised deep learning on whole slide images

Campanella et al., 2019

RL Approach to Pick Pathology Patches

Work with Amir Kiani, Dr. Jeanne Shen, and others at Stanford Medical School

Ongoing work, Stanford ML Group

Performance of DL vs Manual Grading for Detecting Diabetic Retinopathy in India

Gulshan et al., 2019

We are investigating performance of XRay Interpretation in clinic

Work in Stanford ML Group (Anuj Pareek, Sharon Zhou, Mark Sabini, Minh Phu, Chris Wang, Dr. Lungren, and others)

Detection of Brain Activation in Unresponsive Patients with Acute Brain Injury

Claassen et al., NEJM, 2019

Detection of treatment response to antidepressant medications using EEG

Work in Stanford ML Group (Nathan Dass, Vinjai Vale, Jingbo Yang, Dr. Leanne Williams)

Continuous prediction of future acute kidney injury

Tomasev et al., 2019

Prediction of 6-month mortality for palliative care

Avati et al., 2018 (Stanford ML Group with Prof. Nigam Shah and others)

Applications of deep learning in fluorescence microscopy

Belthangady et al., 2019

Can we detect abnormalities with just a few drops of blood?

Ongoing work, Stanford ML Group (with Nathan Dass, Prof. Utkan Demirci, Prof. Matt Lungren and others

Deep learning with Real-time Inference in Cell Sorting and Flow Cytometry

Li et al., 2019

Cell Sorting with Deep Learning

Ongoing work, Stanford ML Group (with Bora Uyumazturk, Jose Giron, Prof. Gozde Durmus, Prof. Utkan Demirci

Genome-wide cell-free DNA fragmentation in patients with cancer

Christiano et al, 2019

Tracking the Leader: Gaze Behavior in Group Interactions

Capozzi et al, 2019

https://doctorpenguin.com

With Eric Topol

Machine Learning Framework

https://www.slideshare.net/LuMa921/deep-learning-a-visual-introduction

Paradigm shift

Feature Extraction might include software to extract corners, Edges. Bottleneck is human time and creativity!

https://cdn-images-1.medium.com/max/1600/0*XdmPfRUMnLUHsGJp.png

Paradigm shift

https://cdn-images-1.medium.com/max/1600/0*XdmPfRUMnLUHsGJp.png

What changed?

https://www.slideshare.net/LuMa921/deep-learning-a-visual-introduction/8

Next Paradigm shift?

ML engineers still involved in design of neural network architectures, and data processing decisions.

Trend is to automate that away

DL detects abnormalities in knee MRIs at the level of radiologists

Rajpurkar & Irvin et al., PLOS Medicine, 2018

DL can detect abnormalities in knee MRIs at the level of radiologists

Rajpurkar & Irvin et al., PLOS Medicine, 2018

Accuracy of experts w/ & w/o assistance can be compared with crossover design

Bien & Rajpurkar et al, PLOS Medicine, 2018

Model assistance improved specificity of detecting ACL tears on knee MRIs

Model assistance resulted in a

mean increase of 0.048 (4.8%) in ACL specificity.

No other significant improvements.

Bien & Rajpurkar et al, PLOS Medicine, 2018

DL detects cerebral aneurysms in head CTAs at the level of radiologists

Park & Chute & Rajpurkar et al., to appear

DL detects cerebral aneurysms in head CTAs at the level of radiologists

Park & Chute & Rajpurkar et al., to appear

Does model assistance with segmentation improve clinicians in detecting aneurysms?

Park & Chute & Rajpurkar et al, JAMA Network Open, 2019

Model assistance improved sensitivity in detecting aneurysms from head CTAs

Park & Chute & Rajpurkar et al, to appear

Clinician performance can be equalized with model assistance

Greatest improvement for clinician with lowest unaugmented score.

Smallest improvement for clinician with highest unaugmented score.

Park & Chute & Rajpurkar et al, to appear

Does model assistance improve cancer subtyping for pathologists?

Cancer Type A: 90%

Cancer Type B: 10%

Assistance did not significantly increase the accuracy of the pathologists

-Improvement in accuracy of the pathologists as a group is not significant (p = 0.184).

Rajpurkar & Uyumazturk & Kiani et al, in submission

Accuracy of the diagnostic assistant is dependent on pathologist experience

-Accuracy in the hands of gastrointestinal (GI) pathology specialists significantly higher compared with non-GI specialists and pathology trainees (p = 0.032).

Rajpurkar & Uyumazturk & Kiani et al, in submission

Can DL improve performance of physicians in South Africa on TB task

Work with Dr. Matt Lungren, Dr. Robyn Ball, and others at Stanford Medical School, and Dr. Tom Boyles at University of Cape Town, SA.

Rajpurkar & O’Connell et al, in submission

Clinicians Assisted is more accurate than unassisted (p = 0.002)

Rajpurkar & O’Connell et al, in submission

Autonomous Algorithm more accurate than Clinicians Assisted (p < 0.001)

Rajpurkar & O’Connell et al, in submission

Imagine snapping a photo of a chest x-ray for immediate analysis

Clinical Background

The majority of chest x-rays in the world are done on film.

With Amir Kiani and others, Stanford ML Group

XRay4All: Building a deployable web app for automated medical imaging

Platform analyzes uploaded images on a secure cloud backend and provides a probabilistic interpretation for different medical conditions.

You can also “teach” our AI agents by providing labeled images or correcting its reported interpretations.”

Work with Dr. Matt Lungren, Dr. Jeanne Shen,

Dr. Terry Fotre, and others at the Stanford Med

School and Hospital

With Amir Kiani and others, Stanford ML Group

We are building for fast inference time and secure storage

With Amir Kiani and others, Stanford ML Group

We are investigating performance of CheXNet when taking photos in clinic

Ongoing work, Stanford ML Group

XRay4All: Live demo

Link

With Amir Kiani and others, Stanford ML Group

We have released labeled datasets needed to train + validate models

Work with Dr. Matt Lungren, Dr. Curt Langlotz and many others at the Stanford Med School Irvin & Rajpurkar et al. 2019, AAAI; Bien & Rajpurkar et al, PLOS Medicine; Rajpurkar & Irvin et al. 2018, MIDL

Accurate labels generated w/ open-sourced labeler for radiology reports

200k Chest X-Rays

Expert Ground Truth Labels

Radiologist Benchmarks

Irvin & Rajpurkar et al. 2019, AAAI; Bien & Rajpurkar et al, PLOS Medicine; Rajpurkar & Irvin et al. 2018, MIDL

Open invite to the world to participate in our competitions w/ hidden test set

Irvin & Rajpurkar et al. 2019, AAAI; Bien & Rajpurkar et al, PLOS Medicine; Rajpurkar & Irvin et al. 2018, MIDL

AI For Healthcare Bootcamp

https://stanfordmlgroup.github.io/programs/aihc-bootcamp/