UNSURE2023
Uncertainty for Safe Utilization of Machine Learning in Medical Imaging

Recent Updates

Important Dates

  • July 6, 2023 Extended Paper Submission Deadline (23:59 PST)
  • June 29, 2023 Paper Submission Deadline (23:59 PST)
  • July 22, 2023 Reviews due
  • July 29, 2023 Publication of decisions
  • August 4, 2023 Camera ready submissions due
  • October 12, 2023 Workshop date

About the MICCAI UNSURE Workshop

Overview

With the rise and influence of machine learning (ML) in medical application and the need to translate newly developed techniques into clinical practice, questions about safety and uncertainty over measurements and reported quantities have gained importance. Obtaining accurate measurements is insufficient, as one needs to establish the circumstances under which these values generalize, or give appropriate error bounds for these measures. This is becoming particularly relevant to patient safety as many research groups and companies have deployed or are aiming to deploy ML technology in clinical practice.

The purpose of this workshop is to develop awareness and encourage research on uncertainty modelling to ensure safety for applications spanning both the MIC and CAI fields. In particular, this workshop invites submissions to cover different facets of this topic, including but not limited to: detection and quantification of algorithmic failures; processes of healthcare risk management (e.g. CAD systems); robustness and adaptation to domain shifts; evaluation of uncertainty estimates; defence against noise and mistakes in data (e.g. bias, label mistakes, measurement noise, inter/intra-observer variability). The workshop aims to encourage contributions in a wide range of applications and types of ML algorithms. The use or development of any relevant ML methods are welcomed, including, but not limited to, probabilistic deep learning, Bayesian nonparametric statistics, graphical models and Gaussian processes. We also aim to ensure broad coverage of applications in the context of both MIC and CAI, which are categorized into reporting problems (descriptions of image contents) such as diagnosis, measurements, segmentation, detection, and enhancement problems (addition of information) such as image synthesis, registration, reconstruction, super-resolution, harmonisation, inpainting and augmented display.

Details

In the last few years, machine learning (ML) techniques have permeated many aspects of the MICCAI community, leading to substantial progress in a wide range of applications ranging from image analysis to surgical assistance. However, in medical applications, algorithms ultimately assist life and death decisions, and translation of such innovations into practice requires a measure of safety. In practice, ML systems often face situations where the correct decision is ambiguous, and therefore principled mechanisms for quantifying uncertainty are required to envision potential practical deployment.

Safety is indeed paramount in medical imaging applications, where images inform scientific conclusions in research, and diagnostic, prognostic or interventional decisions in clinics. However, efforts have mostly focused on improving the accuracy, while systematic approaches ensuring safety of medical imaging-derived automated systems are largely lacking in the existing body of research.

Uncertainty quantification has recently attracted attention in the MICCAI community as a promising approach to provide a reliability metric of the output, and as a mechanism to communicate the knowledge boundary of such ML systems. Spurred on by this emergent interest, the workshop will encourage discussions on the topic of uncertainty modelling and alternative approaches for risk management in a wide range of medical applications. It aims thereby to highlight both fundamental and practical challenges that need to be addressed to achieve safer implementations of ML systems in the clinical world.

Format

The UNSURE workshop will be held as an in-person, full-day satellite event of MICCAI 2023 at the Vancouver Convention Centre. Although, we prefer in person presentation, speakers who are unable to join will be given the option to present virtually. We are currently not planning for virtual attendance for non-speakers, however, video recordings of all talks will be made available subject to the respective speakers' permission.

Call for Papers

Scope

We accept submissions of original, unpublished work on safety and uncertainty in medical imaging, including (but not limited to) the following areas:

  • Uncertainty quantification in any MIC or CAI applications
  • Risk management of ML systems in clinical pipelines
  • Out-of-distribution and anomaly detection
  • Defending against hallucinations in enhancement tasks (e.g. super-resolution, reconstruction, modality translation)
  • Robustness to domain shifts
  • Measurement errors
  • Modelling noise in data (e.g. labels, measurements, inter/intra-observer variability)
  • Validation of uncertainty estimates
  • Active Learning
  • Confidence bounds
  • Posterior inference over point estimates
  • Bayesian deep learning
  • Graphical models
  • Gaussian processes
  • Calibration of uncertainty measures
  • Bayesian decision theory

Submission Format

Our submission guidelines follow the guidelines of the main MICCAI conference. Submissions must be 8-page papers (excluding references) following the Springer LNCS format. Author names, affiliations and acknowledgements, as well as any obvious phrasings or clues that can identify authors must be removed to ensure anonymity. Note that the 8 page limit refers only to the main content. Including references and acknowledgements the submission may exceed 8 pages.


Analogous to the main conference we also allow supplementary materials in the form of a maximum 2-page PDF with tables, figures or proofs, and captions not exceeding 100 words. Multi-media materials such as videos are also allowed. Please refer to Section 5 of the MICCAI conference submission guidelines for more details.

How to submit?

Please submit papers using the paper submission system.

Submission Deadline: June 29, 2023 - 23:59 (PST)


Extended Submission Deadline: July 6, 2023 - 23:59 (PST)

Publication

All accepted papers will be published as part of the MICCAI Satellite Events joint LNCS proceedings to be published by Springer Nature.

Program

Date and Location

Date: October 12th, 2023, 08:00-17:30 Vancouver Time.

Schedule

Timezone Information: All times below are given in Vancouver Time (PDT). Please be mindful of time zone conversions when planning your attendance.

Location: Vancouver Convention Center (East), Meeting Room 9
  • 08:00-08:10 Introduction
  • 08:10-08:55 Keynote - Prof. James Moon
  • 09:00-10:00 Long Oral Session 1
    • 09:00-09:20
      Proper scoring loss functions are simple and effective for uncertainty quantification of White Matter Hyperintensities, Ben R Philps et al.

      09:20-09:40
      Numerical Uncertainty of Convolutional Neural Networks Inference for Structural Brain MRI Analysis, Ines Gonzalez Pepe et al.

      09:40-10:00
      Redesigning Out-of-Distribution Detection on 3D Medical Images, Anton Vasiliuk et al.

  • 10:00-10:30 Coffee Break
  • 10:30-11:30 Spotlight Session
    • 10:30-10:34
      Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentation, Jadie Adams et al.

      10:34-10:38
      Propagation and Attribution of Uncertainty in Medical Imaging Pipelines, Leonhard F Feiner et al.

      10:38-10:42
      Confidence-Aware and Self-Supervised Image Anomaly Localisation, Johanna P Müller et al.

      10:42-10:46
      RR-CP: Reliable-Region-Based Conformal Prediction for Trustworthy Medical Image Classification, Yizhe Zhang et al.

      10:46-10:50
      Bayesian Uncertainty Estimation in Landmark Localization using Convolutional Gaussian Processes, Lawrence Schobs et al.

      10:50-10:54
      Breaking Down Covariate Shift on Pneumothorax Chest X-ray Classification, Bogdan A Bercean et al.

      10:54-10:58
      Pitfalls of Conformal Predictions for Medical Image Classification, Hendrik A. Mehrtens et al.

      10:58-11:02
      Multi-layer Aggregation as a key to Feature-based OOD detection, Benjamin Lambert et al.

      11:02-11:06
      Examining the effects of slice thickness on the reproducibility of CT radiomics for patients with colorectal liver metastases, Jacob J Peoples et al.

      11:06-11:10
      Feature-Based Pipeline for Anomaly Segmentation on Medical Images, Daria Frolova et al.

      11:10-11:14
      Robustness Stress Testing in Medical Image Classification, Mobarakol Islam et al.

      11:14-11:18
      Uncertainty Estimation and Propagation in Accelerated MRI Reconstruction, Paul Fischer et al.

      11:18-11:22
      Uncertainty estimation in liver tumor segmentation using the posterior bootstrap, Shishuai Wang et al.

      11:22-11:26
      Uncertainty-based quality assurance of carotid artery wall segmentation in black-blood MRI, Elina Thibeau-Sutre et al.

      11:26-11:30
      Dimensionality Reduction for Improving Out-of-Distribution Detection in Medical Image Segmentation, McKell Woodland et al.
  • 11:30-14:15 Poster Session & Lunch Break
  • 14:15-15:20 Long Oral Session 2
    • 14:15-14:35
      TriadNet: Sampling-free predictive intervals for lesional volume in 3D brain MR images, Benjamin Lambert et al.

      14:35-14:55
      How inter-rater variability relates to aleatoric and epistemic uncertainty: a case study with deep learning-based paraspinal muscle segmentation, Parinaz Roshanzamir et al.

      14:55-15:15
      On the use of Mahalanobis distance for out-of-distribution detection with neural networks for medical imaging, Harry E J Anthony et al.
  • 15:20-16:00 Coffee Break
  • 16:00-16:45 Keynote - Anastasios Angelopoulos
  • 16:45-17:15 Panel Discussion (with Tal Arbel, William M. Wells III, Anastasios Angelopoulos & James Moon)
  • 17:15-17:30 Concluding Remarks & Best paper award

Keynote Speakers

Prof. James Moon

CMR lead at Barts Heart Centre London, Medical director of Chenies Mews Imaging Centre and CEO of Mycardium AI Ltd.

Anastasios Angelopoulos

Ph.D. student in Electrical Engineering and Computer Science at the University of California, Berkeley.

Recorded Videos

UNSURE 2023 Long Oral: https://youtube.com/playlist?list=PLS3ky3lN6xtqSLgyxpQqL0_oLHV1199VI&si=DrfHpe1T28yeaJnA
UNSURE 2023 Sportlight: https://youtube.com/playlist?list=PLS3ky3lN6xtpxAEieNa8PHzoiWn0Wts0Z&si=P6cm8JXua9UDX69R

UNSURE 2023 Organizing Committee

Organizers

In alphabetical order

Christian F. Baumgartner

Cluster of Excellence: Machine Learning for Science, University of Tubingen

Adrian Dalca

CSAIL, MIT and
MGH, Harvard Medical School

Raghav Mehta

Probabilistic Vision Group, Centre for Intelligent Machines, McGill University

Chen Qin

Imperial College London

Carole Sudre

MRC Unit for Lifelong Health and Ageing, University College London / Centre for Medical Image Computing, University College London / Department of Biomedical Engineering, King's College London

William (Sandy) Wells

Radiology, BWH, Harvard Medical School

On-Site Organizers

Paul Fischerr

Cluster of Excellence: Machine Learning for Science, University of Tubingen

Parinaz Roshanzamir

Concordia University

Program Committee

  • Adrian Galdran (Universitat Pompeu Fabra)
  • Cheng Ouyang (Imperial College London)
  • Chloe He (University College London)
  • Di Qiu (The Chinese University of Hong Kong)
  • Dimitri Hamzaoui (INRIA)
  • Evan Yu (Cornell University)
  • Fons van der Sommen (Eindhoven University of Technology)
  • Hariharan Ravishankar (GE Healthcare)
  • Hongwei Li (Technical University of Munich)
  • Hongxiang Lin (University College London / Zheijiang Lab)
  • Isaac Llorente Saguer (University College London)
  • Ishaan Bhat (University Medical Center Utrecht)
  • Jacob Carse (University of Dundee)
  • Jinwei Zhang (Cornell University)
  • Leo Joskowicz (Hebrew University of Jerusalem)
  • Liane Canas (King's College London)
  • Luke Whitbread (University of Adelaide)
  • Mark Graham (King's College London)
  • Matthew Baugh (Imperial College London)
  • Max-Heinrich Laves (Leibniz Universitat Hannover)
  • Neil Oxtoby (University College London)
  • Parinaz Roshanzamir (Concordia University)
  • Pedro Borges (King's College London)
  • Prasad Sudhakar (GE Healthcare)
  • Prerak Mody (Leiden University Medical Centre)
  • Robin Camarasa (Erasmus Medical Centre)
  • Stephen J. McKenna (University of Dundee)
  • Tim Adler (DKFZ)
  • Tristan Glatard (Concordia University)
  • William Consagra (Harvard Medical School)
  • Won Hwa Kim (POSTECH)
  • Yiming Xiao (Concordia University)
  • Yipeng Hu (University College London)
  • Contact

    For general inquiries please send an email to: unsure@mit.edu