UNSURE2021
Uncertainty for Safe Utilization of Machine Learning in Medical Imaging

Recent Updates

  • September 2021: Proceedings available here.
  • July 2021: Decision due July 29, 2021
  • July 2021: Review period deadline on July 26, 2021
  • June 2021: Deadline extensions by one week to July 2, 2021
  • May 2021: Official call for papers announced and paper submission system activated
  • April 2021: Submission deadline of June 25, 2021 announced
  • April 2021: Website Up! The old event pages have been moved here: UNSURE 2019, UNSURE 2020

Important Dates

  • July 2, 2021 Extended Paper Submission Deadline (23:59 AoE time -- UTC-12)
  • June 25, 2021 Paper Submission Deadline (23:59 AoE time -- UTC-12)
  • July 26, 2021 Reviews due
  • July 29, 2021 Notification of paper decision
  • August 6, 2021 Camera ready papers due
  • October 1, 2021 Workshop date

About the MICCAI UNSURE Workshop

Overview

With the rise and influence of machine learning (ML) in medical application and the need to translate newly developed techniques into clinical practice, questions about safety and uncertainty over measurements and reported quantities have gained importance. Obtaining accurate measurements is insufficient, as one needs to establish the circumstances under which these values generalize, or give appropriate error bounds for these measures. This is becoming particularly relevant to patient safety as many research groups and companies have deployed or are aiming to deploy ML technology in clinical practice.

The purpose of this workshop is to develop awareness and encourage research on uncertainty modelling to ensure safety for applications spanning both the MIC and CAI fields. In particular, this workshop invites submissions to cover different facets of this topic, including but not limited to: detection and quantification of algorithmic failures; processes of healthcare risk management (e.g. CAD systems); robustness and adaptation to domain shifts; evaluation of uncertainty estimates; defence against noise and mistakes in data (e.g. bias, label mistakes, measurement noise, inter/intra-observer variability). The workshop aims to encourage contributions in a wide range of applications and types of ML algorithms. The use or development of any relevant ML methods are welcomed, including, but not limited to, probabilistic deep learning, Bayesian nonparametric statistics, graphical models and Gaussian processes. We also aim to ensure broad coverage of applications in the context of both MIC and CAI, which are categorized into reporting problems (descriptions of image contents) such as diagnosis, measurements, segmentation, detection, and enhancement problems (addition of information) such as image synthesis, registration, reconstruction, super-resolution, harmonisation, inpainting and augmented display.

Details

In the last few years, machine learning (ML) techniques have permeated many aspects of the MICCAI community, leading to substantial progress in a wide range of applications ranging from image analysis to surgical assistance. However, in medical applications, algorithms ultimately assist life and death decisions, and translation of such innovations into practice requires a measure of safety. In practice, ML systems often face situations where the correct decision is ambiguous, and therefore principled mechanisms for quantifying uncertainty are required to envision potential practical deployment.

Safety is indeed paramount in medical imaging applications, where images inform scientific conclusions in research, and diagnostic, prognostic or interventional decisions in clinics. However, efforts have mostly focused on improving the accuracy, while systematic approaches ensuring safety of medical imaging-derived automated systems are largely lacking in the existing body of research.

Uncertainty quantification has recently attracted attention in the MICCAI community as a promising approach to provide a reliability metric of the output, and as a mechanism to communicate the knowledge boundary of such ML systems. Spurred on by this emergent interest, the workshop will encourage discussions on the topic of uncertainty modelling and alternative approaches for risk management in a wide range of medical applications. It aims thereby to highlight both fundamental and practical challenges that need to be addressed to achieve safer implementations of ML systems in the clinical world.

Call for Papers

Scope

We accept submissions of original, unpublished work on safety and uncertainty in medical imaging, including (but not limited to) the following areas:

  • Uncertainty quantification in any MIC or CAI applications
  • Risk management of ML systems in clinical pipelines
  • Defending against hallucinations in enhancement tasks (e.g. super-resolution, reconstruction, modality translation)
  • Robustness to domain shifts
  • Measurement errors
  • Modelling noise in data (e.g. labels, measurements, inter/intra-observer variability)
  • Validation of uncertainty estimates
  • Active Learning
  • Confidence bounds
  • Posterior inference over point estimates
  • Bayesian deep learning
  • Graphical models
  • Gaussian processes
  • Calibration of uncertainty measures
  • Bayesian decision theory

Submission Format

Submissions must be 8-page papers (excluding references) following the Springer LNCS format. Author names, affiliations and acknowledgements, as well as any obvious phrasings or clues that can identify authors must be removed to ensure anonymity. Note that in contrast to the requirements to the main MICCAI conference, the 8 page limit refers only to the main content. Including references and acknowledgements the submission may exceed 8 pages.

How to submit?

Please submit papers using the paper submission system.
Submission Deadline: June 25, 2021 - 23:59 (AoE time)
Extended Submission Deadline: July 2, 2021 - 23:59 (AoE time)

Publication

All accepted papers will be published as part of the MICCAI Satellite Events joint LNCS proceedings to be published by Springer Nature.

Resources

Best paper award

The UNSURE 2021 Best Paper award was presented to Christoph Berger and colleagues for their work: Confidence based out of distribution detection: a comparative study and analysis.

Best paper runner-up

The runner-up award for best paper was presented to Alexandru Tifrea and collegues for their work: Novel disease detection using ensembles with regularized disagreement.

Proceedings

The full Springer proceedings for the UNSURE 2021 workshop are available here: https://link.springer.com/book/10.1007/978-3-030-87735-4

Program

Schedule

Timezone Information: All times below are given in UTC Time. Please be mindful of time zone conversions when planning your attendance. The proceedings for the workshop are available here.
  • 09:00-09:10 Introduction
  • 09:10-10:00 Keynote - Prof. Roland Wiest
  • 10:00-10:35 Spotlight Session 1
    • Suvodeep Sinha 10:00-10:05
      "Model Uncertainty Estimation for Medical Imaging Based Diagnosis", Di Qiu et al.

      10:05-10:10
      "Leveraging Uncertainty Estimates to improve Segmentation performance in cardiac MR", Tewodros W Arega et al.

      10:10-10:15
      "Task-agnostic out of distribution detection using kernel density estimation", Ertunc Erdil et al.

      10:15-10:20
      "Out of distribution detection for medical images", Oliver Zhang et al.

      10:20-10:25
      "Uncertainty-aware deep learning based deformable registration", Irina Grigorescu et al.
      10:25-10:30
      "Robust Selective classification of skin lesions with asymmetric costs", Stephen J McKenna et al.
      10:30-10:35
      "Monte Carlo Concrete DropPath for epistemic uncertainty estimation in brain tumor segmentation", Natalia E Khanzhina et al.
  • 10:35-10:50 Break
  • 10:50-11:50 Long Oral Session 1
    • 10:50-11:05
      "Accurate simulation of operating system updates in neuroimaging using Monte-Carlo arithmetic", Ali Salari et al.

      11:05-11:20
      "Novel disease detection using ensembles with regularized disagreement", Alexandru Tifrea et al.
      11:20-11:35
      "Improving the reliability of semantic segmentation of medical images by uncertainty modelling with bayesian deep networks and curriculum learning", Bisser Raytchev et al.
      11:35-11:50
      Q&A Session
  • 11:50-13:00 Lunch Break
  • 13:00-14:00 Long Oral Session 2
    • 13:00-13:15
      "Unpaired MR Image Homogeneisation by disentangled representations and its uncertainty", Hongwei Li et al.

      13:15-13:30
      "Confidence based out of distribution detection: a comparative study and analysis", Christoph Berger et al.
      13:30-13:45
      "Improving aleatoric uncertainty quantification in multi-annotated medial image segmentation with normalizing flows", Amaan Valiuddin et al.
      13:45-14:00
      Q&A Session
  • 14:00-14:50 Keynote - Prof. Ender Konukoglu
  • 14:50-15:10 Break
  • 15:10-17:30 Poster Session
    • Poster Room A

    • Poster A1: Model Uncertainty Estimation for Medical Imaging Based Diagnosis -- Di Qiu et al.
    • Poster A2: Leveraging Uncertainty Estimates to improve Segmentation performance in cardiac MR -- Tewodros W Arega et al.
    • Poster A3: Task-agnostic out of distribution detection using kernel density estimation -- Ertunc Erdil et al.
    • Poster A4: Out of distribution detection for medical images -- Oliver Zhang et al.
    • Poster A5: Uncertainty-aware deep learning based deformable registration -- Irina Grigorescu et al.
    • Poster A6: Robust Selective classification of skin lesions with asymmetric costs -- Stephen J McKenna et al.
    • Poster Room B

    • Poster B1: Accurate simulation of operating system updates in neuroimaging using Monte-Carlo arithmetic -- Ali Salari et al.
    • Poster B2: Novel disease detection using ensembles with regularized disagreement -- Alexandru Tifrea et al.
    • Poster B3: Improving the reliability of semantic segmentation of medical images by uncertainty modelling with bayesian deep networks and curriculum learning -- Bisser Raytchev et al.
    • Poster B4: Unpaired MR Image Homogeneisation by disentangled representations and its uncertainty -- Hongwei Li et al.
    • Poster B5: Monte Carlo Concrete DropPath for epistemic uncertainty estimation in brain tumor segmentation -- Natalia E Khanzhina et al.
    • Poster B6: Improving aleatoric uncertainty quantification in multi-annotated medial image segmentation with normalizing flows - Amaan Valiuddin et al.
  • 17:30-17:40 Concluding Remarks & Best paper award

Keynote Speakers

Prof. Dr. Roland Wiest

Institute for Diagnostic and Interventional Neuroradiology, University Hospital Bern, and leader of the Support Center for Advanced Neuroimaging (SCAN) research group

UNSURE2021 Organizing Committee

Organizers

Christian Baumgartner

Cluster of Excellence: Machine Learning for Science, University of Tubingen

Adrian Dalca

CSAIL, MIT and
MGH, Harvard Medical School

Carole Sudre

MRC Unit for Lifelong Health and Ageing, University College London / Centre for Medical Image Computing, University College London / Department of Biomedical Engineering, King's College London

Ryutaro Tanno

Department of Computer Science, University College London

Koen Van Leemput

Technical University of Denmark and
Harvard Medical School

William (Sandy) Wells

Radiology, BWH, Harvard Medical School

Program Committee

  • Alejandro Granados (King's College London)
  • Alireza Mehrtash (Brigham and Women's Hospital)
  • Arunkumar Kannan (University of British Columbia)
  • Azat Garifullin (Lappeeranta University of Technology)
  • Daniel Grzech (Imperial College London)
  • Daniel Coelho (Imperial College London)
  • Eleni Chiou (University College London)
  • Evan Yu (Cornell University)
  • Felix Bragman (King's College London)
  • Hongxiang Lin (University College London / Zheijiang Lab)
  • Ivor Simpson (University of Sussex)
  • Jinwei Zhang (Cornell University)
  • Jorge Cardoso (King's College London)
  • Leo Joskowicz (Hebrew University of Jerusalem)
  • Liane Canas (King's College London)
  • Malte Hoffmann (Harvard Medical School)
  • Mark Graham (King's College London)
  • Max-Heinrich Laves (Hamburg University of Technology)
  • Pedro Borges (King's College London)
  • Pieter Van Molle (Ghent University)
  • Raghav Mehta (McGill University)
  • Reuben Dorent (King's College London)
  • Robin Camarasa (Erasmus Medical Centre)
  • Roger Soberanis-Mukul (Technische Universitat Munchen)
  • Tanya Nair (Imagia)
  • Thomas Varsavsky (King's College London)
  • Tim Adler (DKFZ)
  • Yukun Ding (University of Notre-Dame)
  • Zhilu Zhang (Cornell University)

Contact

For general inquiries please send an email to: unsure@mit.edu