Cookies on this website
We use cookies to ensure that we give you the best experience on our website. If you click 'Continue' we'll assume that you are happy to receive all cookies and you won't see this message again. Click 'Find out more' for information on how to change your cookie settings.

In a correspondence to Nature Medicine, a team of Oxford-led academics describe upcoming new guidelines to improve the reporting of early clinical stage (or first-with-human) evaluation of decision support systems driven by artificial intelligence.

Red light with silhouette of a man

As an increasing number of clinical decision-support systems driven by artificial intelligence progress from development to implementation, better guidance on the reporting of human factors (ergonomics) and early-stage clinical evaluation is needed.

Therefore, Dr Baptiste Vasey and Professor Peter McCulloch from the Nuffield Department of Surgical Sciences, together with a steering group of experts in the fields of machine learning, human factors, and guidelines development from Oxford University and other universities in the UK and abroad, aim to develop new guidelines to improve the reporting of the early-stage clinical evaluation of AI-based decision support systems.

“We are convinced that human clinicians should and will remain at the centre of patient care, and therefore aiming to improve the way in which AI-based clinical decision support systems are evaluated when used to enhance rather than replace human intelligence. A critical phase of this process is when such systems are assessed when first used by clinicians in real life settings,” says Dr Vasey.

The article outlines a Delphi process, comprising two rounds of experts’ feedback and one consensus meeting, and the rationale for the new Developmental and Exploratory Clinical Investigation of Decision support systems based on Artificial Intelligence (DECIDE-AI) guidelines being developed. The authors hope to address some of the important issues hindering the translation from in silico algorithm development to clinical impact.

The current literature is lacking of clear recommendations for this stage of clinical algorithm development, which has led to low quality research and poor reporting, as well as a gap between the number of algorithms proposed in scientific publications and the number of algorithms actually impacting on patient care.

The focus of the DECIDE-AI guidelines will be on the algorithm’s actual performance when used by clinicians at small scale, its safety profile, the human factors evaluation, and collecting the algorithm’s characteristics necessary to inform the study design of larger-scale trials.

“The early clinical stage of evaluation takes place between the purely computational validation of an algorithm and its large-scale evaluation through clinical trials. It can be compared to phase I/II trials in the drugs development pathway or IDEAL stage 2a/2b studies for surgical innovation,” says Professor McCulloch.

A Delphi process was launched at the end of January 2021. The DECIDE-AI Steering Group welcome expression of interest from experts in related fields wishing to contribute. Please contact Dr Baptiste Vasey.

Read the correspondence to Nature Medicine 

Meet the NDS researchers

Similar stories

Cochrane ENT awarded NIHR funding for COVID-19 project

Awards and appointments Coronavirus COVID-19 Research

Cochrane ENT at the Nuffield Department of Surgical Sciences has been awarded funding by the National Institute for Health Research (NIHR) to complete a suite of living systematic reviews investigating the effectiveness and safety of interventions to prevent and treat loss of smell after COVID-19 infection.

First peer-reviewed results of phase 3 human trials of Oxford coronavirus vaccine demonstrate efficacy

Coronavirus COVID-19 Publication Research

Today University of Oxford and AstraZeneca researchers present a pooled analysis of Phase 3 trials of a vaccine against SARS-CoV-2 across two different dose regimens, resulting in an average efficacy of 70.4%.

Applications for Oxford-Wolfson-Marriott-Patel Scholarship now open

Awards and appointments Department Research

The Nuffield Department of Surgical Sciences (NDS) are pleased to confirm that one scholarship will be available for Home (UK) and Republic of Ireland (ROI) candidates for the DPhil in Surgical Sciences, starting in Michaelmas Term 2021.

Oxford University breakthrough on global COVID-19 vaccine

Coronavirus COVID-19 Research

The University of Oxford, in collaboration with AstraZeneca plc, today announces interim trial data from its Phase III trials that show its candidate vaccine, ChAdOx1 nCoV-2019, is effective at preventing COVID-19 (SARS-CoV-2) and offers a high level of protection.

Transplant kidney survival rate improves with new method of transportation

Publication Research

New research by the Consortium for Organ Preservation in Europe (COPE) has found that a new method to transport donor kidneys, which uses a combination of cold perfusion with oxygen, can significantly improve transplant results, with less graft failure, better function and lower rejection of the kidney when compared to cold perfusion alone. For patients who need a kidney transplant this is a big step forward. The study, carried out in Belgium, The Netherlands and the UK, was published today in The Lancet.

Breast cancer study wins 2020 ONS Research Excellence Award

Awards and appointments Research

Dr Gurdeep Mannu from the Nuffield Department of Surgical Sciences, University of Oxford, has won the Office for National Statistics (ONS) Research Excellence Award for a study which highlighted early risk factors for breast cancer patients.