Cookies on this website
We use cookies to ensure that we give you the best experience on our website. If you click 'Continue' we'll assume that you are happy to receive all cookies and you won't see this message again. Click 'Find out more' for information on how to change your cookie settings.

In a correspondence to Nature Medicine, a team of Oxford-led academics describe upcoming new guidelines to improve the reporting of early clinical stage (or first-with-human) evaluation of decision support systems driven by artificial intelligence.

Red light with silhouette of a man

As an increasing number of clinical decision-support systems driven by artificial intelligence progress from development to implementation, better guidance on the reporting of human factors (ergonomics) and early-stage clinical evaluation is needed.

Therefore, Dr Baptiste Vasey and Professor Peter McCulloch from the Nuffield Department of Surgical Sciences, together with a steering group of experts in the fields of machine learning, human factors, and guidelines development from Oxford University and other universities in the UK and abroad, aim to develop new guidelines to improve the reporting of the early-stage clinical evaluation of AI-based decision support systems.

“We are convinced that human clinicians should and will remain at the centre of patient care, and therefore aiming to improve the way in which AI-based clinical decision support systems are evaluated when used to enhance rather than replace human intelligence. A critical phase of this process is when such systems are assessed when first used by clinicians in real life settings,” says Dr Vasey.

The article outlines a Delphi process, comprising two rounds of experts’ feedback and one consensus meeting, and the rationale for the new Developmental and Exploratory Clinical Investigation of Decision support systems based on Artificial Intelligence (DECIDE-AI) guidelines being developed. The authors hope to address some of the important issues hindering the translation from in silico algorithm development to clinical impact.

The current literature is lacking of clear recommendations for this stage of clinical algorithm development, which has led to low quality research and poor reporting, as well as a gap between the number of algorithms proposed in scientific publications and the number of algorithms actually impacting on patient care.

The focus of the DECIDE-AI guidelines will be on the algorithm’s actual performance when used by clinicians at small scale, its safety profile, the human factors evaluation, and collecting the algorithm’s characteristics necessary to inform the study design of larger-scale trials.

“The early clinical stage of evaluation takes place between the purely computational validation of an algorithm and its large-scale evaluation through clinical trials. It can be compared to phase I/II trials in the drugs development pathway or IDEAL stage 2a/2b studies for surgical innovation,” says Professor McCulloch.

A Delphi process was launched at the end of January 2021. The DECIDE-AI Steering Group welcome expression of interest from experts in related fields wishing to contribute. Please contact Dr Baptiste Vasey.

Read the correspondence to Nature Medicine 

Meet the NDS researchers

Similar stories

Dr Matthew Bottomley wins Stewart Cameron Science Award

Congratulations to Dr Matthew Bottomley, who has been announced as the joint winner of the Stewart Cameron Science Award by the Royal Society of Medicine (RSM).

Clare Verrill receives Turing Fellowship

Thirty-three University of Oxford researchers have been named Turing Fellows for the 2021/22 academic year, including Associate Professor Clare Verrill from the Nuffield Department of Surgical Sciences (NDS).

Researchers awarded Wellcome Innovator Grant to investigate role of brainstem nucleus in human consciousness

Researchers at Oxford University have received a prestigious Wellcome Innovator Grant for investigating the role of the pedunculopontine nucleus (PPN) – a brainstem nucleus – in human consciousness.

Discovered gene patterns can predict prostate cancer treatment response

Nearly 40,000 men are diagnosed with prostate cancer each year in the UK. Perhaps the most significant clinical challenge today is deciding which type of treatment will work best for different patient groups.

Regent Lee wins top UKRI Future Leaders Fellowship

Dr Regent Lee of the Nuffield Department of Surgical Sciences is one of eight Oxford University academics who have been awarded significant financial funding from the UK Research and Innovation’s (UKRI) Future Leaders Fellowships Scheme.

Largest trial of carotid artery surgery and stenting finds similar long-term effects on stroke risk

Results from a major clinical trial demonstrate that both stenting and surgery are low-risk and similarly effective procedures for treating carotid artery disease.

Blog posts

Wrap up of 2021

After a brilliant year of hard work and dedication, the SITU team has done some reflection on 2021, focusing on key events, trial progression, and more. Read on to discover how the year 2021 went for the SITU team...

Celebrating Anti-Bullying Week with words of kindness

Keeva Heap, who is undertaking a communications work experience placement at the Nuffield Department of Surgical Sciences (NDS), shares our words of kindness In honour of Anti-Bullying Week.

My virtual work experience with NDS and NDORMS

Louise Tan, a Year 12 student from Ballyclare in County Antrim, Northern Ireland, recently attended the joint NDS and NDORMS Virtual Work Experience. In this guest blog, Louise reflects on her experience.

Celebrating women of NDS

To celebrate 100 years since women were admitted as full members of the University and on the occasion of International Women's Day, a group of inspirational women in the Nuffield Department of Surgical Sciences (NDS) reflect on their journeys, their place in Medical Sciences and their vision for the next 100 years.

The life of a research nurse: supporting the Oxford COVID-19 Vaccine Trial

Research nurses in the NHS are playing a crucial role in helping to trial new coronavirus treatments and vaccines. Three NDS research nurses stepped up to help with the fight against this new disease. Here Bhumika Patel shares her experience of working on the Oxford COVID-19 Vaccine Trial.