Cookies on this website
We use cookies to ensure that we give you the best experience on our website. If you click 'Continue' we'll assume that you are happy to receive all cookies and you won't see this message again. Click 'Find out more' for information on how to change your cookie settings.

New reporting guidelines, jointly published in Nature Medicine and the BMJ by Oxford researchers, will ensure that early studies on using Artificial Intelligence (AI) to treat real patients will give researchers the information needed to develop AI systems safely and effectively.

© Shutterstock

Artificial Intelligence in medicine has shown promising results in numerous simulation studies, but very few AI systems have yet been used in patient care. But as a growing number of AI-driven clinical decision-support systems are progressing from development to implementation, better guidance on the reporting of human factors and early-stage clinical evaluation is needed.

Researchers from Oxford have led the development of a new guideline, DECIDE-AI, which aims to improve the reporting of research on AI systems when they are used for the first time in real clinical settings by doctors treating actual patients. It was developed based on opinions and feedback from over 150 experts across 18 countries, including computer scientists, clinicians, ethicists, patient representatives, and entrepreneurs. 

Baptiste Vasey of Oxford University’s Nuffield Department of Surgical Sciences (NDS) said: 'While AI has shown promising results for clinical application in simulations, very few AI systems have had any significant effect on patient care so far. A major problem has been a lack of robust scientific evidence to back widespread use in clinical practice.

‘This study is the first to clearly define the minimum reporting standards for the early stage evaluation of AI-based decision support systems in clinical settings.’

Professor Peter McCulloch, the senior author, also based at NDS, said: 'These guidelines take the principles developed for evaluating new surgical operations by the Oxford-based IDEAL Collaboration, and apply them to an important new field where guidance is lacking. The idea that innovation and evaluation should proceed in tandem at every stage in the development of new treatments applies just as much to AI as it does to surgery. 

Benjamin Hornsby, Project Coordinator for the IDEAL Collaboration at NDS, said: ‘DECIDE-AI is only the second reporting guideline dedicated to AI systems research, and the first ever to focus on the development stage of this technology. 

‘We chose to focus on early-stage evaluation results to emphasise the need for researchers to report transparently about four key aspects of AI research, to build confidence in the technology and highlight the practical benefits and improved efficiencies the technology can bring to healthcare.’

The factors DECIDE-AI evaluates are:

  • proof of clinical value in a live care environment
  • the AI system’s safety profile and potential risks for patients
  • human factors around the use of the system, such as how the AI system is integrated into the existing care pathway and how it influences clinicians’ behaviour
  • the steps needed to develop the larger-scale trials which will be needed to provide conclusive evidence necessary for an AI system to be accepted into routine patient care.

The full paper, ‘Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI’, can be read in Nature Medicine and the BMJ.

Similar stories

Oxford's largest ever study into varicose veins shows need for surgery is linked to genetics

Varicose veins are a very common manifestation of chronic venous disease, affecting over 30% of the population in Western countries. In America, chronic venous disease affects over 11 million men and 22 million women aged 40–80 years old. Left untreated it can escalate to multiple health complications including leg ulcers and ultimately amputations. A new international study by Oxford researchers published on 2 June 2022 in Nature Communications establishes for the first time, a critical genetic risk score to predict the likelihood of patients suffering with varicose veins to require surgery, as well as pointing the way towards potential new therapies.

Results of the REF 2021: congratulations and thank you

Today, the UK funding bodies have published the results of the UK’s most recent national research assessment exercise, the Research Excellence Framework (REF) 2021.

NDS projects awarded CRUK Oxford Cancer Development Fund

Two research projects at the Nuffield Department of Surgical Sciences (NDS) have been awarded pump-prime funding from this year’s CRUK Oxford Centre Development Fund.

Blog posts

Oxford MedSci goes silver: 10 Years of Athena SWAN

The Medical Sciences Division is celebrating 10 years since its first Athena Swan bronze application, and the first year in which all 16 of its departments have achieved a silver award. The silver award recognises commitment to gender equality, understanding culture and context, and more. Read about our department’s hard work and innovation.

Lights, camera, action! My journey into video production

Dr Hannah McGivern provides a 'behind-the-scenes' account of her experience producing the video 'Journey of a QUOD Sample: Donating to Transplant Research', supported by the funds from the University of Oxford Public Engagement with Research (PER) Seed Fund.

Mentoring in practice

NDS has launched a new, interdepartmental mentoring scheme called RECOGNISE. In this podcast, Gemma Horbatowski (HR Advisor) interviews Monica Dolton (Programme Manager and Research Project Manager) about her experiences of mentor-mentee relationships.