AIDA - Annotation of Image Data by Assignments
An in-house annotation platform, developed in Oxford by: Alan Aberdeen, Nasullah Khalid Alham, Clare Verrill, Jens Rittscher
The validation of quantitative image analysis algorithms as well as the development of strongly or weakly supervised machine learning methods depends on well-curated collections of annotated images. Especially the automated analysis of histology images requires the input of highly trained experts who are able to discern the high visual complexity and relevance of minute details in such images.
While a number of annotation systems have been developed, we propose an approach that structures the image annotation task into a set of well-defined assignments. Our server based annotation system allows the definition of such a prescribed annotation task for a given study. The user interface has been designed so that it is intuitive and does not require any special training. We pay special attention to the fact that human experts often disagree and provide a set of tools that allow highlighting of differences in annotation patterns and extracting a consensus from various experts.
Although the system has been designed for algorithm validation and method development we demonstrate the utility of the proposed approach in a specific annotation study.
The ‘’Immunoscore’’ is a concept developed in colorectal adenocarcinoma samples, that evaluates inflammatory cell infiltrates in CD3 and CD8 immunohistochemical stained sections at the invasive margin (IM) and centre of the tumour (CT). Guidance is not given on the definition of the IM or CT. The Immunoscore potentially provides additional prognostic information, above that provided by traditional TNM staging.
6 pathologists will be asked to assess 8 radical prostatectomy H&E images and draw the following regions of interest: tumour (centre of tumour - CT), tumour (invasive margin - IM) and non-tumour. They were also asked to annotate glands. The annotations provided by individual pathologists will then be compared to one another in order to determine how the IM and CT had been interpreted.