Publications
Proceedings, refereed
Man vs. AI: An in silico study of polyp detection performance
In 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS). L'Aquila, Italy: IEEE, 2023.Status: Published
Man vs. AI: An in silico study of polyp detection performance
AI-based colon polyp detection systems have received much attention, and several products and prototypes report good results. In silico verification is a crucial step when developing such systems, but very few compare human versus AI performance. This paper, therefore, describes methods and results for an in silico test of an AI model with two different versions for polyp detection in colonoscopy and compares them to the performance of endoscopist doctors who reviewed the same colonoscopy video clips. The two versions have different thresholds for false positive rate reduction. Our models perform polyp detection within the range of the endoscopists' performance, although faster, showing a potential for use in a clinical setting. For the AI and the endoscopists alike, the results show a trade-off between high sensitivity and high specificity; to achieve perfect detection, one will also get abundance of false positives. This can cause alarm fatigue in a clinical setting.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2023 |
Conference Name | 2023 IEEE 36th International Symposium on Computer-Based Medical Systems (CBMS) |
Publisher | IEEE |
Place Published | L'Aquila, Italy |
URL | https://ieeexplore.ieee.org/document/10178833 |
DOI | 10.1109/CBMS58004.2023.00307 |
Journal Article
A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging
Medical Image Analysis 70 (2021): 102007.Status: Published
A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging
Gastrointestinal endoscopy has been an active field of research motivated by the large number of highly lethal GI cancers. Early GI cancer precursors are often missed during the endoscopic surveillance. The high missed rate of such abnormalities during endoscopy is thus a critical bottleneck. Lack of attentiveness due to tiring procedures, and requirement of training are few contributing factors. An automatic GI disease classification system can help reduce such risks by flagging suspicious frames and lesions. GI endoscopy consists of several multi-organ surveillance, therefore, there is need to develop methods that can generalize to various endoscopic findings. In this realm, we present a comprehensive analysis of the Medico GI challenges: Medical Multimedia Task at MediaEval 2017, Medico Multimedia Task at MediaEval 2018, and BioMedia ACM MM Grand Challenge 2019. These challenges are initiative to set-up a benchmark for different computer vision methods applied to the multi-class endoscopic images and promote to build new approaches that could reliably be used in clinics. We report the performance of 21 participating teams over a period of three consecutive years and provide a detailed analysis of the methods used by the participants, highlighting the challenges and shortcomings of the current approaches and dissect their credibility for the use in clinical settings. Our analysis revealed that the participants achieved an improvement on maximum Mathew correlation coefficient (MCC) from 82.68% in 2017 to 93.98% in 2018 and 95.20% in 2019 challenges, and a significant increase in computational speed over consecutive years.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Medical Image Analysis |
Volume | 70 |
Pagination | 102007 |
Publisher | Elsevier |
Keywords | Artificial intelligence, BioMedia 2019 Grand Challenge, Computer-aided detection and diagnosis, Gastrointestinal endoscopy challenges, Medical imaging, Medico Task 2017, Medico Task 2018 |
DOI | 10.1016/j.media.2021.102007 |
A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation
IEEE Journal of Biomedical and Health Informatics 25, no. 6 (2021): 2029-2040.Status: Published
A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation
Colonoscopy is considered the gold standard for detection of colorectal cancer and its precursors. Existing examination methods are, however, hampered by high overall miss-rate, and many abnormalities are left undetected. Computer-Aided Diagnosis systems based on advanced machine learning algorithms are touted as a game-changer that can identify regions in the colon overlooked by the physicians during endoscopic examinations, and help detect and characterize lesions. In previous work, we have proposed the ResUNet++ architecture and demonstrated that it produces more efficient results compared with its counterparts U-Net and ResUNet. In this paper, we demonstrate that further improvements to the overall prediction performance of the ResUNet++ architecture can be achieved by using Conditional Random Field (CRF) and Test-Time Augmentation (TTA). We have performed extensive evaluations and validated the improvements using six publicly available datasets: Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS-Larib PolypDB, ASU-Mayo Clinic Colonoscopy Video Database, and CVC-VideoClinicDB. Moreover, we compare our proposed architecture and resulting model with other State-of-the-art methods. To explore the generalization capability of ResUNet++ on different publicly available polyp datasets, so that it could be used in a real-world setting, we performed an extensive cross-dataset evaluation. The experimental results show that applying CRF andTTA improves the performance on various polyp segmentation datasets both on the same dataset and cross-dataset. To check the model’s performance on difficult to detect polyps, we selected, with the help of an expert gastroenterologist,196sessile or flat polyps that are less than ten millimeters in size. This additional data has been made available as a subset of Kvasir-SEG. Our approaches showed good results for flat or sessile and smaller polyps, which are known to be one of the major reasons for high polyp miss-rates. This is one of the significant strengths of our work and indicates that our methods should be investigated further for use in clinical practice.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | IEEE Journal of Biomedical and Health Informatics |
Volume | 25 |
Issue | 6 |
Pagination | 2029 - 2040 |
Publisher | IEEE |
Keywords | colonoscopy, conditional random field, generalization, Polyp segmentation, ResUNet++, test-time augmentation |
DOI | 10.1109/JBHI.2021.3049304 |
Kvasir-Capsule, a video capsule endoscopy dataset
Scientific Data 8, no. 1 (2021): 142.Status: Published
Kvasir-Capsule, a video capsule endoscopy dataset
Artificial intelligence (AI) is predicted to have profound effects on the future of video capsule endoscopy (VCE) technology. The potential lies in improving anomaly detection while reducing manual labour. Existing work demonstrates the promising benefits of AI-based computer-assisted diagnosis systems for VCE. They also show great potential for improvements to achieve even better results. Also, medical data is often sparse and unavailable to the research community, and qualified medical personnel rarely have time for the tedious labelling work. We present Kvasir-Capsule, a large VCE dataset collected from examinations at a Norwegian Hospital. Kvasir-Capsule consists of 117 videos which can be used to extract a total of 4,741,504 image frames. We have labelled and medically verified 47,238 frames with a bounding box around findings from 14 different classes. In addition to these labelled images, there are 4,694,266 unlabelled frames included in the dataset. The Kvasir-Capsule dataset can play a valuable role in developing better algorithms in order to reach true potential of VCE technology.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Scientific Data |
Volume | 8 |
Issue | 1 |
Pagination | 142 |
Publisher | Springer Nature |
URL | http://www.nature.com/articles/s41597-021-00920-z |
DOI | 10.1038/s41597-021-00920-z |
Journal Article
HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy
Scientific Data 7, no. 1 (2020): 1-14.Status: Published
HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy
Artificial intelligence is currently a hot topic in medicine. However, medical data is often sparse and hard to obtain due to legal restrictions and lack of medical personnel for the cumbersome and tedious process to manually label training data. These constraints make it difficult to develop systems for automatic analysis, like detecting disease or other lesions. In this respect, this article presents HyperKvasir, the largest image and video dataset of the gastrointestinal tract available today. The data is collected during real gastro- and colonoscopy examinations at Bærum Hospital in Norway and partly labeled by experienced gastrointestinal endoscopists. The dataset contains 110,079 images and 374 videos, and represents anatomical landmarks as well as pathological and normal findings. The total number of images and video frames together is around 1 million. Initial experiments demonstrate the potential benefits of artificial intelligence-based computer-assisted diagnosis systems. The HyperKvasir dataset can play a valuable role in developing better algorithms and computer-assisted examination systems not only for gastro- and colonoscopy, but also for other fields in medicine.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2020 |
Journal | Scientific Data |
Volume | 7 |
Issue | 1 |
Pagination | 1-14 |
Date Published | 08/2020 |
Publisher | Springer Nature |
Keywords | dataset, GI, Machine learning |
URL | http://www.nature.com/articles/s41597-020-00622-y |
DOI | 10.1038/s41597-020-00622-y |
Proceedings, refereed
Kvasir-SEG: A Segmented Polyp Dataset
In International Conference on Multimedia Modeling. Daejeon, Korea: Springer, 2020.Status: Published
Kvasir-SEG: A Segmented Polyp Dataset
Pixel-wise image segmentation is a highly demanding task in medical image analysis. In practice, it is difficult to find annotated medical images with corresponding segmentation masks. In this paper, we present Kvasir-SEG: an open-access dataset of gastrointestinal polyp images and corresponding segmentation masks, manually annotated and verified by an experienced gastroenterologist. We demonstrate the use of our dataset with a traditional segmentation approach and a modern deep learning based CNN approach. This work will be valuable for researchers to reproduce results and compare their methods in the future. By adding segmentation masks to the Kvasir dataset, which until today only consisted of framewise annotations, we enable multimedia and computer vision researchers to contribute in the field of polyp segmentation and automatic analysis of colonoscopy videos.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | International Conference on Multimedia Modeling |
Pagination | 451-462 |
Publisher | Springer |
Place Published | Daejeon, Korea |
Keywords | Kvasir-SEG dataset, Medical images, Polyp segmentation, ResUNet Fuzzy c-mean clustering, Semantic segmentation |
Proceedings, refereed
ACM Multimedia BioMedia 2019 Grand Challenge Overview
In The ACM International Conference on Multimedia (ACM MM). New York, New York, USA: ACM Press, 2019.Status: Published
ACM Multimedia BioMedia 2019 Grand Challenge Overview
The BioMedia 2019 ACM Multimedia Grand Challenge is the first in a series of competitions focusing on the use of multimedia for different medical use-cases. In this year’s challenge, the participants are asked to develop efficient algorithms which automatically detect a variety of findings commonly identified in the gastrointestinal (GI) tract (a part of the human digestive system). The purpose of this task is to develop methods to aid medical doctors performing routine endoscopy inspections of the GI tract. In this paper, we give a detailed description of the four different tasks of this year’s challenge, present the datasets used for training and testing, and discuss how each submission is evaluated both qualitatively and quantitatively.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | The ACM International Conference on Multimedia (ACM MM) |
Pagination | 2563-2567 |
Date Published | 10/2019 |
Publisher | ACM Press |
Place Published | New York, New York, USA |
ISBN Number | 9781450368896 |
URL | http://dl.acm.org/citation.cfm?doid=3343031http://dl.acm.org/citation.cf... |
DOI | 10.1145/334303110.1145/3343031.3356058 |
ResUNet++: An Advanced Architecture for Medical Image Segmentation
In 2019 IEEE International Symposium on Multimedia (ISM). San Diego, California, USA: IEEE, 2019.Status: Published
ResUNet++: An Advanced Architecture for Medical Image Segmentation
Accurate computer-aided polyp detection and segmentation during colonoscopy examinations can help endoscopists resect abnormal tissue and thereby decrease chances of polyps growing into cancer. Towards developing a fully automated model for pixel-wise polyp segmentation, we propose ResUNet++, which is an improved ResUNet architecture for colonoscopic image segmentation. Our experimental evaluations show that the suggested architecture produces good segmentation results on publicly available datasets. Furthermore, ResUNet++ significantly outperforms U-Net and ResUNet, two key state-of-the-art deep learning architectures, by achieving high evaluation scores with a dice coefficient of 81.33%, and a mean Intersection over Union (mIoU) of 79.27% for the Kvasir-SEG dataset and a dice coefficient of 79.55%, and a mIoU of 79.62% with CVC-612 dataset.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 IEEE International Symposium on Multimedia (ISM) |
Publisher | IEEE |
Place Published | San Diego, California, USA |
Keywords | colonoscopy, deep learning, health informatics, Medical image segmentation, Polyp segmentation, Semantic segmentation |
Journal Article
Deep Learning for Automatic Generation of Endoscopy Reports
Gastrointestinal Endoscopy 89, no. 6 (2019).Status: Published
Deep Learning for Automatic Generation of Endoscopy Reports
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Gastrointestinal Endoscopy |
Volume | 89 |
Issue | 6 |
Date Published | 06/2019 |
Publisher | Elsevier |
Place Published | Gastrointestinal Endoscopy |
DOI | 10.1016/j.gie.2019.04.053 |
Maskinlæringssystemer for gastrointestinale endoskopier
Best Practice Nordic - Gastroenterologi (2019).Status: Published
Maskinlæringssystemer for gastrointestinale endoskopier
Assistert diagnostikk med hjelp av kunstig intelligens (KI) har vært etterspurt lenge og kan bli et viktig hjelpemiddel innen medisin, godt hjulpet av den raske utviklingen innen maskinvare. Denne har gjort innføringen av slike hjelpemidler mulig på relativt kort sikt. Sikrere påvisning og klassifisering av funn og lesjoner innen radiologi og endoskopi er i ferd med å bli et viktig forskningsområde innen KI, og det fokuseres spesielt på maskinlæring. Imidlertid krever vellykket utvikling et komplett system som kan brukes i sanntid i daglig praksis, og som begrenser seg til utvikling av algoritmer. Det kreves også store randomiserte studier for å fastslå om kvaliteten og påliteligheten til systemene er god. Vi deler i denne artikkelen våre erfaringer fra utviklingen av et system for gastrointestinale endoskopier og belyser viktige utfordringer for å skape en effektiv digital assistent.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Best Practice Nordic - Gastroenterologi |
Date Published | 07/2019 |
Publisher | BestPracticeNordic |
URL | https://bestprac.no/maskinlaeringssystemer-for-gastrointestinale-endosko... |
Miscellaneous
Kunstig intelligens for endoskopi – Automatisk deteksjon av lesjoner i sanntid
NGF Nytt, Vol. 26, No 1, March 2019, p. 34: Norsk Gastroenterologisk Forening, 2019.Status: Published
Kunstig intelligens for endoskopi – Automatisk deteksjon av lesjoner i sanntid
BAKGRUNN: I krysningspunktet mellom matematikk, informatikk og statistikk finner vi den vitenskapelige disiplinen kunstig intelligens (KI). Sammen med de siste års eksplosive utvikling innen teknologi har KI muliggjort nye algoritmer, modeller og systemer for maskinassistert diagnostikk. Resultater fra KI basert på dype nevrale nettverk har vist spesielt stort potensiale, også for automatisk deteksjon av lesjoner og anatomiske landemerker i gastrointestinaltraktus under endoskopi. Med sensitivitet og spesifisitet for deteksjon av polypper i tykktarm
på over 90% møter slike metoder nødvendige kliniske krav, men mange eksperimenter er utført på begrensede datasett, eller analysert på feilaktig grunnlag grunnet manglende tilgang og forståelse hos informatikere. For å oppnå best mulig resultat er
et interdisiplinært samarbeid mellom klinikere og informatikere
en forutsetning. Informatikerne trenger medisinske innspill for å lage effektive systemer som fungerer ute i klinikken, og klinikerne trenger forståelse av systemet for å kunne stole på resultatet og stille pålitelige diagnoser. En stor utfordring for denne tilliten er
at fremgangsmåten til en KI-algoritme sees på som en svart boks hvor ingen nøyaktig kan dechiffrere hvordan systemet kom frem
til sin konklusjon.
METODE: Vi har gjennom mange år samlet en stor bilde- database fra endoskopier utført ved Bærum Sykehus, Vestre Viken HF. Bildene er gjennomgått og annotert av tre erfarne endoskopører og fordelt på 16 klasser, inkludert normal Z-linje, øsofagitt, normal cøkum, polypper og ulcerøs colitt. Deretter er bildene brukt til å utvikle, trene og teste KI-modeller. Modellene er basert på maskinlæring og dyp læring, en gren innen KI. Med vårt system Mimir, som kombinerer KI med informasjonssøk og
-gjenfinning, søker vi å lage et helhetlig beslutningsstøttesystem for endoskopører. Algoritmene analyserer videoer i sanntid, finner lesjoner, klassifiserer disse og gir skopøren live feedback om funn under undersøkelsen, slik at funnene kan undersøkes nærmere. Mimir presenterer deretter resultatene i egen programvare, og bruker blant annet “heatmaps” til å forklare hvordan konklusjonen er nådd, og er på den måten et bidrag på veien til å forstå hvordan KI-algoritmene fungerer. Videre jobber vi med å videreutvikle Mimirs støtte for automatisk rapportgenerering, med bilder
og standardtekst basert på funn fra undersøkelsen.
RESULTATER: Deteksjon og klassifisering for de 16 gruppene har vist en sensitivitet på 0,939 og en spesifisitet på 0,996. Algoritmene våre klarer å prosessere bildene i hastigheter på mellom 30 - 1000 bilder per sekund, raskt nok til å kjøre deteksjon i sanntid. En prototype av systemet er i samråd med klinikere testet ved å koble til et koloskopisystem ute i klinikken, og kan
nå analysere videoer i sanntid.
KONKLUSJON: Tester av våre system viser at KI kan bli et viktig hjelpemiddel for å bedre oppdage GI-forandringer, og generere automatiske rapporter i løpet av nærmeste fremtid. Dette kan fungere som viktig beslutningsstøtte for endoskopører, og kan brukes i opplæring av nye endoskopører. Den største begrensningen med KI er at vi per i dag ikke vet hvordan systemet kommer frem til sin konklusjon, som kan påvirke i hvor stor grad vi stoler på resultatet. Vi arbeider derfor med et helhetlig system som ikke bare hjelper legen med diagnostikk, men også forklarer hvordan konklusjonen er nådd, samt å generere automatiske rapporter fra undersøkelsen.
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2019 |
Publisher | Norsk Gastroenterologisk Forening |
Place Published | NGF Nytt, Vol. 26, No 1, March 2019, p. 34 |
Proceedings, refereed
Deep Learning Based Disease Detection Using Domain Specific Transfer Learning
In MediaEval 2018. MediaEval, 2018.Status: Published
Deep Learning Based Disease Detection Using Domain Specific Transfer Learning
In this paper, we present our approach for the Medico Multimedia Task as part of the MediaEval 2018 Benchmark. Our method is based on convolutional neural networks (CNNs), where we compare how fine-tuning, in the context of transfer learning, from different source domains (general versus medical domain) affect classification performance. The preliminary results show that fine-tuning models trained on large and diverse datasets is favorable, even when the model’s source domain has little to no resemblance to the new target.
Afilliation | Machine Learning |
Project(s) | Department of Data Science and Knowledge Discovery |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | MediaEval 2018 |
Publisher | MediaEval |
Keywords | convolutional neural networks, deep learning, Gastrointestinal Disease Detection |
Tradeoffs using Binary and Multiclass Neural Network Classification for Medical Multidisease Detection
In 2018 IEEE International Symposium on Multimedia (ISM). IEEE, 2018.Status: Published
Tradeoffs using Binary and Multiclass Neural Network Classification for Medical Multidisease Detection
The interest in neural networks has increased sig- nificantly, and the application of this type of machine learning is vast, ranging from natural image classification to medical image segmentation. However, many users of neural networks tend to use them as a black box tool. They do not access all of the possible variations, nor take into account the respective classification accuracies and costs. In our work, we focus on multiclass image classification, and in this research, we shed light on the trade-offs between systems using a single multiclass classification and multiple binary classifiers, respectively. We have tested the these classifiers on several modern neural network architectures, including DenseNet, Inception v3, Inception ResNet v2, Xception, NASNet and MobileNet. We have compared several aspects of the performance of these architectures during training and testing using both classification styles. We have compared classification speed and several classification accuracy metrics. Here, we present the results from experiments on a total of 99 networks: 11 multiclass and 88 individual binary networks, for an 8-class classification of medical images. In short, using multiple binary classification networks resulted in a 7% increase in the average F1 score, a 1% increase in average accuracy, a 1% increase in precision, and a 4% increase in average recall. However, on average, such a multi-network style performed the classification 7.6 times slower compared to a single network multiclass implementation. These collective findings show that both approaches can be applied to modern neural network structures. Several binary networks will often give increased classification accuracy, but at the cost of classification speed and resource consumption.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | 2018 IEEE International Symposium on Multimedia (ISM) |
Pagination | 1-8 |
Date Published | 12/2018 |
Publisher | IEEE |
DOI | 10.1109/ISM.2018.00009 |