Projects
Interview training of child-welfare and law-enforcement professionals interviewing maltreated children supported via artificial avatars

The interdisciplinary FRIPRO project aims to improve interviews with maltreated children through a training program using realistic and interactive child avatars.
The department of Holistic Systems (HOST) at SimulaMet will be working with the Faculty of Social Sciences at OsloMet. The project will begin on the 1st of April 2021 and end on the 31st of March in 2024. It is funded by The Research Council of Norway with 12 million NOK and will include three Ph.D. positions.
Maltreatment and abuse of children is a significant societal problem that has serious and damaging effects on children’s behavior, psychological development, and adjustment. Detection and prevention of violence and sexual abuse against children is, therefore, a high priority for Child Protective Services (CPS) and law-enforcement professionals. The conversations and investigative interviews that are conducted with these children must be of high quality. However, both Norwegian and international research shows that despite investments in methodology, the current interview and conversation skills still need to be improved.
By using an empirically informed training system in highly realistic child avatars, this project aims to develop and maintain the advanced skills needed for interviewing maltreated children. They will use data from past investigative interviews with maltreated children and create a real-looking avatar that is capable of expressing emotion and spontaneous responses.
The planned avatar will be a combination of technologies from multiple areas in computer science including AI, computer vision, and natural language processing. The aim is for the child avatars to be a part of an interview-training program that will be implemented in cooperation with the CPS and the police. The training system will be evaluated by the project scientists to judge effectiveness in relation to real-world needs.
The project also involves collaborations with researchers from Griffith University in Australia and the University of Cambridge in the United Kingdom.
Publications
Miscellaneous
ACM Multimedia Grand Challenge on Detecting Cheapfakes
ACM Multimedia Conference (MM): ACM, 2022.Status: Published
ACM Multimedia Grand Challenge on Detecting Cheapfakes
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2022 |
Publisher | ACM |
Place Published | ACM Multimedia Conference (MM) |
Notes | https://2022.acmmm.org/call-for-grand-challenge-submissions/ |
URL | https://detecting-cheapfakes.github.io |
Common Limitations of Image Processing Metrics: A Picture Story
arXiv, 2022.Status: Published
Common Limitations of Image Processing Metrics: A Picture Story
While the importance of automatic image analysis is continuously increasing, recent meta-research revealed major flaws with respect to algorithm validation. Performance metrics are particularly key for meaningful, objective, and transparent performance assessment and validation of the used automatic algorithms, but relatively little attention has been given to the practical pitfalls when using specific metrics for a given image analysis task. These are typically related to (1) the disregard of inherent metric properties, such as the behaviour in the presence of class imbalance or small target structures, (2) the disregard of inherent data set properties, such as the non-independence of the test cases, and (3) the disregard of the actual biomedical domain interest that the metrics should reflect. This living dynamically document has the purpose to illustrate important limitations of performance metrics commonly applied in the field of image analysis. In this context, it focuses on biomedical image analysis problems that can be phrased as image-level classification, semantic segmentation, instance segmentation, or object detection task. The current version is based on a Delphi process on metrics conducted by an international consortium of image analysis experts from more than 60 institutions worldwide.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2022 |
Publisher | arXiv |
Keywords | Computer Vision and Pattern Recognition (cs.CV), electronic engineering, FOS: Computer and information sciences, FOS: Electrical engineering, Image and Video Processing (eess.IV), information engineering |
URL | https://arxiv.org/abs/2104.05642 |
DOI | 10.48550/ARXIV.2104.05642 |
Journal Article
Artificial Intelligence for Colonoscopy: Past, Present, and Future
IEEE Journal of Biomedical and Health Informatics (2022): 1.Status: Published
Artificial Intelligence for Colonoscopy: Past, Present, and Future
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | IEEE Journal of Biomedical and Health Informatics |
Pagination | 1 - 1 |
Date Published | Jan-01-2022 |
Publisher | IEEE |
ISSN | 2168-2194 |
URL | https://ieeexplore.ieee.org/document/9739863/http://xplorestaging.ieee.o... |
DOI | 10.1109/JBHI.2022.3160098 |
Artificial intelligence in dry eye disease
The Ocular Surface 23 (2022): 74-86.Status: Published
Artificial intelligence in dry eye disease
Dry eye disease (DED) has a prevalence of between 5 and 50%, depending on the diagnostic criteria used and population under study. However, it remains one of the most underdiagnosed and undertreated conditions in ophthalmology. Many tests used in the diagnosis of DED rely on an experienced observer for image interpretation, which may be considered subjective and result in variation in diagnosis. Since artificial intelligence (AI) systems are capable of advanced problem solving, use of such techniques could lead to more objective diagnosis. Although the term ‘AI’ is commonly used, recent success in its applications to medicine is mainly due to advancements in the sub-field of machine learning, which has been used to automatically classify images and predict medical outcomes. Powerful machine learning techniques have been harnessed to understand nuances in patient data and medical images, aiming for consistent diagnosis and stratification of disease severity. This is the first literature review on the use of AI in DED. We provide a brief introduction to AI, report its current use in DED research and its potential for application in the clinic. Our review found that AI has been employed in a wide range of DED clinical tests and research applications, primarily for interpretation of interferometry, slit-lamp and meibography images. While initial results are promising, much work is still needed on model development, clinical testing and standardisation.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | The Ocular Surface |
Volume | 23 |
Pagination | 74 - 86 |
Date Published | Jan-01-2022 |
Publisher | Elsevier |
ISSN | 15420124 |
Keywords | artificial intelligence, Dry eye disease, Machine learning |
URL | https://linkinghub.elsevier.com/retrieve/pii/S1542012421001324 |
DOI | 10.1016/j.jtos.2021.11.004 |
Complexity and Variability Analyses of Motor Activity Distinguish Mood States in Bipolar Disorder
PLOS ONE (2022).Status: Accepted
Complexity and Variability Analyses of Motor Activity Distinguish Mood States in Bipolar Disorder
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | PLOS ONE |
Publisher | PLOS ONE |
Deep Tower Networks for Efficient Temperature Forecasting from Multiple Data Sources
Sensors 2232, no. 7 (2022): 2802.Status: Published
Deep Tower Networks for Efficient Temperature Forecasting from Multiple Data Sources
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Sensors |
Volume | 2232 |
Issue | 7 |
Pagination | 2802 |
Date Published | 01-04-2022 |
Publisher | MPDI |
URL | https://www.mdpi.com/1424-8220/22/7/2802https://www.mdpi.com/1424-8220/2... |
DOI | 10.3390/s22072802 |
FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation
IEEE Transactions on Neural Networks and Learning Systems (2022): 1-14.Status: Accepted
FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation
The increase of available large clinical and experimental datasets has contributed to a substantial amount of important contributions in the area of biomedical image analysis. Image segmentation, which is crucial for any quantitative analysis, has especially attracted attention. Recent hardware advancement has led to the success of deep learning approaches. However, although deep learning models are being trained on large datasets, existing methods do not use the information from different learning epochs effectively. In this work, we leverage the information of each training epoch to prune the prediction maps of the subsequent epochs. We propose a novel architecture called feedback attention network (FANet) that unifies the previous epoch mask with the feature map of the current training epoch. The previous epoch mask is then used to provide hard attention to the learned feature maps at different convolutional layers. The network also allows rectifying the predictions in an iterative fashion during the test time. We show that our proposed feedback attention model provides a substantial improvement on most segmentation metrics tested on seven publicly available biomedical imaging datasets demonstrating the effectiveness of FANet. The source code is available at https://github.com/nikhilroxtomar/FANet.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | IEEE Transactions on Neural Networks and Learning Systems |
Pagination | 1 - 14 |
Date Published | Jan-01-2022 |
Publisher | IEEE |
ISSN | 2162-237X |
URL | https://ieeexplore.ieee.org/document/9741842 |
DOI | 10.1109/TNNLS.2022.3159394 |
Meta-learning with implicit gradients in a few-shot setting for medical image segmentation
Computers in Biology and Medicine 14312136320119704317507593739403621582 (2022).Status: Published
Meta-learning with implicit gradients in a few-shot setting for medical image segmentation
Widely used traditional supervised deep learning methods require a large number of training samples but often fail to generalize on unseen datasets. Therefore, a more general application of any trained model is quite limited for medical imaging for clinical practice. Using separately trained models for each unique lesion category or a unique patient population will require sufficiently large curated datasets, which is not practical to use in a real-world clinical set-up. Few-shot learning approaches can not only minimize the need for an enormous number of reliable ground truth labels that are labour-intensive and expensive, but can also be used to model on a dataset coming from a new population. To this end, we propose to exploit an optimization-based implicit model agnostic meta-learning (iMAML) algorithm under few-shot settings for medical image segmentation. Our approach can leverage the learned weights from diverse but small training samples to perform analysis on unseen datasets with high accuracy. We show that, unlike classical few-shot learning approaches, our method improves generalization capability. To our knowledge, this is the first work that exploits iMAML for medical image segmentation and explores the strength of the model on scenarios such as meta-training on unique and mixed instances of lesion datasets. Our quantitative results on publicly available skin and polyp datasets show that the proposed method outperforms the naive supervised baseline model and two recent few-shot segmentation approaches by large margins. In addition, our iMAML approach shows an improvement of 2%–4% in dice score compared to its counterpart MAML for most experiments.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Computers in Biology and Medicine |
Volume | 14312136320119704317507593739403621582 |
Date Published | Jan-04-2022 |
Publisher | Elsevier |
ISSN | 00104825 |
URL | https://linkinghub.elsevier.com/retrieve/pii/S0010482522000191 |
DOI | 10.1016/j.compbiomed.2022.105227 |
MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation
IEEE Journal of Biomedical and Health Informatics (2022).Status: Published
MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation
Methods based on convolutional neural networks have improved the performance of biomedical image segmentation. However, most of these methods cannot efficiently segment objects of variable sizes and train on small and biased datasets, which are common for biomedical use cases. While methods exist that incorporate multi-scale fusion approaches to address the challenges arising with variable sizes, they usually use complex models that are more suitable for general semantic segmentation problems. In this paper, we propose a novel architecture called Multi-Scale Residual Fusion Network (MSRF-Net), which is specially designed for medical image segmentation. The proposed MSRF-Net is able to exchange multi-scale features of varying receptive fields using a Dual-Scale Dense Fusion (DSDF) block. Our DSDF block can exchange information rigorously across two different resolution scales, and our MSRF sub-network uses multiple DSDF blocks in sequence to perform multi-scale fusion. This allows the preservation of resolution, improved information flow and propagation of both high- and low-level features to obtain accurate segmentation maps. The proposed MSRF-Net allows to capture object variabilities and provides improved results on different biomedical datasets. Extensive experiments on MSRF-Net demonstrate that the proposed method outperforms the cutting edge medical image segmentation methods on four publicly available datasets. We achieve the Dice Coefficient (DSC) of 0.9217, 0.9420, and 0.9224, 0.8824 on Kvasir-SEG, CVC-ClinicDB, 2018 Data Science Bowl dataset, and ISIC-2018 skin lesion segmentation challenge dataset respectively. We further conducted generalizability tests that also achieved the highest DSC score with 0.7921 and 0.7575 on CVC-ClinicDB and Kvasir-SEG, respectively.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | IEEE Journal of Biomedical and Health Informatics |
Date Published | 12/2021 |
Publisher | IEEE |
ISSN | 2168-2194 |
URL | https://ieeexplore.ieee.org/document/9662196 |
DOI | 10.1109/JBHI.2021.3138024 |
On evaluation metrics for medical applications of artificial intelligenceAbstract
Nature Scientific Reports 1236825221520218325484051210158697682437, no. 1 (2022).Status: Published
On evaluation metrics for medical applications of artificial intelligenceAbstract
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Nature Scientific Reports |
Volume | 1236825221520218325484051210158697682437 |
Issue | 1 |
Date Published | Jan-12-2022 |
Publisher | Nature |
URL | https://www.nature.com/articles/s41598-022-09954-8https://www.nature.com... |
DOI | 10.1038/s41598-022-09954-8 |
Proceedings, refereed
Automatic Thumbnail Selection for Soccer Videos using Machine Learning
In ACM Multimedia Systems (MMSys) Conference. The ACM Multimedia Systems Conference (MMSys): ACM, 2022.Status: Accepted
Automatic Thumbnail Selection for Soccer Videos using Machine Learning
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | ACM Multimedia Systems (MMSys) Conference |
Publisher | ACM |
Place Published | The ACM Multimedia Systems Conference (MMSys) |
DOI | 10.1145/3524273.3528182 |
HOST-ATS: Automatic Thumbnail Selection with Dashboard-Controlled ML Pipeline and Dynamic User Survey
In ACM Multimedia Systems (MMSys) Conference. The ACM Multimedia Systems Conference (MMSys): ACM, 2022.Status: Accepted
HOST-ATS: Automatic Thumbnail Selection with Dashboard-Controlled ML Pipeline and Dynamic User Survey
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | ACM Multimedia Systems (MMSys) Conference |
Publisher | ACM |
Place Published | The ACM Multimedia Systems Conference (MMSys) |
DOI | 10.1145/3524273.3532908 |
Huldra: A Framework for Collecting Crowdsourced Feedback on Multimedia Assets
In ACM Multimedia Systems (MMSys) Conference. The ACM Multimedia Systems Conference (MMSys): ACM, 2022.Status: Accepted
Huldra: A Framework for Collecting Crowdsourced Feedback on Multimedia Assets
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | ACM Multimedia Systems (MMSys) Conference |
Publisher | ACM |
Place Published | The ACM Multimedia Systems Conference (MMSys) |
DOI | 10.1145/3524273.3532887 |
Investigative Interviews using a Multimodal Virtual Avatar
In American Psychology-Law Society Conference 2022. Denver USA,: American Psychology-Law Society, 2022.Status: Accepted
Investigative Interviews using a Multimodal Virtual Avatar
To meet best-practice standards, we are developing an interactive virtual avatar aiming as a training tool to raise interviewing skills of child-welfare and law-enforcement professionals. Therefore, we present the “Ilma” avatar that recognizes interviewers’ behavior during open-ended, closed and leading questions, and which can automatically respond to the conversation. We conducted a user study in which master students (N=3) and child protective workers (N=8) interviewed “Ilma” and rated their perception of the interaction. The results show that the participants valued the interaction and found the avatar useful. Thus, it has great potential to be an effective training tool.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | American Psychology-Law Society Conference 2022 |
Publisher | American Psychology-Law Society |
Place Published | Denver USA, |
Research proposal: Explainability methods for machine learning systems for multimodal medical datasets
In ACM Multimedia Systems (MMSys) Conference. The ACM Multimedia Systems Conference (MMSys): ACM, 2022.Status: Accepted
Research proposal: Explainability methods for machine learning systems for multimodal medical datasets
This paper contains the research proposal of Andrea M. Storås that was presented at the MMSys 2022 doctoral symposium. Machine learning models have the ability to solve medical tasks with a high level of performance, e.g., classifying medical videos and detecting anomalies using different sources of data. However, many of these models are highly complex and difficult to understand. Lack of interpretability can limit the use of machine learning systems in the medical domain. Explainable artificial intelligence provides explanations regarding the models and their predictions. In this PhD project, we develop machine learning models for automatic analysis of medical data and explain the results using established techniques from the field of explainable artificial intelligence. Current research indicate that there are still open issues to be solved in order for end users to understand multimedia systems powered by machine learning. Consequently, new explanation techniques will also be developed. Different types of medical data are applied in order to investigate the generalizability of the methods.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | ACM Multimedia Systems (MMSys) Conference |
Publisher | ACM |
Place Published | The ACM Multimedia Systems Conference (MMSys) |
ISBN Number | 978-1-4503-9283-9/22/06 |
DOI | 10.1145/3524273.3533925 |
Journal Article
A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging
Medical Image Analysis 70 (2021): 102007.Status: Published
A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging
Gastrointestinal endoscopy has been an active field of research motivated by the large number of highly lethal GI cancers. Early GI cancer precursors are often missed during the endoscopic surveillance. The high missed rate of such abnormalities during endoscopy is thus a critical bottleneck. Lack of attentiveness due to tiring procedures, and requirement of training are few contributing factors. An automatic GI disease classification system can help reduce such risks by flagging suspicious frames and lesions. GI endoscopy consists of several multi-organ surveillance, therefore, there is need to develop methods that can generalize to various endoscopic findings. In this realm, we present a comprehensive analysis of the Medico GI challenges: Medical Multimedia Task at MediaEval 2017, Medico Multimedia Task at MediaEval 2018, and BioMedia ACM MM Grand Challenge 2019. These challenges are initiative to set-up a benchmark for different computer vision methods applied to the multi-class endoscopic images and promote to build new approaches that could reliably be used in clinics. We report the performance of 21 participating teams over a period of three consecutive years and provide a detailed analysis of the methods used by the participants, highlighting the challenges and shortcomings of the current approaches and dissect their credibility for the use in clinical settings. Our analysis revealed that the participants achieved an improvement on maximum Mathew correlation coefficient (MCC) from 82.68% in 2017 to 93.98% in 2018 and 95.20% in 2019 challenges, and a significant increase in computational speed over consecutive years.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Medical Image Analysis |
Volume | 70 |
Pagination | 102007 |
Publisher | Elsevier |
Keywords | Artificial intelligence, BioMedia 2019 Grand Challenge, Computer-aided detection and diagnosis, Gastrointestinal endoscopy challenges, Medical imaging, Medico Task 2017, Medico Task 2018 |
DOI | 10.1016/j.media.2021.102007 |
A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation
IEEE Journal of Biomedical and Health Informatics 25, no. 6 (2021): 2029-2040.Status: Published
A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation
Colonoscopy is considered the gold standard for detection of colorectal cancer and its precursors. Existing examination methods are, however, hampered by high overall miss-rate, and many abnormalities are left undetected. Computer-Aided Diagnosis systems based on advanced machine learning algorithms are touted as a game-changer that can identify regions in the colon overlooked by the physicians during endoscopic examinations, and help detect and characterize lesions. In previous work, we have proposed the ResUNet++ architecture and demonstrated that it produces more efficient results compared with its counterparts U-Net and ResUNet. In this paper, we demonstrate that further improvements to the overall prediction performance of the ResUNet++ architecture can be achieved by using Conditional Random Field (CRF) and Test-Time Augmentation (TTA). We have performed extensive evaluations and validated the improvements using six publicly available datasets: Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS-Larib PolypDB, ASU-Mayo Clinic Colonoscopy Video Database, and CVC-VideoClinicDB. Moreover, we compare our proposed architecture and resulting model with other State-of-the-art methods. To explore the generalization capability of ResUNet++ on different publicly available polyp datasets, so that it could be used in a real-world setting, we performed an extensive cross-dataset evaluation. The experimental results show that applying CRF andTTA improves the performance on various polyp segmentation datasets both on the same dataset and cross-dataset. To check the model’s performance on difficult to detect polyps, we selected, with the help of an expert gastroenterologist,196sessile or flat polyps that are less than ten millimeters in size. This additional data has been made available as a subset of Kvasir-SEG. Our approaches showed good results for flat or sessile and smaller polyps, which are known to be one of the major reasons for high polyp miss-rates. This is one of the significant strengths of our work and indicates that our methods should be investigated further for use in clinical practice.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | IEEE Journal of Biomedical and Health Informatics |
Volume | 25 |
Issue | 6 |
Pagination | 2029 - 2040 |
Publisher | IEEE |
Keywords | colonoscopy, conditional random field, generalization, Polyp segmentation, ResUNet++, test-time augmentation |
DOI | 10.1109/JBHI.2021.3049304 |
AI-Based Video Clipping of Soccer Events
Machine Learning and Knowledge Extraction 3, no. 4 (2021): 990-1008.Status: Published
AI-Based Video Clipping of Soccer Events
The current gold standard for extracting highlight clips from soccer games is the use of manual annotations and clippings, where human operators define the start and end of an event and trim away the unwanted scenes. This is a tedious, time-consuming, and expensive task, to the extent of being rendered infeasible for use in lower league games. In this paper, we aim to automate the process of highlight generation using logo transition detection, scene boundary detection, and optional scene removal. We experiment with various approaches, using different neural network architectures on different datasets, and present two models that automatically find the appropriate time interval for extracting goal events. These models are evaluated both quantitatively and qualitatively, and the results show that we can detect logo and scene transitions with high accuracy and generate highlight clips that are highly acceptable for viewers. We conclude that there is considerable potential in automating the overall soccer video clipping process.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Machine Learning and Knowledge Extraction |
Volume | 3 |
Issue | 4 |
Pagination | 990 - 1008 |
Date Published | 12/2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2504-4990/3/4/49/pdf |
DOI | 10.3390/make3040049 |
Artificial intelligence in the fertility clinic: status, pitfalls and possibilities
Human Reproduction 36, no. 9 (2021): 2429-2442.Status: Published
Artificial intelligence in the fertility clinic: status, pitfalls and possibilities
In recent years, the amount of data produced in the field of ART has increased exponentially. The diversity of data is large, ranging from videos to tabular data. At the same time, artificial intelligence (AI) is progressively used in medical practice and may become a promising tool to improve success rates with ART. AI models may compensate for the lack of objectivity in several critical procedures in fertility clinics, especially embryo and sperm assessments. Various models have been developed, and even though several of them show promising performance, there are still many challenges to overcome. In this review, we present recent research on AI in the context of ART. We discuss the strengths and weaknesses of the presented methods, especially regarding clinical relevance. We also address the pitfalls hampering successful use of AI in the clinic and discuss future possibilities and important aspects to make AI truly useful for ART.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Human Reproduction |
Volume | 36 |
Issue | 9 |
Pagination | 2429 - 2442 |
Date Published | 07/2021 |
Publisher | Oxford Academic |
ISSN | 0268-1161 |
URL | https://academic.oup.com/humrep/article/36/9/2429/6330662 |
DOI | 10.1093/humrep/deab168 |
Automated Event Detection and Classification in Soccer: The Potential of Using Multiple Modalities
Machine Learning and Knowledge Extraction 3, no. 4 (2021): 1030-1054.Status: Published
Automated Event Detection and Classification in Soccer: The Potential of Using Multiple Modalities
Detecting events in videos is a complex task, and many different approaches, aimed at a large variety of use-cases, have been proposed in the literature. Most approaches, however, are unimodal and only consider the visual information in the videos. This paper presents and evaluates different approaches based on neural networks where we combine visual features with audio features to detect (spot) and classify events in soccer videos. We employ model fusion to combine different modalities such as video and audio, and test these combinations against different state-of-the-art models on the SoccerNet dataset. The results show that a multimodal approach is beneficial. We also analyze how the tolerance for delays in classification and spotting time, and the tolerance for prediction accuracy, influence the results. Our experiments show that using multiple modalities improves event detection performance for certain types of events.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Machine Learning and Knowledge Extraction |
Volume | 3 |
Issue | 4 |
Pagination | 1030 - 1054 |
Date Published | 12/2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2504-4990/3/4/51/pdf |
DOI | 10.3390/make3040051 |
Deep learning neural network can measure ECG intervals and amplitudes accurately
Journal of Electrocardiology 69 (2021): 82.Status: Published
Deep learning neural network can measure ECG intervals and amplitudes accurately
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Journal of Electrocardiology |
Volume | 69 |
Pagination | 82 |
Publisher | Elsevier |
DeepFake electrocardiograms using generative adversarial networks are the beginning of the end for privacy issues in medicine
Nature Scientific Reports 11 (2021): 21896.Status: Published
DeepFake electrocardiograms using generative adversarial networks are the beginning of the end for privacy issues in medicine
Recent global developments underscore the prominent role big data have in modern medical science. But privacy issues constitute a prevalent problem for collecting and sharing data between researchers. However, synthetic data generated to represent real data carrying similar information and distribution may alleviate the privacy issue. In this study, we present generative adversarial networks (GANs) capable of generating realistic synthetic DeepFake 10-s 12-lead electrocardiograms (ECGs). We have developed and compared two methods, named WaveGAN* and Pulse2Pulse. We trained the GANs with 7,233 real normal ECGs to produce 121,977 DeepFake normal ECGs. By verifying the ECGs using a commercial ECG interpretation program (MUSE 12SL, GE Healthcare), we demonstrate that the Pulse2Pulse GAN was superior to the WaveGAN* to produce realistic ECGs. ECG intervals and amplitudes were similar between the DeepFake and real ECGs. Although these synthetic ECGs mimic the dataset used for creation, the ECGs are not linked to any individuals and may thus be used freely. The synthetic dataset will be available as open access for researchers at OSF.io and the DeepFake generator available at the Python Package Index (PyPI) for generating synthetic ECGs. In conclusion, we were able to generate realistic synthetic ECGs using generative adversarial neural networks on normal ECGs from two population studies, thereby addressing the relevant privacy issues in medical datasets.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Nature Scientific Reports |
Volume | 11 |
Pagination | 21896 |
Date Published | 09/2021 |
Publisher | Springer nature |
URL | https://www.nature.com/articles/s41598-021-01295-2 |
DOI | 10.1038/s41598-021-01295-2 |
Explaining deep neural networks for knowledge discovery in electrocardiogram analysis
Scientific Reports 11 (2021): 10949.Status: Published
Explaining deep neural networks for knowledge discovery in electrocardiogram analysis
Deep learning-based tools may annotate and interpret medical data more quickly, consistently, and accurately than medical doctors. However, as medical doctors are ultimately responsible for clinical decision-making, any deep learning-based prediction should be accompanied by an explanation that a human can understand. We present an approach called electrocardiogram gradient class activation map (ECGradCAM), which is used to generate attention maps and explain the reasoning behind deep learning-based decision-making in ECG analysis. Attention maps may be used in the clinic to aid diagnosis, discover new medical knowledge, and identify novel features and characteristics of medical tests. In this paper, we showcase how ECGradCAM attention maps can unmask how a novel deep learning model measures both amplitudes and intervals in 12-lead electrocardiograms, and we show an example of how attention maps may be used to develop novel ECG features.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Scientific Reports |
Volume | 11 |
Pagination | 10949 |
Date Published | 05/2021 |
Publisher | Springer Nature |
URL | http://www.nature.com/articles/s41598-021-90285-5h |
DOI | 10.1038/s41598-021-90285-5 |
File System Support for Privacy-Preserving Analysis and Forensics in Low-Bandwidth Edge Environments
Information 12, no. 10 (2021): 430.Status: Published
File System Support for Privacy-Preserving Analysis and Forensics in Low-Bandwidth Edge Environments
In this paper, we present initial results from our distributed edge systems research in the domain of sustainable harvesting of common good resources in the Arctic Ocean. Specifically, we are developing a digital platform for real-time privacy-preserving sustainability management in the domain of commercial fishery surveillance operations. This is in response to potentially privacy-infringing mandates from some governments to combat overfishing and other sustainability challenges. Our approach is to deploy sensory devices and distributed artificial intelligence algorithms on mobile, offshore fishing vessels and at mainland central control centers. To facilitate this, we need a novel data plane supporting efficient, available, secure, tamper-proof, and compliant data management in this weakly connected offshore environment. We have built our first prototype of Dorvu, a novel distributed file system in this context. Our devised architecture, the design trade-offs among conflicting properties, and our initial experiences are further detailed in this paper.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Information |
Volume | 12 |
Issue | 10 |
Pagination | 430 |
Date Published | 10/2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2078-2489/12/10/430 |
DOI | 10.3390/info12100430 |
Impact of Image Resolution on Deep Learning Performance in Endoscopy Image Classification: An Experimental Study Using a Large Dataset of Endoscopic Images
Diagnostics 11, no. 12 (2021).Status: Published
Impact of Image Resolution on Deep Learning Performance in Endoscopy Image Classification: An Experimental Study Using a Large Dataset of Endoscopic Images
Recent trials have evaluated the efficacy of deep convolutional neural network (CNN)-based AI systems to improve lesion detection and characterization in endoscopy. Impressive results are achieved, but many medical studies use a very small image resolution to save computing resources at the cost of losing details. Today, no conventions between resolution and performance exist, and monitoring the performance of various CNN architectures as a function of image resolution provides insights into how subtleties of different lesions on endoscopy affect performance. This can help set standards for image or video characteristics for future CNN-based models in gastrointestinal (GI) endoscopy. This study examines the performance of CNNs on the HyperKvasir dataset, consisting of 10,662 images from 23 different findings. We evaluate two CNN models for endoscopic image classification under quality distortions with image resolutions ranging from 32 × 32 to 512 × 512 pixels. The performance is evaluated using two-fold cross-validation and F1-score, maximum Matthews correlation coefficient (MCC), precision, and sensitivity as metrics. Increased performance was observed with higher image resolution for all findings in the dataset. MCC was achieved at image resolutions between 512 × 512 pixels for classification for the entire dataset after including all subclasses. The highest performance was observed with an MCC value of 0.9002 when the models were trained on the highest resolution and tested on the same resolution. Different resolutions and their effect on CNNs are explored. We show that image resolution has a clear influence on the performance which calls for standards in the field in the future.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Diagnostics |
Volume | 11 |
Issue | 12 |
Date Published | 09/2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2075-4418/11/12/2183 |
DOI | 10.3390/diagnostics11122183 |
Kvasir-Capsule, a video capsule endoscopy dataset
Scientific Data 8, no. 1 (2021): 142.Status: Published
Kvasir-Capsule, a video capsule endoscopy dataset
Artificial intelligence (AI) is predicted to have profound effects on the future of video capsule endoscopy (VCE) technology. The potential lies in improving anomaly detection while reducing manual labour. Existing work demonstrates the promising benefits of AI-based computer-assisted diagnosis systems for VCE. They also show great potential for improvements to achieve even better results. Also, medical data is often sparse and unavailable to the research community, and qualified medical personnel rarely have time for the tedious labelling work. We present Kvasir-Capsule, a large VCE dataset collected from examinations at a Norwegian Hospital. Kvasir-Capsule consists of 117 videos which can be used to extract a total of 4,741,504 image frames. We have labelled and medically verified 47,238 frames with a bounding box around findings from 14 different classes. In addition to these labelled images, there are 4,694,266 unlabelled frames included in the dataset. The Kvasir-Capsule dataset can play a valuable role in developing better algorithms in order to reach true potential of VCE technology.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Scientific Data |
Volume | 8 |
Issue | 1 |
Pagination | 142 |
Publisher | Springer Nature |
URL | http://www.nature.com/articles/s41597-021-00920-z |
DOI | 10.1038/s41597-021-00920-z |
MedAI: Transparency in Medical Image Segmentation
Nordic Machine Intelligence 1, no. 1 (2021): 1-4.Status: Published
MedAI: Transparency in Medical Image Segmentation
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Nordic Machine Intelligence |
Volume | 1 |
Issue | 1 |
Pagination | 1 - 4 |
Date Published | Jan-11-2021 |
Publisher | NMI |
Place Published | Oslo |
URL | https://journals.uio.no/NMI/article/view/9140https://journals.uio.no/NMI... |
DOI | 10.5617/nmi.9140 |
Prediction of Cloud Fractional Cover Using Machine Learning
Big Data and Cognitive Computing 5, no. 4 (2021): 62.Status: Published
Prediction of Cloud Fractional Cover Using Machine Learning
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Big Data and Cognitive Computing |
Volume | 5 |
Issue | 4 |
Pagination | 62 |
Date Published | Jan-12-2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2504-2289/5/4/62https://www.mdpi.com/2504-2289/5/4/... |
DOI | 10.3390/bdcc5040062 |
Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning
IEEE Access 9 (2021): 40496-40510.Status: Published
Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning
Computer-aided detection, localization, and segmentation methods can help improve colonoscopy procedures. Even though many methods have been built to tackle automatic detection and segmentation of polyps, benchmarking of state-of-the-art methods still remains an open problem. This is due to the increasing number of researched computer vision methods that can be applied to polyp datasets. Benchmarking of novel methods can provide a direction to the development of automated polyp detection and segmentation tasks. Furthermore, it ensures that the produced results in the community are reproducible and provide a fair comparison of developed methods. In this paper, we benchmark several recent state-of-the-art methods using Kvasir-SEG, an open-access dataset of colonoscopy images for polyp detection, localization, and segmentation evaluating both method accuracy and speed. Whilst, most methods in literature have competitive performance over accuracy, we show that the proposed ColonSegNet achieved a better trade-off between an average precision of 0.8000 and mean IoU of 0.8100, and the fastest speed of 180 frames per second for the detection and localization task. Likewise, the proposed ColonSegNetachieved a competitive dice coefficient of 0.8206 and the best average speed of 182.38 frames per second for the segmentation task. Our comprehensive comparison with various state-of-the-art methods reveals the importance of benchmarking the deep learning methods for automated real-time polyp identification and delineations that can potentially transform current clinical practices and minimize miss-detection rates.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | IEEE Access |
Volume | 9 |
Pagination | 40496-40510 |
Publisher | IEEE |
Keywords | Medical image segmentation, ColonSegNet, colonoscopy, polyps, deep learning, detection, localization, benchmarking, Kvasir-SEG |
DOI | 10.1109/ACCESS.2021.3063716 |
Towards better explainable deep learning models for embryo selection in ART
Human Reproduction 36, no. Supplement_1 (2021).Status: Published
Towards better explainable deep learning models for embryo selection in ART
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Human Reproduction |
Volume | 36 |
Issue | Supplement_1 |
Date Published | Jun-08-2021 |
Publisher | Oxford Academic |
ISSN | 0268-1161 |
URL | https://academic.oup.com/humrep/article/doi/10.1093/humrep/deab130.259 |
DOI | 10.1093/humrep/deab130.259 |
Unraveling the Impact of Land Cover Changes on Climate using Machine Learning and Explainable Artificial Intelligence
Big Data and Cognitive Computing 5 (2021): 55.Status: Published
Unraveling the Impact of Land Cover Changes on Climate using Machine Learning and Explainable Artificial Intelligence
A general issue in climate science is to handle big data and run complex and computationally heavy simulations. In this paper, we explore the potential of using machine learning (ML) to spare computational time and optimize data usage. The paper analyzes the effects of changes in land cover (LC), such as deforestation or urbanization, on local climate. Along with green house gas emission, LC changes are known to be important causes of climate change. ML methods were trained to learn the relation between LC changes and temperature changes. The results showed that Random Forest (RF) outperformed other ML methods. Explainable Artificial Intelligence (XAI) was further used to interpret the RF method and explain the impact of different LC changes on temperature. The results mainly agree with the climate science literature, demonstrating that ML methods in combination with XAI can be useful in analyzing the climate effects of LC changes. All parts of the analysis pipeline are explained including data pre-processing, feature extraction, ML training, performance evaluation and XAI.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Big Data and Cognitive Computing |
Volume | 5 |
Number | 4 |
Pagination | 55 |
Publisher | Multidisciplinary Digital Publishing Institute |
Using 3D Convolutional Neural Networks for Real-time Detection of Soccer Events
International Journal of Semantic Computing 15, no. 2 (2021): 161-187.Status: Published
Using 3D Convolutional Neural Networks for Real-time Detection of Soccer Events
Developing systems for the automatic detection of events in video is a task which has gained attention in many areas including sports. More specifically, event detection for soccer videos has been studied widely in the literature. However, there are still a number of shortcomings in the state-of-the-art such as high latency, making it challenging to operate at the live edge. In this paper, we present an algorithm to detect events in soccer videos in real time, using 3D convolutional neural networks. We test our algorithm on three different datasets from SoccerNet, the Swedish Allsvenskan, and the Norwegian Eliteserien. Overall, the results show that we can detect events with high recall, low latency, and accurate time estimation. The trade-off is a slightly lower precision compared to the current state-of-the-art, which has higher latency and performs better when a less accurate time estimation can be accepted. In addition to the presented algorithm, we perform an extensive ablation study on how the different parts of the training pipeline affect the final results.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | International Journal of Semantic Computing |
Volume | 15 |
Issue | 2 |
Number | 2 |
Pagination | 161 - 187 |
Date Published | Jan-06-2021 |
Publisher | World Scientific |
ISSN | 1793-351X |
Keywords | 3d CNN, classification, Detection, soccer events, spotting |
URL | https://www.worldscientific.com/doi/abs/10.1142/S1793351X2140002X |
DOI | 10.1142/S1793351X2140002X |
Visual Sentiment Analysis from Disaster Images in Social Media
Sensors (2021).Status: Accepted
Visual Sentiment Analysis from Disaster Images in Social Media
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Sensors |
Publisher | MDPI |
Book Chapter
Artificial Intelligence in Gastroenterology
In Artificial Intelligence in Medicine, 1-20. Cham: Springer International Publishing, 2021.Status: Published
Artificial Intelligence in Gastroenterology
The holy grail in endoscopy examinations has for a long time been assisted diagnosis using Artificial Intelligence (AI). Recent developments in computer hardware are now enabling technology to equip clinicians with promising tools for computer-assisted diagnosis (CAD) systems. However, creating viable models or architectures, training them, and assessing their ability to diagnose at a human level, are complicated tasks. This is currently an active area of research, and many promising methods have been proposed. In this chapter, we give an overview of the topic. This includes a description of current medical challenges followed by a description of the most commonly used methods in the field. We also present example results from research targeting some of these challenges, and a discussion on open issues and ongoing work is provided. Hopefully, this will inspire and enable readers to future develop CAD systems for gastroenterology.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Book Chapter |
Year of Publication | 2021 |
Book Title | Artificial Intelligence in Medicine |
Pagination | 1 - 20 |
Date Published | 09/2021 |
Publisher | Springer International Publishing |
Place Published | Cham |
ISBN Number | 978-3-030-58080-3 |
Keywords | Anomaly detection, artificial intelligence, Gastrointestinal endoscopy, Hand-crafted features, Neural Networks, Performance, Semantic segmentation |
URL | https://link.springer.com/referenceworkentry/10.1007%2F978-3-030-58080-3... |
DOI | 10.1007/978-3-030-58080-3_163-2 |
Poster
Assessment of sperm motility according to WHO classification using convolutional neural networks
ESHRE: ESHRE, 2021.Status: Accepted
Assessment of sperm motility according to WHO classification using convolutional neural networks
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Poster |
Year of Publication | 2021 |
Publisher | ESHRE |
Place Published | ESHRE |
Proceedings, refereed
Automated Clipping of Soccer Events using Machine Learning
In IEEE International Symphosium of Multimedia (ISM). IEEE, 2021.Status: Published
Automated Clipping of Soccer Events using Machine Learning
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | IEEE International Symphosium of Multimedia (ISM) |
Date Published | 12/2021 |
Publisher | IEEE |
DOI | 10.1109/ISM52913.2021.00042 |
Data Augmentation Using Generative Adversarial Networks For Creating Realistic Artificial Colon Polyp Images
In DDW 2021, 2021.Status: Published
Data Augmentation Using Generative Adversarial Networks For Creating Realistic Artificial Colon Polyp Images
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | DDW 2021 |
DDANet: Dual Decoder Attention Network for Automatic Polyp Segmentation
In 25th International Conference on Pattern Recognition. Springer, 2021.Status: Published
DDANet: Dual Decoder Attention Network for Automatic Polyp Segmentation
Colonoscopy is the gold standard for examination and detection of colorectal polyps. Localization and delineation of polyps can play a vital role in treatment (e.g., surgical planning) and prognostic decision making. Polyp segmentation can provide detailed boundary information for clinical analysis. Convolutional neural networks have improved the performance in colonoscopy. However, polyps usually possess various challenges, such as intra-and inter-class variation and noise. While manual labeling for polyp assessment requires time from experts and is prone to human error (e.g., missed lesions), an automated, accurate, and fast segmentation can improve the quality of delineated lesion boundaries and reduce missed rate. The Endotect challenge provides an opportunity to benchmark computer vision methods by training on the publicly available Hyperkvasir and testing on a separate unseen dataset. In this paper, we propose a novel architecture called ``DDANet'' based on a dual decoder attention network. Our experiments demonstrate that the model trained on the Kvasir-SEG dataset and tested on an unseen dataset achieves a dice coefficient of 0.7874, mIoU of 0.7010, recall of 0.7987, and a precision of 0.8577, demonstrating the generalization ability of our model.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 25th International Conference on Pattern Recognition |
Pagination | 307-314 |
Publisher | Springer |
Keywords | Benchmarking, Convolutional neural network, deep learning, Polyp segmentation |
DeepSynthBody: the beginning of the end for data deficiency in medicine
In The International Conference on Applied Artificial Intelligence (ICAPAI). IEEE, 2021.Status: Published
DeepSynthBody: the beginning of the end for data deficiency in medicine
Limited access to medical data is a barrier on developing new and efficient machine learning solutions in medicine such as computer-aided diagnosis, risk assessments, predicting optimal treatments and home-based personal healthcare systems. This paper presents DeepSynthBody: a novel framework that overcomes some of the inherent restrictions and limitations of medical data by using deep generative adversarial networks to produce synthetic data with characteristics similar to the real data, so-called DeepSynth (deep synthetic) data. We show that DeepSynthBody can address two key issues commonly associated with medical data, namely privacy concerns (as a result of data protection rules and regulations) and the high costs of annotations. To demonstrate the full pipeline of applying DeepSynthBody concepts and user-friendly functionalities, we also describe a synthetic medical dataset generated and published using our framework. DeepSynthBody opens a new era of machine learning applications in medicine with a synthetic model of the human body.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | The International Conference on Applied Artificial Intelligence (ICAPAI) |
Publisher | IEEE |
DOI | 10.1109/ICAPAI49758.2021.9462062 |
DivergentNets: Medical Image Segmentation by Network Ensemble
In EndoCV, 2021.Status: Accepted
DivergentNets: Medical Image Segmentation by Network Ensemble
Detection of colon polyps has become a trending topic in the intersecting fields of machine learning and gastrointestinal endoscopy. The focus has mainly been on per-frame classification. More recently, polyp segmentation has gained attention in the medical community. Segmentation has the advantage of being more accurate than per-frame classification or object detection as it can show the affected area in greater detail. For our contribution to the EndoCV 2021 segmentation challenge, we propose two separate approaches. First, a segmentation model named TriUNet composed of three separate UNet models. Second, we combine TriUNet with an ensemble of well-known segmentation models, namely UNet++, FPN, DeepLabv3, and DeepLabv3+, into a model called DivergentNets to produce more generalizable medical image segmentation masks. In addition, we propose a modified Dice loss that calculates loss only for a single class when performing multi-class segmentation, forcing the model to focus on what is most important. Overall, the proposed methods achieved the best average scores for each respective round in the challenge, with TriUNet being the winning model in Round I and DvergentNets being the winning model in Round II of the segmentation generalization challenge at EndoCV 2021
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | EndoCV |
Dutkat: A Multimedia System for Catching Illegal Catchers in a Privacy-Preserving Manner
In Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR'21). New York, NY, USA: ACM, 2021.Status: Published
Dutkat: A Multimedia System for Catching Illegal Catchers in a Privacy-Preserving Manner
Fish crime is considered a global and serious problem for a healthy and sustainable development of one of mankind's important sources of food. Technological surveillance and control solutions are emerging as remedies to combat criminal activities, but such solutions might also come with impractical and negative side-effects and challenges. In this paper, we present the concept and design of a surveillance system in lieu of current surveillance trends striking a delicate balance between privacy of legal actors while simultaneously capturing evidence-based footage, sensory data, and forensic proofs of illicit activities. Our proposed novel approach is to assist human operators in the 24/7 surveillance loop of remote professional fishing activities with a privacy-preserving Artificial Intelligence (AI) surveillance system operating in the same proximity as the activities being surveyed. The system will primarily be using video surveillance data, but also other sensor data captured on the fishing vessel. Additionally, the system correlates with other sources such as reports from other fish catches in the approximate area and time, etc. Only upon true positive flagging of specific potentially illicit activities by the locally executing AI algorithms, can forensic evidence be accessed from this physical edge, the fishing vessel. Besides a more privacy-preserving solution, our edge-based AI system also benefits from much less data that has to be transferred over unreliable, low-bandwidth satellite-based networks.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR'21) |
Pagination | 57-61 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450385299 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3463944 |
DOI | 10.1145/346394410.1145/3463944.3469102 |
Dutkat: A Multimedia System for Catching Illegal Catchers in a Privacy-Preserving Manner
In Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR'21). New York, NY, USA: ACM, 2021.Status: Published
Dutkat: A Multimedia System for Catching Illegal Catchers in a Privacy-Preserving Manner
Fish crime is considered a global and serious problem for a healthy and sustainable development of one of mankind's important sources of food. Technological surveillance and control solutions are emerging as remedies to combat criminal activities, but such solutions might also come with impractical and negative side-effects and challenges. In this paper, we present the concept and design of a surveillance system in lieu of current surveillance trends striking a delicate balance between privacy of legal actors while simultaneously capturing evidence-based footage, sensory data, and forensic proofs of illicit activities. Our proposed novel approach is to assist human operators in the 24/7 surveillance loop of remote professional fishing activities with a privacy-preserving Artificial Intelligence (AI) surveillance system operating in the same proximity as the activities being surveyed. The system will primarily be using video surveillance data, but also other sensor data captured on the fishing vessel. Additionally, the system correlates with other sources such as reports from other fish catches in the approximate area and time, etc. Only upon true positive flagging of specific potentially illicit activities by the locally executing AI algorithms, can forensic evidence be accessed from this physical edge, the fishing vessel. Besides a more privacy-preserving solution, our edge-based AI system also benefits from much less data that has to be transferred over unreliable, low-bandwidth satellite-based networks.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR'21) |
Pagination | 57-61 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450385299 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3463944 |
DOI | 10.1145/346394410.1145/3463944.3469102 |
Exploring Deep Learning Methods for Real-Time Surgical Instrument Segmentation in Laparoscopy
In 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE, 2021.Status: Published
Exploring Deep Learning Methods for Real-Time Surgical Instrument Segmentation in Laparoscopy
Minimally Invasive Surgery (MIS) is a surgical intervention used to examine the organs inside the abdomen and has been widely used due to its effectiveness over open surgery. Due to the hardware improvements such as high definition cameras, this procedure has significantly improved and new software methods have open potential for computer-assisted procedures. However, there exists challenges and requirement to improve detection and tracking of the position of the instruments during these surgical procedures. To this end, we evaluate and compare some popular deep learning methods that can potentially be explored for the automated segmentation of surgical instruments in laparoscopy, an important step towards tool tracking. Our experimental results demonstrate that the Dual decoder attention network (DDANet) produces a superior result compared to other recent deep learning methods. DDANet produces a dice coefficient of 0.8739 and mean intersection over union of 0.8183 for the Robust Medical Instrument Segmentation (ROBUST-MIS) Challenge 2019 dataset, at a real-time speed of 101.36 frames per second which is critical for such procedures.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI) |
Pagination | 1-4 |
Publisher | IEEE |
Keywords | Real-time segmentation, minimally invasive surgery, surgical instruments, laparoscopy, deep learning |
HTAD: A Home-Tasks Activities Dataset with Wrist-accelerometer and Audio Features
In 27th International Conference on Multimedia Modeling. Springer, 2021.Status: Published
HTAD: A Home-Tasks Activities Dataset with Wrist-accelerometer and Audio Features
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 27th International Conference on Multimedia Modeling |
Pagination | 196-205 |
Publisher | Springer |
Keywords | Accelerometer, Activity recognition, Audio, dataset, Sensor fusion |
HYPERAKTIV: An Activity Dataset from Patients with Attention-Deficit/Hyperactivity Disorder (ADHD)
In Proceedings of the 12th ACM Multimedia Systems Conference (MMSys '21). ACM, 2021.Status: Published
HYPERAKTIV: An Activity Dataset from Patients with Attention-Deficit/Hyperactivity Disorder (ADHD)
Machine learning research within healthcare frequently lacks the public data needed to be fully reproducible and comparable. Datasets are often restricted due to privacy concerns and legal requirements that come with patient-related data. Consequentially, many algorithms and models get published on the same topic without a standard benchmark to measure against. Therefore, this paper presents HYPERAKTIV, a public dataset containing health, activity, and heart rate data from patients diagnosed with attention deficit hyperactivity disorder, better known as ADHD. The dataset consists of data collected from 51 patients with ADHD and 52 clinical controls. In addition to the activity and heart rate data, we also include a series of patient attributes such as their age, sex, and information about their mental state, as well as output data from a computerized neuropsychological test. Together with the presented dataset, we also provide baseline experiments using traditional machine learning algorithms to predict ADHD based on the included activity data. We hope that this dataset can be used as a starting point for computer scientists who want to contribute to the field of mental health, and as a common benchmark for future work in ADHD analysis.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 12th ACM Multimedia Systems Conference (MMSys '21) |
Pagination | 314–319 |
Publisher | ACM |
URL | https://dl.acm.org/doi/10.1145/3458305.3478454 |
DOI | 10.1145/3458305.3478454 |
Identification of spermatozoa by unsupervised learning from video data
In 37th Virtual Annual Meeting of the European Society of Human Reproduction and Embryology (ESHRE). Oxford University Press, 2021.Status: Published
Identification of spermatozoa by unsupervised learning from video data
Identification of individual sperm is essential to assess a given sperm sample's motility behavior. Existing computer-aided systems need training data based on annotations by professionals, which is resource demanding. On the other hand, data analysed by unsupervised machine learning algorithms can improve supervised algorithms that are more stable for clinical applications. Therefore, unsupervised sperm identification can improve computer-aided sperm analysis systems predicting different aspects of sperm samples. Other possible applications are assessing kinematics and counting of spermatozoa.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 37th Virtual Annual Meeting of the European Society of Human Reproduction and Embryology (ESHRE) |
Publisher | Oxford University Press |
Impact of Image Resolution on Convolutional Neural Networks Performance in Gastrointestinal Endoscopy
In DDW 2021, 2021.Status: Published
Impact of Image Resolution on Convolutional Neural Networks Performance in Gastrointestinal Endoscopy
Convolutional neural networks (CNNs) are increasingly used to improve and automate processes in gastroenterology, like the detection of polyps during a colonoscopy. An important input to these methods is images and videos. Up until now, no well-defined, common understanding or standard regarding the resolution of the images and video frames has been defined, and to reduce processing time and resource requirements, images are today almost always down-sampled. However, how such down-sampling and the image resolution influence the performance in context with medical data is unknown. In this work, we investigate how the resolution relates to the performance of convolutional neural networks. This can help set standards for image or video characteristics for future CNN based models in gastrointestinal endoscopy.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | DDW 2021 |
Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy
In 27th International Conference on Multimedia Modeling. Vol. LNCS, volume 12573. Springer, 2021.Status: Published
Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy
Gastrointestinal (GI) pathologies are periodically screened, biopsied, and resected using surgical tools. Usually the procedures and the treated or resected areas are not specifically tracked or analysed during or after colonoscopies. Information regarding disease borders, development and amount and size of the resected area get lost. This can lead to poor follow-up and bothersome reassessment difficulties post-treatment. To improve the current standard and also to foster more research on the topic we have released the ``Kvasir Instrument'' dataset which consists of $590$ annotated frames containing GI procedure tools such as snares, balloons and biopsy forceps, etc. Beside of the images, the dataset includes ground truth masks and bounding boxes and has been verified by two expert GI endoscopists. Additionally, we provide a baseline for the segmentation of the GI tools to promote research and algorithm development. We obtained a dice coefficient score of 0.9158 and a Jaccard index of 0.8578 using a classical U-Net architecture. A similar dice coefficient score was observed for DoubleUNet. The qualitative results showed that the model did not work for the images with specularity and the frames with multiple instruments, while the best result for both methods was observed on all other types of images. Both, qualitative and quantitative results show that the model performs reasonably good, but there is a large potential for further improvements. Benchmarking using the dataset provides an opportunity for researchers to contribute to the field of automatic endoscopic diagnostic and therapeutic tool segmentation for GI endoscopy.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 27th International Conference on Multimedia Modeling |
Volume | LNCS, volume 12573 |
Pagination | 218-229 |
Publisher | Springer |
Multimodal Virtual Avatars for Investigative Interviews with Children
In Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR '21). New York, NY, USA: ACM, 2021.Status: Published
Multimodal Virtual Avatars for Investigative Interviews with Children
In this article, we present our ongoing work in the field of training police officers who conduct interviews with abused children. The objectives in this context are to protect vulnerable children from abuse, facilitate prosecution of offenders, and ensure that innocent adults are not accused of criminal acts. There is therefore a need for more data that can be used for improved interviewer training to equip police with the skills to conduct high-quality interviews. To support this important task, we propose to research a training program that utilizes different system components and multimodal data from the field of artificial intelligence such as chatbots, generation of visual content, text-to-speech, and speech-to-text. This program will be able to generate an almost unlimited amount of interview and also training data. The goal of combining all these different technologies and datatypes is to create an immersive and interactive child avatar that responds in a realistic way, to help to support the training of police interviewers, but can also produce synthetic data of interview situations that can be used to solve different problems in the same domain.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR '21) |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450385299 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3463944 |
DOI | 10.1145/346394410.1145/3463944.3469269 |
NanoNet: Real-Time Polyp Segmentation in Video Capsule Endoscopy and Colonoscopy
In 34th IEEE CBMS International Symposium on Computer-Based Medical Systems. IEEE, 2021.Status: Published
NanoNet: Real-Time Polyp Segmentation in Video Capsule Endoscopy and Colonoscopy
Deep learning in gastrointestinal endoscopy can assist to improve clinical performance and be helpful to assess lesions more accurately. To this extent, semantic segmentation methods that can perform automated real-time delineation of a region-of-interest, e.g., boundary identification of cancer or precancerous lesions, can benefit both diagnosis and interventions. However, accurate and real-time segmentation of endoscopic images is extremely challenging due to its high operator dependence and high-definition image quality. To utilize automated methods in clinical settings, it is crucial to design lightweight models with low latency such that they can be integrated with low-end endoscope hardware devices. In this work, we propose NanoNet, a novel architecture for the segmentation of video capsule endoscopy and colonoscopy images. Our proposed architecture allows real-time performance and has higher segmentation accuracy compared to other more complex ones. We use video capsule endoscopy and standard colonoscopy datasets with polyps, and a dataset consisting of endoscopy biopsies and surgical instruments, to evaluate the effectiveness of our approach. Our experiments demonstrate the increased performance of our architecture in terms of a trade-off between model complexity, speed, model parameters, and metric performances. Moreover, the resulting model size is relatively tiny, with only nearly 36,000 parameters compared to traditional deep learning approaches having millions of parameters.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 34th IEEE CBMS International Symposium on Computer-Based Medical Systems |
Publisher | IEEE |
Keywords | colonoscopy, deep learning, segmentation, tool segmentation, Video capsule endoscopy |
Njord: An out-in-the-wild real world fish vessel catch analysis dataset
In Arctic Frontiers. Tromsø, Norway: Arctic Frontiers, 2021.Status: Published
Njord: An out-in-the-wild real world fish vessel catch analysis dataset
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Arctic Frontiers |
Publisher | Arctic Frontiers |
Place Published | Tromsø, Norway |
PAANet: Progressive Alternating Attention for Automatic Medical Image Segmentation
In Proc. of 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART). Paris / Créteil, France: IEEE, 2021.Status: Published
PAANet: Progressive Alternating Attention for Automatic Medical Image Segmentation
Medical image segmentation can provide detailed information for clinical analysis which can be useful for scenarios where the detailed location of a finding is important. Knowing the location of a disease can play a vital role in treatment and decision-making. Convolutional neural network (CNN) based encoder-decoder techniques have advanced the performance of automated medical image segmentation systems. Several such CNN-based methodologies utilize techniques such as spatial- and channel-wise attention to enhance performance. Another technique that has drawn attention in recent years is residual dense blocks (RDBs). The successive convolutional layers in densely connected blocks are capable of extracting diverse features with varied receptive fields and thus, enhancing performance. However, consecutive stacked convolutional operators may not necessarily generate features that facilitate the identification of the target structures. In this paper, we propose a progressive alternating attention network (PAANet). We develop progressive alternating attention dense (PAAD) blocks, which construct a guiding attention map (GAM) after every convolutional layer in the dense blocks using features from all scales. The GAM allows the following layers in the dense blocks to focus on the spatial locations relevant to the target region. Every alternatePAAD block inverts the GAM to generate a reverse attention map which guides ensuing layers to extract boundary and edge-related information, refining the segmentation process. Our experiments on three different biomedical image segmentation datasets exhibit that our PAANet achieves favorable performance when compared to other state-of-the-art methods.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proc. of 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART) |
Date Published | 12/2021 |
Publisher | IEEE |
Place Published | Paris / Créteil, France |
URL | https://ieeexplore.ieee.org/document/9677844 |
DOI | 10.1109/BioSMART54244.2021.9677844 |
Sustainable Commercial Fishing: Digital Inspectors to the Rescue
In Arctic Frontiers. Arctic Frontiers, 2021.Status: Published
Sustainable Commercial Fishing: Digital Inspectors to the Rescue
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Arctic Frontiers |
Publisher | Arctic Frontiers |
Temperature Forecasting using Tower Networks
In Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval. New York, NY, USA: Association for Computing Machinery, 2021.Status: Published
Temperature Forecasting using Tower Networks
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval |
Pagination | 18-23 |
Date Published | 08/2021 |
Publisher | Association for Computing Machinery |
Place Published | New York, NY, USA |
ISBN Number | 9781450385299 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3463944https://dl.acm.org/doi... |
DOI | 10.1145/346394410.1145/3463944.3469099 |
Temperature Forecasting using Tower Networks
In Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval. New York, NY, USA: Association for Computing Machinery, 2021.Status: Published
Temperature Forecasting using Tower Networks
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval |
Pagination | 18-23 |
Date Published | 08/2021 |
Publisher | Association for Computing Machinery |
Place Published | New York, NY, USA |
ISBN Number | 9781450385299 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3463944https://dl.acm.org/doi... |
DOI | 10.1145/346394410.1145/3463944.3469099 |
Miscellaneous
MMSys'21 Grand Challenge on Detecting Cheapfakes
arXiv, 2021.Status: Published
MMSys'21 Grand Challenge on Detecting Cheapfakes
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2021 |
Publisher | arXiv |
Journal Article
A Genetic Attack Against Machine Learning Classifiers to Steal Biometric Actigraphy Profiles from Health Related Sensor Data
Journal of Medical Systems 44, no. 10 (2020): 1-11.Status: Published
A Genetic Attack Against Machine Learning Classifiers to Steal Biometric Actigraphy Profiles from Health Related Sensor Data
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2020 |
Journal | Journal of Medical Systems |
Volume | 44 |
Issue | 10 |
Pagination | 1-11 |
Date Published | Jan-10-2020 |
Publisher | Springer |
ISSN | 0148-5598 |
URL | http://link.springer.com/10.1007/s10916-020-01646-y |
DOI | 10.1007/s10916-020-01646-y |
An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning applied to Gastrointestinal Tract Abnormality Classification
ACM Transactions on Computing for Healthcare 1 (2020): 1-29.Status: Published
An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning applied to Gastrointestinal Tract Abnormality Classification
Precise and efficient automated identification of Gastrointestinal (GI) tract diseases can help doctors treat more patients and improve the rate of disease detection and identification. Currently, automatic analysis of diseases in the GI tract is a hot topic in both computer science and medical-related journals. Nevertheless, the evaluation of such an automatic analysis is often incomplete or simply wrong. Algorithms are often only tested on small and biased datasets, and cross-dataset evaluations are rarely performed. A clear understanding of evaluation metrics and machine learning models with cross datasets is crucial to bring research in the field to a new quality level. Towards this goal, we present comprehensive evaluations of five distinct machine learning models using Global Features and Deep Neural Networks that can classify 16 different key types of GI tract conditions, including pathological findings, anatomical landmarks, polyp removal conditions, and normal findings from images captured by common GI tract examination instruments. Inour evaluation, we introduce performance hexagons using six performance metrics such as recall, precision, specificity, accuracy, F1-score, and Matthews Correlation Coefficient to demonstrate how to determine the real capabilities of models rather than evaluatingthem shallowly. Furthermore, we perform cross-dataset evaluations using different datasets for training and testing. With these cross-dataset evaluations, we demonstrate the challenge of actually building a generalizable model that could be used across different hospitals. Our experiments clearly show that more sophisticated performance metrics and evaluation methods need to be applied to get reliable models rather than depending on evaluations of the splits of the same dataset, i.e., the performance metrics should always be interpreted together rather than relying on a single metric.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2020 |
Journal | ACM Transactions on Computing for Healthcare |
Volume | 1 |
Number | 3 |
Pagination | 1-29 |
Publisher | ACM |
Applying machine learning in motor activity time series of depressed bipolar and unipolar patients compared to healthy controls
PLOS ONE 15, no. 8 (2020): e0231995.Status: Published
Applying machine learning in motor activity time series of depressed bipolar and unipolar patients compared to healthy controls
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2020 |
Journal | PLOS ONE |
Volume | 15 |
Issue | 8 |
Pagination | e0231995 |
Date Published | Dec-08-2021 |
Publisher | PLOS ONE |
URL | https://dx.plos.org/10.1371/journal.pone.0231995 |
DOI | 10.1371/journal.pone.0231995 |
Comparative validation of multi-instance instrument segmentation in endoscopy: results of the ROBUST-MIS 2019 challenge
Medical Image Analysis (2020).Status: Published
Comparative validation of multi-instance instrument segmentation in endoscopy: results of the ROBUST-MIS 2019 challenge
Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video im-ages have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2020 |
Journal | Medical Image Analysis |
Date Published | 11/2020 |
Publisher | Elsevier |
Human Activity Recognition from Multiple Sensors Data Using Multi-fusion Representations and CNNs
ACM Transactions on Multimedia Computing, Communications, and Applications 16, no. 2 (2020): 1-19.Status: Published
Human Activity Recognition from Multiple Sensors Data Using Multi-fusion Representations and CNNs
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2020 |
Journal | ACM Transactions on Multimedia Computing, Communications, and Applications |
Volume | 16 |
Issue | 2 |
Pagination | 1 - 19 |
Date Published | Mar-06-2021 |
Publisher | ACM |
Place Published | New York |
ISSN | 1551-6857 |
URL | https://dl.acm.org/doi/10.1145/3377882 |
DOI | 10.1145/3377882 |
HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy
Scientific Data 7, no. 1 (2020): 1-14.Status: Published
HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy
Artificial intelligence is currently a hot topic in medicine. However, medical data is often sparse and hard to obtain due to legal restrictions and lack of medical personnel for the cumbersome and tedious process to manually label training data. These constraints make it difficult to develop systems for automatic analysis, like detecting disease or other lesions. In this respect, this article presents HyperKvasir, the largest image and video dataset of the gastrointestinal tract available today. The data is collected during real gastro- and colonoscopy examinations at Bærum Hospital in Norway and partly labeled by experienced gastrointestinal endoscopists. The dataset contains 110,079 images and 374 videos, and represents anatomical landmarks as well as pathological and normal findings. The total number of images and video frames together is around 1 million. Initial experiments demonstrate the potential benefits of artificial intelligence-based computer-assisted diagnosis systems. The HyperKvasir dataset can play a valuable role in developing better algorithms and computer-assisted examination systems not only for gastro- and colonoscopy, but also for other fields in medicine.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2020 |
Journal | Scientific Data |
Volume | 7 |
Issue | 1 |
Pagination | 1-14 |
Date Published | 08/2020 |
Publisher | Springer Nature |
Keywords | dataset, GI, Machine learning |
URL | http://www.nature.com/articles/s41597-020-00622-y |
DOI | 10.1038/s41597-020-00622-y |
Proceedings, refereed
ACM Multimedia BioMedia 2020 Grand Challenge Overview
In Proceedings of the 28th ACM International Conference on Multimedia. New York, NY, USA: Association for Computing Machinery, 2020.Status: Published
ACM Multimedia BioMedia 2020 Grand Challenge Overview
The BioMedia 2020 ACM Multimedia Grand Challenge is the second in a series of competitions focusing on the use of multimedia for different medical use-cases. In this year's challenge, participants are asked to develop algorithms that automatically predict the quality of a given human semen sample using a combination of visual, patient-related, and laboratory-analysis-related data. Compared to last year's challenge, participants are provided with a fully multimodal dataset (videos, analysis data, study participant data) from the field of assisted human reproduction. The tasks encourage the use of the different modalities contained within the dataset and finding smart ways of how they may be combined to further improve prediction accuracy. For example, using only video data or combining video data and patient-related data. The ground truth was developed through a preliminary analysis done by medical experts following the World Health Organization's standard for semen quality assessment. The task lays the basis for automatic, real-time support systems for artificial reproduction. We hope that this challenge motivates multimedia researchers to explore more medical-related applications and use their vast knowledge to make a real impact on people's lives.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Proceedings of the 28th ACM International Conference on Multimedia |
Pagination | 4655–4658 |
Publisher | Association for Computing Machinery |
Place Published | New York, NY, USA |
ISBN Number | 9781450379885 |
Keywords | artificial intelligence, Machine learning, male fertility, semen analysis, spermatozoa |
URL | https://doi.org/10.1145/3394171.3416287 |
DOI | 10.1145/3394171.3416287 |
DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation
In CBMS 2020: International Symposium on Computer-Based Medical Systems. IEEE, 2020.Status: Published
DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | CBMS 2020: International Symposium on Computer-Based Medical Systems |
Publisher | IEEE |
Notes | This paper was nominated for the best paper award at CBMS 2020. |
ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Internet Applications
In European Conference on Information Retrieval. Cham: Springer International Publishing, 2020.Status: Published
ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Internet Applications
This paper presents an overview of the 2020 ImageCLEF lab that will be organized as part of the Conference and Labs of the Evaluation Forum—CLEF Labs 2020 in Thessaloniki, Greece. ImageCLEF is an ongoing evaluation initiative (run since 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval of visual data with the aim of providing information access to large collections of images in various usage scenarios and domains. In 2020, the 18th edition of ImageCLEF will organize four main tasks: (i) a Lifelog task (videos, images and other sources) about daily activity understanding, retrieval and summarization, (ii) a Medical task that groups three previous tasks (caption analysis, tuberculosis prediction, and medical visual question answering) with new data and adapted tasks, (iii) a Coral task about segmenting and labeling collections of coral images for 3D modeling, and a new (iv) Web user interface task addressing the problems of detecting and recognizing hand drawn website UIs (User Interfaces) for generating automatic code. The strong participation, with over 235 research groups registering and 63 submitting over 359 runs for the tasks in 2019 shows an important interest in this benchmarking campaign. We expect the new tasks to attract at least as many researchers for 2020.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | European Conference on Information Retrieval |
Pagination | 533 - 541 |
Publisher | Springer International Publishing |
Place Published | Cham |
ISBN Number | 978-3-030-45441-8 |
ISSN Number | 0302-9743 |
URL | https://link.springer.com/chapter/10.1007/978-3-030-45442-5_69 |
DOI | 10.1007/978-3-030-45442-5_69 |
Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy
In International Conference on Multimedia Modeling. Springer, 2020.Status: Published
Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy
Gastrointestinal pathologies are periodically screened, biopsied, and resected using surgical tools. Usually, the procedures and the treated or resected areas are not specifically tracked or analysed during or after colonoscopies. Information regarding disease borders, development, amount, and size of the resected area get lost. This can lead to poor follow-up and bothersome reassessment difficulties post-treatment. To improve the current standard and also to foster more research on the topic, we have released the ``Kvasir-Instrument'' dataset, which consists of 590 annotated frames containing GI procedure tools such as snares, balloons, and biopsy forceps, etc. Besides the images, the dataset includes ground truth masks and bounding boxes and has been verified by two expert GI endoscopists. Additionally, we provide a baseline for the segmentation of the GI tools to promote research and algorithm development. We obtained a dice coefficient score of 0.9158 and a Jaccard index of 0.8578 using a classical U-Net architecture. A similar dice coefficient score was observed for DoubleUNet. The qualitative results showed that the model did not work for the images with specularity and the frames with multiple tools, while the best result for both methods was observed on all other types of images. Both qualitative and quantitative results show that the model performs reasonably good, but there is potential for further improvements. Benchmarking using the dataset provides an opportunity for researchers to contribute to the field of automatic endoscopic diagnostic and therapeutic tool segmentation for GI endoscopy.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | International Conference on Multimedia Modeling |
Publisher | Springer |
Keywords | Benchmarking, Convolutional neural network, Gastrointestinal endoscopy, Tool segmentation |
URL | https://www.springerprofessional.de/en/kvasir-instrument-diagnostic-and-... |
Kvasir-SEG: A Segmented Polyp Dataset
In International Conference on Multimedia Modeling. Daejeon, Korea: Springer, 2020.Status: Published
Kvasir-SEG: A Segmented Polyp Dataset
Pixel-wise image segmentation is a highly demanding task in medical image analysis. In practice, it is difficult to find annotated medical images with corresponding segmentation masks. In this paper, we present Kvasir-SEG: an open-access dataset of gastrointestinal polyp images and corresponding segmentation masks, manually annotated and verified by an experienced gastroenterologist. We demonstrate the use of our dataset with a traditional segmentation approach and a modern deep learning based CNN approach. This work will be valuable for researchers to reproduce results and compare their methods in the future. By adding segmentation masks to the Kvasir dataset, which until today only consisted of framewise annotations, we enable multimedia and computer vision researchers to contribute in the field of polyp segmentation and automatic analysis of colonoscopy videos.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | International Conference on Multimedia Modeling |
Pagination | 451-462 |
Publisher | Springer |
Place Published | Daejeon, Korea |
Keywords | Kvasir-SEG dataset, Medical images, Polyp segmentation, ResUNet Fuzzy c-mean clustering, Semantic segmentation |
LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification
In The Joint International Conference PDCAT-PAAP 2020. Springer, 2020.Status: Published
LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification
Deep Neural Networks (DNNs) have become the de-facto standard in computer vision, as well as in many other pattern recognition tasks. A key drawback of DNNs is that the training phase can be very computationally expensive. Organizations or individuals that cannot afford purchasing state-of-the-art hardware or tapping into cloud-hosted infrastructures may face a long waiting time before the training completes or might not be able to train a model at all. Investigating novel ways to reduce the training time could be a potential solution to alleviate this drawback, and thus enabling more rapid development of new algorithms and models. In this paper, we propose LightLayers, a method for reducing the number of trainable parameters in deep neural networks (DNN). The proposed LightLayers consists of LightDense andLightConv2D layer that are as efficient as regular Conv2D and Dense layers, but uses less parameters. We resort to Matrix Factorization to reduce the complexity of the DNN models resulting into lightweight DNNmodels that require less computational power, without much loss in the accuracy. We have tested LightLayers on MNIST, Fashion MNIST, CI-FAR 10, and CIFAR 100 datasets. Promising results are obtained for MNIST, Fashion MNIST, CIFAR-10 datasets whereas CIFAR 100 shows acceptable performance by using fewer parameters.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | The Joint International Conference PDCAT-PAAP 2020 |
Publisher | Springer |
Keywords | CIFAR-10, Convolutional neural network, Deep Learning, Fashion MNIST, Lightweight model, MNIST, Weight decomposition |
Medico Multimedia Task at MediaEval 2020: Automatic Polyp Segmentation
In Medico MediaEval 2020. CEUR, 2020.Status: Published
Medico Multimedia Task at MediaEval 2020: Automatic Polyp Segmentation
Colorectal cancer is the third most common cause of cancer worldwide. According to Global cancer statistics 2018, the incidence of colorectal cancer is increasing in both developing and developed countries. Early detection of colon anomalies such as polyps is important for cancer prevention, and automatic polyp segmentation can play a crucial role for this. Regardless of the recent advancement in early detection and treatment options, the estimated polyp miss rate is still around 20\%. Support via an automated computer-aided diagnosis system could be one of the potential solutions for the overlooked polyps. Such detection systems can help low-cost design solutions and save doctors time, which they could for example use to perform more patient examinations. In this paper, we introduce the 2020 Medico challenge, provide some information on related work and the dataset, describe the task and evaluation metrics, and discuss the necessity of organizing the Medico challenge.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Medico MediaEval 2020 |
Publisher | CEUR |
Organiser Team at ImageCLEFlifelog 2020: A Baseline Approach for Moment Retrieval and Athlete Performance Prediction using Lifelog Data
In CLEF2020. CEUR Workshop Proceedings, 2020.Status: Published
Organiser Team at ImageCLEFlifelog 2020: A Baseline Approach for Moment Retrieval and Athlete Performance Prediction using Lifelog Data
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | CLEF2020 |
Publisher | CEUR Workshop Proceedings |
Overview of ImageCLEF lifelog 2020: lifelog moment retrieval and sport performance lifelog
In CLEF2020 . CEUR Workshop Proceedings, 2020.Status: Published
Overview of ImageCLEF lifelog 2020: lifelog moment retrieval and sport performance lifelog
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | CLEF2020 |
Publisher | CEUR Workshop Proceedings |
Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications
In CLEF 2020. Vol. 12260. Cham: Springer International Publishing, 2020.Status: Published
Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications
This paper presents an overview of the 2020 ImageCLEF lab that will be organized as part of the Conference and Labs of the Evaluation Forum—CLEF Labs 2020 in Thessaloniki, Greece. ImageCLEF is an ongoing evaluation initiative (run since 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval of visual data with the aim of providing information access to large collections of images in various usage scenarios and domains. In 2020, the 18th edition of ImageCLEF will organize four main tasks: (i) a Lifelog task (videos, images and other sources) about daily activity understanding, retrieval and summarization, (ii) a Medical task that groups three previous tasks (caption analysis, tuberculosis prediction, and medical visual question answering) with new data and adapted tasks, (iii) a Coral task about segmenting and labeling collections of coral images for 3D modeling, and a new (iv) Web user interface task addressing the problems of detecting and recognizing hand drawn website UIs (User Interfaces) for generating automatic code. The strong participation, with over 235 research groups registering and 63 submitting over 359 runs for the tasks in 2019 shows an important interest in this benchmarking campaign. We expect the new tasks to attract at least as many researchers for 2020.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | CLEF 2020 |
Volume | 12260 |
Pagination | 311 - 341 |
Publisher | Springer International Publishing |
Place Published | Cham |
ISBN Number | 978-3-030-58218-0 |
ISSN Number | 0302-9743 |
URL | https://doi.org/10.1007/978-3-030-58219-7_22 |
DOI | 10.1007/978-3-030-58219-710.1007/978-3-030-58219-7_22 |
PCC arena: a benchmark platform for point cloud compression algorithms
In MMSys '20: 11th ACM Multimedia Systems ConferenceProceedings of the 12th ACM International Workshop on Immersive Mixed and Virtual Environment Systems. New York, NY, USA: ACM, 2020.Status: Published
PCC arena: a benchmark platform for point cloud compression algorithms
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | MMSys '20: 11th ACM Multimedia Systems ConferenceProceedings of the 12th ACM International Workshop on Immersive Mixed and Virtual Environment Systems |
Date Published | 06/2020 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450379472 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3386293https://dl.acm.org/doi... |
DOI | 10.1145/338629310.1145/3386293.3397112 |
PMData: a sports logging dataset
In Proceedings of the 11th ACM Multimedia Systems Conference. ACM, 2020.Status: Published
PMData: a sports logging dataset
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Proceedings of the 11th ACM Multimedia Systems Conference |
Pagination | 231-236 |
Publisher | ACM |
PSYKOSE: A Motor Activity Database of Patients with Schizophrenia
In 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS). Rochester, MN, USA: IEEE, 2020.Status: Published
PSYKOSE: A Motor Activity Database of Patients with Schizophrenia
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS) |
Publisher | IEEE |
Place Published | Rochester, MN, USA |
URL | https://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=9169740http... |
DOI | 10.1109/CBMS49503.202010.1109/CBMS49503.2020.00064 |
Pyramid-Focus-Augmentation: Medical Image Segmentation with Step-Wise Focus
In MediaEval 2020. CEUR, 2020.Status: Published
Pyramid-Focus-Augmentation: Medical Image Segmentation with Step-Wise Focus
Segmentation of findings in the gastrointestinal tract is a challenging but also an important task which is an important building stone for sufficient automatic decision support systems. In this work, we present our solution for the Medico 2020 task, which focused on the problem of colon polyp segmentation. We present our simple but efficient idea of using an augmentation method that uses grids in a pyramid-like manner (large to small) for segmentation. Our results show that the proposed methods work as indented and can also lead to comparable results when competing with other methods.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | MediaEval 2020 |
Publisher | CEUR |
Real-Time Detection of Events in Soccer Videos using 3D Convolutional Neural Networks
In 2020 IEEE International Symposium on Multimedia (ISM). IEEE, 2020.Status: Published
Real-Time Detection of Events in Soccer Videos using 3D Convolutional Neural Networks
In this paper, we present an algorithm for automatically detecting events in soccer videos using 3D convolutional neural networks. The algorithm uses a sliding window approach to scan over a given video to detect events such as goals, yellow/red cards, and player substitutions. We test the method on three different datasets from SoccerNet, the Swedish Allsvenskan, and the Norwegian Eliteserien. Overall, the results show that we can detect events with high recall, low latency, and accurate time estimation. The trade-off is a slightly lower precision compared to the current state-of-the-art, which has higher latency and performs better when a less accurate time estimation can be accepted. In addition to the presented algorithm, we perform an extensive ablation study on how the different parts of the training pipeline affect the final results.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | 2020 IEEE International Symposium on Multimedia (ISM) |
Publisher | IEEE |
DOI | 10.1109/ISM.2020.00030 |
Reproducibility Companion Paper: Instance of Interest Detection
In Proceedings of the 28th ACM International Conference on Multimedia. New York, NY, USA: ACM, 2020.Status: Published
Reproducibility Companion Paper: Instance of Interest Detection
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Proceedings of the 28th ACM International Conference on Multimedia |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450379885 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3394171https://dl.acm.org/doi... |
DOI | 10.1145/339417110.1145/3394171.3414811 |
Reproducibility Companion Paper: Selective Deep Convolutional Features for Image Retrieval
In Proceedings of the 28th ACM International Conference on Multimedia. New York, NY, USA: ACM, 2020.Status: Published
Reproducibility Companion Paper: Selective Deep Convolutional Features for Image Retrieval
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Proceedings of the 28th ACM International Conference on Multimedia |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450379885 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3394171https://dl.acm.org/doi... |
DOI | 10.1145/339417110.1145/3394171.3414814 |
Scalable Infrastructure for Efficient Real-Time Sports Analytics
In Companion Publication of the 2020 International Conference on Multimodal Interaction. New York, NY, USA: ACM, 2020.Status: Published
Scalable Infrastructure for Efficient Real-Time Sports Analytics
Recent technological advances are adapted in sports to improve performance, avoid injuries, and make advantageous decisions. In this paper, we describe our ongoing efforts to develop and deploy PMSys, our smartphone-based athlete monitoring and reporting system. We describe our first attempts to gain insight into some of the data we have collected. Experiences so far are promising, both on the technical side and for athlete performance development. Our initial application of artificial-intelligence methods for prediction is encouraging and indicative.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Companion Publication of the 2020 International Conference on Multimodal Interaction |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450380027 |
Keywords | algorithmic analysis, artificial intelligence, Machine learning, privacy-preserving data collection, Sports performance logging |
URL | https://dl.acm.org/doi/proceedings/10.1145/3395035https://dl.acm.org/doi... |
DOI | 10.1145/339503510.1145/3395035.3425300 |
The EndoTect 2020 Challenge: Evaluation and Comparison of Classification, Segmentation and Inference Time for Endoscopy
In 25th International Conference on Pattern Recognition (ICPR). IEEE, 2020.Status: Published
The EndoTect 2020 Challenge: Evaluation and Comparison of Classification, Segmentation and Inference Time for Endoscopy
The EndoTect challenge at the International Conference on Pattern Recognition 2020 aims to motivate the development of algorithms that aid medical experts in finding anomalies that commonly occur in the gastrointestinal tract. Using HyperKvasir, a large dataset containing images taken from several endoscopies, the participants competed in three tasks. Each task focuses on a specific requirement for making it useful in a real-world medical scenario. The tasks are (i) high classification performance in terms of prediction accuracy, (ii) efficient classification measured by the number of images classified per second, and (iii) pixel-level segmentation of specific anomalies. Hopefully, this can motivate different computer science researchers to help benchmark a crucial component of a future computer-aided diagnosis system, which in turn, could potentially save human lives.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | 25th International Conference on Pattern Recognition (ICPR) |
Publisher | IEEE |
Toadstool: A Dataset for Training Emotional IntelligentMachines Playing Super Mario Bros
In The ACM Multimedia Systems Conference (MMSys). The ACM Multimedia Systems Conference (MMSys): ACM, 2020.Status: Published
Toadstool: A Dataset for Training Emotional IntelligentMachines Playing Super Mario Bros
Games are often defined as engines of experience, and they are heavily relying on emotions, they arouse in players. In this paper, we present a dataset called Toadstool as well as a reproducible methodology to extend on the dataset. The dataset consists of video, sensor, and demographic data collected from ten participants playing Super Mario Bros, an iconic and famous video game. The sensor data is collected through an Empatica E4 wristband, which provides high-quality measurements and is graded as a medical device. In addition to the dataset and the methodology for data collection, we present a set of baseline experiments which show that we can use video game frames together with the facial expressions to predict the blood volume pulse of the person playing Super Mario Bros. With the dataset and the collection methodology we aim to contribute to research on emotionally aware machine learning algorithms, focusing on reinforcement learning and multimodal data fusion. We believe that the presented dataset can be interesting for a manifold of researchers to explore exciting new interdisciplinary questions.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | The ACM Multimedia Systems Conference (MMSys) |
Publisher | ACM |
Place Published | The ACM Multimedia Systems Conference (MMSys) |
URL | https://dl.acm.org/doi/10.1145/3339825.3394939 |
DOI | 10.1145/3339825.3394939 |
Vid2Pix - A Framework for Generating High-Quality Synthetic Videos
In 2020 IEEE International Symposium on Multimedia (ISM). IEEE, 2020.Status: Published
Vid2Pix - A Framework for Generating High-Quality Synthetic Videos
Data is arguably the most important resource today as it fuels the algorithms powering services we use every day. However, in fields like medicine, publicly available datasets are few, and labeling medical datasets require tedious efforts from trained specialists. Generated synthetic data can be to future successful healthcare clinical intelligence. Here, we present a GAN-based video generator demonstrating promising results.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | 2020 IEEE International Symposium on Multimedia (ISM) |
Publisher | IEEE |
Poster
Big data is not always better – prediction of live birth using machine learning on time-lapse videos of human embryos
ESHRE virtual 36th Annual Meeting: ESHRE, 2020.Status: Published
Big data is not always better – prediction of live birth using machine learning on time-lapse videos of human embryos
Afilliation | Machine Learning |
Project(s) | No Simula project |
Publication Type | Poster |
Year of Publication | 2020 |
Publisher | ESHRE |
Place Published | ESHRE virtual 36th Annual Meeting |
Proceedings, refereed
A Web-Based Software for Training and Quality Assessment in the Image Analysis Workflow for Cardiac T1 Mapping MRI
In 2019 IEEE International Symposium on Multimedia (ISM). IEEE, 2019.Status: Published
A Web-Based Software for Training and Quality Assessment in the Image Analysis Workflow for Cardiac T1 Mapping MRI
Medical practice makes significant use of imaging scans such as Ultrasound or MRI as a diagnostic tool. They are used in the visual inspection or quantification of medical parameters computed from the images in post-processing. However, the value of such parameters depends much on the user's variability, device, and algorithmic differences. In this paper, we focus on quantifying the variability due to the human factor, which can be primarily addressed by the structured training of a human operator. We focus on a specific emerging cardiovascular \gls{mri} methodology, the T1 mapping, that has proven useful to identify a range of pathological alterations of the myocardial tissue structure. Training, especially in emerging techniques, is typically not standardized, varying dramatically across medical centers and research teams. Additionally, training assessment is mostly based on qualitative approaches. Our work aims to provide a software tool combining traditional clinical metrics and convolutional neural networks to aid the training process by gathering contours from multiple trainees, quantifying discrepancy from local gold standard or standardized guidelines, classifying trainees output based on critical parameters that affect contours variability.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 IEEE International Symposium on Multimedia (ISM) |
Publisher | IEEE |
DOI | 10.1109/ISM46123.2019.00047 |
ACM Multimedia BioMedia 2019 Grand Challenge Overview
In The ACM International Conference on Multimedia (ACM MM). New York, New York, USA: ACM Press, 2019.Status: Published
ACM Multimedia BioMedia 2019 Grand Challenge Overview
The BioMedia 2019 ACM Multimedia Grand Challenge is the first in a series of competitions focusing on the use of multimedia for different medical use-cases. In this year’s challenge, the participants are asked to develop efficient algorithms which automatically detect a variety of findings commonly identified in the gastrointestinal (GI) tract (a part of the human digestive system). The purpose of this task is to develop methods to aid medical doctors performing routine endoscopy inspections of the GI tract. In this paper, we give a detailed description of the four different tasks of this year’s challenge, present the datasets used for training and testing, and discuss how each submission is evaluated both qualitatively and quantitatively.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | The ACM International Conference on Multimedia (ACM MM) |
Pagination | 2563-2567 |
Date Published | 10/2019 |
Publisher | ACM Press |
Place Published | New York, New York, USA |
ISBN Number | 9781450368896 |
URL | http://dl.acm.org/citation.cfm?doid=3343031http://dl.acm.org/citation.cf... |
DOI | 10.1145/334303110.1145/3343031.3356058 |
Automatic Hyperparameter Optimization for Transfer Learning on Medical Image Datasets Using Bayesian Optimization
In 13th International Symposium on Medical Information and Communication Technology (ISMICT). IEEE, 2019.Status: Published
Automatic Hyperparameter Optimization for Transfer Learning on Medical Image Datasets Using Bayesian Optimization
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 13th International Symposium on Medical Information and Communication Technology (ISMICT) |
Pagination | 1-6 |
Publisher | IEEE |
DOI | 10.1109/ISMICT.2019.8743779 |
Estimating Downlink Throughput from End-User Measurements in Mobile Broadband Networks
In IEEE World of Wireless, Mobile and Multimedia Networks (WoWMoM). IEEE, 2019.Status: Published
Estimating Downlink Throughput from End-User Measurements in Mobile Broadband Networks
Afilliation | Communication Systems |
Project(s) | MONROE: Measuring Mobile Broadband Networks in Europe |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | IEEE World of Wireless, Mobile and Multimedia Networks (WoWMoM) |
Publisher | IEEE |
Extracting temporal features into a spatial domain using autoencoders for sperm video analysis
In MediaEval 2019. CEUR Workshop Proceedings, 2019.Status: Published
Extracting temporal features into a spatial domain using autoencoders for sperm video analysis
In this paper, we present a two-step deep learning method that is used to predict sperm motility and morphology based on video recordings of human spermatozoa. First, we use an autoencoder to extract temporal features from a given semen video and plot these into image-space, which we call feature-images. Second, these feature-images are used to perform transfer learning to predict the motility and morphology values of human sperm. The presented method shows it's capability to extract temporal information into spatial domain feature-images which can be used with traditional convolutional neural networks. Furthermore, the accuracy of the predicted motility of a given semen sample shows that a deep learning-based model can capture the temporal information of microscopic recordings of human semen.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | MediaEval 2019 |
Date Published | 10/2019 |
Publisher | CEUR Workshop Proceedings |
Fusion of multiple representations extracted from a single sensor’s data for activity recognition using CNNs
In 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.Status: Published
Fusion of multiple representations extracted from a single sensor’s data for activity recognition using CNNs
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 International Joint Conference on Neural Networks (IJCNN) |
Pagination | 1–6 |
Publisher | IEEE |
GameStory Task at MediaEval 2019
In Proceedings of MediaEval 2019. CEUR Workshop Proceedings (CEUR-WS.org), 2019.Status: Published
GameStory Task at MediaEval 2019
Game video streams are watched by millions, so that, meanwhile, one can make a living from broadcasting and commenting video games, whereas some have become professional e-sports athletes. E-sports leagues and tournaments have emerged worldwide, where players compete in controlled environments, streaming the matches online, and allowing the audience to discuss and criticize the game- play. In the GameStory task, held for the second time at MediaEval, we foster research into this exciting domain. Our focus is on an- alyzing and summarizing video game streams. With the help of ZNIPE.tv, we compiled a high-quality dataset of a Counter-Strike: Global Offensive tournament alongside ground truth labels for two analysis tasks, forming a basis for summarization.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | Proceedings of MediaEval 2019 |
Date Published | 10/2019 |
Publisher | CEUR Workshop Proceedings (CEUR-WS.org) |
GANEx: A complete pipeline of training, inference and benchmarking GAN experiments
In 2019 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 2019.Status: Published
GANEx: A complete pipeline of training, inference and benchmarking GAN experiments
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 International Conference on Content-Based Multimedia Indexing (CBMI) |
Pagination | 1-4 |
Publisher | IEEE |
Keywords | GAN, GANEx, Generative Adversarial Network |
Heart Rate Prediction from Head Movement during Virtual Reality Treatment for Social Anxiety
In 2019 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 2019.Status: Published
Heart Rate Prediction from Head Movement during Virtual Reality Treatment for Social Anxiety
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 International Conference on Content-Based Multimedia Indexing (CBMI) |
Pagination | 1–5 |
Publisher | IEEE |
ImageCLEF 2019: Multimedia Retrieval in Lifelogging, Medical, Nature, and Security Applications
In European Conference on Information Retrieval. Springer, 2019.Status: Published
ImageCLEF 2019: Multimedia Retrieval in Lifelogging, Medical, Nature, and Security Applications
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | European Conference on Information Retrieval |
Pagination | 301–308 |
Date Published | 02/2019 |
Publisher | Springer |
ImageCLEF 2019: Multimedia retrieval in medicine, lifelogging, security and nature
In International Conference of the Cross-Language Evaluation Forum for European Languages. Cham: Springer, 2019.Status: Published
ImageCLEF 2019: Multimedia retrieval in medicine, lifelogging, security and nature
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | International Conference of the Cross-Language Evaluation Forum for European Languages |
Pagination | 358–386 |
Publisher | Springer |
Place Published | Cham |
LIFER 2.0: Discovering Personal Lifelog Insights using an Interactive Lifelog Retrieval System
In CLEF2019 Working Notes. CEUR Workshop Proceedings (CEUR-WS.org), 2019.Status: Published
LIFER 2.0: Discovering Personal Lifelog Insights using an Interactive Lifelog Retrieval System
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | CLEF2019 Working Notes |
Publisher | CEUR Workshop Proceedings (CEUR-WS.org) |
Medical Multimedia Systems and Applications
In Proceedings of the 27th ACM International Conference on Multimedia - MM '19. New York, NY, USA: ACM Press, 2019.Status: Published
Medical Multimedia Systems and Applications
In recent years, we have observed a rise of interest in the multimedia community towards research topics related to health. It can be observed that this goes into two interesting directions. One is personal health with a larger focus on well-being and everyday healthy living. The other direction focuses more on multimedia challenges within the health-care systems, for example, how can multimedia content produced in hospitals be used efficiently but also on the user perspective of patients and health-care personal. Challenges and requirements in this interesting and challenging direction are similar to classic multimedia research, but with some additional pitfalls and challenges. This tutorial aims to give a general introduction to the research area; to provide an overview of specific requirements, pitfalls and challenges; to discuss existing and possible future work; and to elaborate on how machine learning approaches can help in multimedia-related challenges to improve the health-care quality for patients and support medical experts in their daily work.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | Proceedings of the 27th ACM International Conference on Multimedia - MM '19 |
Pagination | 2711-2713 |
Date Published | 1072019 |
Publisher | ACM Press |
Place Published | New York, NY, USA |
ISBN Number | 9781450368896 |
URL | http://dl.acm.org/citation.cfm?doid=3343031http://dl.acm.org/citation.cf... |
DOI | 10.1145/3343031.3351319 |
Medico Multimedia Task at MediaEval 2019
In MediaEval. CEUR Workshop Proceedings, 2019.Status: Published
Medico Multimedia Task at MediaEval 2019
The Medico: Multimedia for Medicine Task is running for the third time as part of MediaEval 2019. This year, we have changed the task from anomaly detection in images of the gastrointestinal tract to focus on the automatic prediction of human semen quality based on videos. The purpose of this task is to aid in the assessment of male reproductive health by providing a quick and consisted method of analyzing human semen. In this paper, we describe the task in detail, give a brief description of the provided dataset, and discuss the evaluation process and the metrics used to rank the submissions of the participants.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | MediaEval |
Publisher | CEUR Workshop Proceedings |
Multi-Modal Machine Learning for Flood Detection in News, Social Media and Satellite Sequences
In Multimediaeval Benchmark 2019. CEUR Workshop Proceedings, 2019.Status: Published
Multi-Modal Machine Learning for Flood Detection in News, Social Media and Satellite Sequences
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | Multimediaeval Benchmark 2019 |
Date Published | 10/2019 |
Publisher | CEUR Workshop Proceedings |
One-Dimensional Convolutional Neural Networks on Motor Activity Measurements in Detection of Depression
In Proceedings of the 4th International Workshop on Multimedia for Personal Health & Health Care - HealthMedia '19. New York, NY, USA: ACM Press, 2019.Status: Published
One-Dimensional Convolutional Neural Networks on Motor Activity Measurements in Detection of Depression
Nowadays, it has become possible to measure different human activities using wearable devices. Besides measuring the number of daily steps or calories burned, these datasets have much more potential since different activity levels are also collected. Such data would be helpful in the field of psychology because it can relate to various mental health issues such as changes in mood and stress. In this paper, we present a machine learning approach to detect depression using a dataset with motor activity recordings of one group of people with depression and one group without, i.e., the condition group includes 23 unipolar and bipolar persons, and the control group includes 32 persons without depression. We use convolutional neural networks to classify the depressed and nondepressed patients. Moreover, different levels of depression were classified. Finally, we trained a model that predicts MontgomeryÅsberg Depression Rating Scale scores. We achieved an average F1-score of 0.70 for detecting the control and condition groups. The mean squared error for score prediction was approximately 4.0.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | Proceedings of the 4th International Workshop on Multimedia for Personal Health & Health Care - HealthMedia '19 |
Pagination | 9-15 |
Date Published | 10/2019 |
Publisher | ACM Press |
Place Published | New York, NY, USA |
ISBN Number | 9781450369145 |
URL | http://dl.acm.org/citation.cfm?doid=3347444http://dl.acm.org/citation.cf... |
DOI | 10.1145/334744410.1145/3347444.3356238 |
Overview of ImageCLEFlifelog 2019: solve my life puzzle and lifelog moment retrieval
In CLEF2019 Working Notes. Vol. 2380. CEUR Workshop Proceedings (CEUR-WS.org), 2019.Status: Published
Overview of ImageCLEFlifelog 2019: solve my life puzzle and lifelog moment retrieval
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | CLEF2019 Working Notes |
Volume | 2380 |
Pagination | 09–12 |
Publisher | CEUR Workshop Proceedings (CEUR-WS.org) |
Performance of Data Enhancements and Training Optimization for Neural Network – A Polyp Detection Case Study
In IEEE CBMS International Symposium on Computer-Based Medical Systems (CBMS). IEEE, 2019.Status: Published
Performance of Data Enhancements and Training Optimization for Neural Network – A Polyp Detection Case Study
Deep learning using neural networks is becoming more and more popular. It is frequently used in areas like video analysis, image retrieval, traffic forecast and speech recognition. In this respect, the learning and training process usually requires a lot of data. However, in many areas, data is scarce which is definitely the case in our medical application scenario, i.e., polyp detection in the gastrointestinal tract. Here, colorectal cancer is on the list of most common cancer types, and often, the cancer arises from benign, adenomatous polyps containing dysplastic cells. Detection and removal of polyps can therefore prevent the development of cancer. Due to high cost, time consumption, patient discomfort and in-accuracy of existing procedures, researchers have started to explore systems for automatic polyp detection to assist and automate current examination procedures. Following the current gained traction for neural networks, and the typical lack of medical data, we explore how data enhancements affect the training and evaluation of the networks in terms of polyp detection accuracy and particularly if it can be used to increase the detection rate. We also experiment with how various training techniques can be used to increase performance. Our experimental results show how data enhancement and training optimization can be used to increase different aspects of the performance, but we also point out mechanisms that have no and even a negative effect.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | IEEE CBMS International Symposium on Computer-Based Medical Systems (CBMS) |
Publisher | IEEE |
Predicting Peek Readiness-to-Train of Soccer Players Using Long Short-Term Memory Recurrent Neural Networks
In 2019 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 2019.Status: Published
Predicting Peek Readiness-to-Train of Soccer Players Using Long Short-Term Memory Recurrent Neural Networks
We are witnessing the emergence of a myriad of hardware and software systems that quantifies sport and physical activities. These are frequently touted as game changers and important for future sport developments. The vast amount of generated data is often visualized in graphs and dashboards, for use by coaches and other sports professionals to make decisions on training and match strategies. Modern machine- learning methods have the potential to further fuel this process by deriving useful insights that are not easily observable in the raw data streams.
This paper tackles the problem of deriving peaks in soccer players’ ability to perform from subjective self-reported wellness data collected using the PMSys system. For this, we train a long short-term memory recurrent neural network model using data from two professional Norwegian soccer teams. We show that our model can predict performance peaks in most scenarios with a precision and recall of at least 90%. Equipped with such insight, coaches and trainers can better plan individual and team training sessions, and perhaps avoid over training and injuries.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 International Conference on Content-Based Multimedia Indexing (CBMI) |
Publisher | IEEE |
DOI | 10.1109/CBMI.2019.8877406 |
Predicting Sperm Motility and Morphology using Deep Learning and Handcrafted Features
In MediaEval. ceur ws org, 2019.Status: Published
Predicting Sperm Motility and Morphology using Deep Learning and Handcrafted Features
This paper presents the approach proposed by the organizer team (SimulaMet) for MediaEval 2019 Multimedia for Medicine: The Medico Task. The approach uses a data preparation method which is based on global features extracted from multiple frames within each video and then combines this with information about the patient in order to create a compressed representation of each video. The goal is to create a less hardware expensive data representation that still retains the temporal information of the video and related patient data. Overall, the results need some improvement before being a viable option for clinical use.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | MediaEval |
Publisher | ceur ws org |
Real-time Analysis of Physical Performance Parameters in Elite Soccer
In 2019 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 2019.Status: Published
Real-time Analysis of Physical Performance Parameters in Elite Soccer
Technology is having vast impact on the sports in- dustry, and in particular soccer. All over the world, soccer teams are adapting digital information systems to quantify performance metrics. The goal is to assess strengths and weaknesses of indi- vidual players, training regimes, and play strategies; to improve performance and win games. However, most existing methods rely on post-game analytic. This limits coaches to review games in retrospect without any means to do changes during sessions. In collaboration with an elite soccer club, we have developed Metrix which is a computerized toolkit for coaches to perform real- time monitoring and analysis of the players’ performance. Using sensor technology to track movement, performance parameters are instantly available to coaches through a mobile phone client. Metrix provides coaches with a toolkit to individualize training load to different playing positions on the field, or to the player himself. Our results show that Metrix is able to quantify player performance and propagate it to coaches in real-time during a match or practice, i.e., latency is below 100 ms on the field. In our initial user evaluation, the coaches express that this is a valuable asset in day-to-day work.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 International Conference on Content-Based Multimedia Indexing (CBMI) |
Publisher | IEEE |
DOI | 10.1109/CBMI.2019.8877422 |
ResUNet++: An Advanced Architecture for Medical Image Segmentation
In 2019 IEEE International Symposium on Multimedia (ISM). San Diego, California, USA: IEEE, 2019.Status: Published
ResUNet++: An Advanced Architecture for Medical Image Segmentation
Accurate computer-aided polyp detection and segmentation during colonoscopy examinations can help endoscopists resect abnormal tissue and thereby decrease chances of polyps growing into cancer. Towards developing a fully automated model for pixel-wise polyp segmentation, we propose ResUNet++, which is an improved ResUNet architecture for colonoscopic image segmentation. Our experimental evaluations show that the suggested architecture produces good segmentation results on publicly available datasets. Furthermore, ResUNet++ significantly outperforms U-Net and ResUNet, two key state-of-the-art deep learning architectures, by achieving high evaluation scores with a dice coefficient of 81.33%, and a mean Intersection over Union (mIoU) of 79.27% for the Kvasir-SEG dataset and a dice coefficient of 79.55%, and a mIoU of 79.62% with CVC-612 dataset.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 IEEE International Symposium on Multimedia (ISM) |
Publisher | IEEE |
Place Published | San Diego, California, USA |
Keywords | colonoscopy, deep learning, health informatics, Medical image segmentation, Polyp segmentation, Semantic segmentation |
Saga: An Open Source Platform for Training Machine Learning Models and Community-driven Sharing of Techniques
In International Conference on Content-Based Multimedia Indexing (CBMI 2019). IEEE, 2019.Status: Published
Saga: An Open Source Platform for Training Machine Learning Models and Community-driven Sharing of Techniques
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | International Conference on Content-Based Multimedia Indexing (CBMI 2019) |
Pagination | 1-4 |
Publisher | IEEE |
DOI | 10.1109/CBMI.2019.8877455 |
Semantic Analysis of Soccer News for Automatic Game Event Classification
In 2019 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 2019.Status: Published
Semantic Analysis of Soccer News for Automatic Game Event Classification
We are today overwhelmed with information, of which an important part is news. Sports news, in particular, has become very popular, where soccer makes up a big part of this coverage. For sports fans, it can be a time consuming and tedious to keep up with the news that they really care about. In this paper, we present different machine learning methods applied to soccer news from a Norwegian newspaper and a TV station's news site to summarize the content in a short and digestible manner. We present a system to collect, index, label, analyze, and present the collected news articles based on the content. We perform a thorough comparison between deep learning and traditional machine learning algorithms on text classification. Furthermore, we present a dataset of soccer news which was collected from two different Norwegian news sites and shared online.
Afilliation | Machine Learning |
Project(s) | Simula Metropolitan Center for Digital Engineering, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 International Conference on Content-Based Multimedia Indexing (CBMI) |
Publisher | IEEE |
Stacked dense optical flows and dropout layers to predict sperm motility and morphology
In MediaEval 2019, 27-29 October 2019, Sophia Antipolis, France, 2019.Status: Published
Stacked dense optical flows and dropout layers to predict sperm motility and morphology
In this paper, we analyse two deep learning methods to predict sperm motility and sperm morphology from sperm videos. We use two different inputs: stacked pure frames of videos and dense optical flows of video frames. To solve this regression task of predicting motility and morphology, stacked dense optical flows and extracted original frames from sperm videos were used with the modified state of the art convolution neural networks. For modifications of the selected models, we have introduced an additional multi-layer perceptron to overcome the problem of over-fitting. The method which had an additional multi-layer perceptron with dropout layers, shows the best results when the inputs consist of both dense optical flows and an original frame of videos.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | MediaEval 2019, 27-29 October 2019, Sophia Antipolis, France |
Date Published | 10/2019 |
Summarizing E-Sports Matches and Tournaments: The Example of Counter-Strike: Global Offensive
In International Workshop on Immersive Mixed and Virtual Environment Systems (MMVE). ACM, 2019.Status: Published
Summarizing E-Sports Matches and Tournaments: The Example of Counter-Strike: Global Offensive
That video and computer games have reached the masses is a well known fact. Furthermore, game streaming and watching other people play video games is another phenomenon that has outgrown its small beginning by far, and game streams, be it live or recorded, are today viewed by millions. E-sports is the result of organized leagues and tournaments in which players can compete in controlled environments and viewers can experience the matches, discuss and criticize, just like in physical sports. However, as traditional sports, e-sports matches may be long and contain less interesting parts, introducing the challenge of producing well directed summaries and highlights. In this paper, we describe our efforts to approach the game streaming and e-sports phenomena from a multimedia research point of view. We focus on the challenge of summarizing matches from specific relevant game, Counter-Strike: Global Offensive (CS:GO). We survey related work, describe the rules and structure of the game and identify the main challenges for summarizing e-sports matches. With this contribution, we aim to foster multimedia research in the area of e-sports and game streaming.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | International Workshop on Immersive Mixed and Virtual Environment Systems (MMVE) |
Publisher | ACM |
THREAT: A Large Annotated Corpus for Detection of Violent Threats
In 2019 International Conference on Content-Based Multimedia Indexing (CBMI). IEEE, 2019.Status: Published
THREAT: A Large Annotated Corpus for Detection of Violent Threats
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | 2019 International Conference on Content-Based Multimedia Indexing (CBMI) |
Pagination | 1–5 |
Publisher | IEEE |
Unsupervised Preprocessing to Improve Generalisation for Medical Image Classification
In IEEE 13th International Symposium on Medical Information and Communication Technology (ISMICT). IEEE, 2019.Status: Published
Unsupervised Preprocessing to Improve Generalisation for Medical Image Classification
Automated disease detection in videos and images from the gastrointestinal (GI) tract has received much attention in the last years. However, the quality of image data is often reduced due to overlays of text and positional data.
In this paper, we present different methods of preprocessing such images and we describe our approach to GI disease classification for the Kvasir v2 dataset.
We propose multiple approaches to inpaint problematic areas in the images to improve the anomaly classification, and we discuss the effect that such preprocessing does to the input data.
In short, our experiments show that the proposed methods improve the Matthews correlation coefficient by approximately 7% in terms of better classification of GI anomalies.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems, Simula Metropolitan Center for Digital Engineering |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | IEEE 13th International Symposium on Medical Information and Communication Technology (ISMICT) |
Publisher | IEEE |
DOI | 10.1109/ISMICT.2019.8743979 |
Using 2D and 3D Convolutional Neural Networks to Predict Semen Quality
In MediaEval. CEUR Workshop Proceedings, 2019.Status: Published
Using 2D and 3D Convolutional Neural Networks to Predict Semen Quality
In this paper, we present the approach of team Jmag to solve this year's Medico Multimedia Task as part of the MediaEval 2019 Benchmark. This year, the task focuses on automatically determining quality characteristics of human sperm through the analysis of microscopic videos of human semen and associated patient data. Our approach is based on deep convolutional neural networks (CNNs) of varying sizes and dimensions. Here, we aim to analyze both the spatial and temporal information present in the videos. The results show that the method holds promise for predicting the motility of sperm, but predicting morphology appears to be more difficult.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | MediaEval |
Publisher | CEUR Workshop Proceedings |
Using Deep Learning to Predict Motility and Morphology of Human Sperm
In MediaEval 2019. CEUR Workshop Proceedings, 2019.Status: Published
Using Deep Learning to Predict Motility and Morphology of Human Sperm
In the Medico Task 2019, the main focus is to predict sperm quality based on videos and other related data. In this paper, we present the approach of team LesCats which is based on deep convolution neural networks, where we experiment with different data preprocessing methods to predict the morphology and motility of human sperm. The achieved results show that deep learning is a promising method for human sperm analysis. Out best method achieves a mean absolute error of 8.962 for the motility task and a mean absolute error of 5.303 for the morphology task.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | MediaEval 2019 |
Publisher | CEUR Workshop Proceedings |
Using Mr. MAPP for Lower Limb Phantom Pain Management
In Proceedings of the 27th ACM International Conference on Multimedia. ACM, 2019.Status: Published
Using Mr. MAPP for Lower Limb Phantom Pain Management
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | Proceedings of the 27th ACM International Conference on Multimedia |
Pagination | 1071–1075 |
Publisher | ACM |
VISEM: a multimodal video dataset of human spermatozoa
In Proceedings of the 10th ACM Multimedia Systems Conference. ACM, 2019.Status: Published
VISEM: a multimodal video dataset of human spermatozoa
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2019 |
Conference Name | Proceedings of the 10th ACM Multimedia Systems Conference |
Pagination | 261–266 |
Publisher | ACM |
Public outreach
An interview with Géraldine Morin
In ACM SIGMultimedia Records. Vol. 10. ACM, 2019.Status: Published
An interview with Géraldine Morin
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Public outreach |
Year of Publication | 2019 |
Secondary Title | ACM SIGMultimedia Records |
Volume | 10 |
Pagination | 4 |
Publisher | ACM |
Report from the SIGMM Emerging Leaders Symposium 2018
In ACM SIGMultimedia Records. Vol. 10. ACM, 2019.Status: Published
Report from the SIGMM Emerging Leaders Symposium 2018
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Public outreach |
Year of Publication | 2019 |
Secondary Title | ACM SIGMultimedia Records |
Volume | 10 |
Pagination | 2 |
Publisher | ACM |
Journal Article
Artificial intelligence as a tool in predicting sperm motility and morphology: P-116
Human Reproduction 34 (2019).Status: Published
Artificial intelligence as a tool in predicting sperm motility and morphology: P-116
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Human Reproduction |
Volume | 34 |
Publisher | Oxford Academic |
Artificial intelligence predicts sperm motility from sperm fatty acids: P-120
Human Reproduction 34 (2019).Status: Published
Artificial intelligence predicts sperm motility from sperm fatty acids: P-120
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Human Reproduction |
Volume | 34 |
Publisher | Oxford Academic |
Automatic detection of passable roads after floods in remote sensed and social media data
Signal Processing: Image Communication 74 (2019): 110-118.Status: Published
Automatic detection of passable roads after floods in remote sensed and social media data
This paper addresses the problem of floods classification and floods aftermath detection based onboth social media and satellite imagery. Automatic detection of disasters such as floods is still a very challenging task. The focus lies on identifying passable routes or roads during floods. Two novel solutions are presented, which were developed for two corresponding tasks at the MediaEval 2018 benchmarking challenge. The tasks are (i) identification of images providing evidence for road passability and (ii) differentiation and detection of passable and non-passable roads in images from two complementary sources of information. For the first challenge, we mainly rely on object and scene-level features extracted through multiple deep models pre-trained on the ImageNet and Places datasets. The object and scene-level features are then combined using early, late and double fusion techniques. To identify whether or not it is possible for a vehicle to pass a road in satellite images, we rely on Convolutional Neural Networks and a transfer learning-based classification approach. The evaluation of the proposed methods is carried out on the large-scale datasets provided for the benchmark competition. The results demonstrate significant improvement in the performance over the recent state-of-art approaches.
Afilliation | Communication Systems |
Project(s) | UMOD: Understanding and Monitoring Digital Wildfires, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Signal Processing: Image Communication |
Volume | 74 |
Pagination | 110-118 |
Publisher | Elsevier |
Keywords | convolutional neural networks, Flood detection, Multimedia Indexing and Retrieval, Natural Disasters, Satellite Imagery, Social Media |
DOI | 10.1016/j.image.2019.02.002 |
Bleeding detection in wireless capsule endoscopy videos—Color versus texture features
Journal of applied clinical medical physics 20, no. 8 (2019): 141-154.Status: Published
Bleeding detection in wireless capsule endoscopy videos—Color versus texture features
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Journal of applied clinical medical physics |
Volume | 20 |
Issue | 8 |
Pagination | 141-154 |
Publisher | Wiley Online Library |
Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018)
ITE Transactions on Media Technology and Applications 7 (2019): 46-59.Status: Published
Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018)
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | ITE Transactions on Media Technology and Applications |
Volume | 7 |
Number | 2 |
Pagination | 46–59 |
Publisher | The Institute of Image Information and Television Engineers |
Deep Learning for Automatic Generation of Endoscopy Reports
Gastrointestinal Endoscopy 89, no. 6 (2019).Status: Published
Deep Learning for Automatic Generation of Endoscopy Reports
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Gastrointestinal Endoscopy |
Volume | 89 |
Issue | 6 |
Date Published | 06/2019 |
Publisher | Elsevier |
Place Published | Gastrointestinal Endoscopy |
DOI | 10.1016/j.gie.2019.04.053 |
Efficient Live and On-Demand Tiled HEVC 360 VR Video Streaming
International Journal of Semantic Computing 13, no. 3 (2019): 367-391.Status: Published
Efficient Live and On-Demand Tiled HEVC 360 VR Video Streaming
360 panorama video displayed through Virtual reality (VR) glasses or large screens o®ers immersive user experiences, but as such technology becomes commonplace, the need for e±cient streaming methods of such high-bitrate videos arises. In this respect, the attention that 360panorama video has received lately is huge. Many methods have already been proposed, and in this paper, we shed more light on the di®erent trade-o®s in order to save bandwidth while preserving the video quality in the user's ̄eld-of-view (FoV). Using 360 VR content delivered to a Gear VR head-mounted display with a Samsung Galaxy S7 and to a Huawei Q22 set-top- box, we have tested various tiling schemes analyzing the tile layout, the tiling and encoding overheads, mechanisms for faster quality switching beyond the DASH segment boundaries and quality selection con ̄gurations. In this paper, we present an e±cient end-to-end design and real-world implementation of such a 360 streaming system. Furthermore, in addition to researching an on-demand system, we also go beyond the existing on-demand solutions and present a live streaming system which strikes a trade-o® between bandwidth usage and the video quality in the user's FoV. We have created an architecture that combines RTP and DASH, and our system multiplexes a single HEVC hardware decoder to provide faster quality switching than at the traditional GOP boundaries. We demonstrate the performance and illustrate the trade-o®s through real-world experiments where we can report comparable bandwidth savings to existing on-demand approaches, but with faster quality switches when the FoV changes.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | International Journal of Semantic Computing |
Volume | 13 |
Issue | 3 |
Number | 3 |
Pagination | 367-391 |
Publisher | World Scientific |
Flexible device compositions and dynamic resource sharing in PCIe interconnected clusters using Device Lending
Cluster Computing 22, no. 86 (2019): 1-24.Status: Published
Flexible device compositions and dynamic resource sharing in PCIe interconnected clusters using Device Lending
Modern workloads often exceed the processing and I/O capabilities provided by resource virtualization, requiring direct access to the physical hardware in order to reduce latency and computing overhead. For computers interconnected in a cluster, access to remote hardware resources often requires facilitation both in hardware and specialized drivers with virtualization support. This limits the availability of resources to specific devices and drivers that are supported by the virtualization technology being used, as well as what the interconnection technology supports. For PCI Express (PCIe) clusters, we have previously proposed Device Lending as a solution for enabling direct low latency access to remote devices. The method has extremely low computing overhead and does not require any application- or device-specific distribution mechanisms. Any PCIe device, such as network cards disks, and GPUs, can easily be shared among the connected hosts. In this work, we have extended our solution with support for a virtual machine (VM) hypervisor. Physical remote devices can be “passed through” to VM guests, enabling direct access to physical resources while still retaining the flexibility of virtualization. Additionally, we have also implemented multi-device support, enabling shortest-path peer-to-peer transfers between remote devices residing in different hosts. Our experimental results prove that multiple remote devices can be used, achieving bandwidth and latency close to native PCIe, and without requiring any additional support in device drivers. I/O intensive workloads run seamlessly using both local and remote resources. With our added VM and multi-device support, Device Lending offers highly customizable configurations of remote devices that can be dynamically reassigned and shared to optimize resource utilization, thus enabling a flexible composable I/O infrastructure for VMs as well as bare-metal machines.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, LADIO: Live Action Data Input/Output, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Cluster Computing |
Volume | 22 |
Issue | 86 |
Pagination | 1-24 |
Date Published | 09/2019 |
Publisher | Springer |
ISSN | 1573-7543 |
URL | https://link.springer.com/article/10.1007/s10586-019-02988-0 |
DOI | 10.1007/s10586-019-02988-0 |
Machine Learning-Based Analysis of Sperm Videos and Participant Data for Male Fertility Prediction
Nature Scientific Reports 9, no. 1 (2019).Status: Published
Machine Learning-Based Analysis of Sperm Videos and Participant Data for Male Fertility Prediction
Methods for automatic analysis of clinical data are usually targeted towards a specific modality and do not make use of all relevant data available. In the field of male human reproduction, clinical and biological data
are not used to its fullest potential. Manual evaluation of a semen sample using a microscope is time-consuming and requires extensive training. Furthermore, the validity of manual semen analysis has been questioned due to limited reproducibility, and often high inter-personnel variation. The existing computer-aided sperm analyzer systems are not recommended for routine clinical use due to methodological challenges caused by the consistency of the semen sample. Thus, there is a need for an improved methodology.
We use modern and classical machine learning techniques together with a dataset consisting of 85 videos of human semen samples and related participant data to automatically predict sperm motility. Used techniques include simple linear regression and more sophisticated methods using convolutional neural networks. Our results indicate that sperm motility prediction based on deep learning using sperm motility videos is rapid to perform and consistent. The algorithms performed worse when participant data was added. In conclusion, machine learning-based automatic analysis may become a valuable tool in male infertility investigation and research.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Nature Scientific Reports |
Volume | 9 |
Issue | 1 |
Number | 16770 |
Publisher | Springer Nature |
Maskinlæringssystemer for gastrointestinale endoskopier
Best Practice Nordic - Gastroenterologi (2019).Status: Published
Maskinlæringssystemer for gastrointestinale endoskopier
Assistert diagnostikk med hjelp av kunstig intelligens (KI) har vært etterspurt lenge og kan bli et viktig hjelpemiddel innen medisin, godt hjulpet av den raske utviklingen innen maskinvare. Denne har gjort innføringen av slike hjelpemidler mulig på relativt kort sikt. Sikrere påvisning og klassifisering av funn og lesjoner innen radiologi og endoskopi er i ferd med å bli et viktig forskningsområde innen KI, og det fokuseres spesielt på maskinlæring. Imidlertid krever vellykket utvikling et komplett system som kan brukes i sanntid i daglig praksis, og som begrenser seg til utvikling av algoritmer. Det kreves også store randomiserte studier for å fastslå om kvaliteten og påliteligheten til systemene er god. Vi deler i denne artikkelen våre erfaringer fra utviklingen av et system for gastrointestinale endoskopier og belyser viktige utfordringer for å skape en effektiv digital assistent.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Best Practice Nordic - Gastroenterologi |
Date Published | 07/2019 |
Publisher | BestPracticeNordic |
URL | https://bestprac.no/maskinlaeringssystemer-for-gastrointestinale-endosko... |
Natural disasters detection in social media and satellite imagery: a survey
Multimedia Tools and Applications 78, no. 22 (2019): 31267-31302.Status: Published
Natural disasters detection in social media and satellite imagery: a survey
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Multimedia Tools and Applications |
Volume | 78 |
Issue | 22 |
Pagination | 31267 - 31302 |
Date Published | Jan-11-2019 |
Publisher | Springer |
ISSN | 1380-7501 |
DOI | 10.1007/s11042-019-07942-1 |
Social media and satellites: Disaster event detection, linking and summarization
Multimedia Tools and Applications 78 (2019): 2837-2875.Status: Published
Social media and satellites: Disaster event detection, linking and summarization
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Multimedia Tools and Applications |
Volume | 78 |
Number | 3 |
Pagination | 2837–2875 |
Publisher | Springer |
Place Published | Netherlands |
The detecting and predicting mood transitions in bipolar disorder study protocol (a sub-study of the Introducing Mental Health through Adaptive Technology (INTROMAT) project)
Bipolar Disorders 21 (2019): 68-69.Status: Published
The detecting and predicting mood transitions in bipolar disorder study protocol (a sub-study of the Introducing Mental Health through Adaptive Technology (INTROMAT) project)
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Bipolar Disorders |
Volume | 21 |
Pagination | 68–69 |
Publisher | Wiley |
User-adaptive models for activity and emotion recognition using deep transfer learning and data augmentation
User Modeling and User-Adapted Interaction (2019): 1-29.Status: Published
User-adaptive models for activity and emotion recognition using deep transfer learning and data augmentation
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | User Modeling and User-Adapted Interaction |
Pagination | 1–29 |
Publisher | Springer Netherlands |
映像情報メディア学会英語論文誌
映像情報メディア学会英語論文誌 7 (2019): 46-59.Status: Submitted
映像情報メディア学会英語論文誌
Afilliation | Machine Learning |
Project(s) | No Simula project |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | 映像情報メディア学会英語論文誌 |
Volume | 7 |
Number | 2 |
Pagination | 46–59 |
Publisher | {一般社団法人 映像情報メディア学会 |
Place Published | Japan |
Book Chapter
Challenges for Multimedia Research in E-Sports Using Counter-Strike Global Offensive as an Example
In Savegame, 197-206. Vol. 4. Wiesbaden: Springer Fachmedien Wiesbaden, 2019.Status: Published
Challenges for Multimedia Research in E-Sports Using Counter-Strike Global Offensive as an Example
That video and computer games have reached the masses is a well-known fact. However, game streaming and, therefore, watching other people play videogames has also outgrown its humble beginnings by far. Game streams, be it live or recorded, are viewed by millions. Many of the streams are broadcasting competitive multiplayer games. This is called e-sports and it is very similar to sports broadcasting. E-sports is organized in leagues and tournaments in which players can compete in controlled environments and viewers can experience the matches, discuss and criticize just like in physical sports. In this paper, we look into the challenges for computer science in general and multimedia research in particular. The multimedia research community has done a lot of work on video streaming, broadcasting and analyzing the audience, but has missed the opportunity to investigate e-sports in detail. We focus on one particular game we deem representative for e-sports, Counter-Strike: Global Offensive, and investigate how the audience consumes game streams from competitive tournaments.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Book Chapter |
Year of Publication | 2019 |
Book Title | Savegame |
Volume | 4 |
Pagination | 197 - 206 |
Publisher | Springer Fachmedien Wiesbaden |
Place Published | Wiesbaden |
ISBN Number | 978-3-658-27394-1 |
ISBN | 2524-3241 |
URL | http://link.springer.com/10.1007/978-3-658-27395-8_13 |
DOI | 10.1007/978-3-658-27395-8_13 |
Image Retrieval Evaluation in Specific Domains
In Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF, 275-305. Vol. INRE, volume 41. Springer, 2019.Status: Published
Image Retrieval Evaluation in Specific Domains
Image retrieval was, and still is, a hot topic in research. It comes with many challenges that changed over the years with the emergence of more advanced methods for analysis and enormous growth of images created, shared and consumed. This chapter gives an overview of domain-specific image retrieval evaluation approaches, which were part of the ImageCLEF evaluation campaign . Specifically, the robot vision, photo retrieval, scalable image annotation and lifelogging tasks are presented. The ImageCLEF medical activity is described in a separate chapter in this volume. Some of the presented tasks have been available for several years, whereas others are quite new (like lifelogging). This mix of new and old topics has been chosen to give the reader an idea about the development and trends within image retrieval. For each of the tasks, the datasets, participants, techniques used and lessons learned are presented and discussed leading to a comprehensive summary.
Afilliation | Machine Learning |
Project(s) | Simula Metropolitan Center for Digital Engineering, Department of Holistic Systems |
Publication Type | Book Chapter |
Year of Publication | 2019 |
Book Title | Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF |
Volume | INRE, volume 41 |
Pagination | 275-305 |
Publisher | Springer |
DOI | 10.1007/978-3-030-22948-1_12 |
Poster
Efficient Processing of Medical Videos in a Multi-auditory Environment Using Gpu Lending
NVIDIA's GPU Technology Conference (GTC), 2019.Status: Published
Efficient Processing of Medical Videos in a Multi-auditory Environment Using Gpu Lending
Afilliation | Software Engineering |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Poster |
Year of Publication | 2019 |
Place Published | NVIDIA's GPU Technology Conference (GTC) |
Miscellaneous
Kunstig intelligens for endoskopi – Automatisk deteksjon av lesjoner i sanntid
NGF Nytt, Vol. 26, No 1, March 2019, p. 34: Norsk Gastroenterologisk Forening, 2019.Status: Published
Kunstig intelligens for endoskopi – Automatisk deteksjon av lesjoner i sanntid
BAKGRUNN: I krysningspunktet mellom matematikk, informatikk og statistikk finner vi den vitenskapelige disiplinen kunstig intelligens (KI). Sammen med de siste års eksplosive utvikling innen teknologi har KI muliggjort nye algoritmer, modeller og systemer for maskinassistert diagnostikk. Resultater fra KI basert på dype nevrale nettverk har vist spesielt stort potensiale, også for automatisk deteksjon av lesjoner og anatomiske landemerker i gastrointestinaltraktus under endoskopi. Med sensitivitet og spesifisitet for deteksjon av polypper i tykktarm
på over 90% møter slike metoder nødvendige kliniske krav, men mange eksperimenter er utført på begrensede datasett, eller analysert på feilaktig grunnlag grunnet manglende tilgang og forståelse hos informatikere. For å oppnå best mulig resultat er
et interdisiplinært samarbeid mellom klinikere og informatikere
en forutsetning. Informatikerne trenger medisinske innspill for å lage effektive systemer som fungerer ute i klinikken, og klinikerne trenger forståelse av systemet for å kunne stole på resultatet og stille pålitelige diagnoser. En stor utfordring for denne tilliten er
at fremgangsmåten til en KI-algoritme sees på som en svart boks hvor ingen nøyaktig kan dechiffrere hvordan systemet kom frem
til sin konklusjon.
METODE: Vi har gjennom mange år samlet en stor bilde- database fra endoskopier utført ved Bærum Sykehus, Vestre Viken HF. Bildene er gjennomgått og annotert av tre erfarne endoskopører og fordelt på 16 klasser, inkludert normal Z-linje, øsofagitt, normal cøkum, polypper og ulcerøs colitt. Deretter er bildene brukt til å utvikle, trene og teste KI-modeller. Modellene er basert på maskinlæring og dyp læring, en gren innen KI. Med vårt system Mimir, som kombinerer KI med informasjonssøk og
-gjenfinning, søker vi å lage et helhetlig beslutningsstøttesystem for endoskopører. Algoritmene analyserer videoer i sanntid, finner lesjoner, klassifiserer disse og gir skopøren live feedback om funn under undersøkelsen, slik at funnene kan undersøkes nærmere. Mimir presenterer deretter resultatene i egen programvare, og bruker blant annet “heatmaps” til å forklare hvordan konklusjonen er nådd, og er på den måten et bidrag på veien til å forstå hvordan KI-algoritmene fungerer. Videre jobber vi med å videreutvikle Mimirs støtte for automatisk rapportgenerering, med bilder
og standardtekst basert på funn fra undersøkelsen.
RESULTATER: Deteksjon og klassifisering for de 16 gruppene har vist en sensitivitet på 0,939 og en spesifisitet på 0,996. Algoritmene våre klarer å prosessere bildene i hastigheter på mellom 30 - 1000 bilder per sekund, raskt nok til å kjøre deteksjon i sanntid. En prototype av systemet er i samråd med klinikere testet ved å koble til et koloskopisystem ute i klinikken, og kan
nå analysere videoer i sanntid.
KONKLUSJON: Tester av våre system viser at KI kan bli et viktig hjelpemiddel for å bedre oppdage GI-forandringer, og generere automatiske rapporter i løpet av nærmeste fremtid. Dette kan fungere som viktig beslutningsstøtte for endoskopører, og kan brukes i opplæring av nye endoskopører. Den største begrensningen med KI er at vi per i dag ikke vet hvordan systemet kommer frem til sin konklusjon, som kan påvirke i hvor stor grad vi stoler på resultatet. Vi arbeider derfor med et helhetlig system som ikke bare hjelper legen med diagnostikk, men også forklarer hvordan konklusjonen er nådd, samt å generere automatiske rapporter fra undersøkelsen.
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2019 |
Publisher | Norsk Gastroenterologisk Forening |
Place Published | NGF Nytt, Vol. 26, No 1, March 2019, p. 34 |
Journal Article
An interview with Miriam Redi
ACM SIGMultimedia Records 10, no. 1 (2018): 2.Status: Published
An interview with Miriam Redi
Afilliation | Communication Systems, Machine Learning |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | ACM SIGMultimedia Records |
Volume | 10 |
Issue | 1 |
Pagination | 2 - 2 |
Date Published | Nov-04-2019 |
Publisher | ACM |
URL | http://dl.acm.org/citation.cfm?doid=3210241http://dl.acm.org/citation.cf... |
DOI | 10.1145/321024110.1145/3210241.3210243 |
An interview with Prof. Alan Smeaton
ACM SIGMultimedia Records 9, no. 3 (2018): 1.Status: Published
An interview with Prof. Alan Smeaton
Afilliation | Communication Systems, Machine Learning |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | ACM SIGMultimedia Records |
Volume | 9 |
Issue | 3 |
Pagination | 1 - 1 |
Date Published | Sep-01-2018 |
Publisher | ACM |
URL | http://dl.acm.org/citation.cfm?doid=3178422http://dl.acm.org/citation.cf... |
DOI | 10.1145/317842210.1145/3178422.3178423 |
Mental Health Monitoring with Multimodal Sensing and Machine Learning: A Survey
Pervasive and Mobile Computing 51 (2018): 1-26.Status: Published
Mental Health Monitoring with Multimodal Sensing and Machine Learning: A Survey
Personal and ubiquitous sensing technologies such as smartphones have allowed the continuous collection of data in an unobtrusive manner. Machine learning methods have been applied to continuous sensor data to predict user contextual information such as location, mood, physical activity, etc. Recently, there has been a growing interest in leveraging ubiquitous sensing technologies for mental health care applications, thus, allowing the automatic continuous monitoring of different mental conditions such as depression, anxiety, stress, and so on. This paper surveys recent research works in mental health monitoring systems (MHMS) using sensor data and machine learning. We focused on research works about mental disorders/conditions such as: depression, anxiety, bipolar disorder, stress, etc. We propose a classification taxonomy to guide the review of related works and present the overall phases of MHMS. Moreover, research challenges in the field and future opportunities are also discussed.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | Pervasive and Mobile Computing |
Volume | 51 |
Pagination | 1-26 |
Date Published | 12/2018 |
Publisher | Elsevier |
DOI | 10.1016/j.pmcj.2018.09.003 |
Methodology to develop machine learning algorithms to improve performance in gastrointestinal endoscopy
World Journal of Gastroenterology 45, no. 24 (2018): 5057-5062.Status: Published
Methodology to develop machine learning algorithms to improve performance in gastrointestinal endoscopy
Assisted diagnosis using artificial intelligence has been a holy grail in medical research for many years, and recent developments in computer hardware have enabled the narrower area of machine learning to equip clinicians with potentially useful tools for computer assisted diagnosis (CAD) systems. However, training and assessing a computer’s ability to diagnose like a human are complex tasks, and successful outcomes depend on various factors. We have focused our work on gastrointestinal (GI) endoscopy because it is a cornerstone for diagnosis and treatment of diseases of the GI tract. About 2.8 million luminal GI (esophageal, stomach, colorectal) cancers are detected globally every year, and although substantial technical improvements in endoscopes have been made over the last 10-15 years, a major limitation of endoscopic examinations remains operator variation. This translates into a substantial inter-observer variation in the detection and assessment of mucosal lesions, causing among other things an average polyp miss-rate of 20% in the colon and thus the subsequent development of a number of post-colonoscopy colorectal cancers. CAD systems might eliminate this variation and lead to more accurate diagnoses. In this editorial, we point out some of the current challenges in the development of efficient computer-based digital assistants. We give examples of proposed tools using various techniques, identify current challenges, and give suggestions for the development and assessment of future CAD systems.
Afilliation | Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | World Journal of Gastroenterology |
Volume | 45 |
Issue | 24 |
Pagination | 5057-5062 |
Date Published | 12/2018 |
Publisher | Baishideng Publishing Group Inc |
URL | https://www.wjgnet.com/1007-9327/abstract/v24/i45/5057.htm |
DOI | 10.3748/wjg.v24.i45.5057 |
Multimodal analysis of user behavior and browsed content under different image search intents
International Journal of Multimedia Information Retrieval 7 (2018): 29-41.Status: Published
Multimodal analysis of user behavior and browsed content under different image search intents
The motivation or intent of a search for content may vary between users and use-cases. Knowledge and understanding of these underlying objectives may therefore be important in order to return appropriate search results, and studies of user search intent are emerging in information retrieval to understand why a user is searching for a particular type of content. In the context of image search, our work targets automatic recognition of users’ intent in an early stage of a search session. We have designed seven different search scenarios under the intent conditions of finding items, re-finding items and entertainment. Moreover, we have collected facial expressions, physiological responses, eye gaze and implicit user interactions from 51 participants who performed seven different search tasks on a custom-built image retrieval platform, and we have analyzed the users’ spontaneous and explicit reactions under different intent conditions. Finally, we trained different machine learning models to predict users’ search intent from the visual content of the visited images, the user interactions and the spontaneous responses. Our experimental results show that after fusing the visual and user interaction features, our system achieved the F-1 score of 0.722 for classifying three classes in a user-independent cross-validation. Eye gaze and implicit user interactions, including mouse movements and keystrokes are the most informative features for intent recognition. In summary, the most promising results are obtained by modalities that can be captured unobtrusively and online, and the results therefore demonstrate the potential of including intent-based methods in multimedia retrieval platforms.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | International Journal of Multimedia Information Retrieval |
Volume | 7 |
Pagination | 29 - 41 |
Date Published | Jan-03-2018 |
Publisher | Springer |
ISSN | 2192-6611 |
URL | http://link.springer.com/10.1007/s13735-018-0150-6http://link.springer.c... |
DOI | 10.1007/s13735-018-0150-6 |
Sharing and reproducibility in ACM SIGMM
ACM SIGMultimedia Records 10 (2018): 1.Status: Published
Sharing and reproducibility in ACM SIGMM
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | ACM SIGMultimedia Records |
Volume | 10 |
Number | 2 |
Pagination | 1 |
Publisher | ACM |
URL | https://dl.acm.org/citation.cfm?id=3264707 |
DOI | 10.1145/3264706.3264707 |
Social Media and Satellites. Disaster event detection, linking and summarization
Multimedia Tools and Applications 78, no. 3 (2018): 2837-2875.Status: Published
Social Media and Satellites. Disaster event detection, linking and summarization
Being able to automatically link social media and satellite imagery holds large opportunities for research, with a potentially considerable impact on society. The possibility of integrating different information sources opens in fact to new scenarios where the wide coverage of satellite imaging can be used as a collector of the fine-grained details provided by the social media. Remote-sensed data and social media data can well complement each other, integrating the wide perspective provided by the satellite view with the information collected locally, being it textual, audio, or visual. Among the possible applications, natural disasters are certainly one of the most interesting scenarios, where global and local perspectives are needed at the same time.
In this paper, we present a system called JORD that is able to autonomously collect social media data (including the text analysis in local languages) about technological and environmental disasters, and link it automatically to remote-sensed data. Moreover, in order to ensure the quality of retrieved information, JORD is equipped with a hierarchical filtering mechanism relying on the temporal information and the content analysis of retrieved multimedia data.
To show the capabilities of the system, we present a large number of disaster events detected by the system, and we evaluate both the quality of the provided information about the events and the usefulness of JORD from potential users viewpoint, using crowdsourcing.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | Multimedia Tools and Applications |
Volume | 78 |
Issue | 3 |
Pagination | 2837–2875 |
Publisher | Springer |
Place Published | US |
Keywords | Event Detection, Information retrieval, Natural Disaster, Social Media |
DOI | 10.1007/s11042-018-5982-9 |
Top-Down Saliency Detection Driven by Visual Classification
Computer Vision and Image Understanding 172 (2018): 67-76.Status: Published
Top-Down Saliency Detection Driven by Visual Classification
This paper presents an approach for saliency detection able to emulate the integration of the top-down (task-controlled) and bottom-up (sensory information) processes involved in human visual attention. In particular, we first learn how to generate saliency when a specific visual task has to be accomplished. Afterwards, we investigate if and to what extent
the learned saliency maps can support visual classification in nontrivial cases. To achieve this, we propose SalClass-
Net, a CNN framework consisting of two networks jointly trained: a) the first one computing top-down saliency maps
from input images, and b) the second one exploiting the computed saliency maps for visual classification.
To test our approach, we collected a dataset of eye-gaze maps, using a Tobii T60 eye tracker, by asking several subjects
to look at images from the Stanford Dogs dataset, with the objective of distinguishing dog breeds.
Performance analysis on our dataset and other saliency benchmarking datasets, such as POET, showed that Sal-
ClassNet outperforms state-of-the-art saliency detectors, such as SalNet and SALICON. Finally, we also analyzed
the performance of SalClassNet in a fine-grained recognition task and found out that it yields enhanced classification
accuracy compared to Inception and VGG-19 classifiers. The achieved results, thus, demonstrate that 1) condition-
ing saliency detectors with object classes reaches state-of-the-art performance, and 2) explicitly providing top-down
saliency maps to visual classifiers enhances accuracy.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | Computer Vision and Image Understanding |
Volume | 172 |
Pagination | 67-76 |
Publisher | Elsevier |
DOI | 10.1016/j.cviu.2018.03.005 |
Talk, keynote
Automatic Detection of Angiectasia: Evaluation of Deep Learning and Handcrafted Approaches
In IEEE Conference on Biomedical and Health Informatics (BHI) 2018, 2018.Status: Published
Automatic Detection of Angiectasia: Evaluation of Deep Learning and Handcrafted Approaches
Angiectasia, formerly called angiodysplasia, is one of the most frequent vascular lesions and often the cause of gastrointestinal bleedings. Medical specialists assessing videos of examinations reach a detection performance of 16% for the detection of bleeding to 69% for the detection of angiectasia. In this paper, we present several machine-learning-based approaches for angiectasia detection in wireless video capsule endoscopy images. The most promising results for pixel-wise localization and frame-wise detection are obtained by the proposed deep learning approach using generative adversarial networks (GANs) with a sensitivity of 88% and specificity of 99.9% for pixel-wise localization and a sensitivity of 98% and a specificity of 100% for frame-wise detection, which fits the requirements for automatic angiectasia detection in real clinical settings.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Talk, keynote |
Year of Publication | 2018 |
Location of Talk | IEEE Conference on Biomedical and Health Informatics (BHI) 2018 |
Proceedings, refereed
Automatic Hyperparameter Optimization in Keras for the MediaEval 2018 Medico Multimedia Task
In Working Notes Proceedings of the MediaEval 2018 Workshop. CEUR Workshop Proceedings (CEUR-WS.org), 2018.Status: Published
Automatic Hyperparameter Optimization in Keras for the MediaEval 2018 Medico Multimedia Task
This paper details the approach to the MediaEval 2018 Medico Multimedia Task made by the Rune team. The decided upon approach uses a work-in-progress hyperparameter optimization system called Saga. Saga is a system for creating the best hyperparameter finding in Keras, a popular machine learning framework, using Bayesian optimization and transfer learning. In addition to optimizing the Keras classifier configuration, we try manipulating the dataset by adding extra images in a class lacking in images and splitting a commonly misclassified class into two classes.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Working Notes Proceedings of the MediaEval 2018 Workshop |
Publisher | CEUR Workshop Proceedings (CEUR-WS.org) |
Keywords | automatic hyperparameter optimization, Bayesian optimization, CNN, convolutional neural networks, dataset manipulation, gpyopt, hyperparameter optimization, keras, saga, tensorflow, Transfer Learning |
Challenges and Opportunities within Personal Life Archives
In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval. New York, NY, USA: ACM Press, 2018.Status: Published
Challenges and Opportunities within Personal Life Archives
Nowadays, almost everyone holds some form or other of a personal life archive. Automatically maintaining such an archive is an activity that is becoming increasingly common, however without automatic support the users will quickly be overwhelmed by the volume of data and will miss out on the potential benefits that lifelogs provide. In this paper we give an overview of the current status of lifelog research and propose a concept for exploring these archives. We motivate the need for new methodologies for indexing data, organizing content and supporting information access. Finally we will describe challenges to be addressed and give an overview of initial steps that have to be taken, to address the challenges of organising and searching personal life archives.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval |
Pagination | 335-343 |
Date Published | 07/2018 |
Publisher | ACM Press |
Place Published | New York, NY, USA |
ISBN Number | 9781450350464 |
URL | http://dl.acm.org/citation.cfm?doid=3206025 |
DOI | 10.1145/3206025.3206040 |
Comprehensible Reasoning and Automated Reporting of Medical Examinations Based on Deep Learning Analysis
In Proceedings of the 9th ACM Multimedia Systems Conference. Amsterdam, Netherlands: ACM, 2018.Status: Published
Comprehensible Reasoning and Automated Reporting of Medical Examinations Based on Deep Learning Analysis
Afilliation | Communication Systems |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Proceedings of the 9th ACM Multimedia Systems Conference |
Pagination | 490-493 |
Publisher | ACM |
Place Published | Amsterdam, Netherlands |
ISBN Number | 978-1-4503-5192-8 |
DOI | 10.1145/3204949.3208113 |
Deep Learning and Hand-crafted Feature Based Approaches for Polyp Detection in Medical Videos
In 31st IEEE CBMS International Symposium on Computer-Based Medical Systems. Karlstad, Sweden: IEEE, 2018.Status: Published
Deep Learning and Hand-crafted Feature Based Approaches for Polyp Detection in Medical Videos
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | 31st IEEE CBMS International Symposium on Computer-Based Medical Systems |
Pagination | 381-386 |
Publisher | IEEE |
Place Published | Karlstad, Sweden |
ISSN Number | 2372-9198 |
DOI | 10.1109/CBMS.2018.00073 |
Deep Learning and Handcrafted Feature Based Approaches for Automatic Detection of Angiectasia
In 2018 IEEE Conference on Biomedical and Health Informatics (BHI). IEEE, 2018.Status: Published
Deep Learning and Handcrafted Feature Based Approaches for Automatic Detection of Angiectasia
Angiectasia, formerly called angiodysplasia, is one of the most frequent vascular lesions and often the cause of gastrointestinal bleedings. Medical specialists assessing videos or images of examinations reach a detection performance of 16% for the detection of bleeding to 69% for the detection of angiectasia. This shows that automatic detection to support medical experts can be useful. In this paper, we present several machine learning-based approaches for angiectasia detection in wireless video capsule endoscopy frames. In summary, the most promising results for pixel-wise localization and framewise detection are obtained by the proposed deep learning method using generative adversarial networks (GANs). Using this approach, we achieve a sensitivity of 88% and specificity of 99.9% for pixel-wise localization, and a sensitivity of 98% and a specificity of 100% for frame-wise detection. Thus, the results demonstrate the capability of using deep learning for automatic angiectasia detection in real clinical settings.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | 2018 IEEE Conference on Biomedical and Health Informatics (BHI) |
Pagination | 365-368 |
Publisher | IEEE |
Keywords | Angiectasia, computer aided diagnosis, deep learning, Machine learning, video capsular endoscopy |
DOI | 10.1109/BHI.2018.8333444 |
Deep learning approaches for flood classification and flood aftermath detection
In Working Notes Proceedings of the MediaEval 2018 Workshop. Vol. 2283. Sophia Antipolis, France: CEUR-WS.org, 2018.Status: Published
Deep learning approaches for flood classification and flood aftermath detection
This paper presents the method proposed by team UTAOS for MediaEval 2018 Multimedia Satellite Task: Emergency Response for Flooding Events. In the first challenge, we mainly rely on object and scene level features extracted through multiple deep models pre-trained on the ImageNet and Places datasets. The object and scene-level features are combined using early, late and double fusion techniques achieving an average F1-score of 65.03%, 60.59% and 63.58%, respectively. For the second challenge, we rely on a convolutional neural networks (CNNs) and a transfer learning-based classification approach achieving an average F1-score of 62.30% and 61.02% for run 1 and run 2, respectively.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Working Notes Proceedings of the MediaEval 2018 Workshop |
Volume | 2283 |
Publisher | CEUR-WS.org |
Place Published | Sophia Antipolis, France |
Deep Learning Based Disease Detection Using Domain Specific Transfer Learning
In MediaEval 2018. MediaEval, 2018.Status: Published
Deep Learning Based Disease Detection Using Domain Specific Transfer Learning
In this paper, we present our approach for the Medico Multimedia Task as part of the MediaEval 2018 Benchmark. Our method is based on convolutional neural networks (CNNs), where we compare how fine-tuning, in the context of transfer learning, from different source domains (general versus medical domain) affect classification performance. The preliminary results show that fine-tuning models trained on large and diverse datasets is favorable, even when the model’s source domain has little to no resemblance to the new target.
Afilliation | Machine Learning |
Project(s) | Department of Data Science and Knowledge Discovery |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | MediaEval 2018 |
Publisher | MediaEval |
Keywords | convolutional neural networks, deep learning, Gastrointestinal Disease Detection |
Depresjon: A Motor Activity Database of Depression Episodes in Unipolar and Bipolar Patients
In Proceedings of the 9th ACM Multimedia Systems Conference. New York, NY, USA: ACM Press, 2018.Status: Published
Depresjon: A Motor Activity Database of Depression Episodes in Unipolar and Bipolar Patients
Wearable sensors measuring different parts of people's activity are a common technology nowadays. In research, data collected using these devices also draws attention. Nevertheless, datasets containing sensor data in the field of medicine are rare. Often, data is non-public and only results are published. This makes it hard for other researchers to reproduce and compare results or even collaborate. In this paper we present a unique dataset containing sensor data collected from patients suffering from depression. The dataset contains motor activity recordings of 23 unipolar and bipolar depressed patients and 32 healthy controls. For each patient we provide sensor data over several days of continuous measuring and also some demographic data. The severity of the patients' depressive state was labeled using ratings done by medical experts on the Montgomery-Asberg Depression Rating Scale (MADRS). In this respect, the here presented dataset can be useful to explore and understand the association between depression and motor activity better. By making this dataset available, we invite and enable interested researchers the possibility to tackle this challenging and important societal problem.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Proceedings of the 9th ACM Multimedia Systems Conference |
Pagination | 472-477 |
Date Published | 06/2018 |
Publisher | ACM Press |
Place Published | New York, NY, USA |
ISBN Number | 9781450351928 |
DOI | 10.1145/3204949.3208125 |
Dissecting Deep Neural Networks for Better Medical Image Classification and Classification Understanding
In 31st IEEE CBMS International Symposium on Computer-Based Medical Systems. Karlstad, Sweden: IEEE, 2018.Status: Published
Dissecting Deep Neural Networks for Better Medical Image Classification and Classification Understanding
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | 31st IEEE CBMS International Symposium on Computer-Based Medical Systems |
Publisher | IEEE |
Place Published | Karlstad, Sweden |
ISSN Number | 2372-9198 |
DOI | 10.1109/CBMS.2018.00070 |
Efficient Live and on-Demand Tiled HEVC 360 VR Video Streaming
In 2018 IEEE International Symposium on Multimedia (ISM). Taichung, Taiwan: IEEE, 2018.Status: Published
Efficient Live and on-Demand Tiled HEVC 360 VR Video Streaming
With 360◦ panorama video technology becoming commonplace, the need for efficient streaming methods for such videos arises. We go beyond the existing on-demand solutions and present a live streaming system which strikes a trade-off between bandwidth usage and the video quality in the user’s field-of-view. We have created an architecture that combines RTP and DASH to deliver 360◦ VR content to a Huawei set-top-box and a Samsung Galaxy S7. Our system multiplexes a single HEVC hardware decoder to provide faster quality switching than at the traditional GOP boundaries. We demonstrate the performance and illustrate the trade-offs through real-world experiments where we can report comparable bandwidth savings to existing on-demand approaches, but with faster quality switches when the field-of- view changes.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | 2018 IEEE International Symposium on Multimedia (ISM) |
Pagination | 81-88 |
Date Published | 12/2018 |
Publisher | IEEE |
Place Published | Taichung, Taiwan |
DOI | 10.1109/ISM.2018.00022 |
GameStory Task at MediaEval 2018
In Proceeding of the MediaEval Benchmarking Initiative for Multimedia Evaluation. CEUR Workshop Proceedings, 2018.Status: Published
GameStory Task at MediaEval 2018
That video games have reached the masses is well known. Moreover, game streaming and watching other people play video games is a phenomenon that has outgrown its small beginnings. Game video streams, be it live or recorded, are viewed by millions. E-sports is the result of organized leagues and tournaments in which players can compete in controlled environments and viewers can experience the matches, discuss and criticize, just like in physical sports. In the GameStory task, taking place the first time in 2018, we approach the game streaming and e-sports phenomena from a multimedia research side. We focus on the task of summarizing matches using a specific relevant game, Counter-Strike: Global Offensive, as a case study. With the help of ZNIPE.tv, we provide a data set of high quality data and meta data from competitive tournaments and aim to foster research in the area of e-sports and game streaming.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Proceeding of the MediaEval Benchmarking Initiative for Multimedia Evaluation |
Date Published | 10/2018 |
Publisher | CEUR Workshop Proceedings |
HINDSIGHT: An R-Based Framework Towards Long Short Term Memory (LSTM) Optimization
In Multimedia Systems Conference (MMSys). ACM, 2018.Status: Published
HINDSIGHT: An R-Based Framework Towards Long Short Term Memory (LSTM) Optimization
Afilliation | Communication Systems |
Project(s) | MONROE: Measuring Mobile Broadband Networks in Europe |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Multimedia Systems Conference (MMSys) |
Publisher | ACM |
Medico Multimedia Task at MediaEval 2018
In Working Notes Proceedings of the MediaEval 2018 Workshop. CEUR Workshop Proceedings, 2018.Status: Published
Medico Multimedia Task at MediaEval 2018
The Medico: Multimedia for Medicine Task, running for the second time as part of MediaEval 2018, focuses on detecting abnormalities, diseases, anatomical landmarks and other findings in images captured by medical devices in the gastrointestinal tract. The task is described, including the use case and its challenges, the dataset with ground truth, the required participant runs and the evaluation metrics.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Working Notes Proceedings of the MediaEval 2018 Workshop |
Publisher | CEUR Workshop Proceedings |
Mimir: An Automatic Reporting and Reasoning System for Deep Learning based Analysis in the Medical Domain
In Proceedings of the 9th ACM Multimedia Systems Conference. Amsterdam, Netherlands: ACM, 2018.Status: Published
Mimir: An Automatic Reporting and Reasoning System for Deep Learning based Analysis in the Medical Domain
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Proceedings of the 9th ACM Multimedia Systems Conference |
Pagination | 369-374 |
Publisher | ACM |
Place Published | Amsterdam, Netherlands |
ISBN Number | 978-1-4503-5192-8 |
DOI | 10.1145/3204949.3208129 |
Motor Activity Based Classification of Depression in Unipolar and Bipolar Patients
In 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS). Karlstad, Sweden: IEEE, 2018.Status: Published
Motor Activity Based Classification of Depression in Unipolar and Bipolar Patients
Wearable sensors measuring different parts of people's activity are a common technology nowadays. Data created using these devices holds a lot of potential besides measuring the quantity of daily steps or calories burned, since continuous recordings of heart rate and activity levels usually are collected. Furthermore, there is an increasing awareness in the field of psychiatry on how these activity data relates to various mental health issues such as changes in mood, personality, inability to cope with daily problems or stress and withdrawal from friends and activities. In this paper we present the analysis of a unique dataset containing sensor data collected from patients suffering from depression. The dataset contains motor activity recordings of 23 unipolar and bipolar depressed patients and 32 healthy controls. We apply machine learning to classify patients into depressed and nondepressed. For evaluation of the algorithms, leave one patient out validation is performed. The best results achieved are an F1 score of 0.73 and a MCC of 0.44. The overall findings show that sensor data contains information that can be used to determine the depression status of a person.
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | 2018 IEEE 31st International Symposium on Computer-Based Medical Systems (CBMS) |
Pagination | 316-321 |
Date Published | 06/2018 |
Publisher | IEEE |
Place Published | Karlstad, Sweden |
ISSN Number | 2372-9198 |
URL | https://ieeexplore.ieee.org/document/8417257/http://xplorestaging.ieee.o... |
DOI | 10.1109/CBMS.2018.00062 |
OpenSea - Open Search Based Classification Tool
In Proceedings of the 9th ACM Multimedia Systems Conference. Amsterdam, Netherlands: ACM, 2018.Status: Published
OpenSea - Open Search Based Classification Tool
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Proceedings of the 9th ACM Multimedia Systems Conference |
Pagination | 363-368 |
Publisher | ACM |
Place Published | Amsterdam, Netherlands |
ISBN Number | 978-1-4503-5192-8 |
DOI | 10.1145/3204949.3208128 |
Overview of ImageCLEF 2018: Challenges, datasets and evaluation
In ImageCLEF 2018. Springer, 2018.Status: Published
Overview of ImageCLEF 2018: Challenges, datasets and evaluation
This paper presents an overview of the ImageCLEF 2018 evaluation campaign, an event that was organized as part of the CLEF (Conference and Labs of the Evaluation Forum) Labs 2018. ImageCLEF is an ongoing initiative (it started in 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval with the aim of providing information access to collections of images in various usage scenarios and domains. In 2018, the 16th edition of ImageCLEF ran three main tasks and a pilot task: (1) a caption prediction task that aims at predicting the caption of a figure from the biomedical literature based only on the figure image; (2) a tuberculosis task that aims at detecting the tuberculosis type, severity and drug resistance from CT (Computed Tomography) volumes of the lung; (3) a LifeLog task (videos, images and other sources) about daily activities understanding and moment retrieval, and (4) a pilot task on visual question answering where systems are tasked with answering medical questions. The strong participation, with over 100 research groups registering and 31 submitting results for the tasks, shows an increasing interest in this benchmarking campaign.
Afilliation | Communication Systems, Machine Learning |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | ImageCLEF 2018 |
Date Published | 08/2018 |
Publisher | Springer |
URL | https://link.springer.com/chapter/10.1007/978-3-319-98932-7_28 |
Overview of ImageCLEFlifelog 2018: daily living understanding and lifelog moment retrieval
In Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum. Avignon, France: CEUR Workshop Proceedings, 2018.Status: Published
Overview of ImageCLEFlifelog 2018: daily living understanding and lifelog moment retrieval
Afilliation | Communication Systems, Machine Learning |
Project(s) | No Simula project |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Working Notes of CLEF 2018 - Conference and Labs of the Evaluation Forum |
Date Published | 09/2018 |
Publisher | CEUR Workshop Proceedings |
Place Published | Avignon, France |
URL | http://ceur-ws.org/Vol-2125/ |