Projects
TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet

The Internet as we know it today has been optimised to transmit large amounts of data or “greedy streams” - the type of transmission involved in downloading large files or watching online TV. Up to recently, Internet research has primarily focused on speeding up transmission by increasing bandwidth so that more data can be transferred at a given time. The most common Internet protocol for transmitting data, TCP, works by apportioning available bandwidth among the users present at any given time. The downside is that this can cause latency, or delay, in data transmissions. For time-dependant applications such as Internet telephony and online gaming, time lags as short as a few hundred milliseconds can create big problems.
In real-time gaming against other players online, data is transmitted only when an action such as moving around or shooting at someone is performed. The same principle applies for stock market programs when placing orders or requesting share prices, for example, via the trading systems in use the Norwegian Stock Exchange. In such cases it is essential to avoid any delay.
Applications like these often generate what are called thin data streams. With thin streams only small amounts of data are transmitted at a time and there can be extended periods between data packages. Such thin streams cannot compete with greedy traffic for bandwidth. Thin streams almost invariably come up short against greedy traffic and users are left to cope with the resulting lag. We want a more balanced Internet where thin streams don’t always lose out. This can be achieved by adding speed to the mix, instead of only thinking about maximising throughput.
Final goal:
Locate the sources of increased delay and loss rates for thin streams along the entire path of a data packet. Develop mechanisms that reduce the latency for time-dependent thin-stream applications without having to change current Internet infrastructure.
Funding source:
The Research Council of Norway
All partners:
- Coop. Assoc. for Internet Data Analysis (CAIDA)
- FUNCOM OSLO AS
- Universitetet i Oslo
- Karlstads Universitet
- UNINETT AS
- University of Kaiserslautern
- CISCO SYSTEMS NORWAY AS
Publications for TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet
Technical reports
Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture
Internet Engineering Task Force, 2017.Status: Submitted
Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Architecture
This document describes the L4S architecture for the provision of a new service that the Internet could provide to eventually replace best efforts for all traffic: Low Latency, Low Loss, Scalable throughput (L4S). It is becoming common for all (or most) applications being run by a user at any one time to require low latency. However, the only solution the IETF can offer for ultra-low queuing delay is Diffserv, which only favours a minority of packets at the expense of others. In extensive testing the new L4S service keeps average queuing delay under a millisecond for all applications even under very heavy load, without sacrificing utilization; and it keeps congestion loss to zero. It is becoming widely recognized that adding more access capacity gives diminishing returns, because latency is becoming the critical problem. Even with a high capacity broadband access, the reduced latency of L4S remarkably and consistently improves performance under load for applications such as interactive video, conversational video, voice, Web, gaming, instant messaging, remote desktop and cloud-based apps (even when all being used at once over the same access link). The insight is that the root cause of queuing delay is in TCP, not in the queue. By fixing the sending TCP (and other transports) queuing latency becomes so much better than today that operators will want to deploy the network part of L4S to enable new products and services. Further, the network part is simple to deploy - incrementally with zero-config. Both parts, sender and network, ensure coexistence with other legacy traffic. At the same time L4S solves the long-recognized problem with the future scalability of TCP throughput.
This document describes the L4S architecture, briefly describing the different components and how the work together to provide the aforementioned enhanced Internet service.
Afilliation | Communication Systems |
Project(s) | TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Technical reports |
Year of Publication | 2017 |
Number | draft-briscoe-tsvwg-l4s-arch-01 |
Date Published | 03/2017 |
Publisher | Internet Engineering Task Force |
Keywords | Algorithms, Analysis, AQM, Congestion Avoidance, congestion control, Data Communication, Design, Evaluation, Internet, latency, networks, Performance, QoS, Scaling, tcp |
Notes | (Work in Progress) |
URL | https://tools.ietf.org/html/draft-briscoe-tsvwg-l4s-arch |
Adding Explicit Congestion Notification (ECN) to TCP Control Packets
Internet Engineering Task Force, 2017.Status: Submitted
Adding Explicit Congestion Notification (ECN) to TCP Control Packets
This documents explores the possibility of adding ECN support to TCP control packets.
Afilliation | Communication Systems |
Project(s) | TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Technical reports |
Year of Publication | 2017 |
Number | draft-bagnulo-tcpm-generalized-ecn-00 |
Date Published | 04/2017 |
Publisher | Internet Engineering Task Force |
Keywords | congestion control, Data Communication, Internet, latency, networks, Protocols, QoS, Quality of Service, Rate Control, Security, Signalling, Standards |
Notes | (Work in Progress) |
URL | https://tools.ietf.org/html/draft-bagnulo-tcpm-generalized-ecn |
TRILL: ECN (Explicit Congestion Notification) Support
Internet Engineering Task Force, 2017.Status: Accepted
TRILL: ECN (Explicit Congestion Notification) Support
Explicit congestion notification (ECN) allows a forwarding element to notify downstream devices, including the destination, of the onset of congestion without having to drop packets. This document extends this capability to TRILL switches, including integration with IP ECN.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency, TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Technical reports |
Year of Publication | 2017 |
Number | draft-ietf-trill-ecn-support-02 |
Date Published | 03/2017 |
Publisher | Internet Engineering Task Force |
Keywords | Architecture, congestion, Control, Data Communication, Encapsulation, Explicit Notification, Incremental Deployment, Internet, Layering, networks, Protocol Engineering, QoS, Tunnels |
Notes | (Work in Progress) |
URL | http://tools.ietf.org/html/draft-ietf-trill-ecn-support |
Proceedings, refereed
PI2 : A Linearized AQM for both Classic and Scalable TCP
In Proc. ACM CoNEXT 2016. New York, NY, USA: ACM, 2016.Status: Published
PI2 : A Linearized AQM for both Classic and Scalable TCP
This paper concerns the use of Active Queue Management (AQM) to reduce queuing delay. It offers insight into why it has proved hard for a Proportional Integral (PI) controller to remain both responsive and stable while controlling `Classic' TCP flows, such as TCP Reno and Cubic. Due to their non-linearity, the controller's adjustments have to be smaller when the target drop probability is lower. The PI Enhanced (PIE) algorithm attempts to solve this problem by scaling down the adjustments of the controller using a look-up table. Instead, we control an internal variable that is by definition linearly proportional to the load, then post-process it into the required Classic drop probability---in fact we show that the output simply needs to be squared. This allows tighter control, giving responsiveness and stability better or no worse than PIE achieves, but without all its corrective heuristics.
With suitable packet classification, it becomes simple to extend this PI2 AQM to support coexistence between Classic and Scalable congestion controls in the public Internet. A Scalable congestion control ensures sufficient feedback at any flow rate, an example being Data Centre TCP (DCTCP). A Scalable control is linear, so we can use the internal variable directly without any squaring, by omitting the post-processing stage.
We implemented PI2 as a Linux qdisc to extensively test our claims using Classic and Scalable TCPs.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency, TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Proceedings, refereed |
Year of Publication | 2016 |
Conference Name | Proc. ACM CoNEXT 2016 |
Pagination | 105-119 |
Date Published | 12/2016 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 978-1-4503-4297-1 |
Keywords | Algorithms, Analysis, AQM, Congestion Avoidance, congestion control, Data Communication, Design, Evaluation, Internet, latency, networks, Performance, QoS, Scaling, tcp |
URL | http://dl.acm.org/citation.cfm?doid=2999572.2999578 |
DOI | 10.1145/2999572.2999578 |
Technical reports
Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Problem Statement
Internet Engineering Task Force, 2016.Status: Submitted
Low Latency, Low Loss, Scalable Throughput (L4S) Internet Service: Problem Statement
This document motivates a new service that the Internet could provide to eventually replace best efforts for all traffic: Low Latency, Low Loss, Scalable throughput (L4S). It is becoming common for all (or most) applications being run by a user at any one time to require low latency. However, the only solution the IETF can offer for ultra-low queuing delay is Diffserv, which only favours a minority of packets at the expense of others. In extensive testing the new L4S service keeps average queuing delay under a millisecond for all applications even under very heavy load, without sacrificing utilization; and it keeps congestion loss to zero. It is becoming widely recognized that adding more access capacity gives diminishing returns, because latency is becoming the critical problem. Even with a high capacity broadband access, the reduced latency of L4S remarkably and consistently improves performance under load for applications such as interactive video, conversational video, voice, Web, gaming, instant messaging, remote desktop and cloud-based apps (even when all being used at once over the same access link). The insight is that the root cause of queuing delay is in TCP, not in the queue. By fixing the sending TCP (and other transports) queuing latency becomes so much better than today that operators will want to deploy the network part of L4S to enable new products and services. Further, the network part is simple to deploy - incrementally with zero-config. Both parts, sender and network, ensure coexistence with other legacy traffic. At the same time L4S solves the long-recognized problem with the future scalability of TCP throughput.
This document explains the underlying problems that have been preventing the Internet from enjoying such performance improvements. It then outlines the parts necessary for a solution and the steps that will be needed.
Afilliation | Communication Systems |
Project(s) | TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Technical reports |
Year of Publication | 2016 |
Number | draft-briscoe-tsvwg-aqm-tcpm-rmcat-l4s-problem-02 |
Date Published | 07/2016 |
Publisher | Internet Engineering Task Force |
Keywords | Algorithms, Analysis, AQM, Congestion Avoidance, congestion control, Data Communication, Design, Evaluation, Internet, latency, networks, Performance, QoS, Scaling, tcp |
Notes | (Work in Progress) |
URL | https://tools.ietf.org/html/draft-briscoe-tsvwg-aqm-tcpm-rmcat-l4s-problem |
Propagating Explicit Congestion Notification Across IP Tunnel Headers Separated by a Shim
Internet Engineering Task Force, 2016.Status: Accepted
Propagating Explicit Congestion Notification Across IP Tunnel Headers Separated by a Shim
RFC 6040 on "Tunnelling of Explicit Congestion Notification" made the rules for propagation of ECN consistent for all forms of IP in IP tunnel. This specification extends the scope of RFC 6040 to include tunnels where two IP headers are separated by a shim header that cannot stand alone.
Afilliation | Communication Systems |
Project(s) | TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Technical reports |
Year of Publication | 2016 |
Number | g-rfc6040update-shim-00 |
Date Published | 11/2016 |
Publisher | Internet Engineering Task Force |
Keywords | Architecture, congestion, Control, Data Communication, Encapsulation, Explicit Notification, Internet, Layering, Management, Monitoring, networks, Protocol Engineering, QoS, Tunnels |
Notes | (Work in Progress) |
URL | http://tools.ietf.org/html/draft-briscoe-tsvwg-rfc6040update-shim |
More Accurate ECN Feedback in TCP
Internet Engineering Task Force, 2016.Status: Accepted
More Accurate ECN Feedback in TCP
Explicit Congestion Notification (ECN) is a mechanism where network nodes can mark IP packets instead of dropping them to indicate incipient congestion to the end-points. Receivers with an ECN-capable transport protocol feed back this information to the sender. ECN is specified for TCP in such a way that only one feedback signal can be transmitted per Round-Trip Time (RTT). Recently, new TCP mechanisms like Congestion Exposure (ConEx) or Data Center TCP (DCTCP) need more accurate ECN feedback information whenever more than one marking is received in one RTT. This document specifies an experimental scheme to provide more than one feedback signal per RTT in the TCP header. Given TCP header space is scarce, it overloads the three existing ECN-related flags in the TCP header and provides additional information in a new TCP option.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency, TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Technical reports |
Year of Publication | 2016 |
Number | draft-ietf-tcpm-accurate-ecn-02 |
Date Published | 10/2016 |
Publisher | Internet Engineering Task Force |
Keywords | Architecture, congestion control, Data Communication, Internet, networks, Protocols, QoS, Quality of Service, Rate Control, Security, Signalling, Standards |
Notes | (Work in Progress) |
URL | http://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn |
DualQ Coupled AQM for Low Latency, Low Loss and Scalable Throughput
Internet Engineering Task Force, 2016.Status: Submitted
DualQ Coupled AQM for Low Latency, Low Loss and Scalable Throughput
Data Centre TCP (DCTCP) was designed to provide predictably low queuing latency, near-zero loss, and throughput scalability using explicit congestion notification (ECN) and an extremely simple marking behaviour on switches. However, DCTCP does not co-exist with existing TCP traffic---throughput starves. So, until now, DCTCP could only be deployed where a clean-slate environment could be arranged, such as in private data centres. This specification defines `DualQ Coupled Active Queue Management (AQM)' to allow scalable congestion controls like DCTCP to safely co-exist with classic Internet traffic. The Coupled AQM ensures that a flow runs at about the same rate whether it uses DCTCP or TCP Reno/Cubic, but without inspecting transport layer flow identifiers. When tested in a residential broadband setting, DCTCP achieved sub-millisecond average queuing delay and zero congestion loss under a wide range of mixes of DCTCP and `Classic' broadband Internet traffic, without compromising the performance of the Classic traffic. The solution also reduces network complexity and eliminates network configuration.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency, TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Technical reports |
Year of Publication | 2016 |
Number | draft-briscoe-tsvwg-aqm-dualq-coupled-00 |
Date Published | 10/2016 |
Publisher | Internet Engineering Task Force |
Keywords | Algorithms, Analysis, AQM, Congestion Avoidance, congestion control, Data Communication, Design, Evaluation, Internet, latency, networks, Performance, QoS, Scaling, tcp |
Notes | (Work in Progress) |
URL | http://tools.ietf.org/html/draft-briscoe-tsvwg-aqm-dualq-coupled |
RITE: Reducing Internet Transport Latency

RITE proposes to remove the root causes of unnecessary latency over the Internet. Whilst time-of-flight delay is inevitable, greater delays can result from interactions between transport protocols and buffers. It is this that RITE are tackling.
In RITE we aim to explore and develop changes to the existing infrastructure, so that the changes can be deployed without having to redesign the Internet. Latency can arise from a myriad of different reasons. As an example, setting up all connections for loading a dynamic web page, including DNS and background database connections, may delay the loading of the page with several extra RTTs. This will, unnecessarily, reduce the load speed. In RITE we analyse network traces and investigate network components and operating systems to locate the bottlenecks. RITE researchers use what we learn from this investigation to develop mechanisms that can be deployed in the existing infrastructure to improve the latency for generic Internet use, as well as for our chosen use-cases.
RITE has a strong focus on standardisation, and works through the IETF to transform the results of the project into standards. The partners also aim to contribute new code to the Linux kernel, making the improvements available to the public.
The project is driven by three specific use-cases: Financial applications, networked games and interactive video. These are applications with very different characteristics, but all have strict latency requirements. The industry partners will benefit from reduced Internet latency in a wide range of applications that they either provide infrastructure or hardware for, ultimately benefitting their customers.
Reducing latency may pave the way for new, exiting uses of the Internet. Applications that to this point have been unthinkable may be realised with consistent low-latency service. Also, lower latency for our time-dependent use-cases will improve the experience for the users drastically, giving our industry partners a competitive benefit.
Final goal:
The ultimate goal of RITE is to bring stable, low-latency services to Internet users and businesses, and lay the groundwork for consistent low latency in Internet communication.
Starting by better understanding how latency is traded off for throughput and what generates latency in the Internet, we will investigate the Internet systems to find the latency-inducing bottlenecks, both in the end-systems and in the network. Finally we will design mechanisms that can be integrated with the existing infrastructure to improve the experienced latency.
Funding source:
- The European Commission
- Funding scheme: STREP Total Cost: € 4,910,044.00 EC Contribution: € 3,569,000.00 Contract Number: CNECT-ICT-317700
All partners:
- Simula Research Laboratory (NO)
- British Telecommunications (UK)
- Alcatel-Lucent Bell (BE)
- Megapop Games (NO)
- University of Oslo (NO)
- Karlstad University (SE)
- Institut Mines-Telecom (FR)
- The University Court of the University of Aberdeen (UK)
Publications for RITE: Reducing Internet Transport Latency
Journal Article
Online Identification of Groups of Flows Sharing a Network Bottleneck
IEEE/ACM Transactions on Networking 28 (2020): 2229-2242.Status: Published
Online Identification of Groups of Flows Sharing a Network Bottleneck
Abstract—Most Internet hosts today support multiple access technologies and network interfaces. Multipath transport protocols, like MPTCP, are being deployed (e.g., in smartphones), allowing transparent simultaneous use of multiple links. Besides providing increased resilience to link failures, multipath trans- ports may better exploit available (aggregate) capacity across all interfaces. The safest way to ensure fairness is to assume that any subflows of a multipath end-to-end connection may share bottleneck links, but knowledge of non-shared bottlenecks could allow multipath senders to exploit more capacity without being unfair to other flows. The problem of reliably detecting the existence of (non)-shared bottlenecks is not trivial and is compounded by the fact that bottlenecks may change due to traffic dynamics. In this paper we focus on practical methods to reliably group flows that share, possibly dynamic, bottlenecks online and in a passive manner (i.e., without injecting measurement traffic). We introduce a novel dynamic clustering algorithm that we apply to update our previous shared bottleneck flow grouping (SBFG) method standardized by the IETF, based on delay statistics. We also adapt an offline SBFG method based on wavelet filters to enable it for online operation. These SBFG methods are evaluated by a simple testbed, rigorous simulation and real-world Internet experiments in a testbed comprised of multihomed hosts. Our results suggest that there is no clear winner, and selection of the “best” SBFG method will have to consider tradeoffs regarding accuracy, lag, and application requirements.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency, Department of Mobile Systems and Analytics, The Center for Resilient Networks and Applications |
Publication Type | Journal Article |
Year of Publication | 2020 |
Journal | IEEE/ACM Transactions on Networking |
Volume | 28 |
Number | 5 |
Pagination | 2229--2242 |
Publisher | IEEE/ACM |
ISSN | Print ISSN: 1063-6692, Electronic ISSN: 1558-2566 |
Keywords | dynamic clustering, Internet congestion control, multipath congestion control., shared bottleneck detection |
Notes | Published in the Early Access area on IEEE Xplore. The content is final as presented with the exception of pagination and |
DOI | 10.1109/TNET.2020.3007346 |
Miscellaneous
Shared Bottleneck Detection for Coupled Congestion Control for RTP Media
In RFC 8382. Internet Requests for Comments ed. RFC Editor, 2018.Status: Published
Shared Bottleneck Detection for Coupled Congestion Control for RTP Media
This document describes a mechanism to detect whether end-to-end data flows share a common bottleneck. This mechanism relies on summary statistics that are calculated based on continuous measurements and used as input to a grouping algorithm that runs wherever the knowledge is needed
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency |
Publication Type | Miscellaneous |
Year of Publication | 2018 |
Publisher | RFC Editor |
Notes | Internet Engineering Task Force (IETF) Request for Comments: 8382 Category: Experimental ISSN: 2070-1721 |
URL | https://www.rfc-editor.org/info/rfc8382 |
DOI | 10.17487/RFC8382 |
TR-Number | 8382 |
Journal Article
Operating ranges, tunability and performance of CoDel and PIE
Computer Communications 103 (2017): 74-82.Status: Published
Operating ranges, tunability and performance of CoDel and PIE
Bufferbloat is excessive delay due to the accumulation of packets in a router’s oversized queues. CoDel and PIE are two recent Active Queue Management (AQM) algorithms that have been proposed to address bufferbloat by reducing the queuing delay while trying to maintain a high bottleneck utilization. This paper fills a gap by outlining what are the operating ranges, that is the network characteristics (in terms of round-trip times and bottleneck capacity), for which these algorithms achieve their design goals. This new approach to the problem lets us identify deployment scenarios where both AQM schemes result in poor performance when used with default parameters. Because PIE and CoDel have been proposed with RED’s deployment issues in mind, it was essential to evaluate to what extent we can tune them to achieve various trade-offs and let them control the queuing delay outside their default operating range. We find that, by appropriate tuning (1) the amount of buffering can easily be controlled with PIE, (2) the Round Trip Time (RTT) sensitivity of CoDel can be reduced. Also, we observe there is more correlation between the congestion level, the achieved queuing delay and the targeted delay with CoDel than with PIE. This paper therefore concludes there is no single overall best AQM scheme, as each scheme proposes a specific trade-off.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency |
Publication Type | Journal Article |
Year of Publication | 2017 |
Journal | Computer Communications |
Volume | 103 |
Pagination | 74-82 |
Date Published | 05/2017 |
Publisher | Elsevier |
Keywords | AQM, Bufferbloat, CoDel, congestion control, PIE |
DOI | 10.1016/j.comcom.2016.07.013 |
Proceedings, refereed
Alternative Backoff: Achieving Low Latency and High Throughput with ECN and AQM
In IFIP Networking. IFIP, 2017.Status: Published
Alternative Backoff: Achieving Low Latency and High Throughput with ECN and AQM
A number of recently proposed Active Queue Management (AQM) mechanisms instantiate shallow buffers with burst tolerance to minimise the time that packets spend enqueued at a bottleneck. However, shallow buffering causes noticeable TCP performance degradation as a path’s underlying round trip time (RTT) heads above typical intra-country levels. Using less-aggressive multiplicative backoffs in TCP can compensate for shallow bottleneck buffering. AQM mechanisms may either drop packets or mark them using Explicit Congestion Notification (ECN), depending on whether the sender marked packets as ECN-capable. While a drop may therefore stem from any type of queue, an ECN-mark indicates that an AQM mechanism has done its job, and therefore the queue is likely to be shallow. We propose ABE: “Alternative Backoff with ECN”, which consists of enabling ECN and letting individual TCP senders back off less aggressively in reaction to ECN-marks from AQM-enabled bottlenecks. Using controlled testbed experiments with standard NewReno and CUBIC flows, we show significant performance gains in lightly-multiplexed scenarios, without losing the delay-reduction benefits of deploying AQM. ABE is a sender-side-only modification that can be deployed across networks incrementally (requiring no flag-day) and offers a compelling reason to deploy and enable ECN across the Internet.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency |
Publication Type | Proceedings, refereed |
Year of Publication | 2017 |
Conference Name | IFIP Networking |
Publisher | IFIP |
Keywords | AQM, congestion control, ECN, low latency, tcp |
Notes | Best Paper Award |
URL | http://dl.ifip.org/db/conf/networking/networking2017/1570335770.pdf |
Talks, invited
Multi-Path Transport with OMNeT++ and the INET Framework
In Albacete, Castilla-La Mancha, Spain. Albacete, Castilla-La Mancha, Spain, 2017.Status: Published
Multi-Path Transport with OMNeT++ and the INET Framework
In order to evaluate the performance of multi-path transport protocols, a straightforward initial step is to perform simulations. OMNeT++, together with the INET Framework, provide a powerful Open Source platform for running network simulations. This talk provides an overview of simulating multi-path transport with OMNeT++ and the INET Framework. Particular focus is on the Concurrent Multipath Transfer extension for the Stream Control Transmission Protocol (SCTP). Furthermore, useful additions like the NetPerfMeter application model, the extended network auto-configurator as well as the Simulation Processing Tool-Chain (SimProcTC) are explained.
Afilliation | Communication Systems |
Project(s) | NorNet, RITE: Reducing Internet Transport Latency |
Publication Type | Talks, invited |
Year of Publication | 2017 |
Location of Talk | Albacete, Castilla-La Mancha, Spain |
Place Published | Albacete, Castilla-La Mancha, Spain |
Keywords | CMT, CMT-SCTP, Concurrent Multipath Transfer, INET Framework, Multi-Path Transport, NetPerfMeter, OMNeT++, SCTP, SimProcTC, Stream Control Transmission Protocol |
Technical reports
TRILL: ECN (Explicit Congestion Notification) Support
Internet Engineering Task Force, 2017.Status: Accepted
TRILL: ECN (Explicit Congestion Notification) Support
Explicit congestion notification (ECN) allows a forwarding element to notify downstream devices, including the destination, of the onset of congestion without having to drop packets. This document extends this capability to TRILL switches, including integration with IP ECN.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency, TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Technical reports |
Year of Publication | 2017 |
Number | draft-ietf-trill-ecn-support-02 |
Date Published | 03/2017 |
Publisher | Internet Engineering Task Force |
Keywords | Architecture, congestion, Control, Data Communication, Encapsulation, Explicit Notification, Incremental Deployment, Internet, Layering, networks, Protocol Engineering, QoS, Tunnels |
Notes | (Work in Progress) |
URL | http://tools.ietf.org/html/draft-ietf-trill-ecn-support |
Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP
IETF, 2017.Status: Accepted
Guidelines for Adding Congestion Notification to Protocols that Encapsulate IP
Afilliation | Communication Systems |
Project(s) | The Center for Resilient Networks and Applications, RITE: Reducing Internet Transport Latency |
Publication Type | Technical reports |
Year of Publication | 2017 |
Number | draft-ietf-tsvwg-ecn-encap-guidelines-08 |
Publisher | IETF |
Keywords | Architecture, congestion, Control, Data Communication, Encapsulation, Explicit Notification, Internet, Layering, Management, Monitoring, networks, Protocol Engineering, QoS, Tunnels |
Notes | (Work in Progress) |
URL | http://tools.ietf.org/html/draft-ietf-tsvwg-ecn-encap-guidelines |
Journal Article
Is Multi-Path Transport Suitable for Latency Sensitive Traffic?
Computer Networks (COMNET) 105 (2016): 1-21.Status: Published
Is Multi-Path Transport Suitable for Latency Sensitive Traffic?
This paper assesses whether multi-path communication can help latency-sensitive applications to satisfy the requirements of their users. We consider Concurrent Multi-path Transfer for SCTP (CMT-SCTP) and Multi-path TCP (MPTCP) and evaluate their proficiency in transporting video, gaming, and web traffic over combinations of WLAN and 3G interfaces. To ensure the validity of our evaluation, several experimental approaches were used including simulation, emulation and live experiments. When paths are symmetric in terms of capacity, delay and loss rate, we find that the experienced latency is significantly reduced, compared to using a single path. Using multiple asymmetric paths does not affect latency -- applications do not experience any increase or decrease, but might benefit from other advantages of multi-path communication. In the light of our conclusions, multi-path transport is suitable for latency-sensitive traffic and mature enough to be widely deployed.
Afilliation | , Communication Systems, Communication Systems |
Project(s) | NorNet, RITE: Reducing Internet Transport Latency, The Center for Resilient Networks and Applications |
Publication Type | Journal Article |
Year of Publication | 2016 |
Journal | Computer Networks (COMNET) |
Volume | 105 |
Pagination | 1-21 |
Date Published | 08/2016 |
Publisher | Elsevier |
Keywords | CMT-SCTP, Internet, latency, MPTCP, Multi-Path Communication, transport protocols |
DOI | 10.1016/j.comnet.2016.05.008 |
Proceedings, refereed
Managing real-time media flows through a flow state exchange
In NOMS 2016 IEEE/IFIP Network Operations and Management Symposium, 2016.Status: Published
Managing real-time media flows through a flow state exchange
When multiple congestion controlled flows traverse the same network path, their resulting rate is usually an outcome of their competition at the bottleneck. The WebRTC / RTCWeb suite of standards for inter-browser communication is required to allow prioritization. This is addressed by our previously presented mechanism for coupled congestion control, called the Flow State Exchange (FSE). Here, we present our first simulation results using two mechanisms that have been proposed for IETF standardization: Google Congestion Control (GCC) and Network-Assisted Dynamic Adaptation (NADA). These two mechanisms exhibit aspects that allow us to use a simpler “passive” algorithm in our FSE. Passive coupling allows a less time-constrained request-response style of signaling between congestion control mechanisms and the FSE, which enables the FSE to run as a stand-alone management tool.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency |
Publication Type | Proceedings, refereed |
Year of Publication | 2016 |
Conference Name | NOMS 2016 IEEE/IFIP Network Operations and Management Symposium |
Pagination | 112--120 |
Date Published | 04/2016 |
DOI | 10.1109/NOMS.2016.7502803 |
PI2 : A Linearized AQM for both Classic and Scalable TCP
In Proc. ACM CoNEXT 2016. New York, NY, USA: ACM, 2016.Status: Published
PI2 : A Linearized AQM for both Classic and Scalable TCP
This paper concerns the use of Active Queue Management (AQM) to reduce queuing delay. It offers insight into why it has proved hard for a Proportional Integral (PI) controller to remain both responsive and stable while controlling `Classic' TCP flows, such as TCP Reno and Cubic. Due to their non-linearity, the controller's adjustments have to be smaller when the target drop probability is lower. The PI Enhanced (PIE) algorithm attempts to solve this problem by scaling down the adjustments of the controller using a look-up table. Instead, we control an internal variable that is by definition linearly proportional to the load, then post-process it into the required Classic drop probability---in fact we show that the output simply needs to be squared. This allows tighter control, giving responsiveness and stability better or no worse than PIE achieves, but without all its corrective heuristics.
With suitable packet classification, it becomes simple to extend this PI2 AQM to support coexistence between Classic and Scalable congestion controls in the public Internet. A Scalable congestion control ensures sufficient feedback at any flow rate, an example being Data Centre TCP (DCTCP). A Scalable control is linear, so we can use the internal variable directly without any squaring, by omitting the post-processing stage.
We implemented PI2 as a Linux qdisc to extensively test our claims using Classic and Scalable TCPs.
Afilliation | Communication Systems |
Project(s) | RITE: Reducing Internet Transport Latency, TimeIn: Traffic behaviour of interactive time-dependent thin streams on the modern Internet |
Publication Type | Proceedings, refereed |
Year of Publication | 2016 |
Conference Name | Proc. ACM CoNEXT 2016 |
Pagination | 105-119 |
Date Published | 12/2016 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 978-1-4503-4297-1 |
Keywords | Algorithms, Analysis, AQM, Congestion Avoidance, congestion control, Data Communication, Design, Evaluation, Internet, latency, networks, Performance, QoS, Scaling, tcp |
URL | http://dl.acm.org/citation.cfm?doid=2999572.2999578 |
DOI | 10.1145/2999572.2999578 |
Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources

In the current and future industry and society, there will be an increasing number of systems storing and processing large amounts of data. This is the next frontier for innovation, competition and productivity with ongoing large initiatives both in the EU and the US. Areas where processing of large amounts of unstructured data is applied include medicine, meteorology, genomics,
As such, the aim of the EONS research project is to perform basic research in the area of system and tools support for both, parallel programming and parallel processing, in the context of future distributed large-scale heterogeneous systems. EONS will develop concepts and mechanisms that enable the development of software for these next-generation big-data
The following points are investigated by EONS:
- Formalization of a high level parallel programming model that is compatible with those programming models and languages that developer today know. There are already several approaches to specify potential parallelism, but for workloads with processing and/or time dependencies, we need to add notions of deadlines and execution orders.
- Compiler and multi-core run-time system. Many run-time systems have been built and are in use, but there are large potentials for more efficient execution and run-time support for the dependencies must be added. Scheduling and mapping of tasks to processing engines will here be important. At the core of this plan is the common exploitation of knowledge that can be retained from the compilation step with knowledge that can be gained at runtime during execution on a multi-core system.
- Distributed implementation and high-level scheduler optimization. Adding support for multiple machines makes the previous item more complex. The heterogeneity and complexity increase and the communication costs vary more. A high-level scheduler therefore must take this into account, i.e., in addition to the competition for resources from different concurrent workloads.
Final goal:
Funding source:
All partners:
Publications for Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources
Book Chapter
Camera Synchronization for Panoramic Videos
In MediaSync, 565-592. Springer, 2018.Status: Published
Camera Synchronization for Panoramic Videos
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Book Chapter |
Year of Publication | 2018 |
Book Title | MediaSync |
Pagination | 565-592 |
Date Published | 03/2018 |
Publisher | Springer |
URL | https://doi.org/10.1007/978-3-319-65840-7_20 |
DOI | 10.1007/978-3-319-65840-7_20 |
Journal Article
Social Media and Satellites. Disaster event detection, linking and summarization
Multimedia Tools and Applications 78, no. 3 (2018): 2837-2875.Status: Published
Social Media and Satellites. Disaster event detection, linking and summarization
Being able to automatically link social media and satellite imagery holds large opportunities for research, with a potentially considerable impact on society. The possibility of integrating different information sources opens in fact to new scenarios where the wide coverage of satellite imaging can be used as a collector of the fine-grained details provided by the social media. Remote-sensed data and social media data can well complement each other, integrating the wide perspective provided by the satellite view with the information collected locally, being it textual, audio, or visual. Among the possible applications, natural disasters are certainly one of the most interesting scenarios, where global and local perspectives are needed at the same time.
In this paper, we present a system called JORD that is able to autonomously collect social media data (including the text analysis in local languages) about technological and environmental disasters, and link it automatically to remote-sensed data. Moreover, in order to ensure the quality of retrieved information, JORD is equipped with a hierarchical filtering mechanism relying on the temporal information and the content analysis of retrieved multimedia data.
To show the capabilities of the system, we present a large number of disaster events detected by the system, and we evaluate both the quality of the provided information about the events and the usefulness of JORD from potential users viewpoint, using crowdsourcing.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | Multimedia Tools and Applications |
Volume | 78 |
Issue | 3 |
Pagination | 2837–2875 |
Publisher | Springer |
Place Published | US |
Keywords | Event Detection, Information retrieval, Natural Disaster, Social Media |
DOI | 10.1007/s11042-018-5982-9 |
Multimodal analysis of user behavior and browsed content under different image search intents
International Journal of Multimedia Information Retrieval 7 (2018): 29-41.Status: Published
Multimodal analysis of user behavior and browsed content under different image search intents
The motivation or intent of a search for content may vary between users and use-cases. Knowledge and understanding of these underlying objectives may therefore be important in order to return appropriate search results, and studies of user search intent are emerging in information retrieval to understand why a user is searching for a particular type of content. In the context of image search, our work targets automatic recognition of users’ intent in an early stage of a search session. We have designed seven different search scenarios under the intent conditions of finding items, re-finding items and entertainment. Moreover, we have collected facial expressions, physiological responses, eye gaze and implicit user interactions from 51 participants who performed seven different search tasks on a custom-built image retrieval platform, and we have analyzed the users’ spontaneous and explicit reactions under different intent conditions. Finally, we trained different machine learning models to predict users’ search intent from the visual content of the visited images, the user interactions and the spontaneous responses. Our experimental results show that after fusing the visual and user interaction features, our system achieved the F-1 score of 0.722 for classifying three classes in a user-independent cross-validation. Eye gaze and implicit user interactions, including mouse movements and keystrokes are the most informative features for intent recognition. In summary, the most promising results are obtained by modalities that can be captured unobtrusively and online, and the results therefore demonstrate the potential of including intent-based methods in multimedia retrieval platforms.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | International Journal of Multimedia Information Retrieval |
Volume | 7 |
Pagination | 29 - 41 |
Date Published | Jan-03-2018 |
Publisher | Springer |
ISSN | 2192-6611 |
URL | http://link.springer.com/10.1007/s13735-018-0150-6http://link.springer.c... |
DOI | 10.1007/s13735-018-0150-6 |
Top-Down Saliency Detection Driven by Visual Classification
Computer Vision and Image Understanding 172 (2018): 67-76.Status: Published
Top-Down Saliency Detection Driven by Visual Classification
This paper presents an approach for saliency detection able to emulate the integration of the top-down (task-controlled) and bottom-up (sensory information) processes involved in human visual attention. In particular, we first learn how to generate saliency when a specific visual task has to be accomplished. Afterwards, we investigate if and to what extent
the learned saliency maps can support visual classification in nontrivial cases. To achieve this, we propose SalClass-
Net, a CNN framework consisting of two networks jointly trained: a) the first one computing top-down saliency maps
from input images, and b) the second one exploiting the computed saliency maps for visual classification.
To test our approach, we collected a dataset of eye-gaze maps, using a Tobii T60 eye tracker, by asking several subjects
to look at images from the Stanford Dogs dataset, with the objective of distinguishing dog breeds.
Performance analysis on our dataset and other saliency benchmarking datasets, such as POET, showed that Sal-
ClassNet outperforms state-of-the-art saliency detectors, such as SalNet and SALICON. Finally, we also analyzed
the performance of SalClassNet in a fine-grained recognition task and found out that it yields enhanced classification
accuracy compared to Inception and VGG-19 classifiers. The achieved results, thus, demonstrate that 1) condition-
ing saliency detectors with object classes reaches state-of-the-art performance, and 2) explicitly providing top-down
saliency maps to visual classifiers enhances accuracy.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2018 |
Journal | Computer Vision and Image Understanding |
Volume | 172 |
Pagination | 67-76 |
Publisher | Elsevier |
DOI | 10.1016/j.cviu.2018.03.005 |
Proceedings, refereed
Deep learning approaches for flood classification and flood aftermath detection
In Working Notes Proceedings of the MediaEval 2018 Workshop. Vol. 2283. Sophia Antipolis, France: CEUR-WS.org, 2018.Status: Published
Deep learning approaches for flood classification and flood aftermath detection
This paper presents the method proposed by team UTAOS for MediaEval 2018 Multimedia Satellite Task: Emergency Response for Flooding Events. In the first challenge, we mainly rely on object and scene level features extracted through multiple deep models pre-trained on the ImageNet and Places datasets. The object and scene-level features are combined using early, late and double fusion techniques achieving an average F1-score of 65.03%, 60.59% and 63.58%, respectively. For the second challenge, we rely on a convolutional neural networks (CNNs) and a transfer learning-based classification approach achieving an average F1-score of 62.30% and 61.02% for run 1 and run 2, respectively.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Working Notes Proceedings of the MediaEval 2018 Workshop |
Volume | 2283 |
Publisher | CEUR-WS.org |
Place Published | Sophia Antipolis, France |
Transfer learning with prioritized classification and training dataset equalization for medical objects detection
In Working Notes Proceedings of the MediaEval 2018 Workshop. CEUR Workshop Proceedings, 2018.Status: Published
Transfer learning with prioritized classification and training dataset equalization for medical objects detection
This paper presents the method proposed by the organizer team (SIMULA) for MediaEval 2018 Multimedia for Medicine: the Medico Task. We utilized the recent transfer-learning-based image classification methodology and focused on how easy it is to implement multi-class image classifiers in general and how to improve the classification performance without deep neural network model redesign. The goal for this was both to provide a baseline for the Medico task and to show the performance of out-of-the-box classifiers for the medical use-case scenario.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Working Notes Proceedings of the MediaEval 2018 Workshop |
Publisher | CEUR Workshop Proceedings |
Medico Multimedia Task at MediaEval 2018
In Working Notes Proceedings of the MediaEval 2018 Workshop. CEUR Workshop Proceedings, 2018.Status: Published
Medico Multimedia Task at MediaEval 2018
The Medico: Multimedia for Medicine Task, running for the second time as part of MediaEval 2018, focuses on detecting abnormalities, diseases, anatomical landmarks and other findings in images captured by medical devices in the gastrointestinal tract. The task is described, including the use case and its challenges, the dataset with ground truth, the required participant runs and the evaluation metrics.
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Working Notes Proceedings of the MediaEval 2018 Workshop |
Publisher | CEUR Workshop Proceedings |
Dissecting Deep Neural Networks for Better Medical Image Classification and Classification Understanding
In 31st IEEE CBMS International Symposium on Computer-Based Medical Systems. Karlstad, Sweden: IEEE, 2018.Status: Published
Dissecting Deep Neural Networks for Better Medical Image Classification and Classification Understanding
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | 31st IEEE CBMS International Symposium on Computer-Based Medical Systems |
Publisher | IEEE |
Place Published | Karlstad, Sweden |
ISSN Number | 2372-9198 |
DOI | 10.1109/CBMS.2018.00070 |
Deep Learning and Hand-crafted Feature Based Approaches for Polyp Detection in Medical Videos
In 31st IEEE CBMS International Symposium on Computer-Based Medical Systems. Karlstad, Sweden: IEEE, 2018.Status: Published
Deep Learning and Hand-crafted Feature Based Approaches for Polyp Detection in Medical Videos
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | 31st IEEE CBMS International Symposium on Computer-Based Medical Systems |
Pagination | 381-386 |
Publisher | IEEE |
Place Published | Karlstad, Sweden |
ISSN Number | 2372-9198 |
DOI | 10.1109/CBMS.2018.00073 |
Mimir: An Automatic Reporting and Reasoning System for Deep Learning based Analysis in the Medical Domain
In Proceedings of the 9th ACM Multimedia Systems Conference. Amsterdam, Netherlands: ACM, 2018.Status: Published
Mimir: An Automatic Reporting and Reasoning System for Deep Learning based Analysis in the Medical Domain
Afilliation | Communication Systems |
Project(s) | Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Proceedings of the 9th ACM Multimedia Systems Conference |
Pagination | 369-374 |
Publisher | ACM |
Place Published | Amsterdam, Netherlands |
ISBN Number | 978-1-4503-5192-8 |
DOI | 10.1145/3204949.3208129 |
Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization

PCI Express (PCIe) is today widely used for static local input/output (I/O) expansion and is gaining momentum as a host-to-host high speed interconnect. However these services are currently not able to integrate onto a single hardware infrastructure, two separate PCIe networks are needed.
Dolphin has developed interconnect solutions since the early 1990s and its current product line relies exclusively on PCIe-based interface cards and switches and competes with alternative high-speed communication technologies. A great advantage of Dolphin’s PCIe products is the reduced protocol overhead compared to technologies like 10Gb Ethernet, InfiniBand and other proprietary interconnect technologies. Dolphin is the market leader in providing fast, optimized and easy to use software and hardware products enabling PCIe to be used as a high speed interconnect.
I/O devices are typically statically assigned to a single root complex (host), hot-add, device migration, device sharing and remote access are not supported in flexible way.
In principle, all I/O devices on a shared PCIe fabric can be accessed directly using the PCIe Non Transparent Bridging (NTB) addressing techniques by any connected remote computer, but the required software structure, OS interfaces and implementations do not exist.
Device sharing functionality is addressed in the PCIe Single Root I/O Virtualization (SR-IOV) and Multi Root I/O Virtualization (MR-IOV) specifications but this is still a static approach and requires PCIe devices to be developed according to these specifications.
As I/O devices are normally owned by one compute node, I/O data are normally relayed to remote servers though the host using traditional networking services. Using PCIe NTB techniques, it is possible to enable I/O devices to directly transfer data to a remote node. This will significantly reduce latency and overhead during data transfers.
Final goal:
The goal with this project is to develop a new framework for the operating system and virtual machines that will enable remote discover, addressing, access and use of standard PCIe devices. The framework will enable standard PCIe devices to be re-allocated and shared by computer nodes in the PCIe network with no or minimum changes to the applications and device drivers. On top of this framework we will develop services to validate and demonstrate the framework such as a fast cluster file system, legacy device driver access to I/O devices on a remote node, and clustering of accelerator cards such as the Intel Xeon Phi co-processor and Nvidia Tesla graphics processing units.\
Funding source:
The Research Council of Norway
All partners:
Dolphin Interconnect Solutions
Publications for Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization
PhD Thesis
SmartIO: Device sharing and memory disaggregation in PCIe clusters using non-transparent bridging
In The University of Oslo. Vol. PhD. University of Oslo (UiO), 2022.Status: Published
SmartIO: Device sharing and memory disaggregation in PCIe clusters using non-transparent bridging
Distributed and parallel computing applications are becoming increasingly compute-heavy and data-driven, accelerating the need for disaggregation solutions that enable sharing of I/O resources between networked machines. For example, in a heterogeneous computing cluster, different machines may have different devices available to them, but distributing I/O resources in a way that maximizes both resource utilization and overall cluster performance is a challenge. To facilitate device sharing and memory disaggregation among machines connected using PCIe non-transparent bridges, we present SmartIO. SmartIO makes all machines in the cluster, including their internal devices and memory, part of a common PCIe domain. By leveraging the memory mapping capabilities of non-transparent bridges, remote resources may be used directly, as if these resources were local to the machines using them. Whether devices are local or remote is made transparent by SmartIO. NVMes, GPUs, FPGAs, NICs, and any other PCIe device can be dynamically shared with and distributed to remote machines, and it is even possible to disaggregate devices and memory, in order to share component parts with multiple machines at the same time. Software is entirely removed from the performance-critical path, allowing remote resources to be used with native PCIe performance. To demonstrate that SmartIO is an efficient solution, we have performed a comprehensive evaluation consisting of a wide range of performance experiments, including both synthetic benchmarks and realistic, large-scale workloads. Our experimental results show that remote resources can be used without any performance overhead compared to using local resources, in terms of throughput and latency. Thus, compared to existing disaggregation solutions, SmartIO provides more efficient, low-cost resource sharing, increasing the overall system performance and resource utilization.
Afilliation | Communication Systems |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | PhD Thesis |
Year of Publication | 2022 |
Degree awarding institution | The University of Oslo |
Degree | PhD |
Number of Pages | 236 |
Date Published | 10/2022 |
Publisher | University of Oslo (UiO) |
Thesis Type | Paper Collection |
URL | https://www.duo.uio.no/handle/10852/97351 |
Proceedings, refereed
Host Bypassing: Let your GPU speak Ethernet
In IEEE 8th International Conference on Network Softwarization (NetSoft). IEEE, 2022.Status: Published
Host Bypassing: Let your GPU speak Ethernet
Hardware acceleration of network functions is essential to meet the challenging Quality of Service requirements in nowadays computer networks. Graphical Processing Units (GPU) are a widely deployed technology that can also be used for computing tasks, including acceleration of network functions. In this work, we demonstrate how commodity GPUs, which do not provide any network interfaces, can be used to accelerate network functions. Our approach leverages PCIe peer-to-peer capabilities and allows the GPU to control the network interface card directly, without any assistance from the operating system or control application. The presented evaluation results demonstrate the feasibility of our approach and its performance of up to 10 Gbit/s, even for small packets.
Afilliation | Communication Systems |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | IEEE 8th International Conference on Network Softwarization (NetSoft) |
Pagination | 85-90 |
Date Published | 06/2022 |
Publisher | IEEE |
ISBN Number | 978-1-6654-0694-9 |
URL | https://ieeexplore.ieee.org/document/9844090 |
DOI | 10.1109/NetSoft54395.2022.9844090 |
Journal Article
SmartIO: Zero-overhead Device Sharing through PCIe Networking
ACM Transactions on Computer Systems 38, no. 1-2 (2021): 1-78.Status: Published
SmartIO: Zero-overhead Device Sharing through PCIe Networking
The large variety of compute-heavy and data-driven applications accelerate the need for a distributed I/O solution that enables cost-effective scaling of resources between networked hosts. For example, in a cluster system, different machines may have various devices available at different times, but moving workloads to remote units over the network is often costly and introduces large overheads compared to accessing local resources. To facilitate I/O disaggregation and device sharing among hosts connected using Peripheral Component Interconnect Express (PCIe) non-transparent bridges, we present SmartIO. NVMes, GPUs, network adapters, or any other standard PCIe device may be borrowed and accessed directly, as if they were local to the remote machines. We provide capabilities beyond existing disaggregation solutions by combining traditional I/O with distributed shared-memory functionality, allowing devices to become part of the same global address space as cluster applications. Software is entirely removed from the data path, and simultaneous sharing of a device among application processes running on remote hosts is enabled. Our experimental results show that I/O devices can be shared with remote hosts, achieving native PCIe performance. Thus, compared to existing device distribution mechanisms, SmartIO provides more efficient, low-cost resource sharing, increasing the overall system performance
Afilliation | Communication Systems, Machine Learning |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | ACM Transactions on Computer Systems |
Volume | 38 |
Issue | 1-2 |
Number | 2 |
Pagination | 1–78 |
Date Published | 07/2021 |
Publisher | Association for Computing Machinery |
Place Published | New York, NY, United States |
ISSN | 0734-2071 |
URL | https://dl.acm.org/doi/10.1145/3462545 |
DOI | 10.1145/3462545 |
Proceedings, refereed
Host Bypassing: Direct Data Piping from the Network to the Hardware Accelerator
In IEEE 14th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC). IEEE, 2021.Status: Published
Host Bypassing: Direct Data Piping from the Network to the Hardware Accelerator
Computer networks have become very important and influential over the last years for many common services such as Internet connectivity as well as time-sensitive applications such as videotelephony. Furthermore, approaches like in-network computing enable the offloading of latency-critical and high-performance network functions into the network, e.g. 5G network functions, to enable such time-sensitive applications. In this work, we show how FPGAs in PCIe-based systems, which are typically used as hardware accelerators for latency-critical in-network functions, can be integrated into the data path. Our approach, named host bypassing, allows direct data transfer from the network interface to the accelerator and accomplishes substantial performance benefits over existing state-of-the-art approaches. Our detailed evaluation results demonstrate the possibility of achieving deterministic low latency while operating under heavy load without any packet loss. In addition, fewer CPU resources are required.
Afilliation | Communication Systems |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | IEEE 14th International Symposium on Embedded Multicore/Many-core Systems-on-Chip (MCSoC) |
Pagination | 23-30 |
Date Published | 12/2021 |
Publisher | IEEE |
ISBN Number | 978-1-6654-3860-5 |
URL | https://ieeexplore.ieee.org/document/9691977 |
DOI | 10.1109/MCSoC51149.2021.00012 |
Journal Article
Flexible device compositions and dynamic resource sharing in PCIe interconnected clusters using Device Lending
Cluster Computing 22, no. 86 (2019): 1-24.Status: Published
Flexible device compositions and dynamic resource sharing in PCIe interconnected clusters using Device Lending
Modern workloads often exceed the processing and I/O capabilities provided by resource virtualization, requiring direct access to the physical hardware in order to reduce latency and computing overhead. For computers interconnected in a cluster, access to remote hardware resources often requires facilitation both in hardware and specialized drivers with virtualization support. This limits the availability of resources to specific devices and drivers that are supported by the virtualization technology being used, as well as what the interconnection technology supports. For PCI Express (PCIe) clusters, we have previously proposed Device Lending as a solution for enabling direct low latency access to remote devices. The method has extremely low computing overhead and does not require any application- or device-specific distribution mechanisms. Any PCIe device, such as network cards disks, and GPUs, can easily be shared among the connected hosts. In this work, we have extended our solution with support for a virtual machine (VM) hypervisor. Physical remote devices can be “passed through” to VM guests, enabling direct access to physical resources while still retaining the flexibility of virtualization. Additionally, we have also implemented multi-device support, enabling shortest-path peer-to-peer transfers between remote devices residing in different hosts. Our experimental results prove that multiple remote devices can be used, achieving bandwidth and latency close to native PCIe, and without requiring any additional support in device drivers. I/O intensive workloads run seamlessly using both local and remote resources. With our added VM and multi-device support, Device Lending offers highly customizable configurations of remote devices that can be dynamically reassigned and shared to optimize resource utilization, thus enabling a flexible composable I/O infrastructure for VMs as well as bare-metal machines.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, LADIO: Live Action Data Input/Output, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Journal Article |
Year of Publication | 2019 |
Journal | Cluster Computing |
Volume | 22 |
Issue | 86 |
Pagination | 1-24 |
Date Published | 09/2019 |
Publisher | Springer |
ISSN | 1573-7543 |
URL | https://link.springer.com/article/10.1007/s10586-019-02988-0 |
DOI | 10.1007/s10586-019-02988-0 |
Talks, invited
Dynamic Sharing of GPUs and IO in a PCIe Network
In GPU Technology Conference, San Jose, CA, USA. Nvidia, 2019.Status: Published
Dynamic Sharing of GPUs and IO in a PCIe Network
Learn how GPUs, NVMe drives and other IO devices can be efficiently shared in a PCI Express network using SmartIO from Dolphin Interconnect Solutions.
Traditionally, IO devices are statically assigned to a single root complex (host machine), and features such as hot-add, device migration and remote access are not supported flexibly without complex software frameworks. SmartIO eliminates these restrictions and provides a flexible framework for handling PCIe devices and systems. Devices such as GPUs, NVMe drives and other IO devices can be flexibly accessed from remote systems.
We demonstrate how SmartIO is implemented using standard PCIe and Non-Transparent Bridging, show that our system got near-native performance when moving data borrowed GPUs and NVMe drives. We also show how we can dynamically add more GPUs to scale performance.
Afilliation | Communication Systems |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Talks, invited |
Year of Publication | 2019 |
Location of Talk | GPU Technology Conference, San Jose, CA, USA |
Publisher | Nvidia |
Proceedings, refereed
Flexible Device Sharing in PCIe Clusters using Device Lending
In International Conference on Parallel Processing Companion (ICPP'18 Comp). ACM, 2018.Status: Published
Flexible Device Sharing in PCIe Clusters using Device Lending
Processing workloads may have very high IO demands, exceeding the capabilities provided by resource virtualization and requiring direct access to the physical hardware. For computers that are interconnected in PCI Express (PCIe) networks, we have previously proposed Device Lending as a solution for assigning devices to remote hosts. In this paper, we explain how we have extended our implementation with support for the Linux Kernel-based Virtual Machine (KVM) hypervisor. Using our extended Device Lending, it becomes possible to dynamically “pass through” physical remote devices to VM guests while still retaining the flexibility of virtualization, something that previously required extensive facilitation in both hypervisor and device drivers in the form of paravirtualization.
We have also improved our original implementation with sup- port for interoperability between remote devices. We show that it is possible to use multiple devices residing in different hosts, while still achieving the same bandwidth and latency as native PCIe, and without requiring any additional support in device drivers.
Afilliation | Communication Systems |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, LADIO: Live Action Data Input/Output, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | International Conference on Parallel Processing Companion (ICPP'18 Comp) |
Date Published | 08/2018 |
Publisher | ACM |
ISBN Number | 978-1-4503-6523-9/18/08 |
DOI | 10.1145/3229710.3229759 |
Autonomic Adaptation of Multimedia Content Adhering to Application Mobility
In Distributed Applications and Interoperable Systems (DAIS 2018). Lecture Notes in Computer Science ed. Vol. 10853. Springer, Cham, 2018.Status: Published
Autonomic Adaptation of Multimedia Content Adhering to Application Mobility
Today,manyusersofmultimediaapplicationsaresurrounded by a changing set of multimedia-capable devices. However, users can move their running multimedia applications only to a pre-defined set of devices. Application mobility is the paradigm where users can move their running applications (or parts of) to heterogeneous devices in a seamless manner. In order to continue multimedia processing under the implied context changes in application mobility, applications need to adapt the presentation of multimedia content and their internal configuration. We propose the system DAMPAT that implements an adaptation control loop to adapt multimedia pipelines. Exponential combinatorial growth of possible pipeline configurations is controlled by architectural constraints specified as high-level goals by application developers. Our evaluation shows that the pipeline only needs to be interrupted a few tens of milliseconds to perform the reconfiguration. Thus, production or consumption of multimedia content can continue across heterogeneous devices and user context changes in a seamless manner.
Afilliation | Communication Systems |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | Distributed Applications and Interoperable Systems (DAIS 2018) |
Volume | 10853 |
Edition | Lecture Notes in Computer Science |
Pagination | 153-168 |
Date Published | 06/2018 |
Publisher | Springer, Cham |
ISBN Number | 978-3-319-93766-3 |
DOI | 10.1007/978-3-319-93767-0_11 |
Dynamic Adaptation of Multimedia Presentations for Videoconferencing in Application Mobility
In IEEE International Conference on Multimedia and Expo (ICME). San Diego, CA, USA: IEEE, 2018.Status: Published
Dynamic Adaptation of Multimedia Presentations for Videoconferencing in Application Mobility
Application mobility is the paradigm where users can move their running applications to heterogeneous devices in a seamless manner. This mobility involves dynamic context changes of hardware, network resources, user environment, and user preferences. In order to continue multimedia processing under these context changes, applications need to adapt not only the collection of media streams, i.e., multimedia presentation, but also their internal configuration to work on different hardware. We present the performance analysis to adapt a video-conferencing prototype application in a proposed adaptation control loop to autonomously adapt multimedia pipelines. Results show that the time spent to create an adaptation plan and execute it is in the order of hundreds of milliseconds. The reconfiguration of pipelines, compared to building them from scratch, is approximately 1000 times faster when re-utilizing already instantiated hardware-dependent components. Therefore, we conclude that the adaptation of multimedia pipelines is a feasible approach for multimedia applications that adhere to application mobility.
Afilliation | Communication Systems |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Efficient EONS: Execution of Large Workloads on Elastic Heterogeneous Resources, Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2018 |
Conference Name | IEEE International Conference on Multimedia and Expo (ICME) |
Date Published | 07/2018 |
Publisher | IEEE |
Place Published | San Diego, CA, USA |
ISSN Number | 1945-7871 |
DOI | 10.1109/ICME.2018.8486565 |
Talks, contributed
SmartIO: Dynamic Sharing of GPUs and IO in a PCIe Cluster
In GPU Technology Conference, San Jose, CA, USA. Nvidia, 2018.Status: Published
SmartIO: Dynamic Sharing of GPUs and IO in a PCIe Cluster
Learn how GPUs, NVMe drives and other IO devices can be efficiently shared in a PCI Express cluster using SmartIO from Dolphin Interconnect Solutions.Traditionally, IO devices have been statically assigned to a single root complex (host machine), and features such as hot-add, device migration and remote access is not supported in a flexible way without complex software frameworks. Dolphin SmartIO eliminates these restrictions and provide a flexible framework for handling PCIe devices and systems. Devices such as GPUs, NVMe drives and other IO devices can be flexibly accessed from remote systems. We demonstrate how SmartIO is implemented using standard PCIe and Non-Transparent Bridging, show that our system gets near native performance when moving data from local GPUs to remote NVMe drives, and how we can dynamically add more GPUs to scale performance.
Afilliation | Communication Systems |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Talks, contributed |
Year of Publication | 2018 |
Location of Talk | GPU Technology Conference, San Jose, CA, USA |
Publisher | Nvidia |
Department of Holistic Systems
Our goal is to investigate and solve real-world problems in the area of intelligent distributed systems.
Today, everyone is talking about big-data analytics, video streaming, machine learning, web search, mobile apps, etc., and we all take it for granted that these "services" work perfectly. However, underneath such applications, there are large systems, and individual components of these systems are often “only” a part of a big ecosystem of integrated building blocks needed to enable a functional service. In this respect, the HOST department addresses challenges potentially covering all components of the entire system, from data creation to visualization of the results.
Using a combination of primary and applied research, we are targeting numerous applications, many in the field of sports and medicine, where we have a holistic view of the system and perform basic research, do experimental prototyping, and run experiments in the intended environments.
Currently, machine learning is an essential component of our research where we, for example, in real-time, aim to analyze athlete performance and detect diseases in medical videos. In our holistic systems view, not only the accuracy of the machine learning analysis is of importance, but also the complete pipe-line integration and the system performance (e.g., resource consumption and scalability). We architect complete systems and optimize for particular application requirements, both functional and non-functional, to provide the best possible quality of the service and the lowest possible resource consumption. Finally, we have a goal to put the results into use for the society where we contribute to open source projects and have spun off new industries.

Publications for Department of Holistic Systems
Journal Article
Approximate Bayesian Inference Based on Expected Evaluation
Bayesian Analysis 1, no. 1 (2023).Status: Published
Approximate Bayesian Inference Based on Expected Evaluation
Approximate Bayesian computing (ABC) and Bayesian Synthetic likelihood (BSL) are two popular families of methods to evaluate the posterior distribution when the likelihood function is not available or tractable. For existing variants of ABC and BSL, the focus is usually first put on the simulation algorithm, and after that the form of the resulting approximate posterior distribution comes as a consequence of the algorithm. In this paper we turn this around and firstly define a reasonable approximate posterior distribution by studying the distributional properties of the expected discrepancy, or more generally an expected evaluation, with respect to generated samples from the model. The resulting approximate posterior distribution will be on a simple and interpretable form compared to ABC and BSL.
Secondly a Markov chain Monte Carlo (MCMC) algorithm is developed to simulate from the resulting approximate posterior distribution. The algorithm was evaluated on a synthetic data example and on the Stepping Stone population genetics model, demonstrating that the proposed scheme has real world applicability. The algorithm demonstrates competitive results with the BSL and sequential Monte Carlo ABC algorithms, but is outperformed by the ABC MCMC.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2023 |
Journal | Bayesian Analysis |
Volume | 1 |
Issue | 1 |
Date Published | Jan-01-2023 |
Publisher | Project euclid |
URL | https://projecteuclid.org/journals/bayesian-analysis/volume--1/issue--1/... |
DOI | 10.1214/23-BA1368 |
Training Performance Indications for Amateur Athletes Based on Nutrition and Activity Lifelogs
Algorithms, no. 1 (2023): 30.Status: Published
Training Performance Indications for Amateur Athletes Based on Nutrition and Activity Lifelogs
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2023 |
Journal | Algorithms |
Issue | 1 |
Pagination | 30 |
Date Published | Jan-01-2023 |
Publisher | MDPI |
URL | https://www.mdpi.com/1999-4893/16/1/30 |
DOI | 10.3390/a16010030 |
A multi-center polyp detection and segmentation dataset for generalisability assessment
Nature Scientific Data 10 (2023).Status: Published
A multi-center polyp detection and segmentation dataset for generalisability assessment
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2023 |
Journal | Nature Scientific Data |
Volume | 10 |
Publisher | Nature |
URL | https://doi.org/10.1038/s41597-023-01981-y |
DOI | 10.1038/s41597-023-01981-y |
Live Streaming Technology and Online Child Sexual Exploitation and Abuse - A Scoping Review
Trauma, Violence, & Abuse (2023).Status: Accepted
Live Streaming Technology and Online Child Sexual Exploitation and Abuse - A Scoping Review
Livestreaming of child sexual abuse is an established form of online child sexual exploitation
and abuse. However, only a limited body of research has examined this issue. The Covid-19
pandemic has accelerated internet use and user knowledge of livestreaming services
emphasising the importance of understanding this crime. In this scoping review, existing
literature was brought together through an iterative search of eight databases containing peer-
reviewed journal articles, as well as grey literature. Records were eligible for inclusion if the
primary focus was on livestream technology and online child sexual exploitation and abuse,
the child being defined as eighteen years or younger. Fourteen of the 2,218 records were
selected. The data were charted and divided into four categories: victims, offenders,
legislation, and technology. Limited research, differences in terminology, study design, and
population inclusion criteria present a challenge to drawing general conclusions on the
current state of livestreaming of child sexual abuse. The records show that victims are
predominantly female. The average livestream offender was found to be older than the
average online child sexual abuse offender. Therefore, it is unclear whether the findings are
representative of the global population of livestream offenders. Furthermore, there appears to
be a gap in what the records show on platforms and payment services used and current digital
trends. The lack of a legal definition and privacy considerations pose a challenge to
investigation, detection, and prosecution. The available data allow some insights into a
potentially much larger issue.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2023 |
Journal | Trauma, Violence, & Abuse |
Publisher | SAGE Publications |
Poster
Concept Explanations for Deep Learning-Based Diabetic Retinopathy Diagnosis
Nordic AI Meet 2023, 2023.Status: Accepted
Concept Explanations for Deep Learning-Based Diabetic Retinopathy Diagnosis
Diabetic retinopathy (DR) is a common complication of diabetes that damages the eye and potentially leads to blindness. The severity and treatment choice of DR depends on the presence of medical findings in fundus images. Much work has been done in developing complex machine learning (ML) models to automatically diagnose DR from fundus images. However, their high level of complexity increases the demand for techniques improving human understanding of the ML models. Explainable artificial intelligence (XAI) methods can detect weaknesses in ML models and increase trust among end users. In the medical field, it is crucial to explain ML models in order to apply them in the clinic. While a plethora of XAI methods exists, heatmaps are typically applied for explaining ML models for DR diagnosis. Heatmaps highlight image areas that are regarded as important for the model when making a prediction. Even though heatmaps are popular, they can be less appropriate in the medical field. Testing with Concept Activation Vectors (TCAV), providing explanations based on human-friendly concepts, can be a more suitable alternative for explaining models for DR diagnosis, but it has not been thoroughly investigated for DR models. We develop a deep neural network for diagnosing DR from fundus images and apply TCAV for explaining the resulting model. Concept generation with and without masking is compared. Based on diagnostic criteria for DR, we evaluate the model’s concept ranking for different severity levels of DR. TCAV can explain individual images to gain insight into a specific case, or an entire class to evaluate overall consistency with diagnostic standards. The most important concepts for the DR model agree with diagnostic criteria for DR. No large differences are detected between the two concept generation approaches. TCAV is a flexible explanation method where human-friendly concepts provide insights and trust in ML models for medical image analyses, and it shows promising results for DR grading.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Poster |
Year of Publication | 2023 |
Place Published | Nordic AI Meet 2023 |
Keywords | concept-based explanations, diabetic retinopathy, Explainable artificial intelligence |
Proceedings, refereed
Identifying Important Proteins in Meibomian Gland Dysfunction with Explainable Artificial Intelligence
In IEEE International Symposium on Computer-Based Medical Systems (IEEE CBMS2023), 2023.Status: Accepted
Identifying Important Proteins in Meibomian Gland Dysfunction with Explainable Artificial Intelligence
Meibomian gland dysfunction is the most common cause of dry eye disease, which is a prevalent condition that can damage the ocular surface and cause reduced vision and substantial pain. Meibum secreted from the meibomian glands makes up the majority of the outer, protective lipid layer of the tear film. Changes in the secreted meibum and markers of glandular damage can be detected through tear sampling.
Several studies have investigated the tear film protein expression in meibomian gland dysfunction, but less work apply machine learning to analyze the protein patterns. We use machine learning and methods from explainable artificial intelligence to detect potential clinically relevant proteins in meibomian gland dysfunction. Two different explainable artificial intelligence methods are compared. Several of the proteins found important in the models have been linked to dry eye disease in the past, while some are novel. Consequently, explainable artificial intelligence methods serve as a promising tool for screening for proteins that are relevant for meibomian gland dysfunction. By doing so, one may be able to discover new biomarkers and treatments, and gain a better understanding of how diseases develop.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2023 |
Conference Name | IEEE International Symposium on Computer-Based Medical Systems (IEEE CBMS2023) |
Keywords | Dry eye disease, Explainable artificial intelligence, Machine learning, meibomian gland dysfunction, proteomics |
Using explainable artificial intelligence (XAI) to explore factors affecting meibomian gland (MG) dropout
In ARVO Annual Meeting, 2023.Status: Accepted
Using explainable artificial intelligence (XAI) to explore factors affecting meibomian gland (MG) dropout
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2023 |
Conference Name | ARVO Annual Meeting |
Multimedia datasets: challenges and future possibilities
In International conference on multimedia modeling, 2023.Status: Accepted
Multimedia datasets: challenges and future possibilities
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2023 |
Conference Name | International conference on multimedia modeling |
Detecting human embryo cleavage stages using YOLO v5 object detection algorithm
In Nordic Artificial Intelligence Research and Development. Springer, 2023.Status: Published
Detecting human embryo cleavage stages using YOLO v5 object detection algorithm
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2023 |
Conference Name | Nordic Artificial Intelligence Research and Development |
Pagination | 81-93 |
Publisher | Springer |
Book
The Influence of Delay on Cloud Gaming Quality of Experience
Cham: Springer, 2022.Status: Published
The Influence of Delay on Cloud Gaming Quality of Experience
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Book |
Year of Publication | 2022 |
Edition | 1 |
Number of Pages | 156 |
Date Published | 06/2022 |
Publisher | Springer |
Place Published | Cham |
ISBN Number | 978-3-030-99868-4 |
URL | https://link.springer.com/book/10.1007/978-3-030-99869-1 |
DOI | 10.1109/QoMEX55416.2022.9900908 |
Interview training of child-welfare and law-enforcement professionals interviewing maltreated children supported via artificial avatars

The interdisciplinary FRIPRO project aims to improve interviews with maltreated children through a training program using realistic and interactive child avatars.
The department of Holistic Systems (HOST) at SimulaMet will be working with the Faculty of Social Sciences at OsloMet. The project will begin on the 1st of April 2021 and end on the 31st of March in 2024. It is funded by The Research Council of Norway with 12 million NOK and will include three Ph.D. positions.
Maltreatment and abuse of children is a significant societal problem that has serious and damaging effects on children’s behavior, psychological development, and adjustment. Detection and prevention of violence and sexual abuse against children is, therefore, a high priority for Child Protective Services (CPS) and law-enforcement professionals. The conversations and investigative interviews that are conducted with these children must be of high quality. However, both Norwegian and international research shows that despite investments in methodology, the current interview and conversation skills still need to be improved.
By using an empirically informed training system in highly realistic child avatars, this project aims to develop and maintain the advanced skills needed for interviewing maltreated children. They will use data from past investigative interviews with maltreated children and create a real-looking avatar that is capable of expressing emotion and spontaneous responses.
The planned avatar will be a combination of technologies from multiple areas in computer science including AI, computer vision, and natural language processing. The aim is for the child avatars to be a part of an interview-training program that will be implemented in cooperation with the CPS and the police. The training system will be evaluated by the project scientists to judge effectiveness in relation to real-world needs.
The project also involves collaborations with researchers from Griffith University in Australia and the University of Cambridge in the United Kingdom.
Publications
Journal Article
A multi-center polyp detection and segmentation dataset for generalisability assessment
Nature Scientific Data 10 (2023).Status: Published
A multi-center polyp detection and segmentation dataset for generalisability assessment
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2023 |
Journal | Nature Scientific Data |
Volume | 10 |
Publisher | Nature |
URL | https://doi.org/10.1038/s41597-023-01981-y |
DOI | 10.1038/s41597-023-01981-y |
Live Streaming Technology and Online Child Sexual Exploitation and Abuse - A Scoping Review
Trauma, Violence, & Abuse (2023).Status: Accepted
Live Streaming Technology and Online Child Sexual Exploitation and Abuse - A Scoping Review
Livestreaming of child sexual abuse is an established form of online child sexual exploitation
and abuse. However, only a limited body of research has examined this issue. The Covid-19
pandemic has accelerated internet use and user knowledge of livestreaming services
emphasising the importance of understanding this crime. In this scoping review, existing
literature was brought together through an iterative search of eight databases containing peer-
reviewed journal articles, as well as grey literature. Records were eligible for inclusion if the
primary focus was on livestream technology and online child sexual exploitation and abuse,
the child being defined as eighteen years or younger. Fourteen of the 2,218 records were
selected. The data were charted and divided into four categories: victims, offenders,
legislation, and technology. Limited research, differences in terminology, study design, and
population inclusion criteria present a challenge to drawing general conclusions on the
current state of livestreaming of child sexual abuse. The records show that victims are
predominantly female. The average livestream offender was found to be older than the
average online child sexual abuse offender. Therefore, it is unclear whether the findings are
representative of the global population of livestream offenders. Furthermore, there appears to
be a gap in what the records show on platforms and payment services used and current digital
trends. The lack of a legal definition and privacy considerations pose a challenge to
investigation, detection, and prosecution. The available data allow some insights into a
potentially much larger issue.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2023 |
Journal | Trauma, Violence, & Abuse |
Publisher | SAGE Publications |
Proceedings, refereed
Identifying Important Proteins in Meibomian Gland Dysfunction with Explainable Artificial Intelligence
In IEEE International Symposium on Computer-Based Medical Systems (IEEE CBMS2023), 2023.Status: Accepted
Identifying Important Proteins in Meibomian Gland Dysfunction with Explainable Artificial Intelligence
Meibomian gland dysfunction is the most common cause of dry eye disease, which is a prevalent condition that can damage the ocular surface and cause reduced vision and substantial pain. Meibum secreted from the meibomian glands makes up the majority of the outer, protective lipid layer of the tear film. Changes in the secreted meibum and markers of glandular damage can be detected through tear sampling.
Several studies have investigated the tear film protein expression in meibomian gland dysfunction, but less work apply machine learning to analyze the protein patterns. We use machine learning and methods from explainable artificial intelligence to detect potential clinically relevant proteins in meibomian gland dysfunction. Two different explainable artificial intelligence methods are compared. Several of the proteins found important in the models have been linked to dry eye disease in the past, while some are novel. Consequently, explainable artificial intelligence methods serve as a promising tool for screening for proteins that are relevant for meibomian gland dysfunction. By doing so, one may be able to discover new biomarkers and treatments, and gain a better understanding of how diseases develop.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2023 |
Conference Name | IEEE International Symposium on Computer-Based Medical Systems (IEEE CBMS2023) |
Keywords | Dry eye disease, Explainable artificial intelligence, Machine learning, meibomian gland dysfunction, proteomics |
Multimedia datasets: challenges and future possibilities
In International conference on multimedia modeling, 2023.Status: Accepted
Multimedia datasets: challenges and future possibilities
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2023 |
Conference Name | International conference on multimedia modeling |
Using explainable artificial intelligence (XAI) to explore factors affecting meibomian gland (MG) dropout
In ARVO Annual Meeting, 2023.Status: Accepted
Using explainable artificial intelligence (XAI) to explore factors affecting meibomian gland (MG) dropout
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2023 |
Conference Name | ARVO Annual Meeting |
Proceedings, refereed
A Comparative Study of Interactive Environments for Investigative Interview of A Virtual Child Avatar
In IEEE international symposium on multimedia (ISM). IEEE, 2022.Status: Published
A Comparative Study of Interactive Environments for Investigative Interview of A Virtual Child Avatar
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | IEEE international symposium on multimedia (ISM) |
Pagination | 194-201 |
Publisher | IEEE |
DOI | 10.1109/ISM55400.2022.00043 |
Automatic Thumbnail Selection for Soccer Videos using Machine Learning
In Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22). ACM, 2022.Status: Published
Automatic Thumbnail Selection for Soccer Videos using Machine Learning
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22) |
Pagination | 73-85 |
Publisher | ACM |
DOI | 10.1145/3524273.3528182 |
Automatic Unsupervised Clustering of Videos of the Intracytoplasmic Sperm Injection (ICSI) Procedure
In NAIS: Symposium of the Norwegian AI Society . Vol. 1650. NAIS 2022, 2022.Status: Published
Automatic Unsupervised Clustering of Videos of the Intracytoplasmic Sperm Injection (ICSI) Procedure
The in vitro fertilization procedure called intracytoplasmic sperm injection can be used to help fertilize an egg by injecting a single sperm cell directly into the cytoplasm of the egg. In order to evaluate, refine and improve the method in the fertility clinic, the procedure is usually observed at the clinic. Alternatively, a video of the procedure can be examined and labeled in a time-consuming process. To reduce the time required for the assessment, we propose an unsupervised method that automatically clusters video frames of the intracytoplasmic sperm injection procedure. Deep features are extracted from the video frames and form the basis for a clustering method. The method provides meaningful clusters representing different stages of the intracytoplasmic sperm injection procedure. The clusters can lead to more efficient examinations and possible new insights that can improve clinical practice. Further on, it may also contribute to improved clinical outcomes due to increased understanding about the technical aspects and better results of the procedure. Despite promising results, the proposed method can be further improved by increasing the amount of data and exploring other types of features.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | NAIS: Symposium of the Norwegian AI Society |
Volume | 1650 |
Pagination | 1-11 |
Publisher | NAIS 2022 |
ISBN Number | 978-3-031-17029-4 |
Keywords | clustering, Computer Vision and Pattern Recognition (cs.CV), human reproduction, medical videos, Unsupervised learning |
DOI | 10.1007/978-3-031-17030-0_9 |
Comparison of Crowdsourced and Remote Subjective User Studies: A Case Study of Investigative Child Interviews
In The 14th International Conference on Quality of Multimedia Experience. IEEE, 2022.Status: Published
Comparison of Crowdsourced and Remote Subjective User Studies: A Case Study of Investigative Child Interviews
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | The 14th International Conference on Quality of Multimedia Experience |
Publisher | IEEE |
URL | https://ieeexplore.ieee.org/document/9900900 |
DOI | 10.1109/QoMEX55416.2022.9900900 |
Estimating Predictive Uncertainty in Gastrointestinal Polyp Segmentation
In 2022 IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS). Shenzen, China: IEEE, 2022.Status: Published
Estimating Predictive Uncertainty in Gastrointestinal Polyp Segmentation
Deep neural networks have achieved state-of-the-art performance on numerous applications in the medical field, with use-cases ranging from automation of mundane tasks to diagnosis of life-threatening diseases. Despite these achievements, deep neural networks are considered “black boxes” due to their complex structure and general lack of transparency in their decision-making process. These attributes make it challenging to incorporate deep learning into existing clinical workflows as decisions often need more support than blind faith in a statistical model. This paper presents an investigation of uncertainty estimation for the detection of colon polyps using deep convolutional neural networks (CNNs). We experiment with two different approaches to measure uncertainty, Monte Carlo (MC) dropout and deep ensembles, and discuss the advantages and disadvantages of both methods in terms of computational efficiency and performance gain. Furthermore, we apply the two uncertainty methods to two different state-of-the-art CNN-based polyp segmentation architectures. The uncertainty is visualized as heatmaps on the input images and can be used to make more informed decisions on whether or not to trust a model's predictions. The results show that the predictive uncertainties provide a comparison between different models' predictions which can be interpreted as contrastive explanations where the values are largely influenced by the degree of independence between the models in the ensemble. We also reveal that MC dropout is shown to lack at providing contrastive uncertainty values due to the high correlation between the models' in the ensemble.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | 2022 IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS) |
Pagination | 44-49 |
Publisher | IEEE |
Place Published | Shenzen, China |
URL | https://ieeexplore.ieee.org/document/9867012/http://xplorestaging.ieee.o... |
DOI | 10.1109/CBMS55023.2022.00015 |
Experiences and Lessons Learned from a Crowdsourced-Remote Hybrid User Survey Framework
In 2022 IEEE International Symposium on Multimedia (ISM). Italy: IEEE, 2022.Status: Published
Experiences and Lessons Learned from a Crowdsourced-Remote Hybrid User Survey Framework
Subjective user studies are important to ensure the fidelity and usability of systems that generate multimedia content. Testing how end-users and domain experts perceive multimedia assets might provide crucial information. In this paper, we present our experiences with the open source hybrid crowdsourced-remote user survey framework called Huldra, which is intended for conducting web-based subjective user studies and aims to integrate the individual benefits associated with traditional, crowdsourced, and remote methods. We disseminate our experiences and insights from two actively deployed use cases and discuss challenges and opportunities associated with using Huldra as a framework for conducting user studies.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | 2022 IEEE International Symposium on Multimedia (ISM) |
Pagination | 161-162 |
Publisher | IEEE |
Place Published | Italy |
URL | https://ieeexplore.ieee.org/document/10019678 |
DOI | 10.1109/ISM55400.2022.00035 |
Explainability methods for machine learning systems for multimodal medical datasets: research proposal
In ACM Multimedia Systems (MMSys) Conference. ACM, 2022.Status: Published
Explainability methods for machine learning systems for multimodal medical datasets: research proposal
This paper contains the research proposal of Andrea M. Storås that was presented at the MMSys 2022 doctoral symposium. Machine learning models have the ability to solve medical tasks with a high level of performance, e.g., classifying medical videos and detecting anomalies using different sources of data. However, many of these models are highly complex and difficult to understand. Lack of interpretability can limit the use of machine learning systems in the medical domain. Explainable artificial intelligence provides explanations regarding the models and their predictions. In this PhD project, we develop machine learning models for automatic analysis of medical data and explain the results using established techniques from the field of explainable artificial intelligence. Current research indicate that there are still open issues to be solved in order for end users to understand multimedia systems powered by machine learning. Consequently, new explanation techniques will also be developed. Different types of medical data are applied in order to investigate the generalizability of the methods.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | ACM Multimedia Systems (MMSys) Conference |
Pagination | 347-351 |
Publisher | ACM |
ISBN Number | 978-1-4503-9283-9/22/06 |
DOI | 10.1145/3524273.3533925 |
HOST-ATS: Automatic Thumbnail Selection with Dashboard-Controlled ML Pipeline and Dynamic User Survey
In Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22). ACM, 2022.Status: Published
HOST-ATS: Automatic Thumbnail Selection with Dashboard-Controlled ML Pipeline and Dynamic User Survey
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22) |
Pagination | 334-340 |
Publisher | ACM |
DOI | 10.1145/3524273.3532908 |
Huldra: A Framework for Collecting Crowdsourced Feedback on Multimedia Assets
In Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22). ACM, 2022.Status: Published
Huldra: A Framework for Collecting Crowdsourced Feedback on Multimedia Assets
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | Proceedings of the 13th ACM Multimedia Systems Conference (MMSys '22) |
Pagination | 203-209 |
Publisher | ACM |
DOI | 10.1145/3524273.3532887 |
Human vs. GPT-3: The challenges of extracting emotions from child responses
In The 14th International Conference on Quality of Multimedia Experience. IEEE, 2022.Status: Published
Human vs. GPT-3: The challenges of extracting emotions from child responses
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | The 14th International Conference on Quality of Multimedia Experience |
Publisher | IEEE |
ISBN Number | 978-1-6654-8794-8 |
ISSN Number | 2472-7814 |
Accession Number | 22114185 |
URL | https://ieeexplore.ieee.org/document/9900885 |
DOI | 10.1109/QoMEX55416.2022.9900885 |
Investigative Interviews using a Multimodal Virtual Avatar
In American Psychology-Law Society Conference 2022. Denver USA,: American Psychology-Law Society, 2022.Status: Accepted
Investigative Interviews using a Multimodal Virtual Avatar
To meet best-practice standards, we are developing an interactive virtual avatar aiming as a training tool to raise interviewing skills of child-welfare and law-enforcement professionals. Therefore, we present the “Ilma” avatar that recognizes interviewers’ behavior during open-ended, closed and leading questions, and which can automatically respond to the conversation. We conducted a user study in which master students (N=3) and child protective workers (N=8) interviewed “Ilma” and rated their perception of the interaction. The results show that the participants valued the interaction and found the avatar useful. Thus, it has great potential to be an effective training tool.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | American Psychology-Law Society Conference 2022 |
Publisher | American Psychology-Law Society |
Place Published | Denver USA, |
Is More Realistic Better? A Comparison of Game Engine and GAN-based Avatars for Investigative Interviews of Children
In ICDAR '22: Proceedings of the 3rd ACM Workshop on Intelligent Cross-Data Analysis and Retrieval. New York, NY, USA: ACM, 2022.Status: Published
Is More Realistic Better? A Comparison of Game Engine and GAN-based Avatars for Investigative Interviews of Children
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | ICDAR '22: Proceedings of the 3rd ACM Workshop on Intelligent Cross-Data Analysis and Retrieval |
Pagination | 41-49 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450392419 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3512731 |
DOI | 10.1145/351273110.1145/3512731.3534209 |
Multimedia streaming analytics: quo vadis?
In MHV '22: Mile-High Video Conference. Denver, Colorado, USA: ACM, 2022.Status: Published
Multimedia streaming analytics: quo vadis?
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | MHV '22: Mile-High Video Conference |
Pagination | 62 - 69 |
Publisher | ACM |
Place Published | Denver, Colorado, USA |
ISBN Number | 9781450392228 |
URL | https://dl.acm.org/doi/10.1145/3510450.3517321 |
DOI | 10.1145/3510450.3517321 |
Njord: a fishing trawler dataset
In Proceedings of the 13th ACM Multimedia Systems Conference (MMSYS). New York, NY, USA: ACM, 2022.Status: Published
Njord: a fishing trawler dataset
Fish is one of the main sources of food worldwide. The commercial fishing industry has a lot of different aspects to consider, ranging from sustainability to reporting. The complexity of the domain also attracts a lot of research from different fields like marine biology, fishery sciences, cybernetics, and computer science. In computer science, detection of fishing vessels via for example remote sensing and classification of fish from images or videos using machine learning or other analysis methods attracts growing attention. Surprisingly, little work has been done that considers what is happening on board the fishing vessels. On the deck of the boats, a lot of data and important information are generated with potential applications, such as automatic detection of accidents or automatic reporting of fish caught. This paper presents Njord, a fishing trawler dataset consisting of surveillance videos from a modern off-shore fishing trawler at sea. The main goal of this dataset is to show the potential and possibilities that analysis of such data can provide. In addition to the data, we provide a baseline analysis and discuss several possible research questions this dataset could help answer.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | Proceedings of the 13th ACM Multimedia Systems Conference (MMSYS) |
Date Published | 08/2022 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450392839 |
URL | https://dl.acm.org/doi/pdf/10.1145/3524273.3532886 |
DOI | 10.1145/3524273.3532886 |
Parallel feature selection based on the trace ratio criterion
In International Joint Conference on Neural Networks (IJCNN). IEEE, 2022.Status: Published
Parallel feature selection based on the trace ratio criterion
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | International Joint Conference on Neural Networks (IJCNN) |
Publisher | IEEE |
PolypConnect: Image inpainting for generating realistic gastrointestinal tract images with polyps
In IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS). IEEE, 2022.Status: Published
PolypConnect: Image inpainting for generating realistic gastrointestinal tract images with polyps
Early identification of a polyp in the lower gastrointestinal (GI) tract can lead to prevention of life-threatening colorectal cancer. Developing computer-aided diagnosis (CAD) systems to detect polyps can improve detection accuracy and efficiency and save the time of the domain experts called endoscopists. Lack of annotated data is a common challenge when building CAD systems. Generating synthetic medical data is an active research area to overcome the problem of having relatively few true positive cases in the medical domain. To be able to efficiently train machine learning (ML) models, which are the core of CAD systems, a considerable amount of data should be used. In this respect, we propose the PolypConnect pipeline, which can convert non-polyp images into polyp images to increase the size of training datasets for training. We present the whole pipeline with quantitative and qualitative evaluations involving endoscopists. The polyp segmentation model trained using synthetic data, and real data shows a 5.1% improvement of mean intersection over union (mIOU), compared to the model trained only using real data. The codes of all the experiments are available on GitHub to reproduce the results.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | IEEE 35th International Symposium on Computer-Based Medical Systems (CBMS) |
Pagination | 66-71 |
Date Published | 07/2022 |
Publisher | IEEE |
DOI | 10.1109/CBMS55023.2022.00019 |
Predicting Tacrolimus Exposure in Kidney Transplanted Patients Using Machine Learning
In 35th IEEE CBMS International Symposium on Computer-Based Medical Systems. IEEE, 2022.Status: Published
Predicting Tacrolimus Exposure in Kidney Transplanted Patients Using Machine Learning
Tacrolimus is one of the cornerstone immunosuppressive drugs in most transplantation centers worldwide following solid organ transplantation. Therapeutic drug monitoring of tacrolimus is necessary in order to avoid rejection of the transplanted organ or severe side effects. However, finding the right dose for a given patient is challenging, even for experienced clinicians. Consequently, a tool that can accurately estimate the drug exposure for individual dose adaptions would be of high clinical value. In this work, we propose a new technique using machine learning to estimate the tacrolimus exposure in kidney transplant recipients. Our models achieve predictive errors that are at the same level as an established population pharmacokinetic model, but are faster to develop and require less knowledge about the pharmacokinetic properties of the drug.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | 35th IEEE CBMS International Symposium on Computer-Based Medical Systems |
Pagination | 38-43 |
Publisher | IEEE |
Keywords | Machine learning, personalized medicine, transplantation |
DOI | 10.1109/CBMS55023.2022.00014 |
Segmentation Consistency Training: Out-of-Distribution Generalization for Medical Image Segmentation
In 2022 IEEE International Symposium on Multimedia (ISM). IEEE, 2022.Status: Published
Segmentation Consistency Training: Out-of-Distribution Generalization for Medical Image Segmentation
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | 2022 IEEE International Symposium on Multimedia (ISM) |
Pagination | 42-49 |
Publisher | IEEE |
DOI | 10.1109/ISM55400.2022.00012 |
Soccer Game Summarization using Audio Commentary, Metadata, and Captions
In NarSUM '22: Proceedings of the 1st Workshop on User-centric Narrative Summarization of Long Videos. New York, NY, USA: ACM, 2022.Status: Published
Soccer Game Summarization using Audio Commentary, Metadata, and Captions
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | NarSUM '22: Proceedings of the 1st Workshop on User-centric Narrative Summarization of Long Videos |
Pagination | 13-22 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450394932 |
URL | https://doi.org/10.1145/3552463.3557019 |
DOI | 10.1145/3552463.3557019 |
Towards an AI-driven talking avatar in virtual reality for investigative interviews of children
In GameSys '22: Proceedings of the 2nd Workshop on Games Systems. New York, NY, USA: ACM, 2022.Status: Published
Towards an AI-driven talking avatar in virtual reality for investigative interviews of children
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | GameSys '22: Proceedings of the 2nd Workshop on Games Systems |
Pagination | 9-15 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450393812 |
URL | https://dl.acm.org/doi/10.1145/3534085.3534340 |
DOI | 10.1145/353408510.1145/3534085.3534340 |
Video Analytics in Elite Soccer: A Distributed Computing Perspective
In IEEE Sensor Array and Multichannel Signal Processing Workshop (SAM). Trondheim, Norway: IEEE, 2022.Status: Published
Video Analytics in Elite Soccer: A Distributed Computing Perspective
Ubiquitous sensors and Internet of Things (IoT) technologies have revolutionized the sports industry, providing new methodologies for planning, effective coordination of training, and match analysis post game. New methods, including machine learning, image and video processing, have been developed for performance evaluation, allowing the analyst to track the performance of a player in real-time. Following FIFA's 2015 approval of electronics performance and tracking system during games, performance data of a single player or the entire team is allowed to be collected using GPS-based wearables. Data from practice sessions outside the sporting arena is being collected in greater numbers than ever before. Realizing the significance of data in professional soccer, this paper presents video analytics, examines recent state-of-the-art literature in elite soccer, and summarizes existing real-time video analytics algorithms. We also discuss real-time crowdsourcing of the obtained data, tactical and technical performance, distributed computing and its importance in video analytics and propose a future research perspective.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | IEEE Sensor Array and Multichannel Signal Processing Workshop (SAM) |
Pagination | 221-225 |
Date Published | 06/2022 |
Publisher | IEEE |
Place Published | Trondheim, Norway |
Keywords | analytics, football, soccer, Video |
URL | https://ieeexplore.ieee.org/document/9827827 |
DOI | 10.1109/SAM53842.2022.9827827 |
Virtual Reality Talking Avatar for Investigative Interviews of Maltreat Children
In 19th International Conference on Content-based Multimedia Indexing. New York, NY, USA: Association for Computing Machinery (ACM), 2022.Status: Published
Virtual Reality Talking Avatar for Investigative Interviews of Maltreat Children
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | 19th International Conference on Content-based Multimedia Indexing |
Pagination | 201-204 |
Publisher | Association for Computing Machinery (ACM) |
Place Published | New York, NY, USA |
ISBN Number | 9781450397209 |
URL | https://doi.org/10.1145/3549555.3549572 |
DOI | 10.1145/3549555.3549572 |
When Every Millisecond Counts: The Impact of Delay in VR Gaming
In The 14th International Conference on Quality of Multimedia Experience. IEEE, 2022.Status: Published
When Every Millisecond Counts: The Impact of Delay in VR Gaming
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2022 |
Conference Name | The 14th International Conference on Quality of Multimedia Experience |
Publisher | IEEE |
URL | https://ieeexplore.ieee.org/document/9900908 |
DOI | 10.1109/QoMEX55416.2022.9900908 |
Miscellaneous
ACM Multimedia Grand Challenge on Detecting Cheapfakes
ACM Multimedia Conference (MM): ACM, 2022.Status: Published
ACM Multimedia Grand Challenge on Detecting Cheapfakes
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2022 |
Publisher | ACM |
Place Published | ACM Multimedia Conference (MM) |
Notes | https://2022.acmmm.org/call-for-grand-challenge-submissions/ |
URL | https://detecting-cheapfakes.github.io |
MMSys'22 Grand Challenge on AI-based Video Production for Soccer
arXiv, 2022.Status: Published
MMSys'22 Grand Challenge on AI-based Video Production for Soccer
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2022 |
Publisher | arXiv |
Other Numbers | arXiv:2202.01031 |
URL | https://arxiv.org/abs/2202.01031 |
Visual explanations for polyp detection: How medical doctors assess intrinsic versus extrinsic explanations
arXiv, 2022.Status: Published
Visual explanations for polyp detection: How medical doctors assess intrinsic versus extrinsic explanations
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2022 |
Publisher | arXiv |
URL | https://arxiv.org/abs/2204.00617 |
DOI | 10.48550/arXiv.2204.00617 |
Journal Article
Áika: A Distributed Edge System for AI Inference
Big Data and Cognitive Computing 6, no. 2 (2022): 68.Status: Published
Áika: A Distributed Edge System for AI Inference
Video monitoring and surveillance of commercial fisheries in world oceans has been proposed by the governing bodies of several nations as a response to crimes such as overfishing. Traditional video monitoring systems may not be suitable due to limitations in the offshore fishing environment, including low bandwidth, unstable satellite network connections and issues of preserving the privacy of crew members. In this paper, we present Áika, a robust system for executing distributed Artificial Intelligence (AI) applications on the edge. Áika provides engineers and researchers with several building blocks in the form of Agents, which enable the expression of computation pipelines and distributed applications with robustness and privacy guarantees. Agents are continuously monitored by dedicated monitoring nodes, and provide applications with a distributed checkpointing and replication scheme. Áika is designed for monitoring and surveillance in privacy-sensitive and unstable offshore environments, where flexible access policies at the storage level can provide privacy guarantees for data transfer and access.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Big Data and Cognitive Computing |
Volume | 6 |
Issue | 2 |
Pagination | 68 |
Date Published | Jan-06-2022 |
Publisher | MDPI |
URL | https://www.mdpi.com/2504-2289/6/2/68 |
DOI | 10.3390/bdcc6020068 |
Artificial intelligence in dry eye disease
The Ocular Surface 23 (2022): 74-86.Status: Published
Artificial intelligence in dry eye disease
Dry eye disease (DED) has a prevalence of between 5 and 50%, depending on the diagnostic criteria used and population under study. However, it remains one of the most underdiagnosed and undertreated conditions in ophthalmology. Many tests used in the diagnosis of DED rely on an experienced observer for image interpretation, which may be considered subjective and result in variation in diagnosis. Since artificial intelligence (AI) systems are capable of advanced problem solving, use of such techniques could lead to more objective diagnosis. Although the term ‘AI’ is commonly used, recent success in its applications to medicine is mainly due to advancements in the sub-field of machine learning, which has been used to automatically classify images and predict medical outcomes. Powerful machine learning techniques have been harnessed to understand nuances in patient data and medical images, aiming for consistent diagnosis and stratification of disease severity. This is the first literature review on the use of AI in DED. We provide a brief introduction to AI, report its current use in DED research and its potential for application in the clinic. Our review found that AI has been employed in a wide range of DED clinical tests and research applications, primarily for interpretation of interferometry, slit-lamp and meibography images. While initial results are promising, much work is still needed on model development, clinical testing and standardisation.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | The Ocular Surface |
Volume | 23 |
Pagination | 74 - 86 |
Date Published | Jan-01-2022 |
Publisher | Elsevier |
ISSN | 15420124 |
Keywords | artificial intelligence, Dry eye disease, Machine learning |
URL | https://linkinghub.elsevier.com/retrieve/pii/S1542012421001324 |
DOI | 10.1016/j.jtos.2021.11.004 |
Automatic Tracking of the ICSI procedure using Deep Learning
Human Reproduction 37, no. Supplement_1 (2022).Status: Published
Automatic Tracking of the ICSI procedure using Deep Learning
Study question
Can deep learning be used to detect and track spermatozoa and the different parts of an ICSI procedure?
Summary answer
Deep learning can be used as a tool to assist and organize the contents of an ICSI procedure.
What is known already
Sperm tracking has been a topic of research and practice for many years, especially in the context of computer-aided sperm analysis (CASA). Recent studies have proposed using deep learning algorithms to track spermatozoa for spermatozoon selection in human and animal samples. One critical part of performing ICSI involves the selection of the “best” spermatozoon for injection, but other parts of the procedure may also be of importance. However, as far as we know, tracking using deep learning has not been applied to the ICSI procedure, where detecting instruments and the oocyte could also be helpful in post-analysis and training.
Study design, size, duration
The study was performed using three anonymized videos of the ICSI procedure. The frames of the videos were manually annotated by data scientists and verified by an embryologist. The annotations were bounding boxes around specific parts of the ICSI procedure, including sperm, pipettes, and the oocyte. We trained a YOLOv5 model on the collected data, where two videos were used for training and one video for validation.
Participants/materials, setting, methods
The videos of the ICSI procedure were captured at 200x magnification with a DeltaPix camera at Fertilitetssenteret in Oslo, Norway. ICSI was performed using a Nikon ECLIPSE TE2000-S microscope connected with Eppendorf TransferMan 4m micromanipulators. The spermatozoa were immobilised in 5 µl Polyvinylpyrrolidone (PVP; CooperSurgical). The videos had a resolution of 1920x1080 and were resized to 640x640 before being processed by the YOLOv5 model. The data will be made public in a later study.
Main results and the role of chance
Mean average precision (mAP) with the threshold of 0.5 (mAP@.5) is the main quantitative parameter measured in the YOLOv5 model. All the experiments were performed using three-fold cross-validation, where we present the average metrics calculated over the three folds. Overall, the method showed an average mAP@.5 of 0.50 across all predicted classes, which means that the method can track the different components with good accuracy. Looking closer at the individual classes, we see that instruments like the holding pipette and ICSI pipette are detected with high accuracy with a mAP@.5 of 0.87 and 0.94, respectively. The oocyte is also easily tracked with a mAP@.5 of 0.92. The first polar body is well detected with a mAP@.5 of 0.65. The model has issues detecting and tracking individual sperm (both outside and within the pipette), where the method achieved a mAP@.5 of 0.46 for tracking sperm outside the pipette and 0.03 for the sperm inside the pipette. The low score of detecting the sperm in the pipette can be explained by the often unclear visibility of the sperm through the pipette and the low number of training samples.
Limitations, reasons for caution
The limited sample size makes the generalizability of the method difficult to determine. A more extensive evaluation is necessary. Moreover, as the currency study focuses on tracking, patient information and clinical outcome were not included in the analysis.
Wider implications of the findings
Deep learning has the potential to aid embryologists to perform successful ICSI through tracking and detection of spermatozoa, pipettes, and the oocyte. This could potentially lead to better internal quality control and teaching possibilities, and hopefully better results.
Trial registration number
not applicable
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Human Reproduction |
Volume | 37 |
Issue | Supplement_1 |
Date Published | 07/2022 |
Publisher | Oxford University Press |
ISSN | 0268-1161 |
URL | https://academic.oup.com/humrep/article/37/Supplement_1/deac107.261/6619904 |
DOI | 10.1093/humrep/deac107.261 |
Exploration of Different Time Series Models for Soccer Athlete Performance Prediction
MDPI Engineering Proceedings 18, no. 1 (2022): 37.Status: Published
Exploration of Different Time Series Models for Soccer Athlete Performance Prediction
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | MDPI Engineering Proceedings |
Volume | 18 |
Issue | 1 |
Pagination | 37 |
Publisher | MDPI |
DOI | 10.3390/engproc2022018037 |
FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation
IEEE Transactions on Neural Networks and Learning Systems (2022): 1-14.Status: Published
FANet: A Feedback Attention Network for Improved Biomedical Image Segmentation
The increase of available large clinical and experimental datasets has contributed to a substantial amount of important contributions in the area of biomedical image analysis. Image segmentation, which is crucial for any quantitative analysis, has especially attracted attention. Recent hardware advancement has led to the success of deep learning approaches. However, although deep learning models are being trained on large datasets, existing methods do not use the information from different learning epochs effectively. In this work, we leverage the information of each training epoch to prune the prediction maps of the subsequent epochs. We propose a novel architecture called feedback attention network (FANet) that unifies the previous epoch mask with the feature map of the current training epoch. The previous epoch mask is then used to provide hard attention to the learned feature maps at different convolutional layers. The network also allows rectifying the predictions in an iterative fashion during the test time. We show that our proposed feedback attention model provides a substantial improvement on most segmentation metrics tested on seven publicly available biomedical imaging datasets demonstrating the effectiveness of FANet. The source code is available at https://github.com/nikhilroxtomar/FANet.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | IEEE Transactions on Neural Networks and Learning Systems |
Pagination | 1 - 14 |
Date Published | Jan-01-2022 |
Publisher | IEEE |
ISSN | 2162-237X |
URL | https://ieeexplore.ieee.org/document/9741842 |
DOI | 10.1109/TNNLS.2022.3159394 |
Meta-learning with implicit gradients in a few-shot setting for medical image segmentation
Computers in Biology and Medicine 143 (2022): 105227.Status: Published
Meta-learning with implicit gradients in a few-shot setting for medical image segmentation
Widely used traditional supervised deep learning methods require a large number of training samples but often fail to generalize on unseen datasets. Therefore, a more general application of any trained model is quite limited for medical imaging for clinical practice. Using separately trained models for each unique lesion category or a unique patient population will require sufficiently large curated datasets, which is not practical to use in a real-world clinical set-up. Few-shot learning approaches can not only minimize the need for an enormous number of reliable ground truth labels that are labour-intensive and expensive, but can also be used to model on a dataset coming from a new population. To this end, we propose to exploit an optimization-based implicit model agnostic meta-learning (iMAML) algorithm under few-shot settings for medical image segmentation. Our approach can leverage the learned weights from diverse but small training samples to perform analysis on unseen datasets with high accuracy. We show that, unlike classical few-shot learning approaches, our method improves generalization capability. To our knowledge, this is the first work that exploits iMAML for medical image segmentation and explores the strength of the model on scenarios such as meta-training on unique and mixed instances of lesion datasets. Our quantitative results on publicly available skin and polyp datasets show that the proposed method outperforms the naive supervised baseline model and two recent few-shot segmentation approaches by large margins. In addition, our iMAML approach shows an improvement of 2%–4% in dice score compared to its counterpart MAML for most experiments.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Computers in Biology and Medicine |
Volume | 143 |
Pagination | 105227 |
Date Published | Jan-04-2022 |
Publisher | Elsevier |
ISSN | 00104825 |
URL | https://linkinghub.elsevier.com/retrieve/pii/S0010482522000191 |
DOI | 10.1016/j.compbiomed.2022.105227 |
MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation
IEEE Journal of Biomedical and Health Informatics 26, no. 5 (2022): 2252-2263.Status: Published
MSRF-Net: A Multi-Scale Residual Fusion Network for Biomedical Image Segmentation
Methods based on convolutional neural networks have improved the performance of biomedical image segmentation. However, most of these methods cannot efficiently segment objects of variable sizes and train on small and biased datasets, which are common for biomedical use cases. While methods exist that incorporate multi-scale fusion approaches to address the challenges arising with variable sizes, they usually use complex models that are more suitable for general semantic segmentation problems. In this paper, we propose a novel architecture called Multi-Scale Residual Fusion Network (MSRF-Net), which is specially designed for medical image segmentation. The proposed MSRF-Net is able to exchange multi-scale features of varying receptive fields using a Dual-Scale Dense Fusion (DSDF) block. Our DSDF block can exchange information rigorously across two different resolution scales, and our MSRF sub-network uses multiple DSDF blocks in sequence to perform multi-scale fusion. This allows the preservation of resolution, improved information flow and propagation of both high- and low-level features to obtain accurate segmentation maps. The proposed MSRF-Net allows to capture object variabilities and provides improved results on different biomedical datasets. Extensive experiments on MSRF-Net demonstrate that the proposed method outperforms the cutting edge medical image segmentation methods on four publicly available datasets. We achieve the Dice Coefficient (DSC) of 0.9217, 0.9420, and 0.9224, 0.8824 on Kvasir-SEG, CVC-ClinicDB, 2018 Data Science Bowl dataset, and ISIC-2018 skin lesion segmentation challenge dataset respectively. We further conducted generalizability tests that also achieved the highest DSC score with 0.7921 and 0.7575 on CVC-ClinicDB and Kvasir-SEG, respectively.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | IEEE Journal of Biomedical and Health Informatics |
Volume | 26 |
Issue | 5 |
Pagination | 2252-2263 |
Date Published | 12/2021 |
Publisher | IEEE |
ISSN | 2168-2194 |
URL | https://ieeexplore.ieee.org/document/9662196 |
DOI | 10.1109/JBHI.2021.3138024 |
On evaluation metrics for medical applications of artificial intelligence
Scientific Reports 12 (2022): 1-9.Status: Published
On evaluation metrics for medical applications of artificial intelligence
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Scientific Reports |
Volume | 12 |
Number | 1 |
Pagination | 1–9 |
Publisher | Nature Publishing Group |
Prediction of Schizophrenia from Activity Data using Hidden Markov Model Parameters
Neural Computing and Applications (2022).Status: Published
Prediction of Schizophrenia from Activity Data using Hidden Markov Model Parameters
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Neural Computing and Applications |
Publisher | Springer |
Real-time deep learning based multi object tracking of spermatozoa in fresh samples
Human Reproduction 37, no. Supplement_1, July 2022, deac107.104 (2022).Status: Published
Real-time deep learning based multi object tracking of spermatozoa in fresh samples
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Human Reproduction |
Volume | 37 |
Issue | Supplement_1, July 2022, deac107.104 |
Date Published | 06/2022 |
Publisher | Oxford University Press |
DOI | 10.1093/humrep/deac107.104 |
SinGAN-Seg: Synthetic training data generation for medical image segmentation
PLOS ONE 17, no. 5 (2022): e0267976.Status: Published
SinGAN-Seg: Synthetic training data generation for medical image segmentation
Analyzing medical data to find abnormalities is a time-consuming and costly task, particularly for rare abnormalities, requiring tremendous efforts from medical experts. Therefore, artificial intelligence has become a popular tool for the automatic processing of medical data, acting as a supportive tool for doctors. However, the machine learning models used to build these tools are highly dependent on the data used to train them. Large amounts of data can be difficult to obtain in medicine due to privacy reasons, expensive and time-consuming annotations, and a general lack of data samples for infrequent lesions. In this study, we present a novel synthetic data generation pipeline, called SinGAN-Seg, to produce synthetic medical images with corresponding masks using a single training image. Our method is different from the traditional generative adversarial networks (GANs) because our model needs only a single image and the corresponding ground truth to train. We also show that the synthetic data generation pipeline can be used to produce alternative artificial segmentation datasets with corresponding ground truth masks when real datasets are not allowed to share. The pipeline is evaluated using qualitative and quantitative comparisons between real data and synthetic data to show that the style transfer technique used in our pipeline significantly improves the quality of the generated data and our method is better than other state-of-the-art GANs to prepare synthetic images when the size of training datasets are limited. By training UNet++ using both real data and the synthetic data generated from the SinGAN-Seg pipeline, we show that the models trained on synthetic data have very close performances to those trained on real data when both datasets have a considerable amount of training data. In contrast, we show that synthetic data generated from the SinGAN-Seg pipeline improves the performance of segmentation models when training datasets do not have a considerable amount of data. All experiments were performed using an open dataset and the code is publicly available on GitHub.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | PLOS ONE |
Volume | 17 |
Issue | 5 |
Pagination | e0267976 |
Date Published | 05/2022 |
Publisher | PLOS |
URL | https://doi.org/10.1371/journal.pone.0267976 |
DOI | 10.1371/journal.pone.0267976 |
Synthesizing a Talking Child Avatar to Train Interviewers Working with Maltreated Children
Big Data and Cognitive Computing 6, no. 2 (2022): 62.Status: Published
Synthesizing a Talking Child Avatar to Train Interviewers Working with Maltreated Children
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Big Data and Cognitive Computing |
Volume | 6 |
Issue | 2 |
Pagination | 62 |
Date Published | Jan-06-2022 |
Publisher | MDPI |
URL | https://www.mdpi.com/2504-2289/6/2/62https://www.mdpi.com/2504-2289/6/2/... |
DOI | 10.3390/bdcc6020062 |
Towards the Neuroevolution of Low-level artificial general intelligence
Frontiers in Robotics and AI 9 (2022).Status: Published
Towards the Neuroevolution of Low-level artificial general intelligence
In this work, we argue that the search for Artificial General Intelligence should start from a much lower level than human-level intelligence. The circumstances of intelligent behavior in nature resulted from an organism interacting with its surrounding environment, which could change over time and exert pressure on the organism to allow for learning of new behaviors or environment models. Our hypothesis is that learning occurs through interpreting sensory feedback when an agent acts in an environment. For that to happen, a body and a reactive environment are needed. We evaluate a method to evolve a biologically- inspired artificial neural network that learns from environment reactions named Neuroevolution of Artificial General Intelligence, a framework for low-level artificial general intelligence. This method allows the evolutionary complexification of a randomly-initialized spiking neural network with adaptive synapses, which controls agents instantiated in mutable environments. Such a configuration allows us to benchmark the adaptivity and generality of the controllers. The chosen tasks in the mutable environments are food foraging, emulation of logic gates, and cart-pole balancing. The three tasks are successfully solved with rather small network topologies and therefore it opens up the possibility of experimenting with more complex tasks and scenarios where curriculum learning is beneficial.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Frontiers in Robotics and AI |
Volume | 9 |
Date Published | 10/2022 |
Publisher | Frontiers |
URL | https://www.frontiersin.org/articles/10.3389/frobt.2022.1007547/full |
DOI | 10.3389/frobt.2022.1007547 |
Visual Sentiment Analysis from Disaster Images in Social Media
Sensors 22 (2022): 3628.Status: Published
Visual Sentiment Analysis from Disaster Images in Social Media
The increasing popularity of social networks and users’ tendency towards sharing their feelings, expressions, and opinions in text, visual, and audio content have opened new opportunities and challenges in sentiment analysis. While sentiment analysis of text streams has been widely explored in the literature, sentiment analysis from images and videos is relatively new. This article focuses on visual sentiment analysis in a societally important domain, namely disaster analysis in social media. To this aim, we propose a deep visual sentiment analyzer for disaster-related images, covering different aspects of visual sentiment analysis starting from data collection, annotation, model selection, implementation, and evaluations. For data annotation and analyzing people’s sentiments towards natural disasters and associated images in social media, a crowd-sourcing study has been conducted with a large number of participants worldwide. The crowd-sourcing study resulted in a large-scale benchmark dataset with four different sets of annotations, each aiming at a separate task. The presented analysis and the associated dataset, which is made public, will provide a baseline/benchmark for future research in the domain. We believe the proposed system can contribute toward more livable communities by helping different stakeholders, such as news broadcasters, humanitarian organizations, as well as the general public.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2022 |
Journal | Sensors |
Volume | 22 |
Number | 10 |
Pagination | 3628 |
Date Published | 05/2022 |
Publisher | MDPI |
URL | https://doi.org/10.3390/s22103628 |
DOI | 10.3390/s22103628 |
Poster
Automatic Thumbnail Selection for Soccer using Machine Learning
NORA Annual Conference, Stavanger, Norway, 2022.Status: Accepted
Automatic Thumbnail Selection for Soccer using Machine Learning
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Poster |
Year of Publication | 2022 |
Place Published | NORA Annual Conference, Stavanger, Norway |
Predicting drug exposure in kidney transplanted patients using machine learning
NORA Annual Conference, Stavanger, Norway, 2022.Status: Published
Predicting drug exposure in kidney transplanted patients using machine learning
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Poster |
Year of Publication | 2022 |
Place Published | NORA Annual Conference, Stavanger, Norway |
Type of Work | Poster presentation |
Talks, invited
Explainable Artificial Intelligence in Medicine
In Nordic AI Meet 2022, 2022.Status: Accepted
Explainable Artificial Intelligence in Medicine
Machine learning (ML) has shown outstanding abilities to solve a large variety of tasks such as image recognition and natural language processing, which has huge relevance for the medical field. Complex ML models, including convolutional neural networks (CNNs), are used to analyse high dimensional data such as images and videos from medical examinations. With increasing model complexity, the demand for techniques improving human understanding of the ML models also increases. If medical doctors do not understand how the models work, they might not know when the models are actually wrong or even refuse to use them. This can hamper the implementation of ML systems in the clinic and negatively affect patients. To promote successful integration of ML systems in the clinic, it is important to provide explanations that establish trust in the models among healthcare personnel. Explainable artificial intelligence (XAI) aims to provide explanations about ML models and their predictions. Several techniques have already been developed. Existing XAI methods often fail to meet the requirements of medical doctors, probably because they are not sufficiently involved in the development of the methods. We develop ML models solving tasks in various medical domains. The resulting models are explained using a selection of existing XAI methods, and the explanations are evaluated by medical experts. Their feedback is used to develop improved XAI methods. We have investigated established techniques for making ML systems more transparent in the fields of gastroenterology, assisted reproductive technology, organ transplantation and cardiology. Experiences from our projects will be used to develop new explanation techniques for clinical practice in close collaboration with medical experts.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Talks, invited |
Year of Publication | 2022 |
Location of Talk | Nordic AI Meet 2022 |
Keywords | Explainable artificial intelligence, Machine learning, medicine |
Soccer Athlete Performance Prediction using Time Series Analysis
In NORA Annual Conference, 2022.Status: Accepted
Soccer Athlete Performance Prediction using Time Series Analysis
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Talks, invited |
Year of Publication | 2022 |
Location of Talk | NORA Annual Conference |
Talk, keynote
From a need, to an idea, to a complete system: a perspective based on real-world applications
In IEEE/ACM International Symposium on Quality of Service (virtual). IEEE, 2022.Status: Published
From a need, to an idea, to a complete system: a perspective based on real-world applications
Researchers and developers have been talking about what determines a good system in various aspects for a long time, and eventually, it is all about quality. We are interested in quality throughout the entire holistic system, from data production to data consumption, and we have for several years aimed to solve real-world problems with a focus on both functional and non-functional requirements. This includes optimizing the various steps of both workflow and computing pipelines.
In this talk, we will address quality at various levels and aspects within a holistic system. We will highlight some examples starting from challenges in society, using sports and health as showcases, via researching solutions where we optimize workflows, system performance and user experiences, to observations of real use and deployments of the systems. In particular, the talk will cover a video and metadata system for commercial sport production and dissemination as an integrated, next generation live streaming and archive system. We will also use an example from an AI-based medical computer assisted diagnosis system to detect anomalies during colonoscopy examinations. The talk will look at various solutions and the state of the art, and will conclude with various open challenges.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Talk, keynote |
Year of Publication | 2022 |
Location of Talk | IEEE/ACM International Symposium on Quality of Service (virtual) |
Date Published | 06/2022 |
Publisher | IEEE |
Type of Talk | Keynote |
URL | https://iwqos2022.ieee-iwqos.org/program/keynote/ |
Book Chapter
Smittestopp analytics: Analysis of position data
In Smittestopp − A Case Study on Digital Contact Tracing, 63-79. Vol. 11. Cham: Springer International Publishing, 2022.Status: Published
Smittestopp analytics: Analysis of position data
Contact tracing applications generally rely on Bluetooth data. This type of data works well to determine whether a contact occurred (smartphones were close to each other) but cannot offer the contextual information GPS data can offer. Did the contact happen on a bus? In a building? And of which type? Are some places recurrent contact locations? By answering such questions, GPS data can help develop more accurate and better-informed contact tracing applications. This chapter describes the ideas and approaches implemented for GPS data within the Smittestopp contact tracing application.We will present the pipeline used and the contribution of GPS data for contextual information, using inferred transport modes and surrounding POIs, showcasing the opportunities in the use of GPS information. Finally,we discuss ethical and privacy considerations, as well as some lessons learned.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Book Chapter |
Year of Publication | 2022 |
Book Title | Smittestopp − A Case Study on Digital Contact Tracing |
Volume | 11 |
Pagination | 63 - 79 |
Date Published | 06/2022 |
Publisher | Springer International Publishing |
Place Published | Cham |
ISBN Number | 978-3-031-05465-5 |
ISBN | 2512-1677 |
URL | https://link.springer.com/content/pdf/10.1007/978-3-031-05466-2_4 |
DOI | 10.1007/978-3-031-05466-2_4 |
Smittestopp Backend
In Smittestopp − A Case Study on Digital Contact Tracing, 29-62. Vol. 11. Cham: Springer International Publishing, 2022.Status: Published
Smittestopp Backend
An efficient backend solution is of great importance for any large-scale system, and Smittestopp is no exception. The Smittestopp backend comprises various components for user and device registration, mobile app data ingestion, database and cloud operations, and web interface support. This chapter describes our journey from a vague idea to a deployed system. We provide an overview of the system internals and design iterations and discuss the challenges that we faced during the development process, along with the lessons learned. The Smittestopp backend handled around 1.5 million registered devices and provided various insights and analyses before being discontinued a few months after its launch.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Book Chapter |
Year of Publication | 2022 |
Book Title | Smittestopp − A Case Study on Digital Contact Tracing |
Volume | 11 |
Pagination | 29 - 62 |
Date Published | 06/2022 |
Publisher | Springer International Publishing |
Place Published | Cham |
ISBN Number | 978-3-031-05465-5 |
ISBN | 2512-1677 |
URL | https://link.springer.com/content/pdf/10.1007/978-3-031-05466-2.pdf |
DOI | 10.1007/978-3-031-05466-2_3 |
Journal Article
A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging
Medical Image Analysis 70 (2021): 102007.Status: Published
A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging
Gastrointestinal endoscopy has been an active field of research motivated by the large number of highly lethal GI cancers. Early GI cancer precursors are often missed during the endoscopic surveillance. The high missed rate of such abnormalities during endoscopy is thus a critical bottleneck. Lack of attentiveness due to tiring procedures, and requirement of training are few contributing factors. An automatic GI disease classification system can help reduce such risks by flagging suspicious frames and lesions. GI endoscopy consists of several multi-organ surveillance, therefore, there is need to develop methods that can generalize to various endoscopic findings. In this realm, we present a comprehensive analysis of the Medico GI challenges: Medical Multimedia Task at MediaEval 2017, Medico Multimedia Task at MediaEval 2018, and BioMedia ACM MM Grand Challenge 2019. These challenges are initiative to set-up a benchmark for different computer vision methods applied to the multi-class endoscopic images and promote to build new approaches that could reliably be used in clinics. We report the performance of 21 participating teams over a period of three consecutive years and provide a detailed analysis of the methods used by the participants, highlighting the challenges and shortcomings of the current approaches and dissect their credibility for the use in clinical settings. Our analysis revealed that the participants achieved an improvement on maximum Mathew correlation coefficient (MCC) from 82.68% in 2017 to 93.98% in 2018 and 95.20% in 2019 challenges, and a significant increase in computational speed over consecutive years.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Medical Image Analysis |
Volume | 70 |
Pagination | 102007 |
Publisher | Elsevier |
Keywords | Artificial intelligence, BioMedia 2019 Grand Challenge, Computer-aided detection and diagnosis, Gastrointestinal endoscopy challenges, Medical imaging, Medico Task 2017, Medico Task 2018 |
DOI | 10.1016/j.media.2021.102007 |
A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation
IEEE Journal of Biomedical and Health Informatics 25, no. 6 (2021): 2029-2040.Status: Published
A Comprehensive Study on Colorectal Polyp Segmentation with ResUNet++, Conditional Random Field and Test-Time Augmentation
Colonoscopy is considered the gold standard for detection of colorectal cancer and its precursors. Existing examination methods are, however, hampered by high overall miss-rate, and many abnormalities are left undetected. Computer-Aided Diagnosis systems based on advanced machine learning algorithms are touted as a game-changer that can identify regions in the colon overlooked by the physicians during endoscopic examinations, and help detect and characterize lesions. In previous work, we have proposed the ResUNet++ architecture and demonstrated that it produces more efficient results compared with its counterparts U-Net and ResUNet. In this paper, we demonstrate that further improvements to the overall prediction performance of the ResUNet++ architecture can be achieved by using Conditional Random Field (CRF) and Test-Time Augmentation (TTA). We have performed extensive evaluations and validated the improvements using six publicly available datasets: Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS-Larib PolypDB, ASU-Mayo Clinic Colonoscopy Video Database, and CVC-VideoClinicDB. Moreover, we compare our proposed architecture and resulting model with other State-of-the-art methods. To explore the generalization capability of ResUNet++ on different publicly available polyp datasets, so that it could be used in a real-world setting, we performed an extensive cross-dataset evaluation. The experimental results show that applying CRF andTTA improves the performance on various polyp segmentation datasets both on the same dataset and cross-dataset. To check the model’s performance on difficult to detect polyps, we selected, with the help of an expert gastroenterologist,196sessile or flat polyps that are less than ten millimeters in size. This additional data has been made available as a subset of Kvasir-SEG. Our approaches showed good results for flat or sessile and smaller polyps, which are known to be one of the major reasons for high polyp miss-rates. This is one of the significant strengths of our work and indicates that our methods should be investigated further for use in clinical practice.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | IEEE Journal of Biomedical and Health Informatics |
Volume | 25 |
Issue | 6 |
Pagination | 2029 - 2040 |
Publisher | IEEE |
Keywords | colonoscopy, conditional random field, generalization, Polyp segmentation, ResUNet++, test-time augmentation |
DOI | 10.1109/JBHI.2021.3049304 |
AI-Based Video Clipping of Soccer Events
Machine Learning and Knowledge Extraction 3, no. 4 (2021): 990-1008.Status: Published
AI-Based Video Clipping of Soccer Events
The current gold standard for extracting highlight clips from soccer games is the use of manual annotations and clippings, where human operators define the start and end of an event and trim away the unwanted scenes. This is a tedious, time-consuming, and expensive task, to the extent of being rendered infeasible for use in lower league games. In this paper, we aim to automate the process of highlight generation using logo transition detection, scene boundary detection, and optional scene removal. We experiment with various approaches, using different neural network architectures on different datasets, and present two models that automatically find the appropriate time interval for extracting goal events. These models are evaluated both quantitatively and qualitatively, and the results show that we can detect logo and scene transitions with high accuracy and generate highlight clips that are highly acceptable for viewers. We conclude that there is considerable potential in automating the overall soccer video clipping process.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Machine Learning and Knowledge Extraction |
Volume | 3 |
Issue | 4 |
Pagination | 990 - 1008 |
Date Published | 12/2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2504-4990/3/4/49/pdf |
DOI | 10.3390/make3040049 |
Artificial intelligence in the fertility clinic: status, pitfalls and possibilities
Human Reproduction 36, no. 9 (2021): 2429-2442.Status: Published
Artificial intelligence in the fertility clinic: status, pitfalls and possibilities
In recent years, the amount of data produced in the field of ART has increased exponentially. The diversity of data is large, ranging from videos to tabular data. At the same time, artificial intelligence (AI) is progressively used in medical practice and may become a promising tool to improve success rates with ART. AI models may compensate for the lack of objectivity in several critical procedures in fertility clinics, especially embryo and sperm assessments. Various models have been developed, and even though several of them show promising performance, there are still many challenges to overcome. In this review, we present recent research on AI in the context of ART. We discuss the strengths and weaknesses of the presented methods, especially regarding clinical relevance. We also address the pitfalls hampering successful use of AI in the clinic and discuss future possibilities and important aspects to make AI truly useful for ART.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Human Reproduction |
Volume | 36 |
Issue | 9 |
Pagination | 2429 - 2442 |
Date Published | 07/2021 |
Publisher | Oxford Academic |
ISSN | 0268-1161 |
URL | https://academic.oup.com/humrep/article/36/9/2429/6330662 |
DOI | 10.1093/humrep/deab168 |
Automated Event Detection and Classification in Soccer: The Potential of Using Multiple Modalities
Machine Learning and Knowledge Extraction 3, no. 4 (2021): 1030-1054.Status: Published
Automated Event Detection and Classification in Soccer: The Potential of Using Multiple Modalities
Detecting events in videos is a complex task, and many different approaches, aimed at a large variety of use-cases, have been proposed in the literature. Most approaches, however, are unimodal and only consider the visual information in the videos. This paper presents and evaluates different approaches based on neural networks where we combine visual features with audio features to detect (spot) and classify events in soccer videos. We employ model fusion to combine different modalities such as video and audio, and test these combinations against different state-of-the-art models on the SoccerNet dataset. The results show that a multimodal approach is beneficial. We also analyze how the tolerance for delays in classification and spotting time, and the tolerance for prediction accuracy, influence the results. Our experiments show that using multiple modalities improves event detection performance for certain types of events.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Machine Learning and Knowledge Extraction |
Volume | 3 |
Issue | 4 |
Pagination | 1030 - 1054 |
Date Published | 12/2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2504-4990/3/4/51/pdf |
DOI | 10.3390/make3040051 |
Deep learning neural network can measure ECG intervals and amplitudes accurately
Journal of Electrocardiology 69 (2021): 82.Status: Published
Deep learning neural network can measure ECG intervals and amplitudes accurately
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Journal of Electrocardiology |
Volume | 69 |
Pagination | 82 |
Publisher | Elsevier |
DeepFake electrocardiograms using generative adversarial networks are the beginning of the end for privacy issues in medicine
Nature Scientific Reports 11 (2021): 21896.Status: Published
DeepFake electrocardiograms using generative adversarial networks are the beginning of the end for privacy issues in medicine
Recent global developments underscore the prominent role big data have in modern medical science. But privacy issues constitute a prevalent problem for collecting and sharing data between researchers. However, synthetic data generated to represent real data carrying similar information and distribution may alleviate the privacy issue. In this study, we present generative adversarial networks (GANs) capable of generating realistic synthetic DeepFake 10-s 12-lead electrocardiograms (ECGs). We have developed and compared two methods, named WaveGAN* and Pulse2Pulse. We trained the GANs with 7,233 real normal ECGs to produce 121,977 DeepFake normal ECGs. By verifying the ECGs using a commercial ECG interpretation program (MUSE 12SL, GE Healthcare), we demonstrate that the Pulse2Pulse GAN was superior to the WaveGAN* to produce realistic ECGs. ECG intervals and amplitudes were similar between the DeepFake and real ECGs. Although these synthetic ECGs mimic the dataset used for creation, the ECGs are not linked to any individuals and may thus be used freely. The synthetic dataset will be available as open access for researchers at OSF.io and the DeepFake generator available at the Python Package Index (PyPI) for generating synthetic ECGs. In conclusion, we were able to generate realistic synthetic ECGs using generative adversarial neural networks on normal ECGs from two population studies, thereby addressing the relevant privacy issues in medical datasets.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Nature Scientific Reports |
Volume | 11 |
Pagination | 21896 |
Date Published | 09/2021 |
Publisher | Springer nature |
URL | https://www.nature.com/articles/s41598-021-01295-2 |
DOI | 10.1038/s41598-021-01295-2 |
DeepFake electrocardiograms: the key for open science for artificial intelligence in medicine
Scientific Reports (2021).Status: Accepted
DeepFake electrocardiograms: the key for open science for artificial intelligence in medicine
Recent global developments underscore the prominent role big data have in modern medical science. Privacy issues are a prevalent problem for collecting and sharing data between researchers. Synthetic data generated to represent real data carrying similar information and distribution may alleviate the privacy issue.In this study, we present generative adversarial networks (GANs) capable of generating realistic synthetic DeepFake 12-lead 10-sec electrocardiograms (ECGs). We have developed and compare two methods, WaveGAN* and Pulse2Pulse GAN. We trained the GANs with 7,233 real normal ECG to produce 121,977 DeepFake normal ECGs. By verifying the ECGs using a commercial ECG interpretation program (MUSE 12SL, GE Healthcare), we demonstrate that the Pulse2Pulse GAN was superior to the WaveGAN to produce realistic ECGs. ECG intervals and amplitudes were similar between the DeepFake and real ECGs. These synthetic ECGs are fully anonymous and cannot be referred to any individual, hence they may be used freely. The synthetic dataset will be available as open access for researchers at OSF.io and the DeepFake generator available at the Python Package Index (PyPI) for generating synthetic ECGs.In conclusion, we were able to generate realistic synthetic ECGs using adversarial neural networks on normal ECGs from two population studies, i.e., there by addressing the relevant privacy issues in medical datasets.Competing Interest StatementThe authors have declared no competing interest.Clinical TrialN/AFunding StatementThis work is funded in part by Novo Nordisk Foundation project number NNF18CC0034900.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:We confirm that all experiments were performed in accordance with Helsinki guidelines and regulations of the Danish Regional Committees for Medical and Health Research Ethics. The data studies were approved by the ethical committee of Region Zealand (SJ-113, SJ-114, SJ-191), ethical committee of Copenhagen Amt (KA 98 155).All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesThe Normal DeepFake ECGs are available at OSF (https://osf.io/6hved/) with corresponding MUSE 12SL ground truth values freely downloadable and usable for ECG algorithm development. The DeepFake generative model is available at https://pypi.org/project/deepfake-ecg/ to generate only synthetic ECGs. https://osf.io/6hved/ https://pypi.org/project/deepfake-ecg/
Afilliation | Scientific Computing, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Scientific Reports |
Publisher | Nature Publishing Group |
URL | https://www.medrxiv.org/content/early/2021/05/10/2021.04.27.21256189.1 |
DOI | 10.1101/2021.04.27.21256189 |
Explaining deep neural networks for knowledge discovery in electrocardiogram analysis
Scientific Reports 11 (2021): 10949.Status: Published
Explaining deep neural networks for knowledge discovery in electrocardiogram analysis
Deep learning-based tools may annotate and interpret medical data more quickly, consistently, and accurately than medical doctors. However, as medical doctors are ultimately responsible for clinical decision-making, any deep learning-based prediction should be accompanied by an explanation that a human can understand. We present an approach called electrocardiogram gradient class activation map (ECGradCAM), which is used to generate attention maps and explain the reasoning behind deep learning-based decision-making in ECG analysis. Attention maps may be used in the clinic to aid diagnosis, discover new medical knowledge, and identify novel features and characteristics of medical tests. In this paper, we showcase how ECGradCAM attention maps can unmask how a novel deep learning model measures both amplitudes and intervals in 12-lead electrocardiograms, and we show an example of how attention maps may be used to develop novel ECG features.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Scientific Reports |
Volume | 11 |
Pagination | 10949 |
Date Published | 05/2021 |
Publisher | Springer Nature |
URL | http://www.nature.com/articles/s41598-021-90285-5h |
DOI | 10.1038/s41598-021-90285-5 |
File System Support for Privacy-Preserving Analysis and Forensics in Low-Bandwidth Edge Environments
Information 12, no. 10 (2021): 430.Status: Published
File System Support for Privacy-Preserving Analysis and Forensics in Low-Bandwidth Edge Environments
In this paper, we present initial results from our distributed edge systems research in the domain of sustainable harvesting of common good resources in the Arctic Ocean. Specifically, we are developing a digital platform for real-time privacy-preserving sustainability management in the domain of commercial fishery surveillance operations. This is in response to potentially privacy-infringing mandates from some governments to combat overfishing and other sustainability challenges. Our approach is to deploy sensory devices and distributed artificial intelligence algorithms on mobile, offshore fishing vessels and at mainland central control centers. To facilitate this, we need a novel data plane supporting efficient, available, secure, tamper-proof, and compliant data management in this weakly connected offshore environment. We have built our first prototype of Dorvu, a novel distributed file system in this context. Our devised architecture, the design trade-offs among conflicting properties, and our initial experiences are further detailed in this paper.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Information |
Volume | 12 |
Issue | 10 |
Pagination | 430 |
Date Published | 10/2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2078-2489/12/10/430 |
DOI | 10.3390/info12100430 |
Impact of Image Resolution on Deep Learning Performance in Endoscopy Image Classification: An Experimental Study Using a Large Dataset of Endoscopic Images
Diagnostics 11, no. 12 (2021).Status: Published
Impact of Image Resolution on Deep Learning Performance in Endoscopy Image Classification: An Experimental Study Using a Large Dataset of Endoscopic Images
Recent trials have evaluated the efficacy of deep convolutional neural network (CNN)-based AI systems to improve lesion detection and characterization in endoscopy. Impressive results are achieved, but many medical studies use a very small image resolution to save computing resources at the cost of losing details. Today, no conventions between resolution and performance exist, and monitoring the performance of various CNN architectures as a function of image resolution provides insights into how subtleties of different lesions on endoscopy affect performance. This can help set standards for image or video characteristics for future CNN-based models in gastrointestinal (GI) endoscopy. This study examines the performance of CNNs on the HyperKvasir dataset, consisting of 10,662 images from 23 different findings. We evaluate two CNN models for endoscopic image classification under quality distortions with image resolutions ranging from 32 × 32 to 512 × 512 pixels. The performance is evaluated using two-fold cross-validation and F1-score, maximum Matthews correlation coefficient (MCC), precision, and sensitivity as metrics. Increased performance was observed with higher image resolution for all findings in the dataset. MCC was achieved at image resolutions between 512 × 512 pixels for classification for the entire dataset after including all subclasses. The highest performance was observed with an MCC value of 0.9002 when the models were trained on the highest resolution and tested on the same resolution. Different resolutions and their effect on CNNs are explored. We show that image resolution has a clear influence on the performance which calls for standards in the field in the future.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Diagnostics |
Volume | 11 |
Issue | 12 |
Date Published | 09/2021 |
Publisher | MDPI |
URL | https://www.mdpi.com/2075-4418/11/12/2183 |
DOI | 10.3390/diagnostics11122183 |
Kvasir-Capsule, a video capsule endoscopy dataset
Scientific Data 8, no. 1 (2021): 142.Status: Published
Kvasir-Capsule, a video capsule endoscopy dataset
Artificial intelligence (AI) is predicted to have profound effects on the future of video capsule endoscopy (VCE) technology. The potential lies in improving anomaly detection while reducing manual labour. Existing work demonstrates the promising benefits of AI-based computer-assisted diagnosis systems for VCE. They also show great potential for improvements to achieve even better results. Also, medical data is often sparse and unavailable to the research community, and qualified medical personnel rarely have time for the tedious labelling work. We present Kvasir-Capsule, a large VCE dataset collected from examinations at a Norwegian Hospital. Kvasir-Capsule consists of 117 videos which can be used to extract a total of 4,741,504 image frames. We have labelled and medically verified 47,238 frames with a bounding box around findings from 14 different classes. In addition to these labelled images, there are 4,694,266 unlabelled frames included in the dataset. The Kvasir-Capsule dataset can play a valuable role in developing better algorithms in order to reach true potential of VCE technology.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Scientific Data |
Volume | 8 |
Issue | 1 |
Pagination | 142 |
Publisher | Springer Nature |
URL | http://www.nature.com/articles/s41597-021-00920-z |
DOI | 10.1038/s41597-021-00920-z |
MedAI: Transparency in Medical Image Segmentation
Nordic Machine Intelligence 1, no. 1 (2021): 1-4.Status: Published
MedAI: Transparency in Medical Image Segmentation
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | Nordic Machine Intelligence |
Volume | 1 |
Issue | 1 |
Pagination | 1 - 4 |
Date Published | Jan-11-2021 |
Publisher | NMI |
Place Published | Oslo |
URL | https://journals.uio.no/NMI/article/view/9140https://journals.uio.no/NMI... |
DOI | 10.5617/nmi.9140 |
Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning
IEEE Access 9 (2021): 40496-40510.Status: Published
Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning
Computer-aided detection, localization, and segmentation methods can help improve colonoscopy procedures. Even though many methods have been built to tackle automatic detection and segmentation of polyps, benchmarking of state-of-the-art methods still remains an open problem. This is due to the increasing number of researched computer vision methods that can be applied to polyp datasets. Benchmarking of novel methods can provide a direction to the development of automated polyp detection and segmentation tasks. Furthermore, it ensures that the produced results in the community are reproducible and provide a fair comparison of developed methods. In this paper, we benchmark several recent state-of-the-art methods using Kvasir-SEG, an open-access dataset of colonoscopy images for polyp detection, localization, and segmentation evaluating both method accuracy and speed. Whilst, most methods in literature have competitive performance over accuracy, we show that the proposed ColonSegNet achieved a better trade-off between an average precision of 0.8000 and mean IoU of 0.8100, and the fastest speed of 180 frames per second for the detection and localization task. Likewise, the proposed ColonSegNetachieved a competitive dice coefficient of 0.8206 and the best average speed of 182.38 frames per second for the segmentation task. Our comprehensive comparison with various state-of-the-art methods reveals the importance of benchmarking the deep learning methods for automated real-time polyp identification and delineations that can potentially transform current clinical practices and minimize miss-detection rates.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | IEEE Access |
Volume | 9 |
Pagination | 40496-40510 |
Publisher | IEEE |
Keywords | Medical image segmentation, ColonSegNet, colonoscopy, polyps, deep learning, detection, localization, benchmarking, Kvasir-SEG |
DOI | 10.1109/ACCESS.2021.3063716 |
SmartIO: Zero-overhead Device Sharing through PCIe Networking
ACM Transactions on Computer Systems 38, no. 1-2 (2021): 1-78.Status: Published
SmartIO: Zero-overhead Device Sharing through PCIe Networking
The large variety of compute-heavy and data-driven applications accelerate the need for a distributed I/O solution that enables cost-effective scaling of resources between networked hosts. For example, in a cluster system, different machines may have various devices available at different times, but moving workloads to remote units over the network is often costly and introduces large overheads compared to accessing local resources. To facilitate I/O disaggregation and device sharing among hosts connected using Peripheral Component Interconnect Express (PCIe) non-transparent bridges, we present SmartIO. NVMes, GPUs, network adapters, or any other standard PCIe device may be borrowed and accessed directly, as if they were local to the remote machines. We provide capabilities beyond existing disaggregation solutions by combining traditional I/O with distributed shared-memory functionality, allowing devices to become part of the same global address space as cluster applications. Software is entirely removed from the data path, and simultaneous sharing of a device among application processes running on remote hosts is enabled. Our experimental results show that I/O devices can be shared with remote hosts, achieving native PCIe performance. Thus, compared to existing device distribution mechanisms, SmartIO provides more efficient, low-cost resource sharing, increasing the overall system performance
Afilliation | Communication Systems, Machine Learning |
Project(s) | Unified PCIe IO: Unified PCI Express for Distributed Component Virtualization, Department of Holistic Systems, Department of High Performance Computing |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | ACM Transactions on Computer Systems |
Volume | 38 |
Issue | 1-2 |
Number | 2 |
Pagination | 1–78 |
Date Published | 07/2021 |
Publisher | Association for Computing Machinery |
Place Published | New York, NY, United States |
ISSN | 0734-2071 |
URL | https://dl.acm.org/doi/10.1145/3462545 |
DOI | 10.1145/3462545 |
Using 3D Convolutional Neural Networks for Real-time Detection of Soccer Events
International Journal of Semantic Computing 15, no. 2 (2021): 161-187.Status: Published
Using 3D Convolutional Neural Networks for Real-time Detection of Soccer Events
Developing systems for the automatic detection of events in video is a task which has gained attention in many areas including sports. More specifically, event detection for soccer videos has been studied widely in the literature. However, there are still a number of shortcomings in the state-of-the-art such as high latency, making it challenging to operate at the live edge. In this paper, we present an algorithm to detect events in soccer videos in real time, using 3D convolutional neural networks. We test our algorithm on three different datasets from SoccerNet, the Swedish Allsvenskan, and the Norwegian Eliteserien. Overall, the results show that we can detect events with high recall, low latency, and accurate time estimation. The trade-off is a slightly lower precision compared to the current state-of-the-art, which has higher latency and performs better when a less accurate time estimation can be accepted. In addition to the presented algorithm, we perform an extensive ablation study on how the different parts of the training pipeline affect the final results.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Journal Article |
Year of Publication | 2021 |
Journal | International Journal of Semantic Computing |
Volume | 15 |
Issue | 2 |
Number | 2 |
Pagination | 161 - 187 |
Date Published | Jan-06-2021 |
Publisher | World Scientific |
ISSN | 1793-351X |
Keywords | 3d CNN, classification, Detection, soccer events, spotting |
URL | https://www.worldscientific.com/doi/abs/10.1142/S1793351X2140002X |
DOI | 10.1142/S1793351X2140002X |
Book Chapter
Artificial Intelligence in Gastroenterology
In Artificial Intelligence in Medicine, 1-20. Cham: Springer International Publishing, 2021.Status: Published
Artificial Intelligence in Gastroenterology
The holy grail in endoscopy examinations has for a long time been assisted diagnosis using Artificial Intelligence (AI). Recent developments in computer hardware are now enabling technology to equip clinicians with promising tools for computer-assisted diagnosis (CAD) systems. However, creating viable models or architectures, training them, and assessing their ability to diagnose at a human level, are complicated tasks. This is currently an active area of research, and many promising methods have been proposed. In this chapter, we give an overview of the topic. This includes a description of current medical challenges followed by a description of the most commonly used methods in the field. We also present example results from research targeting some of these challenges, and a discussion on open issues and ongoing work is provided. Hopefully, this will inspire and enable readers to future develop CAD systems for gastroenterology.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Book Chapter |
Year of Publication | 2021 |
Book Title | Artificial Intelligence in Medicine |
Pagination | 1 - 20 |
Date Published | 09/2021 |
Publisher | Springer International Publishing |
Place Published | Cham |
ISBN Number | 978-3-030-58080-3 |
Keywords | Anomaly detection, artificial intelligence, Gastrointestinal endoscopy, Hand-crafted features, Neural Networks, Performance, Semantic segmentation |
URL | https://link.springer.com/referenceworkentry/10.1007%2F978-3-030-58080-3... |
DOI | 10.1007/978-3-030-58080-3_163-2 |
Proceedings, refereed
Automated Clipping of Soccer Events using Machine Learning
In IEEE International Symphosium of Multimedia (ISM). IEEE, 2021.Status: Published
Automated Clipping of Soccer Events using Machine Learning
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | IEEE International Symphosium of Multimedia (ISM) |
Date Published | 12/2021 |
Publisher | IEEE |
DOI | 10.1109/ISM52913.2021.00042 |
Data Augmentation Using Generative Adversarial Networks For Creating Realistic Artificial Colon Polyp Images
In DDW 2021, 2021.Status: Published
Data Augmentation Using Generative Adversarial Networks For Creating Realistic Artificial Colon Polyp Images
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | DDW 2021 |
DDANet: Dual Decoder Attention Network for Automatic Polyp Segmentation
In 25th International Conference on Pattern Recognition. Springer, 2021.Status: Published
DDANet: Dual Decoder Attention Network for Automatic Polyp Segmentation
Colonoscopy is the gold standard for examination and detection of colorectal polyps. Localization and delineation of polyps can play a vital role in treatment (e.g., surgical planning) and prognostic decision making. Polyp segmentation can provide detailed boundary information for clinical analysis. Convolutional neural networks have improved the performance in colonoscopy. However, polyps usually possess various challenges, such as intra-and inter-class variation and noise. While manual labeling for polyp assessment requires time from experts and is prone to human error (e.g., missed lesions), an automated, accurate, and fast segmentation can improve the quality of delineated lesion boundaries and reduce missed rate. The Endotect challenge provides an opportunity to benchmark computer vision methods by training on the publicly available Hyperkvasir and testing on a separate unseen dataset. In this paper, we propose a novel architecture called ``DDANet'' based on a dual decoder attention network. Our experiments demonstrate that the model trained on the Kvasir-SEG dataset and tested on an unseen dataset achieves a dice coefficient of 0.7874, mIoU of 0.7010, recall of 0.7987, and a precision of 0.8577, demonstrating the generalization ability of our model.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 25th International Conference on Pattern Recognition |
Pagination | 307-314 |
Publisher | Springer |
Keywords | Benchmarking, Convolutional neural network, deep learning, Polyp segmentation |
DeepSynthBody: the beginning of the end for data deficiency in medicine
In The International Conference on Applied Artificial Intelligence (ICAPAI). IEEE, 2021.Status: Published
DeepSynthBody: the beginning of the end for data deficiency in medicine
Limited access to medical data is a barrier on developing new and efficient machine learning solutions in medicine such as computer-aided diagnosis, risk assessments, predicting optimal treatments and home-based personal healthcare systems. This paper presents DeepSynthBody: a novel framework that overcomes some of the inherent restrictions and limitations of medical data by using deep generative adversarial networks to produce synthetic data with characteristics similar to the real data, so-called DeepSynth (deep synthetic) data. We show that DeepSynthBody can address two key issues commonly associated with medical data, namely privacy concerns (as a result of data protection rules and regulations) and the high costs of annotations. To demonstrate the full pipeline of applying DeepSynthBody concepts and user-friendly functionalities, we also describe a synthetic medical dataset generated and published using our framework. DeepSynthBody opens a new era of machine learning applications in medicine with a synthetic model of the human body.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | The International Conference on Applied Artificial Intelligence (ICAPAI) |
Publisher | IEEE |
DOI | 10.1109/ICAPAI49758.2021.9462062 |
DivergentNets: Medical Image Segmentation by Network Ensemble
In EndoCV, 2021.Status: Accepted
DivergentNets: Medical Image Segmentation by Network Ensemble
Detection of colon polyps has become a trending topic in the intersecting fields of machine learning and gastrointestinal endoscopy. The focus has mainly been on per-frame classification. More recently, polyp segmentation has gained attention in the medical community. Segmentation has the advantage of being more accurate than per-frame classification or object detection as it can show the affected area in greater detail. For our contribution to the EndoCV 2021 segmentation challenge, we propose two separate approaches. First, a segmentation model named TriUNet composed of three separate UNet models. Second, we combine TriUNet with an ensemble of well-known segmentation models, namely UNet++, FPN, DeepLabv3, and DeepLabv3+, into a model called DivergentNets to produce more generalizable medical image segmentation masks. In addition, we propose a modified Dice loss that calculates loss only for a single class when performing multi-class segmentation, forcing the model to focus on what is most important. Overall, the proposed methods achieved the best average scores for each respective round in the challenge, with TriUNet being the winning model in Round I and DvergentNets being the winning model in Round II of the segmentation generalization challenge at EndoCV 2021
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | EndoCV |
Dutkat: A Multimedia System for Catching Illegal Catchers in a Privacy-Preserving Manner
In Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR'21). New York, NY, USA: ACM, 2021.Status: Published
Dutkat: A Multimedia System for Catching Illegal Catchers in a Privacy-Preserving Manner
Fish crime is considered a global and serious problem for a healthy and sustainable development of one of mankind's important sources of food. Technological surveillance and control solutions are emerging as remedies to combat criminal activities, but such solutions might also come with impractical and negative side-effects and challenges. In this paper, we present the concept and design of a surveillance system in lieu of current surveillance trends striking a delicate balance between privacy of legal actors while simultaneously capturing evidence-based footage, sensory data, and forensic proofs of illicit activities. Our proposed novel approach is to assist human operators in the 24/7 surveillance loop of remote professional fishing activities with a privacy-preserving Artificial Intelligence (AI) surveillance system operating in the same proximity as the activities being surveyed. The system will primarily be using video surveillance data, but also other sensor data captured on the fishing vessel. Additionally, the system correlates with other sources such as reports from other fish catches in the approximate area and time, etc. Only upon true positive flagging of specific potentially illicit activities by the locally executing AI algorithms, can forensic evidence be accessed from this physical edge, the fishing vessel. Besides a more privacy-preserving solution, our edge-based AI system also benefits from much less data that has to be transferred over unreliable, low-bandwidth satellite-based networks.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR'21) |
Pagination | 57-61 |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450385299 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3463944 |
DOI | 10.1145/346394410.1145/3463944.3469102 |
Exploring Deep Learning Methods for Real-Time Surgical Instrument Segmentation in Laparoscopy
In 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE, 2021.Status: Published
Exploring Deep Learning Methods for Real-Time Surgical Instrument Segmentation in Laparoscopy
Minimally Invasive Surgery (MIS) is a surgical intervention used to examine the organs inside the abdomen and has been widely used due to its effectiveness over open surgery. Due to the hardware improvements such as high definition cameras, this procedure has significantly improved and new software methods have open potential for computer-assisted procedures. However, there exists challenges and requirement to improve detection and tracking of the position of the instruments during these surgical procedures. To this end, we evaluate and compare some popular deep learning methods that can potentially be explored for the automated segmentation of surgical instruments in laparoscopy, an important step towards tool tracking. Our experimental results demonstrate that the Dual decoder attention network (DDANet) produces a superior result compared to other recent deep learning methods. DDANet produces a dice coefficient of 0.8739 and mean intersection over union of 0.8183 for the Robust Medical Instrument Segmentation (ROBUST-MIS) Challenge 2019 dataset, at a real-time speed of 101.36 frames per second which is critical for such procedures.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 2021 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI) |
Pagination | 1-4 |
Publisher | IEEE |
Keywords | Real-time segmentation, minimally invasive surgery, surgical instruments, laparoscopy, deep learning |
HTAD: A Home-Tasks Activities Dataset with Wrist-accelerometer and Audio Features
In 27th International Conference on Multimedia Modeling. Springer, 2021.Status: Published
HTAD: A Home-Tasks Activities Dataset with Wrist-accelerometer and Audio Features
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 27th International Conference on Multimedia Modeling |
Pagination | 196-205 |
Publisher | Springer |
Keywords | Accelerometer, Activity recognition, Audio, dataset, Sensor fusion |
HYPERAKTIV: An Activity Dataset from Patients with Attention-Deficit/Hyperactivity Disorder (ADHD)
In Proceedings of the 12th ACM Multimedia Systems Conference (MMSys '21). ACM, 2021.Status: Published
HYPERAKTIV: An Activity Dataset from Patients with Attention-Deficit/Hyperactivity Disorder (ADHD)
Machine learning research within healthcare frequently lacks the public data needed to be fully reproducible and comparable. Datasets are often restricted due to privacy concerns and legal requirements that come with patient-related data. Consequentially, many algorithms and models get published on the same topic without a standard benchmark to measure against. Therefore, this paper presents HYPERAKTIV, a public dataset containing health, activity, and heart rate data from patients diagnosed with attention deficit hyperactivity disorder, better known as ADHD. The dataset consists of data collected from 51 patients with ADHD and 52 clinical controls. In addition to the activity and heart rate data, we also include a series of patient attributes such as their age, sex, and information about their mental state, as well as output data from a computerized neuropsychological test. Together with the presented dataset, we also provide baseline experiments using traditional machine learning algorithms to predict ADHD based on the included activity data. We hope that this dataset can be used as a starting point for computer scientists who want to contribute to the field of mental health, and as a common benchmark for future work in ADHD analysis.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 12th ACM Multimedia Systems Conference (MMSys '21) |
Pagination | 314–319 |
Publisher | ACM |
URL | https://dl.acm.org/doi/10.1145/3458305.3478454 |
DOI | 10.1145/3458305.3478454 |
Identification of spermatozoa by unsupervised learning from video data
In 37th Virtual Annual Meeting of the European Society of Human Reproduction and Embryology (ESHRE). Oxford University Press, 2021.Status: Published
Identification of spermatozoa by unsupervised learning from video data
Identification of individual sperm is essential to assess a given sperm sample's motility behavior. Existing computer-aided systems need training data based on annotations by professionals, which is resource demanding. On the other hand, data analysed by unsupervised machine learning algorithms can improve supervised algorithms that are more stable for clinical applications. Therefore, unsupervised sperm identification can improve computer-aided sperm analysis systems predicting different aspects of sperm samples. Other possible applications are assessing kinematics and counting of spermatozoa.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 37th Virtual Annual Meeting of the European Society of Human Reproduction and Embryology (ESHRE) |
Publisher | Oxford University Press |
Impact of Image Resolution on Convolutional Neural Networks Performance in Gastrointestinal Endoscopy
In DDW 2021, 2021.Status: Published
Impact of Image Resolution on Convolutional Neural Networks Performance in Gastrointestinal Endoscopy
Convolutional neural networks (CNNs) are increasingly used to improve and automate processes in gastroenterology, like the detection of polyps during a colonoscopy. An important input to these methods is images and videos. Up until now, no well-defined, common understanding or standard regarding the resolution of the images and video frames has been defined, and to reduce processing time and resource requirements, images are today almost always down-sampled. However, how such down-sampling and the image resolution influence the performance in context with medical data is unknown. In this work, we investigate how the resolution relates to the performance of convolutional neural networks. This can help set standards for image or video characteristics for future CNN based models in gastrointestinal endoscopy.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | DDW 2021 |
Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy
In 27th International Conference on Multimedia Modeling. Vol. LNCS, volume 12573. Springer, 2021.Status: Published
Kvasir-Instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy
Gastrointestinal (GI) pathologies are periodically screened, biopsied, and resected using surgical tools. Usually the procedures and the treated or resected areas are not specifically tracked or analysed during or after colonoscopies. Information regarding disease borders, development and amount and size of the resected area get lost. This can lead to poor follow-up and bothersome reassessment difficulties post-treatment. To improve the current standard and also to foster more research on the topic we have released the ``Kvasir Instrument'' dataset which consists of $590$ annotated frames containing GI procedure tools such as snares, balloons and biopsy forceps, etc. Beside of the images, the dataset includes ground truth masks and bounding boxes and has been verified by two expert GI endoscopists. Additionally, we provide a baseline for the segmentation of the GI tools to promote research and algorithm development. We obtained a dice coefficient score of 0.9158 and a Jaccard index of 0.8578 using a classical U-Net architecture. A similar dice coefficient score was observed for DoubleUNet. The qualitative results showed that the model did not work for the images with specularity and the frames with multiple instruments, while the best result for both methods was observed on all other types of images. Both, qualitative and quantitative results show that the model performs reasonably good, but there is a large potential for further improvements. Benchmarking using the dataset provides an opportunity for researchers to contribute to the field of automatic endoscopic diagnostic and therapeutic tool segmentation for GI endoscopy.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 27th International Conference on Multimedia Modeling |
Volume | LNCS, volume 12573 |
Pagination | 218-229 |
Publisher | Springer |
Medico Multimedia Task at MediaEval 2021: Transparency in Medical Image Segmentation
In Mediaeval Medico 2021. Mediaeval 2021, 2021.Status: Published
Medico Multimedia Task at MediaEval 2021: Transparency in Medical Image Segmentation
The Medico Multimedia Task focuses on providing multimedia researchers with the opportunity to contribute to different areas of medicine using multimedia data to solve several subtasks. This year, the task focuses on transparency within machine learning-based medical segmentation systems, where the use case is gastrointestinal endoscopy. In this paper, we motivate the organization of this task, describe the development and test dataset, and present the evaluation process used to assess the participants' submissions.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Mediaeval Medico 2021 |
Publisher | Mediaeval 2021 |
URL | https://2021.multimediaeval.com/ |
Multimodal Virtual Avatars for Investigative Interviews with Children
In Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR '21). New York, NY, USA: ACM, 2021.Status: Published
Multimodal Virtual Avatars for Investigative Interviews with Children
In this article, we present our ongoing work in the field of training police officers who conduct interviews with abused children. The objectives in this context are to protect vulnerable children from abuse, facilitate prosecution of offenders, and ensure that innocent adults are not accused of criminal acts. There is therefore a need for more data that can be used for improved interviewer training to equip police with the skills to conduct high-quality interviews. To support this important task, we propose to research a training program that utilizes different system components and multimodal data from the field of artificial intelligence such as chatbots, generation of visual content, text-to-speech, and speech-to-text. This program will be able to generate an almost unlimited amount of interview and also training data. The goal of combining all these different technologies and datatypes is to create an immersive and interactive child avatar that responds in a realistic way, to help to support the training of police interviewers, but can also produce synthetic data of interview situations that can be used to solve different problems in the same domain.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proceedings of the 2021 Workshop on Intelligent Cross-Data Analysis and Retrieval (ICDAR '21) |
Publisher | ACM |
Place Published | New York, NY, USA |
ISBN Number | 9781450385299 |
URL | https://dl.acm.org/doi/proceedings/10.1145/3463944 |
DOI | 10.1145/346394410.1145/3463944.3469269 |
NanoNet: Real-Time Polyp Segmentation in Video Capsule Endoscopy and Colonoscopy
In 34th IEEE CBMS International Symposium on Computer-Based Medical Systems. IEEE, 2021.Status: Published
NanoNet: Real-Time Polyp Segmentation in Video Capsule Endoscopy and Colonoscopy
Deep learning in gastrointestinal endoscopy can assist to improve clinical performance and be helpful to assess lesions more accurately. To this extent, semantic segmentation methods that can perform automated real-time delineation of a region-of-interest, e.g., boundary identification of cancer or precancerous lesions, can benefit both diagnosis and interventions. However, accurate and real-time segmentation of endoscopic images is extremely challenging due to its high operator dependence and high-definition image quality. To utilize automated methods in clinical settings, it is crucial to design lightweight models with low latency such that they can be integrated with low-end endoscope hardware devices. In this work, we propose NanoNet, a novel architecture for the segmentation of video capsule endoscopy and colonoscopy images. Our proposed architecture allows real-time performance and has higher segmentation accuracy compared to other more complex ones. We use video capsule endoscopy and standard colonoscopy datasets with polyps, and a dataset consisting of endoscopy biopsies and surgical instruments, to evaluate the effectiveness of our approach. Our experiments demonstrate the increased performance of our architecture in terms of a trade-off between model complexity, speed, model parameters, and metric performances. Moreover, the resulting model size is relatively tiny, with only nearly 36,000 parameters compared to traditional deep learning approaches having millions of parameters.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | 34th IEEE CBMS International Symposium on Computer-Based Medical Systems |
Publisher | IEEE |
Keywords | colonoscopy, deep learning, segmentation, tool segmentation, Video capsule endoscopy |
Njord: An out-in-the-wild real world fish vessel catch analysis dataset
In Arctic Frontiers. Tromsø, Norway: Arctic Frontiers, 2021.Status: Published
Njord: An out-in-the-wild real world fish vessel catch analysis dataset
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Arctic Frontiers |
Publisher | Arctic Frontiers |
Place Published | Tromsø, Norway |
PAANet: Progressive Alternating Attention for Automatic Medical Image Segmentation
In Proc. of 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART). Paris / Créteil, France: IEEE, 2021.Status: Published
PAANet: Progressive Alternating Attention for Automatic Medical Image Segmentation
Medical image segmentation can provide detailed information for clinical analysis which can be useful for scenarios where the detailed location of a finding is important. Knowing the location of a disease can play a vital role in treatment and decision-making. Convolutional neural network (CNN) based encoder-decoder techniques have advanced the performance of automated medical image segmentation systems. Several such CNN-based methodologies utilize techniques such as spatial- and channel-wise attention to enhance performance. Another technique that has drawn attention in recent years is residual dense blocks (RDBs). The successive convolutional layers in densely connected blocks are capable of extracting diverse features with varied receptive fields and thus, enhancing performance. However, consecutive stacked convolutional operators may not necessarily generate features that facilitate the identification of the target structures. In this paper, we propose a progressive alternating attention network (PAANet). We develop progressive alternating attention dense (PAAD) blocks, which construct a guiding attention map (GAM) after every convolutional layer in the dense blocks using features from all scales. The GAM allows the following layers in the dense blocks to focus on the spatial locations relevant to the target region. Every alternatePAAD block inverts the GAM to generate a reverse attention map which guides ensuing layers to extract boundary and edge-related information, refining the segmentation process. Our experiments on three different biomedical image segmentation datasets exhibit that our PAANet achieves favorable performance when compared to other state-of-the-art methods.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Proc. of 2021 4th International Conference on Bio-Engineering for Smart Technologies (BioSMART) |
Date Published | 12/2021 |
Publisher | IEEE |
Place Published | Paris / Créteil, France |
URL | https://ieeexplore.ieee.org/document/9677844 |
DOI | 10.1109/BioSMART54244.2021.9677844 |
Sustainable Commercial Fishing: Digital Inspectors to the Rescue
In Arctic Frontiers. Arctic Frontiers, 2021.Status: Published
Sustainable Commercial Fishing: Digital Inspectors to the Rescue
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Arctic Frontiers |
Publisher | Arctic Frontiers |
Transfer Learning in Polyp and Endoscopic Tool Segmentation from Colonoscopy Images
In Nordic Machine Intelligence. Vol. 1. Nordic Machine Intelligence, 2021.Status: Published
Transfer Learning in Polyp and Endoscopic Tool Segmentation from Colonoscopy Images
Colorectal cancer is one of the deadliest and most widespread types of cancer in the world. Colonoscopy is the procedure used to detect and diagnose polyps from the colon, but today's detection rate shows a significant error rate that affects diagnosis and treatment. An automatic image segmentation algorithm may help doctors to improve the detection rate of pathological polyps in the colon. Furthermore, segmenting endoscopic tools in images taken during colonoscopy may contribute towards robotic assisted surgery. In this study, we trained and validated both pre-trained and not pre-trained segmentation models on two different data sets, containing images of polyps and endoscopic tools. Finally, we applied the models on two separate test sets and the best polyp model got a dice score 0.857 and the test instrument model got a dice score 0.948. Moreover, we found that pre-training of the models increased the performance in segmenting polyps and endoscopic tools.
Afilliation | Scientific Computing |
Project(s) | No Simula project |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | Nordic Machine Intelligence |
Volume | 1 |
Number of Volumes | 1 |
Edition | 1 |
Pagination | 32 - 34 |
Date Published | 11/2021 |
Publisher | Nordic Machine Intelligence |
ISSN Number | 2703-9196 |
Keywords | convolutional neural networks, Hyperkvasir, MedAI challenge, Polyp segmentation, Transfer Learning |
URL | https://journals.uio.no/NMI/article/view/9132 |
DOI | 10.5617/nmi.9132 |
Visual Sentiment Analysis: A Natural Disaster Use-case Task at MediaEval 2021
In MediaEval 2021. Ceurws, 2021.Status: Published
Visual Sentiment Analysis: A Natural Disaster Use-case Task at MediaEval 2021
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2021 |
Conference Name | MediaEval 2021 |
Publisher | Ceurws |
Diagnosing Schizophrenia from Activity Records using Hidden Markov Model Parameters
In 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS)2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS). Aveiro, Portugal: IEEE, 2021.Status: Published
Diagnosing Schizophrenia from Activity Records using Hidden Markov Model Parameters
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type |
Proceedings, non-refereed
Improving generalizibilty in polyp segmentation using ensemble convolutional neural network
In 3rd International Workshop and Challenge on Computer Vision in Endoscopy (EndoCV2021). Vol. 2886. CEUR Workshop Proceedings, 2021.Status: Published
Improving generalizibilty in polyp segmentation using ensemble convolutional neural network
Medical image segmentation is a crucial task in medical image analysis. Despite near expert-label performance with the application of the deep learning method in medical image segmentation, the generalization of such models in the clinical environment remains a significant challenge. Transfer learning from a large medical dataset from the same domain is a common technique to address generalizability. However, it is difficult to find a similar large medical dataset. To address generalizability in polyp segmentation, we have used an ensemble of four MultiResUNet architectures, each trained on the combination of the different centered datasets provided by the challenge organizers. Our method achieved a decent performance of 0.6172 ± 0.0778 for the multi-centered dataset. Our study shows that significant work needs to be done to develop a computer-aided diagnosis system to detect and localize polyp of the multi-center datasets, which is essential for improving the quality of the colonoscopy.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, non-refereed |
Year of Publication | 2021 |
Conference Name | 3rd International Workshop and Challenge on Computer Vision in Endoscopy (EndoCV2021) |
Volume | 2886 |
Publisher | CEUR Workshop Proceedings |
Keywords | colonoscopy, Convolutional neural network, health informatics, Polyp segmentation |
Miscellaneous
MMSys'21 Grand Challenge on Detecting Cheapfakes
arXiv, 2021.Status: Published
MMSys'21 Grand Challenge on Detecting Cheapfakes
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Miscellaneous |
Year of Publication | 2021 |
Publisher | arXiv |
Proceedings, refereed
ACM Multimedia BioMedia 2020 Grand Challenge Overview
In Proceedings of the 28th ACM International Conference on Multimedia. New York, NY, USA: Association for Computing Machinery, 2020.Status: Published
ACM Multimedia BioMedia 2020 Grand Challenge Overview
The BioMedia 2020 ACM Multimedia Grand Challenge is the second in a series of competitions focusing on the use of multimedia for different medical use-cases. In this year's challenge, participants are asked to develop algorithms that automatically predict the quality of a given human semen sample using a combination of visual, patient-related, and laboratory-analysis-related data. Compared to last year's challenge, participants are provided with a fully multimodal dataset (videos, analysis data, study participant data) from the field of assisted human reproduction. The tasks encourage the use of the different modalities contained within the dataset and finding smart ways of how they may be combined to further improve prediction accuracy. For example, using only video data or combining video data and patient-related data. The ground truth was developed through a preliminary analysis done by medical experts following the World Health Organization's standard for semen quality assessment. The task lays the basis for automatic, real-time support systems for artificial reproduction. We hope that this challenge motivates multimedia researchers to explore more medical-related applications and use their vast knowledge to make a real impact on people's lives.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Proceedings of the 28th ACM International Conference on Multimedia |
Pagination | 4655–4658 |
Publisher | Association for Computing Machinery |
Place Published | New York, NY, USA |
ISBN Number | 9781450379885 |
Keywords | artificial intelligence, Machine learning, male fertility, semen analysis, spermatozoa |
URL | https://doi.org/10.1145/3394171.3416287 |
DOI | 10.1145/3394171.3416287 |
DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation
In CBMS 2020: International Symposium on Computer-Based Medical Systems. IEEE, 2020.Status: Published
DoubleU-Net: A Deep Convolutional Neural Network for Medical Image Segmentation
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | CBMS 2020: International Symposium on Computer-Based Medical Systems |
Publisher | IEEE |
Notes | This paper was nominated for the best paper award at CBMS 2020. |
ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Internet Applications
In European Conference on Information Retrieval. Cham: Springer International Publishing, 2020.Status: Published
ImageCLEF 2020: Multimedia Retrieval in Lifelogging, Medical, Nature, and Internet Applications
This paper presents an overview of the 2020 ImageCLEF lab that will be organized as part of the Conference and Labs of the Evaluation Forum—CLEF Labs 2020 in Thessaloniki, Greece. ImageCLEF is an ongoing evaluation initiative (run since 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval of visual data with the aim of providing information access to large collections of images in various usage scenarios and domains. In 2020, the 18th edition of ImageCLEF will organize four main tasks: (i) a Lifelog task (videos, images and other sources) about daily activity understanding, retrieval and summarization, (ii) a Medical task that groups three previous tasks (caption analysis, tuberculosis prediction, and medical visual question answering) with new data and adapted tasks, (iii) a Coral task about segmenting and labeling collections of coral images for 3D modeling, and a new (iv) Web user interface task addressing the problems of detecting and recognizing hand drawn website UIs (User Interfaces) for generating automatic code. The strong participation, with over 235 research groups registering and 63 submitting over 359 runs for the tasks in 2019 shows an important interest in this benchmarking campaign. We expect the new tasks to attract at least as many researchers for 2020.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | European Conference on Information Retrieval |
Pagination | 533 - 541 |
Publisher | Springer International Publishing |
Place Published | Cham |
ISBN Number | 978-3-030-45441-8 |
ISSN Number | 0302-9743 |
URL | https://link.springer.com/chapter/10.1007/978-3-030-45442-5_69 |
DOI | 10.1007/978-3-030-45442-5_69 |
Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy
In International Conference on Multimedia Modeling. Springer, 2020.Status: Published
Kvasir-instrument: Diagnostic and therapeutic tool segmentation dataset in gastrointestinal endoscopy
Gastrointestinal pathologies are periodically screened, biopsied, and resected using surgical tools. Usually, the procedures and the treated or resected areas are not specifically tracked or analysed during or after colonoscopies. Information regarding disease borders, development, amount, and size of the resected area get lost. This can lead to poor follow-up and bothersome reassessment difficulties post-treatment. To improve the current standard and also to foster more research on the topic, we have released the ``Kvasir-Instrument'' dataset, which consists of 590 annotated frames containing GI procedure tools such as snares, balloons, and biopsy forceps, etc. Besides the images, the dataset includes ground truth masks and bounding boxes and has been verified by two expert GI endoscopists. Additionally, we provide a baseline for the segmentation of the GI tools to promote research and algorithm development. We obtained a dice coefficient score of 0.9158 and a Jaccard index of 0.8578 using a classical U-Net architecture. A similar dice coefficient score was observed for DoubleUNet. The qualitative results showed that the model did not work for the images with specularity and the frames with multiple tools, while the best result for both methods was observed on all other types of images. Both qualitative and quantitative results show that the model performs reasonably good, but there is potential for further improvements. Benchmarking using the dataset provides an opportunity for researchers to contribute to the field of automatic endoscopic diagnostic and therapeutic tool segmentation for GI endoscopy.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | International Conference on Multimedia Modeling |
Publisher | Springer |
Keywords | Benchmarking, Convolutional neural network, Gastrointestinal endoscopy, Tool segmentation |
URL | https://www.springerprofessional.de/en/kvasir-instrument-diagnostic-and-... |
Kvasir-SEG: A Segmented Polyp Dataset
In International Conference on Multimedia Modeling. Daejeon, Korea: Springer, 2020.Status: Published
Kvasir-SEG: A Segmented Polyp Dataset
Pixel-wise image segmentation is a highly demanding task in medical image analysis. In practice, it is difficult to find annotated medical images with corresponding segmentation masks. In this paper, we present Kvasir-SEG: an open-access dataset of gastrointestinal polyp images and corresponding segmentation masks, manually annotated and verified by an experienced gastroenterologist. We demonstrate the use of our dataset with a traditional segmentation approach and a modern deep learning based CNN approach. This work will be valuable for researchers to reproduce results and compare their methods in the future. By adding segmentation masks to the Kvasir dataset, which until today only consisted of framewise annotations, we enable multimedia and computer vision researchers to contribute in the field of polyp segmentation and automatic analysis of colonoscopy videos.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | International Conference on Multimedia Modeling |
Pagination | 451-462 |
Publisher | Springer |
Place Published | Daejeon, Korea |
Keywords | Kvasir-SEG dataset, Medical images, Polyp segmentation, ResUNet Fuzzy c-mean clustering, Semantic segmentation |
LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification
In The Joint International Conference PDCAT-PAAP 2020. Springer, 2020.Status: Published
LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification
Deep Neural Networks (DNNs) have become the de-facto standard in computer vision, as well as in many other pattern recognition tasks. A key drawback of DNNs is that the training phase can be very computationally expensive. Organizations or individuals that cannot afford purchasing state-of-the-art hardware or tapping into cloud-hosted infrastructures may face a long waiting time before the training completes or might not be able to train a model at all. Investigating novel ways to reduce the training time could be a potential solution to alleviate this drawback, and thus enabling more rapid development of new algorithms and models. In this paper, we propose LightLayers, a method for reducing the number of trainable parameters in deep neural networks (DNN). The proposed LightLayers consists of LightDense andLightConv2D layer that are as efficient as regular Conv2D and Dense layers, but uses less parameters. We resort to Matrix Factorization to reduce the complexity of the DNN models resulting into lightweight DNNmodels that require less computational power, without much loss in the accuracy. We have tested LightLayers on MNIST, Fashion MNIST, CI-FAR 10, and CIFAR 100 datasets. Promising results are obtained for MNIST, Fashion MNIST, CIFAR-10 datasets whereas CIFAR 100 shows acceptable performance by using fewer parameters.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | The Joint International Conference PDCAT-PAAP 2020 |
Publisher | Springer |
Keywords | CIFAR-10, Convolutional neural network, Deep Learning, Fashion MNIST, Lightweight model, MNIST, Weight decomposition |
Medico Multimedia Task at MediaEval 2020: Automatic Polyp Segmentation
In Medico MediaEval 2020. CEUR, 2020.Status: Published
Medico Multimedia Task at MediaEval 2020: Automatic Polyp Segmentation
Colorectal cancer is the third most common cause of cancer worldwide. According to Global cancer statistics 2018, the incidence of colorectal cancer is increasing in both developing and developed countries. Early detection of colon anomalies such as polyps is important for cancer prevention, and automatic polyp segmentation can play a crucial role for this. Regardless of the recent advancement in early detection and treatment options, the estimated polyp miss rate is still around 20\%. Support via an automated computer-aided diagnosis system could be one of the potential solutions for the overlooked polyps. Such detection systems can help low-cost design solutions and save doctors time, which they could for example use to perform more patient examinations. In this paper, we introduce the 2020 Medico challenge, provide some information on related work and the dataset, describe the task and evaluation metrics, and discuss the necessity of organizing the Medico challenge.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Medico MediaEval 2020 |
Publisher | CEUR |
Organiser Team at ImageCLEFlifelog 2020: A Baseline Approach for Moment Retrieval and Athlete Performance Prediction using Lifelog Data
In CLEF2020. CEUR Workshop Proceedings, 2020.Status: Published
Organiser Team at ImageCLEFlifelog 2020: A Baseline Approach for Moment Retrieval and Athlete Performance Prediction using Lifelog Data
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | CLEF2020 |
Publisher | CEUR Workshop Proceedings |
Overview of ImageCLEF lifelog 2020: lifelog moment retrieval and sport performance lifelog
In CLEF2020 . CEUR Workshop Proceedings, 2020.Status: Published
Overview of ImageCLEF lifelog 2020: lifelog moment retrieval and sport performance lifelog
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | CLEF2020 |
Publisher | CEUR Workshop Proceedings |
Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications
In CLEF 2020. Vol. 12260. Cham: Springer International Publishing, 2020.Status: Published
Overview of the ImageCLEF 2020: Multimedia Retrieval in Medical, Lifelogging, Nature, and Internet Applications
This paper presents an overview of the 2020 ImageCLEF lab that will be organized as part of the Conference and Labs of the Evaluation Forum—CLEF Labs 2020 in Thessaloniki, Greece. ImageCLEF is an ongoing evaluation initiative (run since 2003) that promotes the evaluation of technologies for annotation, indexing and retrieval of visual data with the aim of providing information access to large collections of images in various usage scenarios and domains. In 2020, the 18th edition of ImageCLEF will organize four main tasks: (i) a Lifelog task (videos, images and other sources) about daily activity understanding, retrieval and summarization, (ii) a Medical task that groups three previous tasks (caption analysis, tuberculosis prediction, and medical visual question answering) with new data and adapted tasks, (iii) a Coral task about segmenting and labeling collections of coral images for 3D modeling, and a new (iv) Web user interface task addressing the problems of detecting and recognizing hand drawn website UIs (User Interfaces) for generating automatic code. The strong participation, with over 235 research groups registering and 63 submitting over 359 runs for the tasks in 2019 shows an important interest in this benchmarking campaign. We expect the new tasks to attract at least as many researchers for 2020.
Afilliation | Communication Systems, Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | CLEF 2020 |
Volume | 12260 |
Pagination | 311 - 341 |
Publisher | Springer International Publishing |
Place Published | Cham |
ISBN Number | 978-3-030-58218-0 |
ISSN Number | 0302-9743 |
URL | https://doi.org/10.1007/978-3-030-58219-7_22 |
DOI | 10.1007/978-3-030-58219-710.1007/978-3-030-58219-7_22 |
PMData: a sports logging dataset
In Proceedings of the 11th ACM Multimedia Systems Conference. ACM, 2020.Status: Published
PMData: a sports logging dataset
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | Proceedings of the 11th ACM Multimedia Systems Conference |
Pagination | 231-236 |
Publisher | ACM |
PSYKOSE: A Motor Activity Database of Patients with Schizophrenia
In 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS). Rochester, MN, USA: IEEE, 2020.Status: Published
PSYKOSE: A Motor Activity Database of Patients with Schizophrenia
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | 2020 IEEE 33rd International Symposium on Computer-Based Medical Systems (CBMS) |
Publisher | IEEE |
Place Published | Rochester, MN, USA |
URL | https://ieeexplore.ieee.org/xpl/mostRecentIssue.jsp?punumber=9169740http... |
DOI | 10.1109/CBMS49503.202010.1109/CBMS49503.2020.00064 |
Pyramid-Focus-Augmentation: Medical Image Segmentation with Step-Wise Focus
In MediaEval 2020. CEUR, 2020.Status: Published
Pyramid-Focus-Augmentation: Medical Image Segmentation with Step-Wise Focus
Segmentation of findings in the gastrointestinal tract is a challenging but also an important task which is an important building stone for sufficient automatic decision support systems. In this work, we present our solution for the Medico 2020 task, which focused on the problem of colon polyp segmentation. We present our simple but efficient idea of using an augmentation method that uses grids in a pyramid-like manner (large to small) for segmentation. Our results show that the proposed methods work as indented and can also lead to comparable results when competing with other methods.
Afilliation | Machine Learning |
Project(s) | Department of Holistic Systems |
Publication Type | Proceedings, refereed |
Year of Publication | 2020 |
Conference Name | MediaEval 2020 |
Publisher | CEUR |