Projects
Forny-Titan
Companies developing families of software-based products that undergo intensive testing phase face the challenge of implementing efficient test strategies able to fit limited resources, low test budgets, or ever-shortening cycle time. Simultaneously, the market constantly drives the increase in product quality and complexity, which in turn increases the volume of testing.
TITAN project is developing a new technology that will reconcile these opposing goals. Through efficient test optimisation techniques, TITAN technology helps enhance product quality, reduce risks and control testing costs. The goal of the project is to develop a commercial solution with these capabilities.
The project is funded by the Research Council of Norway through the FORNY2020 program, which aims at creating value and benefits for society from research results. The program funds the results from projects conducted at publicly-funded research institutions, and helps bringing them closer to the market.
Funding source:
Research Council of Norway
All partners:
Simula Innovation
The Certus Centre (SFI)
The Certus Centre has made a strong impact in the global software engineering community in the areas of variability testing, modelling highly configurable systems, and testing data-intensive systems. Software engineers and researchers have teamed up to create an innovative environment where specialised expertise and creative research combine to produce readily exploitable, takeaway results. Certus has issued a number of high-quality publications and developed several software prototype tools, such as the TITAN tool and the Zen-RUCM editor. Another highlight of Certus' production is the experimental application of our techniques in industrial case studies.
Certus has three core expertise fields:
Model-based engineering for highly configurable systems:
Involving the application of model driven engineering techniques to the modelling and configuration of subsea integrated control systems, which can be regarded as highly variable software systems.
Testing of data-intensive systems:
Looking into the improvement of existing regression testing techniques to handle large database-centric applications.
Testing of real-time embedded systems:
Involving research into managing the testing activities of families of realtime embedded systems.
Publications for The Certus Centre (SFI)
Journal Article
Industry-Academia research collaboration in software engineering: The Certus model
Information and Software Technology 132 (2021).Status: Published
Industry-Academia research collaboration in software engineering: The Certus model
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), T3AS, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | Information and Software Technology |
| Volume | 132 |
| Number | 106473 |
| Publisher | Elsevier |
Journal Article
Adaptive Metamorphic Testing with Contextual Bandits
Journal of Systems and Software 165 (2020).Status: Published
Adaptive Metamorphic Testing with Contextual Bandits
Metamorphic Testing is a software testing paradigm which aims at using necessary properties of a system-under-test, called metamorphic relations, to either check its expected outputs, or to generate new test cases. Metamorphic Testing has been successful to test programs for which a full oracle is not available or to test programs for which there are uncertainties on expected outputs such as learning systems. In this article, we propose Adaptive Metamorphic Testing as a generalization of a simple yet powerful reinforcement learning technique, namely contextual bandits, to select one of the multiple metamorphic relations available for a program. By using contextual bandits, Adaptive Metamorphic Testing learns which metamorphic relations are likely to transform a source test case, such that it has higher chance to discover faults. We present experimental results over two major case studies in machine learning, namely image classification and object detection, and identify weaknesses and robustness boundaries. Adaptive Metamorphic Testing efficiently identifies weaknesses of the tested systems in context of the source test case.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), Department of Validation Intelligence for Autonomous Software Systems, Testing of Learning Robots (T-LARGO) , Testing of Learning Robots (T-Largo) |
| Publication Type | Journal Article |
| Year of Publication | 2020 |
| Journal | Journal of Systems and Software |
| Volume | 165 |
| Date Published | 07/2020 |
| Publisher | Elsevier |
| Other Numbers | arXiv:1910.00262 |
| DOI | 10.1016/j.jss.2020.110574 |
PhD Thesis
Software Testing in Continuous Integration with Machine Learning and Constraint Optimization
In The University of Oslo. Vol. PhD, 2020.Status: Published
Software Testing in Continuous Integration with Machine Learning and Constraint Optimization
Frequent automated software testing is a crucial task for modern software development. It has the goal to evaluate a software's functionality and be confident about its quality after recent changes and before the integration of new features or its deployment into the actual production environment.
Further challenges are introduced when testing software for cyber-physical systems that integrate both software and dedicated hardware components, e.g. industrial robots or embedded devices.
This thesis explores how machine learning and constraint optimization can be leveraged to achieve the desired efficiency and to create an intelligent testing process. Specifically, we contribute new methodology for the test suite optimization process of test case prioritization, test case scheduling, and test case selection and assignment. All of these steps are relevant to decide which test cases are most relevant and when to execute them on which test hardware, e.g. an industrial robot. The results of this thesis have been published in international venues and are in production usage at our industrial partner, ABB Robotics Norway.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | PhD Thesis |
| Year of Publication | 2020 |
| Degree awarding institution | The University of Oslo |
| Degree | PhD |
| Date Published | 09/2020 |
| ISBN Number | 1501-7710 |
| URL | https://www.mn.uio.no/ifi/forskning/aktuelt/arrangementer/disputaser/202... |
Proceedings, refereed
Lessons Learned on Research Co-Creation: Making Industry-Academia Collaboration Work
In The 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). Euromicro, 2020.Status: Published
Lessons Learned on Research Co-Creation: Making Industry-Academia Collaboration Work
| Afilliation | Software Engineering |
| Project(s) | T3AS, The Certus Centre (SFI), Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | The 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) |
| Publisher | Euromicro |
RobTest: A CP Approach to Generate Maximal Test Trajectories for Industrial Robots
In International Conference on Principles and Practice of Constraint Programming. LNCS ed. Vol. 12333. Springer, 2020.Status: Published
RobTest: A CP Approach to Generate Maximal Test Trajectories for Industrial Robots
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), Testing of Learning Robots (T-LARGO) , Department of Validation Intelligence for Autonomous Software Systems, Testing of Learning Robots (T-Largo) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | International Conference on Principles and Practice of Constraint Programming |
| Volume | 12333 |
| Edition | LNCS |
| Pagination | 1-17 |
| Date Published | 08/2020 |
| Publisher | Springer |
Edited books
Proceedings of the Joint 7th International Workshop on Conducting Empirical Studies in Industry and 6th International Workshop on Software Engineering Research and Industrial Practice
In International Conference on Software Engineering (ICSE). ACM/IEEE, 2019.Status: Published
Proceedings of the Joint 7th International Workshop on Conducting Empirical Studies in Industry and 6th International Workshop on Software Engineering Research and Industrial Practice
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Edited books |
| Year of Publication | 2019 |
| Secondary Title | International Conference on Software Engineering (ICSE) |
| Publisher | ACM/IEEE |
Journal Article
FightHPV: Design and Evaluation of a Mobile Game to Raise Awareness About Human Papillomavirus and Nudge People to Take Action Against Cervical Cancer
JMIR serious games 7 (2019): e8540.Status: Published
FightHPV: Design and Evaluation of a Mobile Game to Raise Awareness About Human Papillomavirus and Nudge People to Take Action Against Cervical Cancer
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | JMIR serious games |
| Volume | 7 |
| Number | 2 |
| Pagination | e8540 |
| Publisher | JMIR Publications Inc. |
| Place Published | Toronto, Canada |
Implications of Resurgence in Artificial Intelligence for Research Collaborations in Software Engineering
ACM SIGSOFT Software Engineering Notes 44, no. 3 (2019).Status: Published
Implications of Resurgence in Artificial Intelligence for Research Collaborations in Software Engineering
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | ACM SIGSOFT Software Engineering Notes |
| Volume | 44 |
| Issue | 3 |
| Publisher | ACM |
Good Practices in Aligning Software Engineering Research and Industry Practice
ACM SIGSOFT Software Engineering Notes 44, no. 3 (2019).Status: Published
Good Practices in Aligning Software Engineering Research and Industry Practice
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | ACM SIGSOFT Software Engineering Notes |
| Volume | 44 |
| Issue | 3 |
| Publisher | ACM |
Employing Rule Mining and Multi-Objective Search for Dynamic Test Case Prioritization
Journal of Systems and Software 153 (2019): 86-104.Status: Published
Employing Rule Mining and Multi-Objective Search for Dynamic Test Case Prioritization
Test case prioritization (TP) is widely used in regression testing for optimal reordering of test cases to achieve specific criteria (e.g., higher fault detection capability) as early as possible. In our earlier work, we proposed an approach for black-box dynamic TP using rule mining and multi-objective search (named as REMAP) by defining two objectives (fault detection capability and test case reliance score) and considering test case execution results at runtime. In this paper, we conduct an extensive empirical evaluation of REMAP by employing three different rule mining algorithms and three different multi-objective search algorithms, and we also evaluate REMAP with one additional objective (estimated execution time) for a total of 18 different configurations (i.e., 3 rule mining algorithms × 3 search algorithms × 2 different set of objectives) of REMAP. Specifically, we empirically evaluated the 18 variants of REMAP with 1) two variants of random search while using two objectives and three objectives, 2) three variants of greedy algorithm based on one objective, two objectives, and three objectives, 3) 18 variants of static search-based prioritization approaches, and 4) six variants of rule-based prioritization approaches using two industrial and three open source case studies. Results showed that the two best variants of REMAP with two objectives and three objectives significantly outperformed the best variants of competing approaches by 84.4% and 88.9%, and managed to achieve on average 14.2% and 18.8% higher Average Percentage of Faults Detected per Cost (APFDc) scores.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), Department of Engineering Complex Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | Journal of Systems and Software |
| Volume | 153 |
| Pagination | 86-104 |
| Date Published | 07/2019 |
| Publisher | Elsevier |
SmartMed
Increasing amounts of health data are recorded in health registries, with the strategic initiatives of data sharing and fusion across different registries in Norway. This forms an excellent opportunity for world-class medical research as few countries have such a high-quality infrastructure. However, it also constitutes a very high privacy risk should a security breach occur.
Publicized incidents of leaked medical records pose a significant challenge for the digital trust in eHealth were storing, accessing, and exchanging sensitive patient-related data must comply with several regulations, while remaining accessible to authorized health practitioners. Governmental legislation regarding data privacy, such as the EU's GDPR, present an additional source of concern for healthcare registries which are now faced with severe legal and financial consequences in case data confidentiality is breached.
Our principal approach is to facilitate solutions for health registries by using Smart Contracts and the emerging Blockchain Paradigm. From the data safety, authenticity, and nonrepudiation standpoint, blockchain is a perfect fit for sharing medical records since it provides an easily accessible, immutable, and transparent history of all contract-related data, adequate for building applications with trust and accountability. Use of smart contracts brings several additional advantages for sharing medical data by healthcare registries: consent management, fine-grain privacy control, transparency, and reduced bureaucracy and expenses.
Publications for SmartMed
Journal Article
A Survey on Blockchain for Healthcare: Challenges, Benefits, and Future Directions
IEEE Communications Surveys & Tutorials 25, no. 1 (2023): 386-424.Status: Published
A Survey on Blockchain for Healthcare: Challenges, Benefits, and Future Directions
| Afilliation | Software Engineering |
| Project(s) | SmartMed |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | IEEE Communications Surveys & Tutorials |
| Volume | 25 |
| Issue | 1 |
| Pagination | 386 - 424 |
| Publisher | IEEE |
Journal Article
Blockchain verification and validation: Techniques, challenges, and research directions
Computer Science Review 45 (2022): 100492.Status: Published
Blockchain verification and validation: Techniques, challenges, and research directions
| Afilliation | Software Engineering |
| Project(s) | SmartMed |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | Computer Science Review |
| Volume | 45 |
| Pagination | 100492 |
| Publisher | Elsevier |
Journal Article
LEChain: A blockchain-based lawful evidence management scheme for digital forensics
Journal of Future Generation Computer Systems 115 (2020): 406-420.Status: Published
LEChain: A blockchain-based lawful evidence management scheme for digital forensics
Lawful evidence management in digital forensics is of paramount importance in police investigations because such evidence is used to convict suspects of crimes. Existing studies have adopted cloud computing to collect evidence and then leveraged blockchain to support the transparency, immutability, and auditability of the evidence. Unfortunately, such studies only rely on a weak security model and do not cover the entire life cycle of the evidence or address the key privacy issues, i.e., witness privacy in evidence collection and juror privacy in court trials. In this work, we propose LEChain, a blockchainbased lawful evidence management scheme to supervise the entire evidence flow and all of the court data (e.g., votes and trial results), extending from evidence collection and access during the police investigation to jury voting in the court trials. Specifically, we utilize short randomizable signatures to anonymously authenticate witnesses’ identities to protect the witness privacy. Then, we leverage fine-grained access control based on ciphertext-policy attribute-based encryption for evidence access. Next, we design a secure voting method to protect juror privacy. In addition, we build a consortium blockchain to record evidence transactions. Finally, we formally analyze the security and privacy of LEChain and evaluate its computational costs and communication overhead by implementing a prototype based on a local Ethereum test network.
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, SmartMed |
| Publication Type | Journal Article |
| Year of Publication | 2020 |
| Journal | Journal of Future Generation Computer Systems |
| Volume | 115 |
| Pagination | 406-420 |
| Publisher | Elsevier |
| Keywords | blockchain, Chain of custody, Digital forensics, Lawful evidence, Privacy, Security |
| DOI | 10.1016/j.future.2020.09.038 |
Privacy-preserving Navigation Supporting Similar Queries in Vehicular Networks
IEEE Transactions on Dependable and Secure Computing (2020).Status: Published
Privacy-preserving Navigation Supporting Similar Queries in Vehicular Networks
Traffic-sensitive navigation systems in vehicular networks help drivers avoid traffic jams by providing several realtime navigation routes. However, drivers still encounter privacy concerns because their sensitive locations, i.e., their start point and endpoint, are submitted to an honest-but-curious navigation service provider (NSP). Previous privacy-preserving studies exhibit serious deficiencies under similar queries: if a driver makes several similar queries, i.e., periodically makes requests for the same start point and endpoint to the NSP, these requests will eventually reveal the areas of the two points as well as the route. In this paper, we present a novel privacy-preserving navigation scheme PiSim, which supports similar queries in navigation services. Intuitively, we transform the typical navigation approach into a traffic congestion querying approach. Instead of sending two locations to the NSP and awaiting a navigation route, drivers query the traffic congestion along the navigation route. Specifically, PiSim is characterized by extending anonymous authentication, facilitating privacy-preserving multi-keyword fuzzy search, and constructing weighted proximity graphs. Our scheme protects location privacy and route privacy, and defends against multiple requesting, spurious reporting, and collusion attacks from malicious drivers. Finally, a detailed analysis confirms the privacy and security properties of PiSim. Extensive experiments are conducted to demonstrate the feasibility, performance, and privacy protection level.
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, SmartMed |
| Publication Type | Journal Article |
| Year of Publication | 2020 |
| Journal | IEEE Transactions on Dependable and Secure Computing |
| Publisher | IEEE |
| Keywords | Location Privacy, Navigation Services, Privacy Measurement, Similar Queries, Vehicular Networks |
| DOI | 10.1109/TDSC.2020.3017534 |
LiDL: Localization with early detection of sybil and wormhole attacks in IoT Networks
Computers and Security, Elsevier 94 (2020).Status: Published
LiDL: Localization with early detection of sybil and wormhole attacks in IoT Networks
The Internet of Things (IoT) is recognized as a disruptive innovation that has been led by industry leaders and researchers. IoT promises to improve our daily life based on smart objects interacting with each other, and that can be connected to the Internet. Building a security framework into this new paradigm is a significant technical challenge today. It is mainly due to the low-cost and resource-constrained nature of IoT devices. In most of the IoT application scenarios, the routing is done by the de-facto standard protocol called routing protocol for low power and lossy networks (RPL). The use of RPL is suitable due to its energy-efficient schemes, availability of secure and multiple communication modes, and adaptivity to work in various IoT network scenarios. Hence, many researchers are now focusing on RPL related security issues. To this end, our work provides a concise description of two major threats to RPL called sybil and wormhole attacks. Moreover, we propose two solutions to detect these attacks in RPL-based IoT networks. Specifically, our proposed techniques exploit the concept of Highest Rank Common Ancestor (HRCA) to find a common ancestor with the highest rank among all the ancestors that a pair of nodes have in the target network tree. Our two detection algorithms not only detect an ongoing attack but also localizes the position of the adversary in the network. Thus, it makes the mitigation process lightweight and fast. We implement the two approaches in Cooja, the Contiki network emulator. The results obtained from our experiments demonstrate the feasibility of the proposals concerning true positive rate, detection time, packet loss ratio, memory consumption, and network overhead. Our techniques show promising to cover more complex scenarios in the future.
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, SmartMed |
| Publication Type | Journal Article |
| Year of Publication | 2020 |
| Journal | Computers and Security, Elsevier |
| Volume | 94 |
| Publisher | Elsevier |
| ISSN | 0167-4048 |
| Keywords | 6LoWPAN, Internet of things, IPv6, RPL, Security, Sybil attack, Wormhole attack |
| DOI | 10.1016/j.cose.2020.101849 |
Blockchain-enabled Secure Energy Trading with Verifiable Fairness in Industrial Internet of Things
IEEE Transactions on Industrial Informatics 16, no. 10 (2020): 6564-6574.Status: Published
Blockchain-enabled Secure Energy Trading with Verifiable Fairness in Industrial Internet of Things
Energy trading in Industrial Internet of Things (IIoT), a fundamental approach to realize Industry 4.0, plays a vital role in satisfying energy demands and optimizing system efficiency. Existing research works adopts a utility company to distribute energy to energy nodes with the help of energy brokers. Afterwards, they apply blockchain to provide transparency, immutability, and auditability of peer-to-peer (P2P) energy trading. However, their schemes are constructed on a weak security model and do not consider the cheating attack initiated by energy sellers. Such an attack refers to an energy seller refusing to transfer the negotiated energy to an energy purchaser who already paid money. In this paper, we propose FeneChain, a blockchain-based energy trading scheme to supervise and manage the energy trading process towards building a secure energy trading system and improving the energy quality for Industry 4.0. Specifically, we leverage anonymous authentication to protect user privacy, and we design a timed commitments based mechanism to guarantee the verifiable fairness during energy trading. Moreover, we utilize fine-grained access control for energy trading services. We also build a consortium blockchain among energy brokers to verify and record energy trading transactions. Finally, we formally analyze the security and privacy of FeneChain and evaluate its performance (i.e., computational costs and communication overhead) by implementing a prototype via a local Ethereum test network and Raspberry Pi.
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, SmartMed |
| Publication Type | Journal Article |
| Year of Publication | 2020 |
| Journal | IEEE Transactions on Industrial Informatics |
| Volume | 16 |
| Issue | 10 |
| Pagination | 6564-6574 |
| Publisher | IEEE |
| Keywords | blockchain, energy trading, Industrial Internet of Things, Industry 4.0, Privacy, Security |
| DOI | 10.1109/TII.2020.2974537 |
Technical reports
Blockchain for Healthcare: Opportunities, Challenges, and Future Directions
Simula Research Laboratory, 2020.Status: Published
Blockchain for Healthcare: Opportunities, Challenges, and Future Directions
The continuous generation of large volume of medical data from different sources, such as patient monitoring, clinical trials management, and processing payments and reimbursement claims, makes healthcare a data-intensive domain. This data needs to be shared among different medical facilities for various purposes, such as in-depth data analysis and collaborative research, to achieve innovative advances in the medical treatment procedures and drug developments, and providing personalised healthcare services to the patients. However, the existing data exchange solutions in healthcare domain exhibits several challenges, like data security, patient privacy, data owner consent management, interoperability, and chain-of-custody. Recently, the industry and research community turned its focus on the possible use of blockchain technology to solve one or more of the above challenges in healthcare domain. The blockchain technology along with the support from smart contracts is considered as an adequate solution for secure and efficient medical data sharing. It is due to its unique features, such as decentralization, trustlessness, immutability, traceability, and security. In this paper, we provide a comprehensive survey on the state-of-the-art efforts that envision the usage of blockchain-based solutions to improve one or more aspects in healthcare domain.
To this end, first we categorize the existing works based on a set of key challenge(s) that they address in healthcare by using blockchain technology. The particular challenges that we consider for the categorization of the surveyed articles are as follows: (i) security and privacy, (ii) interoperability, (iii) compliance management, (iv) medical records management, and (v) chain-of-custody. Second, we discuss the benefits and limitations that the surveyed solutions exhibit when incorporating blockchain in healthcare. We also investigate the causal relationships among these challenges to demonstrate how various factors causing these challenges are connected with each other. Third, we identify the practical issues that blockchain-driven healthcare must overcome to become a success, and we suggest the additional future research required to address the identified issues. Our survey shows that there is an exponential increase in the number of efforts towards the use blockchain and smart contracts to improve various functionalities in several healthcare services such as clinical trials, medical data management, and drug supply chain management. However, most of these efforts are still in their initial phases, and a lot of work still remains before their integration in practical healthcare scenarios. This work is currently in-progress, and it is planned to be performed in an accepted research-based innovation project by Research Council of Norway called “SMARTMED – Secure and accountable sharing of medical records using smart contracts and blockchain”.
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, SmartMed |
| Publication Type | Technical reports |
| Year of Publication | 2020 |
| Publisher | Simula Research Laboratory |
| Keywords | blockchain, Cloud computing, Electronic Health Records, Internet of things, Medical data, security and privacy |
T-Sar
In transport infrastructures, vessel traffic services, air traffic management, and connected cars all rely on unauthenticated and unencrypted messages transfer that renders these services vulnerable to cyberattacks. Typical attacks such as False Data Injection Attacks (FDIA) are challenging to detect as they alter the semantics of the data (e.g., by adding/removing/multiplying elements on real-time control equipment), while preserving the syntactical correctness of the messages. Identifying these attacks and classifying them as serious threats or unintentional false data has become a significant challenge of traffic monitoring authorities.
The TSAR project aims at demonstrating that recent advances in Artificial Intelligence (AI) can be leveraged in the automatic detection of FDIA in transport infrastructures. By combining realistic threat data generation based on constraint-based software testing techniques and automatic detection with deep reinforcement learning, TSAR will propose a new technology for automatic FDIA generation and detection. This technology will be empirically evaluated with end-users from the maritime domain and with open and accessible data in two other domains, namely air traffic control, and connected cars. By leveraging automatic detection of FDIA in traffic management systems, TSAR will also prepare the ground for the upcoming revolution in traffic management, which concerns, self-driving vessels, self-driving aircraft, and self-driving cars.
Publications for T-Sar
Poster
Encoding Temporal and Spatial Vessel Context using Self-Supervised Learning Model (Student Abstract)
AAAI Conference on Artificial Intelligence (AAAI-21), Student Abstract and Poster Program, 2021.Status: Published
Encoding Temporal and Spatial Vessel Context using Self-Supervised Learning Model (Student Abstract)
| Afilliation | Software Engineering, Machine Learning |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, T-Sar |
| Publication Type | Poster |
| Year of Publication | 2021 |
| Place Published | AAAI Conference on Artificial Intelligence (AAAI-21), Student Abstract and Poster Program |
Proceedings, refereed
Apprentissage auto-supervisé pour la détection d’actions illégales lors de la surveillance du trafic maritime
In Applications Pratiques de l’Intelligence Artificielle. AFIA, 2021.Status: Published
Apprentissage auto-supervisé pour la détection d’actions illégales lors de la surveillance du trafic maritime
| Afilliation | Software Engineering, Machine Learning |
| Project(s) | T-Sar, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2021 |
| Conference Name | Applications Pratiques de l’Intelligence Artificielle |
| Publisher | AFIA |
Encoding Temporal and Spatial Vessel Context using Self-Supervised Learning Model (Student Abstract)
In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. AAAI, 2021.Status: Published
Encoding Temporal and Spatial Vessel Context using Self-Supervised Learning Model (Student Abstract)
| Afilliation | Software Engineering |
| Project(s) | T-Sar, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2021 |
| Conference Name | Proceedings of the AAAI Conference on Artificial Intelligence |
| Volume | 35 |
| Pagination | 15757-15758 |
| Publisher | AAAI |
Poster
Detection of False Data Injection in AIS Vessels Communication Using Machine Learning
FEMTO-ST Institute, Besancon, France, 2020.Status: Published
Detection of False Data Injection in AIS Vessels Communication Using Machine Learning
| Afilliation | Software Engineering |
| Project(s) | T-Sar, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Poster |
| Year of Publication | 2020 |
| Place Published | FEMTO-ST Institute, Besancon, France |
Proceedings, refereed
Improved Testing of AI-Based Anomaly Detection Systems Using Synthetic Surveillance Data
In OpenSky Symposium. MDPI, 2020.Status: Published
Improved Testing of AI-Based Anomaly Detection Systems Using Synthetic Surveillance Data
| Afilliation | Software Engineering |
| Project(s) | T-Sar, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | OpenSky Symposium |
| Publisher | MDPI |
Talks, contributed
Improved Testing of AI-Based Anomaly Detection Systems Using Synthetic Surveillance Data
In OpenSky Symposium, 2020.Status: Published
Improved Testing of AI-Based Anomaly Detection Systems Using Synthetic Surveillance Data
| Afilliation | Software Engineering, Machine Learning |
| Project(s) | T-Sar, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2020 |
| Location of Talk | OpenSky Symposium |
Talks, contributed
Statistics AIS Dataset from Statsat
In Simula Research Laboratory, 2019.Status: Published
Statistics AIS Dataset from Statsat
| Afilliation | Software Engineering, Machine Learning |
| Project(s) | T-Sar, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2019 |
| Location of Talk | Simula Research Laboratory |
The TSAR Project
In 14th Certus User Partner Workshop, Sep. 2019, Larvik, Norway, 2019.Status: Published
The TSAR Project
| Afilliation | Software Engineering |
| Project(s) | T-Sar |
| Publication Type | Talks, contributed |
| Year of Publication | 2019 |
| Location of Talk | 14th Certus User Partner Workshop, Sep. 2019, Larvik, Norway |
Technical reports
SARCoS: Machine Learning Models for the Selection of Security Test Scenarios
Simula Research Laboratory, 2019.Status: Accepted
SARCoS: Machine Learning Models for the Selection of Security Test Scenarios
| Afilliation | Software Engineering |
| Project(s) | T-Sar |
| Publication Type | Technical reports |
| Year of Publication | 2019 |
| Publisher | Simula Research Laboratory |
AI4EU
AI4EU is the European Union’s landmark Artificial Intelligence project, which seeks to develop a European AI ecosystem, bringing together the knowledge, algorithms, tools, and resources available and making it a compelling solution for users. Involving 80 partners, covering 21 countries, the €20m project kicked off in January 2019 and will run for three years. AI4EU will unify Europe’s Artificial Intelligence community. It will facilitate collective work in AI research, innovation, and business in Europe. By sharing AI expertise, knowledge, and tools with the Platform, AI4EU will make AI available to all.
The AI4EU Platform will establish a world reference, built upon and interoperable with existing AI and data components (e.g., the Acumos open-source framework, QWT search engine..) and platforms. Eight industry-driven AI pilots will demonstrate the value of the platform as an innovation tool. To enhance the platform, research on five key interconnected AI scientific areas will be carried out using platform technologies, and results will be implemented. Sustainability will be ensured via the creation of the AI4EU Foundation. The results will feed a new and comprehensive Strategic Research Innovation Agenda for Europe.
Final goal:
Mobilize the entire European AI community to make AI promises real for European Society and Economy
Create a leading collaborative AI European platform to nurture economic growth.
To achieve these goals, the project includes a diverse set of tasks and an ambitious set of activities:
- Create Europe’s leading AI On-Demand-Platform that is open and sustainable
- Bring stakeholders together through high-profile conferences and virtual events
- Develop a relevant, comprehensive and stimulating Strategic Agenda for European AI
- Establish an Ethics Observatory to ensure the development of human-centred AI
- Roll out of €3m in Cascade Funding
AI4EU will nurture more adequacy between business needs and research results and accelerate growth. It will help the European Community to become a global leader in both highly advanced and human-centered AI, promising cutting-edge breakthroughs in this pivotal technological arena.
Funding source: EU Horizon 2020 (ICT-26-2018-2020)
All partners (link)
Project leader: Thales (France)
Twitter (link)
LinkedIn (link)
Publications for AI4EU
Book Chapter
Testing Industrial Robotic Systems: A New Battlefield!
In Software Engineering for Robotics, 109-137. Cham: Springer Nature, 2021.Status: Published
Testing Industrial Robotic Systems: A New Battlefield!
| Afilliation | Software Engineering |
| Project(s) | Testing of Learning Robots (T-LARGO) , Department of Validation Intelligence for Autonomous Software Systems, AI4EU, Testing of Learning Robots (T-Largo) |
| Publication Type | Book Chapter |
| Year of Publication | 2021 |
| Book Title | Software Engineering for Robotics |
| Chapter | 5 |
| Pagination | 109-137 |
| Date Published | 06/2021 |
| Publisher | Springer Nature |
| Place Published | Cham |
| ISBN Number | ISBN 978-3-030-66493-0 |
| URL | https://www.springer.com/gp/book/9783030664930#aboutBook |
Journal Article
Predictive Machine Learning of Objective Boundaries for Solving COPs
AI 2, no. 4 (2021): 527-551.Status: Published
Predictive Machine Learning of Objective Boundaries for Solving COPs
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, AI4EU, Testing of Learning Robots (T-Largo) |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | AI |
| Volume | 2 |
| Issue | 4 |
| Pagination | 527 - 551 |
| Date Published | 12/2021 |
| Publisher | MDPI |
| Place Published | Basel/Switzerland |
| Other Numbers | arXiv:2111.03160 |
| URL | https://www.mdpi.com/2673-2688/2/4/33 |
| DOI | 10.3390/ai2040033 |
Miscellaneous
AI4EU Robotics Pilot: Vibration sensor measurements in a robotic pump
Zenodo, 2021.Status: Published
AI4EU Robotics Pilot: Vibration sensor measurements in a robotic pump
| Afilliation | Software Engineering |
| Project(s) | AI4EU, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Miscellaneous |
| Year of Publication | 2021 |
| Publisher | Zenodo |
| URL | https://doi.org/10.5281/zenodo.5729187 |
| DOI | 10.5281/zenodo.5729187 |
AI4EU Robotics Pilot: Vibration sensor measurements in a robotic wrist
Zenodo, 2021.Status: Published
AI4EU Robotics Pilot: Vibration sensor measurements in a robotic wrist
| Afilliation | Software Engineering |
| Project(s) | AI4EU, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Miscellaneous |
| Year of Publication | 2021 |
| Publisher | Zenodo |
| URL | https://doi.org/10.5281/zenodo.5729818 |
| DOI | 10.5281/zenodo.5729818 |
Proceedings, refereed
Constraint-Guided Reinforcement Learning: Augmenting the Agent-Environment-Interaction
In International Joint Conference on Neural Networks (IJCNN). IEEE, 2021.Status: Published
Constraint-Guided Reinforcement Learning: Augmenting the Agent-Environment-Interaction
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, AI4EU |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2021 |
| Conference Name | International Joint Conference on Neural Networks (IJCNN) |
| Publisher | IEEE |
| ISBN Number | 978-1-6654-3900-8 |
| ISSN Number | 2161-4407 |
| Other Numbers | arXiv:2104.11918 |
| DOI | 10.1109/IJCNN52387.2021.9533996 |
Talks, invited
Breaking silos in data innovation in Europe: Experiences of AI4EU, EUH4D, and DIH4AI
In Data Week 2021, 2021.Status: Published
Breaking silos in data innovation in Europe: Experiences of AI4EU, EUH4D, and DIH4AI
| Afilliation | Software Engineering |
| Project(s) | AI4EU |
| Publication Type | Talks, invited |
| Year of Publication | 2021 |
| Location of Talk | Data Week 2021 |
| Type of Talk | Contributions to panel discussion - https://www.big-data-value.eu/dw21-agenda/ |
Proceedings, refereed
Opening the Software Engineering Toolbox for the Assessment of Trustworthy AI
In 1st International Workshop on New Foundations for Human-Centered AI @ ECAI 2020. CEUR Workshop Proceedings, 2020.Status: Published
Opening the Software Engineering Toolbox for the Assessment of Trustworthy AI
Trustworthiness is a central requirement for the acceptance and success of human-centered artificial intelligence (AI). To deem an AI system as trustworthy, it is crucial to assess its behaviour and characteristics against a gold standard of Trustworthy AI, consisting of guidelines, requirements, or only expectations. While AI systems are highly complex, their implementations are still based on software. The software engineering community has a long-established toolbox for the assessment of software systems, especially in the context of software testing. In this paper, we argue for the application of software engineering and testing practices for the assessment of trustworthy AI. We make the connection between the seven key requirements as defined by the European Commission's AI high-level expert group and established procedures from software engineering and raise questions for future work.
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, AI4EU |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | 1st International Workshop on New Foundations for Human-Centered AI @ ECAI 2020 |
| Pagination | 67-70 |
| Date Published | 08/2020 |
| Publisher | CEUR Workshop Proceedings |
| Other Numbers | arXiv:2007.07768 |
| URL | http://ceur-ws.org/Vol-2659/ahuja.pdf |
Talks, invited
Requirements collection for the AI4EU platform development
In Big Data Value Forum, 3-5 November, Berlin, Germany, 2020.Status: Published
Requirements collection for the AI4EU platform development
| Afilliation | Software Engineering |
| Project(s) | AI4EU, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2020 |
| Location of Talk | Big Data Value Forum, 3-5 November, Berlin, Germany |
Metamorphic Testing: A Validation Technique for Trustworthy AI?
In Workshop on Lessons learnt on Trustworthy AI made in Europe: Challenges and Answers, 2020.Status: Published
Metamorphic Testing: A Validation Technique for Trustworthy AI?
| Afilliation | Software Engineering |
| Project(s) | AI4EU, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2020 |
| Location of Talk | Workshop on Lessons learnt on Trustworthy AI made in Europe: Challenges and Answers |
| Type of Talk | Online. Nov. 13th, 2020. Video available online: https://www.youtube.com/watch?v=B9rAbqcayok |
AI4EU: Pilot Experiments with the platform
In Big Data Value Forum, Nov. 3-5, Berlin, Germany, 2020.Status: Published
AI4EU: Pilot Experiments with the platform
| Afilliation | Software Engineering |
| Project(s) | AI4EU, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2020 |
| Location of Talk | Big Data Value Forum, Nov. 3-5, Berlin, Germany |
| Type of Talk | https://www.european-big-data-value-forum.eu/ |
T3AS
Autonomous systems are emerging technologies that are impacting a range of industries and many areas of human life nowadays. Driverless cars have started appearing on public roads, and collaborative robots have started working with human workers on the factory floor. Autonomous technologies offer a significant opportunity to enhance the economy and society, but they may also cause fatal harm if they malfunction.
There are great open challenges of testing these autonomous systems, to prevent their malfunctioning, and to ensure their safe and fault-free behavior. T3AS project is developing a novel theoretical foundation based on artificial intelligence, with a set of methods and tools, for making AI-based autonomous systems dependable and safe for their users and the environment.
Publications for T3AS
Journal Article
ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events
IEEE Transactions on Software Engineering (TSE) (2023).Status: Submitted
ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events
Autonomous vehicles are advanced driving systems that are well known for being vulnerable to various adversarial attacks, compromising the vehicle's safety, and posing danger to other road users. Rather than actively training complex adversaries by interacting with the environment, there is a need to first intelligently find and reduce the search space to only those states where autonomous vehicles are found less confident. In this paper, we propose a blackbox testing framework ReMAV using offline trajectories first to analyze the existing behavior of autonomous vehicles and determine appropriate thresholds for finding the probability of failure events. To this end, we introduce a three-step methodology which i) uses offline state action pairs of any autonomous vehicle under test, ii) builds an abstract behavior representation using our designed reward modeling technique in order to analyze states with uncertain driving decisions, and iii) uses a disturbance model for minimal perturbation attacks where the driving decisions are less confident. Our reward modeling technique helps in creating a behavior representation that allows us to highlight regions of likely uncertain behavior even when the baseline autonomous vehicle is performing well. This approach allows for more efficient testing without the need for computational and inefficient active adversarial learning techniques. We perform our experiments in a high-fidelity urban driving environment using three different driving scenarios containing single and multi-agent interactions. Our experiment shows a 35%, 23%, 48%, and 50% increase in occurrences of vehicle collision, road objects collision, pedestrian collision, and offroad steering events respectively by the autonomous vehicle under test, demonstrating a significant increase in failure events. We also perform a comparative analysis with prior testing frameworks and show that they underperform in terms of training-testing efficiency, finding total infractions, and simulation steps to identify the first failure compared to our approach. The results show that the proposed framework can be used to understand existing weaknesses of the autonomous vehicles under test in order to only attack those regions, starting with the simplistic perturbation models. We demonstrate that our approach is capable of finding failure events in terms of increased collision and offroad steering errors compared to the baseline driving behavior.
| Afilliation | Software Engineering, Machine Learning |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | IEEE Transactions on Software Engineering (TSE) |
| Publisher | IEEE |
| Keywords | Autonomous Vehicle Testing, Behavior Modeling, deep reinforcement learning, Inverse Reinforcement Learning |
| URL | https://arxiv.org/abs/2308.14550 |
Comparative study of machine learning test case prioritization for continuous integration testing
Software Quality Journal (2023).Status: Published
Comparative study of machine learning test case prioritization for continuous integration testing
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | Software Quality Journal |
| Date Published | 07/2023 |
| Publisher | Springer |
| DOI | 10.1007/s11219-023-09646-0 |
Journal Article
Industry-Academia Research Collaboration and Knowledge Co-creation: Patterns and Anti-patterns
ACM Transactions on Software Engineering and Methodology 31, no. 3 (2022): 1-52.Status: Published
Industry-Academia Research Collaboration and Knowledge Co-creation: Patterns and Anti-patterns
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Volume | 31 |
| Issue | 3 |
| Pagination | 1-52 |
| Publisher | ACM |
| ISSN | 1049-331X |
| URL | https://dl.acm.org/doi/10.1145/3494519https://dl.acm.org/doi/pdf/10.1145... |
| DOI | 10.1145/3494519 |
Proceedings, refereed
Evaluating the Robustness of Deep Reinforcement Learning for Autonomous Policies in a Multi-agent Urban Driving Environment
In 22nd IEEE International Conference on Software Quality, Reliability, and Security (QRS). 22nd International Conference on Software Quality, Reliability and Security (QRS): IEEE, 2022.Status: Published
Evaluating the Robustness of Deep Reinforcement Learning for Autonomous Policies in a Multi-agent Urban Driving Environment
Background: Deep reinforcement learning is actively used for training autonomous car policies in a simulated driving environment. Due to the large availability of various reinforcement learning algorithms and the lack of their systematic comparison across different driving scenarios, we are unsure of which ones are more effective for training autonomous car software in single-agent as well as multi-agent driving environments. Aims: A benchmarking framework for the comparison of deep reinforcement learning in a vision-based autonomous driving will open up the possibilities for training better autonomous car driving policies. Method: To address these challenges, we provide an open and reusable benchmarking framework for systematic evaluation and comparative analysis of deep reinforcement learning algorithms for autonomous driving in a single- and multi-agent environment. Using the framework, we perform a comparative study of four discrete and two continuous action space deep reinforcement learning algorithms. We also propose a comprehensive multi-objective reward function designed for the evaluation of deep reinforcement learning-based autonomous driving agents. We run the experiments in a vision-only high-fidelity urban driving simulated environments. Results: The results indicate that only some of the deep reinforcement learning algorithms perform consistently better across single and multi-agent scenarios when trained in various multi-agent-only environment settings. For example, A3C- and TD3-based autonomous cars perform comparatively better in terms of more robust actions and minimal driving errors in both single and multi-agent scenarios. Conclusions: We conclude that different deep reinforcement learning algorithms exhibit different driving and testing performance in different scenarios, which underlines the need for their systematic comparative analysis. The benchmarking framework proposed in this paper facilitates such a comparison.
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 22nd IEEE International Conference on Software Quality, Reliability, and Security (QRS) |
| Publisher | IEEE |
| Place Published | 22nd International Conference on Software Quality, Reliability and Security (QRS) |
| Keywords | autonomous cars, autonomous driving, deep reinforcement learning, multi-agent systems, testing autonomous driving |
| URL | https://ieeexplore.ieee.org/abstract/document/10062456 |
| DOI | 10.1109/QRS57517.2022.00084 |
Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies
In 29th Asia-Pacific Software Engineering Conference (APSEC). 29th Asia-Pacific Software Engineering Conference (APSEC): IEEE, 2022.Status: Published
Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies
Autonomous cars are well known for being vulnerable to adversarial attacks that can compromise the safety of the car and pose danger to other road users. To effectively defend against adversaries, it is required to not only test autonomous cars for finding driving errors but to improve the robustness of the cars to these errors. To this end, in this paper, we propose a two-step methodology for autonomous cars that consists of (i) finding failure states in autonomous cars by training the adversarial driving agent, and (ii) improving the robustness of autonomous cars by retraining them with effective adversarial inputs. Our methodology supports testing autonomous cars in a multi-agent environment, where we train and compare adversarial car policy on two custom reward functions to test the driving control decision of autonomous cars. We run experiments in a vision-based high-fidelity urban driving simulated environment. Our results show that adversarial testing can be used for finding erroneous autonomous driving behavior, followed by adversarial training for improving the robustness of deep reinforcement learning-based autonomous driving policies. We demonstrate that the autonomous cars retrained using the effective adversarial inputs noticeably increase the performance of their driving policies in terms of reduced collision and offroad steering errors.
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 29th Asia-Pacific Software Engineering Conference (APSEC) |
| Publisher | IEEE |
| Place Published | 29th Asia-Pacific Software Engineering Conference (APSEC) |
| Keywords | adversarial testing, AI testing, autonomous car, autonomous driving, deep reinforcement learning, multi-agent, Robustness, self-driving car, simulation testing |
| URL | https://ieeexplore.ieee.org/abstract/document/10043282 |
| DOI | 10.1109/APSEC57359.2022.00018 |
Journal Article
Bridging Software Engineering Research and Industrial Practice
ACM SIGSOFT Software Engineering Notes 46, no. 1 (2021).Status: Published
Bridging Software Engineering Research and Industrial Practice
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | ACM SIGSOFT Software Engineering Notes |
| Volume | 46 |
| Issue | 1 |
| Publisher | ACM |
Industry-Academia Research Collaborations During and After COVID-19
ACM SIGSOFT Software Engineering Notes 46, no. 4 (2021).Status: Published
Industry-Academia Research Collaborations During and After COVID-19
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | ACM SIGSOFT Software Engineering Notes |
| Volume | 46 |
| Issue | 4 |
| Publisher | ACM |
Industry-Academia research collaboration in software engineering: The Certus model
Information and Software Technology 132 (2021).Status: Published
Industry-Academia research collaboration in software engineering: The Certus model
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), T3AS, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | Information and Software Technology |
| Volume | 132 |
| Number | 106473 |
| Publisher | Elsevier |
Proceedings, refereed
DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing
In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2021.Status: Published
DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing
Continuous integration testing is an important step in the modern software engineering life cycle. Test prioritization is a method that can improve the efficiency of continuous integration testing by selecting test cases that can detect faults in the early stage of each cycle. As continuous integration testing produces voluminous test execution data, test history is a commonly used artifact in test prioritization. However, existing test prioritization techniques for continuous integration either cannot handle large test history or are optimized for using a limited number of historical test cycles. We show that such a limitation can decrease fault detection effectiveness of prioritized test suites.
This work introduces DeepOrder, a deep learning-based model that works on the basis of regression machine learning. DeepOrder ranks test cases based on the historical record of test executions from any number of previous test cycles. DeepOrder learns failed test cases based on multiple factors including the duration and execution status of test cases. We experimentally show that deep neural networks, as a simple regression model, can be efficiently used for test case prioritization in continuous integration testing. DeepOrder is evaluated with respect to time-effectiveness and fault detection effectiveness in comparison with an industry practice and the state of the art approaches. The results show that DeepOrder outperforms the industry practice and state-of-the-art test prioritization approaches in terms of these two metrics.
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2021 |
| Conference Name | 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) |
| Number of Volumes | 10 |
| Pagination | 525-534 |
| Publisher | IEEE |
| DOI | 10.1109/ICSME52107.2021.00053 |
Edited books
Seventh International Workshop on Software Engineering Research and Industrial Practice
In International Conference on Software Engineering (ICSE). South Korea: ACM, 2020.Status: Published
Seventh International Workshop on Software Engineering Research and Industrial Practice
| Afilliation | Software Engineering |
| Project(s) | T3AS, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Edited books |
| Year of Publication | 2020 |
| Secondary Title | International Conference on Software Engineering (ICSE) |
| Publisher | ACM |
| Place Published | South Korea |
TRANSACT
The overarching goal of the TRANSACT project is to develop a universal, distributed solution architecture for the transformation of safety-critical cyber-physical systems, from localised standalone systems into safe and secure distributed solutions leveraging edge and cloud computing.
Market trends show advanced usage of safety-critical systems with novel services based on smart data analytics. Customers require continuous updates to applications and services and seek lower cost (Bill-of-Material, BoM) and easy to install solutions (maintenance) for safety-critical cyber-physical systems (CPS). To respond to these trends, TRANSACT will leverage edge and cloud technologies and establish business partner eco-systems to enhance safety-critical systems in regulated environments. TRANSACT will transform local safety critical CPS into distributed safety-critical CPS solutions with a heterogeneous architecture composed of components along a device-edge-cloud continuum. The distributed solutions incorporating data and cloud services will simplify the CPS devices, reducing their software footprint, and consequently their BoM and Lower of Cost or Market. Business-wise, system manufacturers thus transform to solution providers. To that end TRANSACT will research distributed reference architectures for safety-critical CPS that rely on edge and cloud computing.
These architectures shall enable seamless mixing of on-device, edge and cloud services while assuring flexible yet safe and secure deployment of new applications, and independent releasing of edge and cloud-based components vs. on-device.
Moreover, safety, performance, cybersecurity and privacy of data will be kept on the same level as on-device only safety critical CPS architectures. By also integrating AI services into distributed CPS, TRANSACT will enable fast development of innovative value-based services and business models leading to faster market introduction in the various multi-billion euro markets addressed by TRANSACT. Encouraged by ARTEMIS’ 2019 publication on embedded intelligence, TRANSACT will be a crucial enabler for Europe to shift towards a solution-oriented market “so as to still matter in the Embedded & Cyber-Physical Systems field of tomorrow’s world.”
TRANSACT has received funding from the ECSEL Joint Undertaking (JU) under grant agreement No 101007260. The JU receives support from the European Union's Horizon 2020 research and innovation programme and Netherlands, Finland, Germany, Poland, Austria, Spain, Belgium, Denmark, and Norway.
Publications for TRANSACT
Journal Article
Reactive buffering window trajectory segmentation: RBW-TS
Journal of Big Data 10 (2023).Status: Published
Reactive buffering window trajectory segmentation: RBW-TS
| Afilliation | Software Engineering |
| Project(s) | TRANSACT |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | Journal of Big Data |
| Volume | 10 |
| Number | 123 |
| Date Published | 07/2023 |
| Publisher | Springer |
| DOI | 10.1186/s40537-023-00799-0 |
Interpolation-Based Inference of Vessel Trajectory Waypoints from Sparse AIS Data in Maritime
Journal of Marine Science and Engineering 11, no. 3 (2023).Status: Published
Interpolation-Based Inference of Vessel Trajectory Waypoints from Sparse AIS Data in Maritime
| Afilliation | Software Engineering |
| Project(s) | TRANSACT |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | Journal of Marine Science and Engineering |
| Volume | 11 |
| Issue | 3 |
| Number | 615 |
| Publisher | Ocean Engineering |
SafeWay: Improving the Safety of Autonomous Waypoint Detection in Maritime using Transformer and Interpolation
Maritime Transport Research 4 (2023).Status: Published
SafeWay: Improving the Safety of Autonomous Waypoint Detection in Maritime using Transformer and Interpolation
| Afilliation | Software Engineering |
| Project(s) | TRANSACT, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | Maritime Transport Research |
| Volume | 4 |
| Number | 100086 |
| Date Published | 03/2023 |
| Publisher | Elsevier |
| ISSN | 2666-822X |
| Notes | |
| DOI | 10.1016/j.martra.2023.100086 |
| Research Notes |
Proceedings, refereed
Discovering Gateway Ports in Maritime using Temporal Graph Neural Network Port Classification
In 35th Canadian Conference on Artificial Intelligence. Canadian Artificial Intelligence Association, 2022.Status: Published
Discovering Gateway Ports in Maritime using Temporal Graph Neural Network Port Classification
Vessel navigation is influenced by various factors, such as dynamic environmental factors that change over time or static features such as vessel type or depth of the ocean. These dynamic and static navigational factors impose limitations on vessels, such as long waiting times in regions outside the actual ports, and we call these waiting regions gateway ports. Identifying gateway ports and their associated features such as congestion and available utilities can enhance vessel navigation by planning on fuel optimization or saving time in cargo operation. In this paper, we propose a novel temporal graph neural network (TGNN) based port classification method to enable vessels to discover gateway ports efficiently, thus optimizing their operations. The proposed method processes vessel trajectory data to build dynamic graphs capturing spatio-temporal dependencies between a set of static and dynamic navigational features in the data, and it is evaluated in terms of port classification accuracy on a real-world data set collected from ten vessels operating in Halifax, NS, Canada. The experimental results indicate that our TGNN-based port classification method provides an f-score of 95% in classifying ports.
| Afilliation | Software Engineering |
| Project(s) | TRANSACT, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 35th Canadian Conference on Artificial Intelligence |
| Publisher | Canadian Artificial Intelligence Association |
| Keywords | actual ports, AIS data, gateway ports, maritime situational awareness, maritime traffic, port area, port classification, port congestion, spatio-temporal, temporal graph neural networks, vessel trajectories |
| Notes | |
| DOI | 10.21428/594757db.a76bcb9d |
| Research Notes |
DYNAPORT: Dynamic Navigation and Port Call Optimisation in Real Time
The DYNAPPORT project targets the problem of decarbonization of maritime transport by developing new tools for ports and ships that reduce the ship's fuel consumption and increase port efficiency.
Efficient and viable voyage and port call optimization requires increased cooperation between parties and overcoming several important barriers such as charter parties and contracts enabling just-in-time arrivals and more explicit risk/benefit sharing, improved transparency and reduced cyber-security risks, and reliable and efficient voyage and port call information sharing among all parties.
Simula's work in DYNAPORT focuses on new optimisation methods for port calls and ship routes, with the end goal of developing an integrated end-to-end voyage optimisation tool that enables a minimum 10% energy use reduction for the ship voyage and port call.
Final goal
DYNAPORT aims to develop new optimization and coordination tools for ports and ships that reduce the ship's fuel consumption and increase port efficiency. The tools will be built on information sharing through internationally accepted protocol standards and communication systems.
Funding source
This project is funded by Horizon Europe under the call HORIZON-CL5-2023-D5-01.
Partners
SINTEF Ocean, coordinator (Norway)
Simula (Norway)
Along with 17 more.
Publications
Journal Article
A Survey on Blockchain for Healthcare: Challenges, Benefits, and Future Directions
IEEE Communications Surveys & Tutorials 25, no. 1 (2023): 386-424.Status: Published
A Survey on Blockchain for Healthcare: Challenges, Benefits, and Future Directions
| Afilliation | Software Engineering |
| Project(s) | SmartMed |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | IEEE Communications Surveys & Tutorials |
| Volume | 25 |
| Issue | 1 |
| Pagination | 386 - 424 |
| Publisher | IEEE |
Comparative study of machine learning test case prioritization for continuous integration testing
Software Quality Journal (2023).Status: Published
Comparative study of machine learning test case prioritization for continuous integration testing
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | Software Quality Journal |
| Date Published | 07/2023 |
| Publisher | Springer |
| DOI | 10.1007/s11219-023-09646-0 |
Interpolation-Based Inference of Vessel Trajectory Waypoints from Sparse AIS Data in Maritime
Journal of Marine Science and Engineering 11, no. 3 (2023).Status: Published
Interpolation-Based Inference of Vessel Trajectory Waypoints from Sparse AIS Data in Maritime
| Afilliation | Software Engineering |
| Project(s) | TRANSACT |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | Journal of Marine Science and Engineering |
| Volume | 11 |
| Issue | 3 |
| Number | 615 |
| Publisher | Ocean Engineering |
Reactive buffering window trajectory segmentation: RBW-TS
Journal of Big Data 10 (2023).Status: Published
Reactive buffering window trajectory segmentation: RBW-TS
| Afilliation | Software Engineering |
| Project(s) | TRANSACT |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | Journal of Big Data |
| Volume | 10 |
| Number | 123 |
| Date Published | 07/2023 |
| Publisher | Springer |
| DOI | 10.1186/s40537-023-00799-0 |
ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events
IEEE Transactions on Software Engineering (TSE) (2023).Status: Submitted
ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events
Autonomous vehicles are advanced driving systems that are well known for being vulnerable to various adversarial attacks, compromising the vehicle's safety, and posing danger to other road users. Rather than actively training complex adversaries by interacting with the environment, there is a need to first intelligently find and reduce the search space to only those states where autonomous vehicles are found less confident. In this paper, we propose a blackbox testing framework ReMAV using offline trajectories first to analyze the existing behavior of autonomous vehicles and determine appropriate thresholds for finding the probability of failure events. To this end, we introduce a three-step methodology which i) uses offline state action pairs of any autonomous vehicle under test, ii) builds an abstract behavior representation using our designed reward modeling technique in order to analyze states with uncertain driving decisions, and iii) uses a disturbance model for minimal perturbation attacks where the driving decisions are less confident. Our reward modeling technique helps in creating a behavior representation that allows us to highlight regions of likely uncertain behavior even when the baseline autonomous vehicle is performing well. This approach allows for more efficient testing without the need for computational and inefficient active adversarial learning techniques. We perform our experiments in a high-fidelity urban driving environment using three different driving scenarios containing single and multi-agent interactions. Our experiment shows a 35%, 23%, 48%, and 50% increase in occurrences of vehicle collision, road objects collision, pedestrian collision, and offroad steering events respectively by the autonomous vehicle under test, demonstrating a significant increase in failure events. We also perform a comparative analysis with prior testing frameworks and show that they underperform in terms of training-testing efficiency, finding total infractions, and simulation steps to identify the first failure compared to our approach. The results show that the proposed framework can be used to understand existing weaknesses of the autonomous vehicles under test in order to only attack those regions, starting with the simplistic perturbation models. We demonstrate that our approach is capable of finding failure events in terms of increased collision and offroad steering errors compared to the baseline driving behavior.
| Afilliation | Software Engineering, Machine Learning |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | IEEE Transactions on Software Engineering (TSE) |
| Publisher | IEEE |
| Keywords | Autonomous Vehicle Testing, Behavior Modeling, deep reinforcement learning, Inverse Reinforcement Learning |
| URL | https://arxiv.org/abs/2308.14550 |
SafeWay: Improving the Safety of Autonomous Waypoint Detection in Maritime using Transformer and Interpolation
Maritime Transport Research 4 (2023).Status: Published
SafeWay: Improving the Safety of Autonomous Waypoint Detection in Maritime using Transformer and Interpolation
| Afilliation | Software Engineering |
| Project(s) | TRANSACT, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | Maritime Transport Research |
| Volume | 4 |
| Number | 100086 |
| Date Published | 03/2023 |
| Publisher | Elsevier |
| ISSN | 2666-822X |
| Notes | |
| DOI | 10.1016/j.martra.2023.100086 |
| Research Notes |
Proceedings, refereed
Measuring the Effect of Causal Disentanglement on the Adversarial Robustness of Neural Network Models
In ACM International Conference on Information and Knowledge Management, 2023.Status: Accepted
Measuring the Effect of Causal Disentanglement on the Adversarial Robustness of Neural Network Models
Causal Neural Network models have shown high levels of robustness to adversarial attacks as well as an increased capacity for generalisation tasks such as few-shot learning and rare-context classification compared to traditional Neural Networks. This robustness is argued to stem from the disentanglement of causal and confounder input signals. However, no quantitative study has yet measured the level of disentanglement achieved by these types of causal models or assessed how this relates to their adversarial robustness.
Existing causal disentanglement metrics are not applicable to deterministic models trained on real-world datasets. We, therefore, utilise metrics of content/style disentanglement from the field of Computer Vision to measure different aspects of the causal disentanglement for four state-of-the-art causal Neural Network models. By re-implementing these models with a common ResNet18 architecture we are able to fairly measure their adversarial robustness on three standard image classification benchmarking datasets under seven common white-box attacks. We find a strong association (r=0.820, p=0.001) between the degree to which models decorrelate causal and confounder signals and their adversarial robustness. Additionally, we find a moderate negative association between the pixel-level information content of the confounder signal and adversarial robustness (r=-0.597, p=0.040).
| Afilliation | Machine Learning |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2023 |
| Conference Name | ACM International Conference on Information and Knowledge Management |
Proceedings, refereed
Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies
In 29th Asia-Pacific Software Engineering Conference (APSEC). 29th Asia-Pacific Software Engineering Conference (APSEC): IEEE, 2022.Status: Published
Adversarial Deep Reinforcement Learning for Improving the Robustness of Multi-agent Autonomous Driving Policies
Autonomous cars are well known for being vulnerable to adversarial attacks that can compromise the safety of the car and pose danger to other road users. To effectively defend against adversaries, it is required to not only test autonomous cars for finding driving errors but to improve the robustness of the cars to these errors. To this end, in this paper, we propose a two-step methodology for autonomous cars that consists of (i) finding failure states in autonomous cars by training the adversarial driving agent, and (ii) improving the robustness of autonomous cars by retraining them with effective adversarial inputs. Our methodology supports testing autonomous cars in a multi-agent environment, where we train and compare adversarial car policy on two custom reward functions to test the driving control decision of autonomous cars. We run experiments in a vision-based high-fidelity urban driving simulated environment. Our results show that adversarial testing can be used for finding erroneous autonomous driving behavior, followed by adversarial training for improving the robustness of deep reinforcement learning-based autonomous driving policies. We demonstrate that the autonomous cars retrained using the effective adversarial inputs noticeably increase the performance of their driving policies in terms of reduced collision and offroad steering errors.
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 29th Asia-Pacific Software Engineering Conference (APSEC) |
| Publisher | IEEE |
| Place Published | 29th Asia-Pacific Software Engineering Conference (APSEC) |
| Keywords | adversarial testing, AI testing, autonomous car, autonomous driving, deep reinforcement learning, multi-agent, Robustness, self-driving car, simulation testing |
| URL | https://ieeexplore.ieee.org/abstract/document/10043282 |
| DOI | 10.1109/APSEC57359.2022.00018 |
Discovering Gateway Ports in Maritime using Temporal Graph Neural Network Port Classification
In 35th Canadian Conference on Artificial Intelligence. Canadian Artificial Intelligence Association, 2022.Status: Published
Discovering Gateway Ports in Maritime using Temporal Graph Neural Network Port Classification
Vessel navigation is influenced by various factors, such as dynamic environmental factors that change over time or static features such as vessel type or depth of the ocean. These dynamic and static navigational factors impose limitations on vessels, such as long waiting times in regions outside the actual ports, and we call these waiting regions gateway ports. Identifying gateway ports and their associated features such as congestion and available utilities can enhance vessel navigation by planning on fuel optimization or saving time in cargo operation. In this paper, we propose a novel temporal graph neural network (TGNN) based port classification method to enable vessels to discover gateway ports efficiently, thus optimizing their operations. The proposed method processes vessel trajectory data to build dynamic graphs capturing spatio-temporal dependencies between a set of static and dynamic navigational features in the data, and it is evaluated in terms of port classification accuracy on a real-world data set collected from ten vessels operating in Halifax, NS, Canada. The experimental results indicate that our TGNN-based port classification method provides an f-score of 95% in classifying ports.
| Afilliation | Software Engineering |
| Project(s) | TRANSACT, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 35th Canadian Conference on Artificial Intelligence |
| Publisher | Canadian Artificial Intelligence Association |
| Keywords | actual ports, AIS data, gateway ports, maritime situational awareness, maritime traffic, port area, port classification, port congestion, spatio-temporal, temporal graph neural networks, vessel trajectories |
| Notes | |
| DOI | 10.21428/594757db.a76bcb9d |
| Research Notes |
Evaluating the Robustness of Deep Reinforcement Learning for Autonomous Policies in a Multi-agent Urban Driving Environment
In 22nd IEEE International Conference on Software Quality, Reliability, and Security (QRS). 22nd International Conference on Software Quality, Reliability and Security (QRS): IEEE, 2022.Status: Published
Evaluating the Robustness of Deep Reinforcement Learning for Autonomous Policies in a Multi-agent Urban Driving Environment
Background: Deep reinforcement learning is actively used for training autonomous car policies in a simulated driving environment. Due to the large availability of various reinforcement learning algorithms and the lack of their systematic comparison across different driving scenarios, we are unsure of which ones are more effective for training autonomous car software in single-agent as well as multi-agent driving environments. Aims: A benchmarking framework for the comparison of deep reinforcement learning in a vision-based autonomous driving will open up the possibilities for training better autonomous car driving policies. Method: To address these challenges, we provide an open and reusable benchmarking framework for systematic evaluation and comparative analysis of deep reinforcement learning algorithms for autonomous driving in a single- and multi-agent environment. Using the framework, we perform a comparative study of four discrete and two continuous action space deep reinforcement learning algorithms. We also propose a comprehensive multi-objective reward function designed for the evaluation of deep reinforcement learning-based autonomous driving agents. We run the experiments in a vision-only high-fidelity urban driving simulated environments. Results: The results indicate that only some of the deep reinforcement learning algorithms perform consistently better across single and multi-agent scenarios when trained in various multi-agent-only environment settings. For example, A3C- and TD3-based autonomous cars perform comparatively better in terms of more robust actions and minimal driving errors in both single and multi-agent scenarios. Conclusions: We conclude that different deep reinforcement learning algorithms exhibit different driving and testing performance in different scenarios, which underlines the need for their systematic comparative analysis. The benchmarking framework proposed in this paper facilitates such a comparison.
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 22nd IEEE International Conference on Software Quality, Reliability, and Security (QRS) |
| Publisher | IEEE |
| Place Published | 22nd International Conference on Software Quality, Reliability and Security (QRS) |
| Keywords | autonomous cars, autonomous driving, deep reinforcement learning, multi-agent systems, testing autonomous driving |
| URL | https://ieeexplore.ieee.org/abstract/document/10062456 |
| DOI | 10.1109/QRS57517.2022.00084 |
Journal Article
Blockchain verification and validation: Techniques, challenges, and research directions
Computer Science Review 45 (2022): 100492.Status: Published
Blockchain verification and validation: Techniques, challenges, and research directions
| Afilliation | Software Engineering |
| Project(s) | SmartMed |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | Computer Science Review |
| Volume | 45 |
| Pagination | 100492 |
| Publisher | Elsevier |
Industry-Academia Research Collaboration and Knowledge Co-creation: Patterns and Anti-patterns
ACM Transactions on Software Engineering and Methodology 31, no. 3 (2022): 1-52.Status: Published
Industry-Academia Research Collaboration and Knowledge Co-creation: Patterns and Anti-patterns
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Volume | 31 |
| Issue | 3 |
| Pagination | 1-52 |
| Publisher | ACM |
| ISSN | 1049-331X |
| URL | https://dl.acm.org/doi/10.1145/3494519https://dl.acm.org/doi/pdf/10.1145... |
| DOI | 10.1145/3494519 |
Journal Article
Bridging Software Engineering Research and Industrial Practice
ACM SIGSOFT Software Engineering Notes 46, no. 1 (2021).Status: Published
Bridging Software Engineering Research and Industrial Practice
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | ACM SIGSOFT Software Engineering Notes |
| Volume | 46 |
| Issue | 1 |
| Publisher | ACM |
Industry-Academia research collaboration in software engineering: The Certus model
Information and Software Technology 132 (2021).Status: Published
Industry-Academia research collaboration in software engineering: The Certus model
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), T3AS, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | Information and Software Technology |
| Volume | 132 |
| Number | 106473 |
| Publisher | Elsevier |
Industry-Academia Research Collaborations During and After COVID-19
ACM SIGSOFT Software Engineering Notes 46, no. 4 (2021).Status: Published
Industry-Academia Research Collaborations During and After COVID-19
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | ACM SIGSOFT Software Engineering Notes |
| Volume | 46 |
| Issue | 4 |
| Publisher | ACM |
Proceedings, refereed
DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing
In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2021.Status: Published
DeepOrder: Deep Learning for Test Case Prioritization in Continuous Integration Testing
Continuous integration testing is an important step in the modern software engineering life cycle. Test prioritization is a method that can improve the efficiency of continuous integration testing by selecting test cases that can detect faults in the early stage of each cycle. As continuous integration testing produces voluminous test execution data, test history is a commonly used artifact in test prioritization. However, existing test prioritization techniques for continuous integration either cannot handle large test history or are optimized for using a limited number of historical test cycles. We show that such a limitation can decrease fault detection effectiveness of prioritized test suites.
This work introduces DeepOrder, a deep learning-based model that works on the basis of regression machine learning. DeepOrder ranks test cases based on the historical record of test executions from any number of previous test cycles. DeepOrder learns failed test cases based on multiple factors including the duration and execution status of test cases. We experimentally show that deep neural networks, as a simple regression model, can be efficiently used for test case prioritization in continuous integration testing. DeepOrder is evaluated with respect to time-effectiveness and fault detection effectiveness in comparison with an industry practice and the state of the art approaches. The results show that DeepOrder outperforms the industry practice and state-of-the-art test prioritization approaches in terms of these two metrics.
| Afilliation | Software Engineering |
| Project(s) | T3AS |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2021 |
| Conference Name | 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) |
| Number of Volumes | 10 |
| Pagination | 525-534 |
| Publisher | IEEE |
| DOI | 10.1109/ICSME52107.2021.00053 |
Improving the Reliability of Autonomous Software Systems through Metamorphic Testing
In Proceedings of the 31st European Safety and Reliability Conference (ESREL). ESREL, 2021.Status: Published
Improving the Reliability of Autonomous Software Systems through Metamorphic Testing
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, Testing of Learning Robots (T-LARGO) , Testing of Learning Robots (T-Largo) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2021 |
| Conference Name | Proceedings of the 31st European Safety and Reliability Conference (ESREL) |
| Pagination | 1-page abstract |
| Publisher | ESREL |
| ISBN Number | ISBN: 978-981-18-2016-8 |
Book Chapter
Testing Industrial Robotic Systems: A New Battlefield!
In Software Engineering for Robotics, 109-137. Cham: Springer Nature, 2021.Status: Published
Testing Industrial Robotic Systems: A New Battlefield!
| Afilliation | Software Engineering |
| Project(s) | Testing of Learning Robots (T-LARGO) , Department of Validation Intelligence for Autonomous Software Systems, AI4EU, Testing of Learning Robots (T-Largo) |
| Publication Type | Book Chapter |
| Year of Publication | 2021 |
| Book Title | Software Engineering for Robotics |
| Chapter | 5 |
| Pagination | 109-137 |
| Date Published | 06/2021 |
| Publisher | Springer Nature |
| Place Published | Cham |
| ISBN Number | ISBN 978-3-030-66493-0 |
| URL | https://www.springer.com/gp/book/9783030664930#aboutBook |
Technical reports
Blockchain for Healthcare: Opportunities, Challenges, and Future Directions
Simula Research Laboratory, 2020.Status: Published
Blockchain for Healthcare: Opportunities, Challenges, and Future Directions
The continuous generation of large volume of medical data from different sources, such as patient monitoring, clinical trials management, and processing payments and reimbursement claims, makes healthcare a data-intensive domain. This data needs to be shared among different medical facilities for various purposes, such as in-depth data analysis and collaborative research, to achieve innovative advances in the medical treatment procedures and drug developments, and providing personalised healthcare services to the patients. However, the existing data exchange solutions in healthcare domain exhibits several challenges, like data security, patient privacy, data owner consent management, interoperability, and chain-of-custody. Recently, the industry and research community turned its focus on the possible use of blockchain technology to solve one or more of the above challenges in healthcare domain. The blockchain technology along with the support from smart contracts is considered as an adequate solution for secure and efficient medical data sharing. It is due to its unique features, such as decentralization, trustlessness, immutability, traceability, and security. In this paper, we provide a comprehensive survey on the state-of-the-art efforts that envision the usage of blockchain-based solutions to improve one or more aspects in healthcare domain.
To this end, first we categorize the existing works based on a set of key challenge(s) that they address in healthcare by using blockchain technology. The particular challenges that we consider for the categorization of the surveyed articles are as follows: (i) security and privacy, (ii) interoperability, (iii) compliance management, (iv) medical records management, and (v) chain-of-custody. Second, we discuss the benefits and limitations that the surveyed solutions exhibit when incorporating blockchain in healthcare. We also investigate the causal relationships among these challenges to demonstrate how various factors causing these challenges are connected with each other. Third, we identify the practical issues that blockchain-driven healthcare must overcome to become a success, and we suggest the additional future research required to address the identified issues. Our survey shows that there is an exponential increase in the number of efforts towards the use blockchain and smart contracts to improve various functionalities in several healthcare services such as clinical trials, medical data management, and drug supply chain management. However, most of these efforts are still in their initial phases, and a lot of work still remains before their integration in practical healthcare scenarios. This work is currently in-progress, and it is planned to be performed in an accepted research-based innovation project by Research Council of Norway called “SMARTMED – Secure and accountable sharing of medical records using smart contracts and blockchain”.
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, SmartMed |
| Publication Type | Technical reports |
| Year of Publication | 2020 |
| Publisher | Simula Research Laboratory |
| Keywords | blockchain, Cloud computing, Electronic Health Records, Internet of things, Medical data, security and privacy |
Journal Article
ITE: A Lightweight Implementation of Stratified Reasoning for Constructive Logical Operators
International Journal on Artificial Intelligence Tools 29, no. 3-4 (2020): 23.Status: Published
ITE: A Lightweight Implementation of Stratified Reasoning for Constructive Logical Operators
| Afilliation | Software Engineering |
| Project(s) | Testing of Learning Robots (T-LARGO) , Testing of Learning Robots (T-Largo), Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2020 |
| Journal | International Journal on Artificial Intelligence Tools |
| Volume | 29 |
| Issue | 3-4 |
| Pagination | 23 |
| Date Published | 06/2020 |
| Publisher | World Scientific Publishing |
| DOI | 10.1142/S0218213020600064. |
Proceedings, refereed
Lessons Learned on Research Co-Creation: Making Industry-Academia Collaboration Work
In The 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA). Euromicro, 2020.Status: Published
Lessons Learned on Research Co-Creation: Making Industry-Academia Collaboration Work
| Afilliation | Software Engineering |
| Project(s) | T3AS, The Certus Centre (SFI), Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | The 46th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) |
| Publisher | Euromicro |
Neural Network Classification for Improving Continuous Regression Testing
In The IEEE Second International Conference On Artificial Intelligence Testing (AITest 2020). IEEE, 2020.Status: Published
Neural Network Classification for Improving Continuous Regression Testing
| Afilliation | Software Engineering |
| Project(s) | T3AS, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | The IEEE Second International Conference On Artificial Intelligence Testing (AITest 2020) |
| Publisher | IEEE |
Opening the Software Engineering Toolbox for the Assessment of Trustworthy AI
In 1st International Workshop on New Foundations for Human-Centered AI @ ECAI 2020. CEUR Workshop Proceedings, 2020.Status: Published
Opening the Software Engineering Toolbox for the Assessment of Trustworthy AI
Trustworthiness is a central requirement for the acceptance and success of human-centered artificial intelligence (AI). To deem an AI system as trustworthy, it is crucial to assess its behaviour and characteristics against a gold standard of Trustworthy AI, consisting of guidelines, requirements, or only expectations. While AI systems are highly complex, their implementations are still based on software. The software engineering community has a long-established toolbox for the assessment of software systems, especially in the context of software testing. In this paper, we argue for the application of software engineering and testing practices for the assessment of trustworthy AI. We make the connection between the seven key requirements as defined by the European Commission's AI high-level expert group and established procedures from software engineering and raise questions for future work.
| Afilliation | Software Engineering |
| Project(s) | Department of Validation Intelligence for Autonomous Software Systems, AI4EU |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | 1st International Workshop on New Foundations for Human-Centered AI @ ECAI 2020 |
| Pagination | 67-70 |
| Date Published | 08/2020 |
| Publisher | CEUR Workshop Proceedings |
| Other Numbers | arXiv:2007.07768 |
| URL | http://ceur-ws.org/Vol-2659/ahuja.pdf |
RobTest: A CP Approach to Generate Maximal Test Trajectories for Industrial Robots
In International Conference on Principles and Practice of Constraint Programming. LNCS ed. Vol. 12333. Springer, 2020.Status: Published
RobTest: A CP Approach to Generate Maximal Test Trajectories for Industrial Robots
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), Testing of Learning Robots (T-LARGO) , Department of Validation Intelligence for Autonomous Software Systems, Testing of Learning Robots (T-Largo) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | International Conference on Principles and Practice of Constraint Programming |
| Volume | 12333 |
| Edition | LNCS |
| Pagination | 1-17 |
| Date Published | 08/2020 |
| Publisher | Springer |
Software Testing for Machine Learning
In The Thirty-Fourth AAAI Conference on Artificial Intelligence. AAAI'20: AAAI, 2020.Status: Published
Software Testing for Machine Learning
| Afilliation | Software Engineering |
| Project(s) | T3AS, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2020 |
| Conference Name | The Thirty-Fourth AAAI Conference on Artificial Intelligence |
| Date Published | 2020 |
| Publisher | AAAI |
| Place Published | AAAI'20 |
Talks, invited
Requirements collection for the AI4EU platform development
In Big Data Value Forum, 3-5 November, Berlin, Germany, 2020.Status: Published
Requirements collection for the AI4EU platform development
| Afilliation | Software Engineering |
| Project(s) | AI4EU, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2020 |
| Location of Talk | Big Data Value Forum, 3-5 November, Berlin, Germany |
Edited books
Seventh International Workshop on Software Engineering Research and Industrial Practice
In International Conference on Software Engineering (ICSE). South Korea: ACM, 2020.Status: Published
Seventh International Workshop on Software Engineering Research and Industrial Practice
| Afilliation | Software Engineering |
| Project(s) | T3AS, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Edited books |
| Year of Publication | 2020 |
| Secondary Title | International Conference on Software Engineering (ICSE) |
| Publisher | ACM |
| Place Published | South Korea |
Journal Article
A learning algorithm for optimizing continuous integration development and testing practice
Software: Practice and Experience 49, no. 2 (2019): 192-213.Status: Published
A learning algorithm for optimizing continuous integration development and testing practice
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | Software: Practice and Experience |
| Volume | 49 |
| Issue | 2 |
| Pagination | 192-213 |
| Publisher | Wiley Online Library |
| Keywords | Continuous Integration, Decision Trees, Machine Leanring, Software Testing |
Good Practices in Aligning Software Engineering Research and Industry Practice
ACM SIGSOFT Software Engineering Notes 44, no. 3 (2019).Status: Published
Good Practices in Aligning Software Engineering Research and Industry Practice
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | ACM SIGSOFT Software Engineering Notes |
| Volume | 44 |
| Issue | 3 |
| Publisher | ACM |
Implications of Resurgence in Artificial Intelligence for Research Collaborations in Software Engineering
ACM SIGSOFT Software Engineering Notes 44, no. 3 (2019).Status: Published
Implications of Resurgence in Artificial Intelligence for Research Collaborations in Software Engineering
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | ACM SIGSOFT Software Engineering Notes |
| Volume | 44 |
| Issue | 3 |
| Publisher | ACM |
Proceedings, refereed
Challenges of Testing Machine Learning Based Systems
In Proceedings of the 1st IEEE Artificial Intelligence Testing Conference (AI Test 2019). San Francisco, CA, USA: IEEE, 2019.Status: Published
Challenges of Testing Machine Learning Based Systems
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), Testing of Learning Robots (T-Largo), T3AS |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2019 |
| Conference Name | Proceedings of the 1st IEEE Artificial Intelligence Testing Conference (AI Test 2019) |
| Publisher | IEEE |
| Place Published | San Francisco, CA, USA |
Edited books
Proceedings of the Joint 7th International Workshop on Conducting Empirical Studies in Industry and 6th International Workshop on Software Engineering Research and Industrial Practice
In International Conference on Software Engineering (ICSE). ACM/IEEE, 2019.Status: Published
Proceedings of the Joint 7th International Workshop on Conducting Empirical Studies in Industry and 6th International Workshop on Software Engineering Research and Industrial Practice
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Edited books |
| Year of Publication | 2019 |
| Secondary Title | International Conference on Software Engineering (ICSE) |
| Publisher | ACM/IEEE |
Technical reports
SARCoS: Machine Learning Models for the Selection of Security Test Scenarios
Simula Research Laboratory, 2019.Status: Accepted
SARCoS: Machine Learning Models for the Selection of Security Test Scenarios
| Afilliation | Software Engineering |
| Project(s) | T-Sar |
| Publication Type | Technical reports |
| Year of Publication | 2019 |
| Publisher | Simula Research Laboratory |
Survey on Testing of Deep Learning Systems
Simula Research Laboratory, 2019.Status: Published
Survey on Testing of Deep Learning Systems
Recent studies have shown that deep learning algorithms used for image classification or object recognition are not sufficiently reliable. These algorithms can be easily fooled by applying perturbations to images or generating artificial images that result in misclassification. In this paper, we provide an overview of software testing methods present in literature to test deep learning systems. We have explored different methods of testing deep neural networks, namely metamorphic testing, mutation testing, differential testing, and adversarial perturbation testing. We present the main findings available from the literature and compare these methods systematically and comprehensively. The results show that systematic testing of deep learning systems can further help to increase the performance of state-of-the-art systems.
| Afilliation | Software Engineering |
| Project(s) | Testing of Learning Robots (T-Largo) |
| Publication Type | Technical reports |
| Year of Publication | 2019 |
| Publisher | Simula Research Laboratory |
Talks, contributed
Statistics AIS Dataset from Statsat
In Simula Research Laboratory, 2019.Status: Published
Statistics AIS Dataset from Statsat
| Afilliation | Software Engineering, Machine Learning |
| Project(s) | T-Sar, Department of Validation Intelligence for Autonomous Software Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2019 |
| Location of Talk | Simula Research Laboratory |
The TSAR Project
In 14th Certus User Partner Workshop, Sep. 2019, Larvik, Norway, 2019.Status: Published
The TSAR Project
| Afilliation | Software Engineering |
| Project(s) | T-Sar |
| Publication Type | Talks, contributed |
| Year of Publication | 2019 |
| Location of Talk | 14th Certus User Partner Workshop, Sep. 2019, Larvik, Norway |
Journal Article
Certus: an organizational effort towards research-based innovation in software verification and validation
International Journal of Systems Assurance Engineering and Management 9, no. 2 (2018): 313-322.Status: Published
Certus: an organizational effort towards research-based innovation in software verification and validation
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2018 |
| Journal | International Journal of Systems Assurance Engineering and Management |
| Volume | 9 |
| Issue | 2 |
| Pagination | 313-322 |
| Publisher | Springer |
Proceedings, refereed
DevOps Enhancement with Continuous Test Optimization
In The 30th International Conference on Software Engineering and Knowledge Engineering (SEKE). KSI Research Inc. and Knowledge Systems Institute Graduate School, 2018.Status: Published
DevOps Enhancement with Continuous Test Optimization
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2018 |
| Conference Name | The 30th International Conference on Software Engineering and Knowledge Engineering (SEKE) |
| Pagination | 535-536 |
| Publisher | KSI Research Inc. and Knowledge Systems Institute Graduate School |
| DOI | 10.18293/SEKE2018-168 |
DevOps Improvements for Reduced Cycle Times with Integrated Test Optimizations for Continuous Integration
In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). IEEE, 2018.Status: Published
DevOps Improvements for Reduced Cycle Times with Integrated Test Optimizations for Continuous Integration
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2018 |
| Conference Name | 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC) |
| Pagination | 22-27 |
| Publisher | IEEE |
| DOI | 10.1109/COMPSAC.2018.00012 |
Practical Selective Regression Testing with Effective Redundancy in Interleaved Tests
In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP'18). ACM, 2018.Status: Published
Practical Selective Regression Testing with Effective Redundancy in Interleaved Tests
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2018 |
| Conference Name | Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP'18) |
| Pagination | 153-162 |
| Publisher | ACM |
| ISBN Number | 978-1-4503-5659-6 |
| DOI | 10.1145/3183519.3183532 |
Stratified Constructive Disjunction and Negation in Constraint Programming
In Proc. of IEEE Int. Conf. on Tools with Artificial Intelligence (ICTAI-18). Volos, Greece. Nov. 2018. IEEE, 2018.Status: Published
Stratified Constructive Disjunction and Negation in Constraint Programming
Constraint Programming (CP) is a powerful declarative programming paradigm combining inference and search in order to find solutions to various type of constraint systems. Dealing with highly disjunctive constraint systems is notoriously difficult in CP. Apart from trying to solve each disjunct independently from each other, there is little hope and effort to succeed in constructing intermediate results combining the knowledge originating from several disjuncts. In this paper, we propose If Then Else (ITE), a lightweight approach for implementing stratified constructive disjunction and negation on top of an existing CP solver, namely SICStus Prolog clp(FD). Although constructive disjunction is known for more than three decades, it does not have straightforward implementations in most CP solvers. ITE is a freely available library proposing stratified and constructive reasoning for various operators, including disjunction and negation, implication and conditional. Our preliminary experimental results show that ITE is competitive with existing approaches that handle disjunctive constraint systems.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2018 |
| Conference Name | Proc. of IEEE Int. Conf. on Tools with Artificial Intelligence (ICTAI-18). Volos, Greece. Nov. 2018 |
| Pagination | 106-113 |
| Publisher | IEEE |
| Other Numbers | arXiv:1811.03906 |
| DOI | 10.1109/ICTAI.2018.00026 |
Edited books
Fifth International Workshop on Software Engineering Research and Industrial Practice
In International Conference on Software Engineering (ICSE). Gothenburg, Sweden: ACM, 2018.Status: Published
Fifth International Workshop on Software Engineering Research and Industrial Practice
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Edited books |
| Year of Publication | 2018 |
| Secondary Title | International Conference on Software Engineering (ICSE) |
| Date Published | May |
| Publisher | ACM |
| Place Published | Gothenburg, Sweden |
Talks, contributed
Practical selective regression testing with effective redundancy in interleaved tests
In International Conference on Software Engineering (ICSE), Gothenburg, Sweden. Software Engineering in Practice, 2018.Status: Published
Practical selective regression testing with effective redundancy in interleaved tests
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Talks, contributed |
| Year of Publication | 2018 |
| Location of Talk | International Conference on Software Engineering (ICSE), Gothenburg, Sweden |
| Place Published | Software Engineering in Practice |
Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
In Gesellschaft für Informatik Software Engineering Conference 2018 (SE18), Ulm, Germany, 2018.Status: Published
Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Talks, contributed |
| Year of Publication | 2018 |
| Location of Talk | Gesellschaft für Informatik Software Engineering Conference 2018 (SE18), Ulm, Germany |
Proceedings, refereed
Detecting and Reducing Redundancy in Software Testing for Highly Configurable Systems
In IEEE International Symposium on High Assurance Systems Engineering. ACM/IEEE, 2017.Status: Published
Detecting and Reducing Redundancy in Software Testing for Highly Configurable Systems
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2017 |
| Conference Name | IEEE International Symposium on High Assurance Systems Engineering |
| Publisher | ACM/IEEE |
| ISBN Number | 978-1-5090-4636-2 |
| DOI | 10.1109/HASE.2017.31 |
Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
In Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis. New York, NY, USA: ACM, 2017.Status: Published
Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2017 |
| Conference Name | Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis |
| Pagination | 12-22 |
| Publisher | ACM |
| Place Published | New York, NY, USA |
| ISBN Number | 978-1-4503-5076-1 |
| Other Numbers | arXiv:1811.04122 |
| DOI | 10.1145/3092703.3092709 |
Test Prioritization with Optimally Balanced Configuration Coverage
In IEEE International Symposium on High Assurance Systems Engineering. ACM/IEEE, 2017.Status: Published
Test Prioritization with Optimally Balanced Configuration Coverage
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2017 |
| Conference Name | IEEE International Symposium on High Assurance Systems Engineering |
| Publisher | ACM/IEEE |
| ISBN Number | 978-1-5090-4636-2 |
| DOI | 10.1109/HASE.2017.26 |
TITAN: Test Suite Optimization for Highly Configurable Software
In International Conference on Software Testing, Verification and Validation (ICST 2017) . IEEE, 2017.Status: Published
TITAN: Test Suite Optimization for Highly Configurable Software
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2017 |
| Conference Name | International Conference on Software Testing, Verification and Validation (ICST 2017) |
| Publisher | IEEE |
| ISBN Number | 978-1-5090-6031-3 |
| DOI | 10.1109/ICST.2017.60 |
Journal Article
Modeling and Verifying Combinatorial Interactions to Test Data Intensive Systems: Experience at the Norwegian Customs Directorate
IEEE Transactions on Reliability 66 (2017): 3-16.Status: Published
Modeling and Verifying Combinatorial Interactions to Test Data Intensive Systems: Experience at the Norwegian Customs Directorate
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2017 |
| Journal | IEEE Transactions on Reliability |
| Volume | 66 |
| Number | 1 |
| Pagination | 3–16 |
| Publisher | IEEE Transactions on Reliability |
| URL | https://doi.org/10.1109/TR.2016.2618121 |
| DOI | 10.1109/TR.2016.2618121 |
Using Global Constraints to Automate Regression Testing
AI Magazine 38, no. Spring (2017).Status: Published
Using Global Constraints to Automate Regression Testing
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2017 |
| Journal | AI Magazine |
| Volume | 38 |
| Issue | Spring |
| Number | 1 |
| Publisher | AAAI |
Book Chapter
Software Product Line Test Suite Reduction with Constraint Optimization
In Software Technologies, 68-87. Vol. 743. Springer International Publishing, 2017.Status: Published
Software Product Line Test Suite Reduction with Constraint Optimization
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Book Chapter |
| Year of Publication | 2017 |
| Book Title | Software Technologies |
| Volume | 743 |
| Pagination | 68-87 |
| Publisher | Springer International Publishing |
Proceedings, refereed
A New Approach to Feature-based Test Suite Reduction in Software Product Line Testing
In ICSOFT-EA 2016, 11th Int. Conf. on Software Engineering and Applications, Lisbon, July 2016, Awarded Best Paper. INSTICC Press, 2016.Status: Published
A New Approach to Feature-based Test Suite Reduction in Software Product Line Testing
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2016 |
| Conference Name | ICSOFT-EA 2016, 11th Int. Conf. on Software Engineering and Applications, Lisbon, July 2016, Awarded Best Paper |
| Date Published | 07/2016 |
| Publisher | INSTICC Press |
Automated Regression Testing Using Constraint Programming
In Twenty-Eighth Conference on Innovative Applications of Artificial Intelligence (IAAI-16), Phoenix, AZ, USA, Feb. 2016, 2016.Status: Published
Automated Regression Testing Using Constraint Programming
In software validation, regression testing aims to check the absence of regression faults in new releases of a software system. Typically, test cases used in regression testing are executed during a limited amount of time and are selected to check a given set of user requirements. When testing large systems the number of regression tests grows quickly over the years, and yet the available time slot stays limited. In order to overcome this problem, an approach known as test suite reduction (TSR), has been developed in software engineering to select a smallest subset of test cases, so that each requirement remains covered at least once. However solving the TSR problem is difficult as the underlying optimization problem is NP-hard, but it is also crucial for vendors interested in reducing the time to market of new software releases.
In this paper, we address regression testing and TSR with Constraint Programming (CP). More specifically, we propose new CP models to solve TSR that exploit global constraints, namely NValue and GCC. We reuse a set of preprocessing rules to reduce a priori each instance, and we introduce a structure-aware search heuristic. We evaluated our CP models and proposed improvements against existing approaches, including a simple greedy approach and MINTS, the state-of-the-art tool of the software engineering community. Our experiments show that CP outperforms both the greedy approach and MINTS when it is interfaced with MiniSAT, in terms of percentage of reduction and execution time. When MINTS is interfaced with CPLEX, we show that our CP model performs better only on percentage of reduction. Finally, by working closely with validation engineers from Cisco Systems, Norway, we integrated our CP model into an industrial regression testing process.
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2016 |
| Conference Name | Twenty-Eighth Conference on Innovative Applications of Artificial Intelligence (IAAI-16), Phoenix, AZ, USA, Feb. 2016 |
| Date Published | 02/2016 |
Coverage-based Test Prioritization for Regression Testing of Configurable Software
In IEEE 27th International Symposium on Software Reliability Engineering (ISSRE), 2016.Status: Published
Coverage-based Test Prioritization for Regression Testing of Configurable Software
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2016 |
| Conference Name | IEEE 27th International Symposium on Software Reliability Engineering (ISSRE) |
Effect of Time Window on the Performance of Continuous Regression Testing
In 32nd IEEE International Conference on Software Maintenance and Evolution (ICSME), 2016.Status: Published
Effect of Time Window on the Performance of Continuous Regression Testing
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2016 |
| Conference Name | 32nd IEEE International Conference on Software Maintenance and Evolution (ICSME) |
Improving Configurable Software Testing with Statistical Test Selection
In 31st IEEE/ACM International Conference on Automated Software Engineering (ASE) Workshops, 2016.Status: Published
Improving Configurable Software Testing with Statistical Test Selection
| Afilliation | Software Engineering, Software Engineering |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2016 |
| Conference Name | 31st IEEE/ACM International Conference on Automated Software Engineering (ASE) Workshops |
Journal Article
Modelling and Verifying Combinatorial Interactions to Test Data Intensive Systems: Experience with Optimal Archiving at the Norwegian Customs and Excise Directorate
IEEE Transaction on Reliability, no. 99 (2016): 1-14.Status: Published
Modelling and Verifying Combinatorial Interactions to Test Data Intensive Systems: Experience with Optimal Archiving at the Norwegian Customs and Excise Directorate
Testing data-intensive systems is paramount to increase
our reliance on information processed in e-governance,
scientific/medical research, and social networks. Data accrued in
these systems often go through several manual and computational
steps involving human inputs in interactive media and complex
batch appications that aim to ensure high quality of data in
terms of validity, correctness, and adherence to business rules. A
common industrial practice in testing data-intensive systems is
to extract test databases from live production streams and verify
the data in them through a checklist of requirements either
by tedious manual observation or by executing complex SQL
queries composed and understood by very few domain experts.
We elevate the specification of such requirements on data by
modelling data interactions between fields cross-cutting the test
database’s schema. These interactions are modelled as test cases
in a classification tree model. The model documents intuitive
expert knowledge about what to expect in the test database
and is given executable semantics using our human-in-the-loop
tool DEPICT. DEPICT verifies if interactions occurred or not
in systematically extracted test databases. Non-occurrence of
expected interactions or occurrence of unexpected interactions
indicate faults in the data. We present experiences on how our
model-driven approach has been successfully applied to verify
test databases in the Norwegian Public Sector. In particular, we
present case studies at (1) the Norwegian Customs and Excise
Directorate for verifying the adherence to customs regulations
and (2) the Cancer Registry of Norway to verify its data quality
management process involving both human coders and complex
legacy batches.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2016 |
| Journal | IEEE Transaction on Reliability |
| Issue | 99 |
| Pagination | 1-14 |
| Publisher | IEEE |
Practical Minimization of Pairwise-Covering Test Configurations Using Constraint Programming
Information and Software Technology 71 (2016): 129-146.Status: Published
Practical Minimization of Pairwise-Covering Test Configurations Using Constraint Programming
Context: Testing highly-configurable software systems is challenging due to a large number of test configurations that have to be carefully selected in order to reduce the testing effort as much as possible, while maintaining high software quality. Finding the smallest set of valid test configurations that ensure sufficient coverage of the system's feature interactions is thus the objective of validation engineers, especially when the execution of test configurations is costly or time-consuming. However, this problem is NP-hard in general and approximation algorithms have often been used to address it in practice.
Objective: In this paper, we explore an alternative exact approach based on constraint programming that will allow engineers to increase the effectiveness of configuration testing while keeping the number of configurations as low as possible.
Method: Our approach consists in using a (time-aware) minimization algorithm based on constraint programming. Given the amount of time, our solution generates a minimized set of valid test configurations that ensure coverage of all pairs of feature values (a.k.a. pairwise coverage). The approach has been implemented in a tool called PACOGEN.
Results: PACOGEN was evaluated on 224 feature models from the standard benchmark repository SPLOT, and compared
in comparison with the two existing tools that are based on a greedy algorithm. For 79% of 224 feature models, PACOGEN generated up to 60% fewer test configurations than the competitor tools. We further evaluated PACOGEN in the case study of an industrial video conferencing product line with a feature model of 169 features, and found 60% fewer configurations compared with the manual approach followed by test engineers. The set of test configurations generated by PACOGEN decreased the time required by test engineers in manual test configuration by 85%, increasing the feature-pairs coverage at the same time
Conclusion: Our experimental evaluation concluded that optimal time-aware minimization of pairwise-covering test configurations is efficiently addressed using constraint programming techniques.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2016 |
| Journal | Information and Software Technology |
| Volume | 71 |
| Pagination | 129-146 |
| Date Published | 03/2016 |
| Publisher | Elsevier |
Journal Article
Certus: Glimpses of a Centre for Research-Based Innovation in Software Verification and Validation
International Journal of System Assurance Engineering and Management (2015): 1-25.Status: Published
Certus: Glimpses of a Centre for Research-Based Innovation in Software Verification and Validation
What is gratifying to a software engineering researcher? Three of many possible answers to this question are (a) the intellectual exercise in developing/disseminating approaches that address emerging and existing challenges, (b) recognition from impact in a community of researchers and (c) widespread use of novel ideas, including software, in the society at large leading to enhancement of human ability and job creation. A culmination of these sources requires an organizational effort. This article presents a detailed account of a research-based innovation centre, Certus, to facilitate such a culmination for software engineering researchers. Certus has established a body of knowledge, methods and tools for the validation and verification of software systems in the Norwegian private and public sector. Certus works in close cooperation with five founding user partners and is hosted by the Simula Research Laboratory. We present the organizational structure of Certus and describe how Certus’s life and health is planned and evaluated on a regular basis as a research-based innovation centre. We expound two successful collaborations, with (a) the private sector entity Cisco systems, Norway and (b) the public sector entity the Norwegian Customs and Excise. We hope that this document will serve as a basis to encourage national/international funding schemes to create call for proposals for long-term research-based innovation centres. This, we believe, is one way to justify use of tax payers resources in creating a win–win situation for the triple helix: government, researchers and industry.
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2015 |
| Journal | International Journal of System Assurance Engineering and Management |
| Pagination | 1-25 |
| Date Published | 03/2015 |
| Publisher | Springer |
Proceedings, refereed
Evaluating Reconfiguration Impact in Self-Adaptive Systems – An Approach Based on Combinatorial Interaction Testing
In The 41st Euromicro Conference on Software Engineering and Advanced Applications (SEAA). Madeira, Portugal, August 26-28. Funchal: IEEE, 2015.Status: Published
Evaluating Reconfiguration Impact in Self-Adaptive Systems – An Approach Based on Combinatorial Interaction Testing
Self-adaptive software adapts its behavior to the operational context via automatic run-time reconfiguration of software components. Particular reconfigurations may negatively affect the system Quality of Service (QoS), and therefore their impact over the system performance needs to be thoroughly evaluated. In this paper, we present an approach, based on Combinatorial Interaction Testing (CIT), that generates a sequence of configurations aimed at evaluating the extent to which reconfigurations affect the system QoS. Specifically, we transform a Classification Tree Models (CTM) of the configurations domain to a Constraint Satisfaction Problem (CSP) in Alloy, whose solution is a sequence of reconfigurations achieving T-wise coverage between system features, and R-wise coverage between configurations in the sequence. The resolution of the CSP is performed by an incremental growth algorithm that divides the generation of the sequence into sub-problems, and merges the results into a final set of test configurations. Preliminary validation in a self-adaptive vision system shows that our methodology effectively identifies critical reconfigurations exhibiting a high variation in QoS. This result encourages the use of CIT as a strategy to evaluate the performance of self-adaptive systems.
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2015 |
| Conference Name | The 41st Euromicro Conference on Software Engineering and Advanced Applications (SEAA). Madeira, Portugal, August 26-28 |
| Pagination | 250 - 254 |
| Date Published | 08/2015 |
| Publisher | IEEE |
| Place Published | Funchal |
| DOI | 10.1109/SEAA.2015.72 |
Multi-perspective Regression Test Prioritization for Time-Constrained Environments
In IEEE International Conference on Software Quality, Reliability and Security. Vancouver, Canada, August 3-5, 2015.Status: Published
Multi-perspective Regression Test Prioritization for Time-Constrained Environments
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2015 |
| Conference Name | IEEE International Conference on Software Quality, Reliability and Security |
| Place Published | Vancouver, Canada, August 3-5 |
Towards More Relational Feature Models
In ICSOFT-EA 2015 - Proceedings of the 10th International Conference on Software Engineering and Applications, Colmar, Alsace, France, 20-22 July. SciTePress, 2015.Status: Published
Towards More Relational Feature Models
Feature modeling is of paramount importance to capture variabilities and commonalities within a software product line. Nevertheless, current feature modeling notations are limited, representing only propositional formulae over attributed variables. This position paper advocates the extension of feature modeling formalisms with richer computational domains and relational operations. In particular, it proposes to extend feature modeling with finite and continuous domain variables, with first-order logic quantifiers, and with N-ary relations between features attributes, and with so-called global constraints. In order to extend the expressiveness while preserving automated analysis facilities, feature models could be semantically interpreted as first-order logic formulae (instead of propositional logic formulae), including global and continuous dependency between features. In simpler words, this paper emphasizes the importance of having more relational feature models and presents next generation applications.
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2015 |
| Conference Name | ICSOFT-EA 2015 - Proceedings of the 10th International Conference on Software Engineering and Applications, Colmar, Alsace, France, 20-22 July. |
| Pagination | 381-386 |
| Date Published | 07/2015 |
| Publisher | SciTePress |
| ISBN Number | 978-989-758-114-4 |
Proceedings, refereed
FLOWER: Optimal Test Suite Reduction As a Network Maximum Flow
In Proceedings of Int. Symp. on Soft. Testing and Analysis (ISSTA'14), San José, CA, USA, Jul. 2014. New York, USA: ACM, 2014.Status: Published
FLOWER: Optimal Test Suite Reduction As a Network Maximum Flow
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2014 |
| Conference Name | Proceedings of Int. Symp. on Soft. Testing and Analysis (ISSTA'14), San José, CA, USA, Jul. 2014 |
| Pagination | 171-180 |
| Date Published | 07/2014 |
| Publisher | ACM |
| Place Published | New York, USA |
| ISBN Number | 978-1-4503-2645-2 |
| Keywords | Conference |
| DOI | 10.1145/2610384.2610416 |
Talks, invited
Variability Testing of Highly-Configurable Software
In The Norwegian Computer Society. .: , 2014.Status: Published
Variability Testing of Highly-Configurable Software
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Talks, invited |
| Year of Publication | 2014 |
| Location of Talk | The Norwegian Computer Society |
| Publisher | . |
| Place Published | . |
Talks, invited
Managing Test Configurations in High-Variability Testing Environments With TITAN and Pure::variants
In pure::variants Solutions Forum, SPLC. .: , 2013.Status: Published
Managing Test Configurations in High-Variability Testing Environments With TITAN and Pure::variants
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Talks, invited |
| Year of Publication | 2013 |
| Location of Talk | pure::variants Solutions Forum, SPLC |
| Publisher | . |
| Place Published | . |
Proceedings, refereed
Practical Pairwise Testing for Software Product Lines
In Proceedings of the International Software Product Line Conference (SPLC). New York, NY, USA: ACM, 2013.Status: Published
Practical Pairwise Testing for Software Product Lines
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2013 |
| Conference Name | Proceedings of the International Software Product Line Conference (SPLC) |
| Pagination | 227-235 |
| Publisher | ACM |
| Place Published | New York, NY, USA |
Test Case Prioritization for Continuous Regression Testing: an Industrial Case Study
In Proceedings of the International Conference on Software Maintenance (ICSM). Eindhoven: IEEE, 2013.Status: Published
Test Case Prioritization for Continuous Regression Testing: an Industrial Case Study
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2013 |
| Conference Name | Proceedings of the International Conference on Software Maintenance (ICSM) |
| Publisher | IEEE |
| Place Published | Eindhoven |
Technical reports
Test Selection Based on Data Interactions in Data-Intensive Systems
Simula Research Laboratory, 2013.Status: Published
Test Selection Based on Data Interactions in Data-Intensive Systems
Testing data-intensive systems is paramount to increase our reliance on information in e-governance, scientific/ medical research, and social networks. Common practice to test these systems is by using a live production database. This testing approach is space and time inefficient and lacks clarity about what test cases or scenarios are covered. In this paper, we leverage classification tree modelling to specify desired test cases as data interactions between a set of fields across multiple tables of an existing database. Our methodology and tool, DEPICT, uses test case specifications in classification tree models to (a) automatically derive a spanning tree representing a relationship between any set of fields for any given database schema (b) generates queries to create an efficient inner join between related tables in the spanning tree (c) extract records from various tables that satisfy data interactions in the classification tree model (d) discovers holes or unsatisfied test cases in the test databases. We perform experiments to show that our approach is fast and scalable to extract test databases. Our experiments are based on selecting test databases from 8000 declarations for 60,000 items from the Norwegian Customs and Excise information system TVINN.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Technical reports |
| Year of Publication | 2013 |
| Number | 2013-03 |
| Date Published | 10/2013 |
| Publisher | Simula Research Laboratory |
| Keywords | Conference |
Proceedings, refereed
A Review of Two Experiences From Applying Model Based Testing in Practice
In International Symposium on Software Reliability Engineering Workshops (ISSREW). Dallas, TX: IEEE, 2012.Status: Published
A Review of Two Experiences From Applying Model Based Testing in Practice
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2012 |
| Conference Name | International Symposium on Software Reliability Engineering Workshops (ISSREW) |
| Publisher | IEEE |
| Place Published | Dallas, TX |
Technical reports
Research report on test configuration generation
Simula Research Laboratory, 2012.Status: Published
Research report on test configuration generation
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI) |
| Publication Type | Technical reports |
| Year of Publication | 2012 |
| Publisher | Simula Research Laboratory |