Projects
MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
The innovation planned in this project is an add-on to the digitization project currently being undertaken by the Cancer Registry of Norway (CR). The project started in 2009 and aims to transform the current paper-based/manual system into an ICT-based Automated Cancer Registry System (ACRS). The planned innovation project aims to develop systematic, automated and cost-effective model-based approaches for ensuring the quality of the evolving ACRS system and therefore significantly improving the efficiency of the patient history registration process. This will positively affect all its end users, including researchers, patients, doctors, and government officials.
Funding source:
Regionale forskningsfond
All partners:
- Simula Research Laboratory
- Cancer Registry of Norway
Project leaders:
- Simula: Tao Yue (PI) and Shaukat Ali (Co-PI),
- Cancer Registry: Jan F. Nygård
Publications for MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System
Journal Article
Automated Refactoring of OCL Constraints with Search
IEEE Transactions on Software Engineering (TSE) (2017).Status: Published
Automated Refactoring of OCL Constraints with Search
Object Constraint Language (OCL) constraints are typically used to provide precise semantics to models developed with the Unified Modeling Language (UML). When OCL constraints evolve regularly, it is essential that they are easy to understand and maintain. For instance, in cancer registries, to ensure the quality of cancer data, more than one thousand medical rules are defined and evolve regularly. Such rules can be specified with OCL. It is, therefore, important to ensure the understandability and maintainability of medical rules specified with OCL. To tackle such a challenge, we propose an automated search-based OCL constraint refactoring approach (SBORA) by defining and applying four semantics-preserving refactoring operators (i.e., Context Change, Swap, Split and Merge) and three OCL quality metrics (Complexity, Coupling, and Cohesion) to measure the understandability and maintainability of OCL constraints. We evaluate SBORA along with six commonly used multi-objective search algorithms (e.g., Indicator-Based Evolutionary Algorithm (IBEA)) by employing four case studies from different domains: healthcare (i.e., cancer registry system from Cancer Registry of Norway (CRN)), Oil&Gas (i.e., subsea production systems), warehouse (i.e., handling systems), and an open source case study named SEPA. Results show: 1) IBEA achieves the best performance among all the search algorithms and 2) the refactoring approach along with IBEA can manage to reduce on average 29.25% Complexity and 39% Coupling and improve 47.75% Cohesion, as compared to the original OCL constraint set from CRN. Furthermore, we conducted a controlled experiment with 96 subjects and results show that the understandability and maintainability of the original constraint set can be improved significantly from the perspectives of the 96 participants of the controlled experiment.
| Afilliation | Software Engineering |
| Project(s) | MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System, The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2017 |
| Journal | IEEE Transactions on Software Engineering (TSE) |
| Publisher | IEEE |
| URL | http://ieeexplore.ieee.org/document/8114267/ |
| DOI | 10.1109/TSE.2017.2774829 |
IOCL: An Interactive Tool for Specifying, Validating and Evaluating OCL Constraints
Science of Computer Programming (SCP) 149 (2017): 3-8.Status: Published
IOCL: An Interactive Tool for Specifying, Validating and Evaluating OCL Constraints
The Object Constraint Language (OCL) is commonly used for specifying additional constraints on models, in addition, to the ones enforced by the semantics of the models. However, a lot of practitioners and even researchers are reluctant in using OCL to some extent due to the lack of sufficient familiarity with OCL. To facilitate practitioners and researchers in specifying OCL constraints, we designed and developed a web-based tool called interactive OCL (iOCL) for interactively specifying constraints on a given model. The core idea behind iOCL is to present and display only relevant details (e.g., operations) of OCL to users at a given step of constraint specification process, in addition to helping modelers with its syntax. We evaluated iOCL using a real-world case study from Cancer Registry of Norway and the results showed that iOCL can significantly reduce the time required to specify OCL constraints and decrease the possibility of making syntactic errors during the specification process. Thus, we conclude that iOCL can facilitate the process of OCL constraint specification.
| Afilliation | Software Engineering |
| Project(s) | MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System, The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2017 |
| Journal | Science of Computer Programming (SCP) |
| Volume | 149 |
| Pagination | 3-8 |
| Date Published | 08/2017 |
| Publisher | Elsevier |
Search and similarity based selection of use case scenarios: An empirical study
Empirical Software Engineering (2017): 1-78.Status: Published
Search and similarity based selection of use case scenarios: An empirical study
Use case modeling is a well-known requirements specification method and has been widely applied in practice. Use case scenarios of use case models are input elements for requirements inspection and analysis, requirements-based testing, and other downstream activities. It is, however, a practical challenge to inspect all use case scenarios that can be obtained from any non-trivial use case model, as such an inspection activity is often performed manually by domain experts. Therefore, it is needed to propose an automated solution for selecting a subset of use case scenarios with the ultimate aim of enabling cost-effective requirements (use case) inspection, analysis, and other relevant activities. Our solution is built on a natural language based, restricted use case modeling methodology (named as RUCM), in the sense that requirements specifications are specified as RUCM use case models. Use case scenarios can be automatically derived from RUCM use case models with the already established Zen-RUCM framework. In this paper, we propose a search-based and similarity-based approach called S3RCUM, through an empirical study, to select most diverse use case scenarios to enable cost-effective use case inspections. The empirical study was designed to evaluate the performance of three search algorithms together with eight similarity functions, through one real-world case study and six case studies from literature. Results show that (1+1) Evolutionary Algorithm together with Needleman-Wunsch similarity function significantly outperformed the other 31 combinations of the search algorithms and similarity functions. The combination managed to select 50% of all the generated RUCM use case scenarios for all the case studies to detect all the seeded defects.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines , MBT4CPS: Model-Based Testing For Cyber-Physical Systems , U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System |
| Publication Type | Journal Article |
| Year of Publication | 2017 |
| Journal | Empirical Software Engineering |
| Pagination | 1-78 |
| Date Published | 04/2017 |
| Publisher | Springer |
Master's thesis
A Large-Scale OCL Constraint Repository And Comprehensive Analysis For Supporting Automated Cancer Registry System
In The Department of Informatics, University of Oslo, 2017.Status: Published
A Large-Scale OCL Constraint Repository And Comprehensive Analysis For Supporting Automated Cancer Registry System
| Afilliation | Software Engineering |
| Project(s) | MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System |
| Publication Type | Master's thesis |
| Year of Publication | 2017 |
| Degree awarding institution | The Department of Informatics, University of Oslo |
A Rule-Based Framework for Supporting Automated Change Impact Analysis in the Cancer Registry of Norway
In The Department of Informatics, University of Oslo, 2017.Status: Published
A Rule-Based Framework for Supporting Automated Change Impact Analysis in the Cancer Registry of Norway
| Afilliation | Software Engineering |
| Project(s) | MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System |
| Publication Type | Master's thesis |
| Year of Publication | 2017 |
| Degree awarding institution | The Department of Informatics, University of Oslo |
Proceedings, refereed
RCIA: Automated Change Impact Analysis to Facilitate a Practical Cancer Registry System
In The International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2017.Status: Published
RCIA: Automated Change Impact Analysis to Facilitate a Practical Cancer Registry System
The Cancer Registry of Norway (CRN) employs a cancer registry system to collect cancer patient data (e.g., diagnosis and treatments) from various medical entities (e.g., clinic hospitals). The collected data are then checked for validity (i.e., validation) and assembled as cancer cases (i.e., aggregation) based on more than 1000 cancer coding rules in the system. However, it is frequent in practice that the collected cancer data changes due to various reasons (e.g., different treatments) and the cancer coding rules can also change/evolve due to new medical knowledge. Thus, such a cancer registry system requires an efficient means to automatically analyze these changes and provide consequent impacts to medical experts for further actions. This paper proposes an automated Rule-based Change Impact Analysis (CIA) approach named RCIA that includes: 1) a change classification to capture the potential changes that can occur at CRN; 2) in total 80 change impact analysis rules including 50 dependency rules and 30 impact rules; and 3) an efficient algorithm to analyze changes and produce consequent impacts. We evaluate RCIA via a case study with 12 real change sets from CRN and a conducted interview. The results showed that RCIA managed to produce 100% actual change impacts and the medical expert at CRN is quite positive to apply RCIA to facilitate their cancer registry system. We also shared a set of lessons learned based on the collaboration with CRN.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2017 |
| Conference Name | The International Conference on Software Maintenance and Evolution (ICSME) |
| Pagination | 603-612 |
| Publisher | IEEE |
Talks, contributed
RCIA: Automated Change Impact Analysis to Facilitate a Practical Cancer Registry System
In The International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China, 2017.Status: Published
RCIA: Automated Change Impact Analysis to Facilitate a Practical Cancer Registry System
The Cancer Registry of Norway (CRN) employs a cancer registry system to collect cancer patient data (e.g., diagnosis and treatments) from various medical entities (e.g., clinic hospitals). The collected data are then checked for validity (i.e., validation) and assembled as cancer cases (i.e., aggregation) based on more than 1000 cancer coding rules in the system. However, it is frequent in practice that the collected cancer data changes due to various reasons (e.g., different treatments) and the cancer coding rules can also change/evolve due to new medical knowledge. Thus, such a cancer registry system requires an efficient means to automatically analyze these changes and provide consequent impacts to medical experts for further actions. This paper proposes an automated Rule-based Change Impact Analysis (CIA) approach named RCIA that includes: 1) a change classification to capture the potential changes that can occur at CRN; 2) in total 80 change impact analysis rules including 50 dependency rules and 30 impact rules; and 3) an efficient algorithm to analyze changes and produce consequent impacts. We evaluate RCIA via a case study with 12 real change sets from CRN and a conducted interview. The results showed that RCIA managed to produce 100% actual change impacts and the medical expert at CRN is quite positive to apply RCIA to facilitate their cancer registry system. We also shared a set of lessons learned based on the collaboration with CRN.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System |
| Publication Type | Talks, contributed |
| Year of Publication | 2017 |
| Location of Talk | The International Conference on Software Maintenance and Evolution (ICSME), Shanghai, China |
Technical reports
A Pilot Experiment to Assess Interactive OCL Specification in a Real Setting
Simula Research Laboratory, 2017.Status: Published
A Pilot Experiment to Assess Interactive OCL Specification in a Real Setting
The Object Constraint Language (OCL) is a formal, declarative, and side-effect free language, standardized by the Object Management Group, for specifying constraints or queries on models specified in the Unified Modeling Language (UML). OCL was designed with the aim to bridge the gap between natural language and traditional formal languages requiring a strong mathematical background to understand and apply. OCL, along with UML, have been applied in practice for various purposes such as facilitating automated model-based testing. In most of such contexts of OCL, engineers with software engineering backgrounds specify OCL constraints. However, it is still a challenge for constraint authors (e.g., medical coders) who have no such background to apply OCL for other purposes (e.g., specifying medical rules). In this direction, in our previous work, we proposed a user-interactive specification framework, named iOCL, for facilitating OCL constraint specification and validation. The aim was to ease its adoption in practice in a wider application scope. In this paper, we present a pilot experiment that was conducted to assess the practical applicability of iOCL in Cancer Registry of Norway with real users of iOCL in terms of specifying medical cancer coding rules with iOCL. Results of the pilot experiment showed that, with iOCL, time to specify OCL constraints can be significantly reduced as compared to directly specifying OCL constraints without the tool support. In addition, participants of the experiment found that iOCL is easy to use.
| Afilliation | Software Engineering |
| Project(s) | MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System, The Certus Centre (SFI) |
| Publication Type | Technical reports |
| Year of Publication | 2017 |
| Publisher | Simula Research Laboratory |
Proceedings, refereed
A Model-Based Approach with Tool Support to Facilitate the Cancer Registration Process in Cancer Registry of Norway
In European Telemedicine Conference (ETC), 2016.Status: Published
A Model-Based Approach with Tool Support to Facilitate the Cancer Registration Process in Cancer Registry of Norway
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2016 |
| Conference Name | European Telemedicine Conference (ETC) |
iOCL: A Interactive Tool for Specifying, Validating and Evaluating OCL Constraints
In Tool Demonstrations Track, ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems (MODELS), 2016.Status: Published
iOCL: A Interactive Tool for Specifying, Validating and Evaluating OCL Constraints
The Object Constraint Language (OCL) is frequently used to specify additional constraints on models, in addition, to the ones enforced by semantics of the models. It is a well- known fact that due to the lack of familiarity with OCL, practitioners and even researcher to some extent are reluctant in using OCL. To help practitioners and researchers in writing OCL constraints for their specific problem at hand, we developed a tool called interactive OCL (iOCL) for interactively specifying constraints on a given model. The basic philosophy behind the tool is to present only those details (e.g., operations) of OCL to modelers that are valid at a given step of constraint specification process, in addition to helping modelers with its syntax. Our ultimate aim is to reduce the effort required to specify constraints, subsequently lowering down training cost and increasing the correctness of the constraints. iOCL is a web-based ap- plication that integrates other tools including Eclipse OCL for validation and evaluation of OCL constraints, and EsOCL for automatically generating valid instances of models that satisfy the specified constraints.
| Afilliation | Software Engineering, Software Engineering, Software Engineering |
| Project(s) | MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System, The Certus Centre (SFI) |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2016 |
| Conference Name | Tool Demonstrations Track, ACM/IEEE 19th International Conference on Model Driven Engineering Languages and Systems (MODELS) |
| Pagination | 1-7 |
| Date Published | 09/2016 |
U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty is intrinsic in Cyber-Physical Systems (CPS) owning to novel interactions among software, embedded systems, networking equipment, cloud infrastructures, and agents (e.g., humans). Such systems have become predominantly visible in critical industrial domains (e.g., healthcare and transportation) and oblige the implementation of proper mechanisms to deal with uncertainty during their real operation. One way to ensure the correct implementation of such mechanisms is with automated testing. The U-Test project aims at ensuring that CPS are tested adequately under uncertainty using systematic and automated techniques such as model and search-based testing to facilitate their reliable operation.
U-Test keeps a full catalog of the project's publications, including publications from other academic institutions than Simula.
Final goal
To improve the dependability of Cyber-Physical Systems (CPS) via cost-effective model-based and search-based testing of CPS under uncertainty, by defining an Uncertainty Taxonomy and holistic modeling and testing frameworks with considerable reliance on standards.
Funding source
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 645463. (ICT-01-2014 - Smart Cyber-Physical Systems)
All partners
- Oslo Medtech (Norway)
- Simula (Norway)
- Technical University of Vienna (Austria)
- Fraunhofer FOKUS (Germany)
- Future Position X (Sweden)
- ULMA Handling Systems (Spain)
- Nordic MedTest (Sweden)
- Easy Global Market (France)
- Ikerlan (Spain)
Technical project leader
Shaukat Ali (PI)
Standardization leader
Tao Yue (Co-PI)
Media presence
Publications for U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies
Journal Article
Uncertainty-wise Test Case Generation and Minimization for CyberPhysical Systems
Journal of Systems and Software 153 (2019).Status: Published
Uncertainty-wise Test Case Generation and Minimization for CyberPhysical Systems
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | Journal of Systems and Software |
| Volume | 153 |
| Date Published | 07/2019 |
| Publisher | Elsevier |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Journal Article
Specifying Uncertainty in Use Case Models
Journal of Systems and Software 144 (2018): 573-603.Status: Published
Specifying Uncertainty in Use Case Models
Context: Latent uncertainty in the context of software-intensive systems (e.g., Cyber-Physical Systems (CPSs)) demands explicit attention right from the start of development. Use case modeling—a commonly used method for specifying requirements in practice, should also be extended for explicitly specifying uncertainty.
Objective: Since uncertainty is a common phenomenon in requirements engineering, it is best to address it explicitly by identifying, qualifying, and, where possible, quantifying uncertainty at the beginning stage. The ultimate aim, though not within the scope of this paper, was to use these use cases as the starting point to create test-ready models to support automated testing of CPSs under uncertainty.
Method: We extend the Restricted Use Case Modeling (RUCM) methodology and its supporting tool to specify uncertainty as part of system requirements. Such uncertainties include those caused by insufficient domain expertise of stakeholders, disagreements among them, and known uncertainties about assumptions about the environment of the system. The extended RUCM, called U-RUCM, inherits the features of RUCM, such as automated analyses and generation of models, to mention but a few. Consequently, U-RUCM provides all the key benefits offered by RUCM (i.e., reducing ambiguities in requirements), but also, it allows specification of uncertainties with the possibilities of reasoning and refining existing ones and even uncovering unknown ones.
Results: We evaluated U-RUCM with two industrial CPS case studies. After refining RUCM models (specifying initial requirements), by applying the U-RUCM methodology, we successfully identified and specified additional 306% and 512% (previously unknown) uncertainty requirements, as compared to the initial requirements specified in RUCM. This showed that, with U-RUCM, we were able to get a significantly better and more precise characterization of uncertainties in requirement engineering.
Conclusion: Evaluation results show that U-RUCM is an effective methodology (with tool support) for dealing with uncertainty in requirements engineering. We present our experience, lessons learned, and future challenges, based on the two industrial case studies.
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies |
| Publication Type | Journal Article |
| Year of Publication | 2018 |
| Journal | Journal of Systems and Software |
| Volume | 144 |
| Pagination | 573-603 |
| Publisher | Elsevier |
| Keywords | Belief, Uncertainty, Use Case Modeling |
| DOI | 10.1016/j.jss.2018.06.075 |
PhD Thesis
Uncertainty-wise Cyber-Physical Systems Testing
In The University of Oslo. Vol. PhD. Norway: The University of Oslo, 2018.Status: Published
Uncertainty-wise Cyber-Physical Systems Testing
A Cyber-Physical Systems (CPS), as an integration of computing, communication, and control for making intelligent and autonomous systems, has been widely applied in various safety-critical domains, e.g., avionics and automotive. However, uncertainty is inherent in CPSs due to various reasons such as unpredictable environment under which the CPSs are operated. And, uncertainties may cause irreparable accidents once they cannot be handled properly by CPSs. Therefore, it is crucial to identify uncertainties in CPSs and test CPSs under the uncertainties, to ensure that CPSs are capable of handling the uncertainties during their actual operations, i.e., making CPSs less uncertain.
Towards this direction, five contributions were made in the thesis corresponding to five papers respectively: (C1) a conceptual model, named as U-Model, for helping develop a systematic and comprehensive understanding of uncertainty in CPSs; (C2) an use case modeling methodology, named as U-RUCM, for identifying, qualifying, and, where possible, quantifying uncertainty in requirements engineering; (C3) a test modeling methodology, named as UncerTum, for supporting the construction of test ready models with the explicit representation of uncertainties in CPSs; (C4) an evolution framework, named as UncerTolve, for interactively evolving test ready models specified with UncerTum based on real operational data; and (C5) a testing framework, named as UncerTest, for testing CPSs in the presence of uncertainties in their operating environments in a cost-effective manner using model-based and search-based testing techniques.
Based on our evaluations of the five contributions with the industrial CPS case studies, we observed that U-Model, as the foundation for this research, is sufficiently complete for characterizing and classifying uncertainties in CPSs. Then, the U-Model based modeling methodologies U-RUCM and UncerTum offer solutions to enable the identification and specification of uncertainties at two critical phases of a system development lifecycle: requirements engineering and testing. Furthermore, UncerTolve can successfully evolve model elements of the test ready models specified with UncerTum and calculate objective uncertainty measurements based on real operational data. Last, UncerTest managed to cost-effectively test CPSs in the presence of uncertainties and proactively identify unknown uncertainties by introducing the sources of the uncertainties into the test environments during test case execution.
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, MBT4CPS: Model-Based Testing For Cyber-Physical Systems , The Certus Centre (SFI) |
| Publication Type | PhD Thesis |
| Year of Publication | 2018 |
| Degree awarding institution | The University of Oslo |
| Degree | PhD |
| Number of Pages | 292 |
| Date Published | 26/06/2018 |
| Publisher | The University of Oslo |
| Place Published | Norway |
| Keywords | Cyber-Physical System, Model-based Testing, Uncertainty |
Book Chapter
Uncertainty-wise Testing of Cyber-Physical Systems
In Advances in Computers, 23-94. Vol. 107. Elsevier, 2017.Status: Published
Uncertainty-wise Testing of Cyber-Physical Systems
As compared with classical software/system testing, uncertainty-wise testing explicitly addresses known uncertainty about the behavior of a System Under Test (SUT), its operating environment, and interactions between the SUT and its operational environment, across all testing phases, including test design, test generation, test optimization, and test execution, with the aim to mainly achieve the following two goals. First, uncertainty-wise testing aims to ensure that the SUT deals with known uncertainty adequately. Second, uncertainty-wise testing should be also capable of learning new (previously unknown) uncertainties such that the SUT’s implementation can be improved to guard against newly learned uncertainties during its operation. The necessity to integrate uncertainty in testing is becoming imperative because of the emergence of new types of intelligent and communicating software-based systems such as Cyber-Physical Systems (CPSs). Intrinsically, such systems are exposed to uncertainty because of their interactions with highly indeterminate physical environments. In this chapter, we provide our understanding and experience of uncertainty-wise testing from the aspects of uncertainty-wise model-based testing, uncertainty-wise modeling and evolution of test ready models, and uncertainty-wise multi-objective test optimization, in the context of testing CPSs under uncertainty. Furthermore, we present our vision about this new testing paradigm and its plausible future research directions.
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, The Certus Centre (SFI) |
| Publication Type | Book Chapter |
| Year of Publication | 2017 |
| Book Title | Advances in Computers |
| Volume | 107 |
| Chapter | 2 |
| Pagination | 23-94 |
| Publisher | Elsevier |
Journal Article
Uncertainty-Wise Cyber-Physical System Test Modeling
Software & Systems Modeling (2017).Status: Published
Uncertainty-Wise Cyber-Physical System Test Modeling
It is important that a Cyber-Physical System (CPS) deals with uncertainty in its behavior caused by its unpredictable operating environment, to ensure its reliable operation. One method to ensure that the CPS will handle such uncertainty during its operation is by testing the CPS with Model-based Testing (MBT) techniques. However, existing MBT techniques do not explicitly capture uncertainty in test ready models i.e., capturing the uncertain expected behavior of a CPS in the presence of environment uncertainty. To fill this gap, we present an Uncertainty-Wise test-modeling framework, named as Uncertum, to create test ready models to support MBT of CPSs facing uncertainty. Uncertum relies on the definition of a UML profile (the UML Uncertainty Profile (UUP)) and a set of UML model libraries extending the UML profile for Modeling and Analysis of Real-Time and Embedded Systems (MARTE). Uncertum also benefits from the UML Testing Profile (UTP) V.2 to support standard-based MBT. Uncertum was evaluated with two industrial CPS case studies, one real-world case study, and one open source CPS case study from the following four perspectives: 1) Completeness and Coverage of the profiles and model libraries in terms of concepts defined in their underlying uncertainty conceptual model for CPSs (i.e., U-Model and MARTE, 2) Effort required to model uncertainty with Uncertum, and 3) Correctness of the developed test ready models, which was assessed via model execution. Based on the evaluation, we can conclude that we were successful in modeling all the uncertainties identified in the four case studies, which gives us an indication that Uncertum is sufficiently complete. In terms of modeling effort, we concluded that on average Uncertum requires18.5% more time to apply stereotypes from UUP on test ready models.
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2017 |
| Journal | Software & Systems Modeling |
| Publisher | Springer |
| ISSN | 1619-1374 |
| Keywords | Cyber-Physical System; UML, Model-based Testing, Uncertainty |
| DOI | 10.1007/s10270-017-0609-6 |
Uncertainty-Wise Evolution of Test Ready Models
Information and Software Technology (IST) 87 (2017): 140-159.Status: Published
Uncertainty-Wise Evolution of Test Ready Models
Context: Cyber-Physical Systems (CPSs), when deployed for operation, are inherently prone to uncertainty. Considering their applications in critical domains (e.g., healthcare), it is important that such CPSs are tested sufficiently, with the explicit consideration of uncertainty. Model-based testing (MBT) involves creating test ready models capturing the expected behavior of a CPS and its operating environment. These test ready models are then used for generating executable test cases. It is, therefore, necessary to develop methods that can continuously evolve, based on real operational data collected during the operation of CPSs, test ready models and uncertainty captured in them, all together termed as Belief Test Ready Models (BMs)
Objective: Our objective is to propose a model evolution framework that can interactively improve the quality of BMs, based on operational data. Such BMs are developed by one or more test modelers (belief agents) with their assumptions about the expected behavior of a CPS, its expected physical environment, and potential future deployments. Thus, these models explicitly contain subjective uncertainty of the test modelers.
Method: We propose a framework (named as UncerTolve) for interactively evolving BMs (specified with extended UML notations) of CPSs with subjective uncertainty developed by test modelers. The key inputs of UncerTolve include initial BMs of CPSs with known subjective uncertainty and real data collected from the operation of CPSs. UncerTolve has three key features: 1) Validating the syntactic correctness and conformance of BMs against real operational data via model execution, 2) Evolving objective uncertainty measurements of BMs via model execution, and 3) Evolving state invariants (modeling test oracles) and guards of transitions (modeling constraints for test data generation) of BMs with a machine learning technique.
Results: As a proof-of-concept, we evaluated UncerTolve with one industrial CPS case study, i.e., GeoSports from the healthcare domain. Using UncerTolve, we managed to evolve 51% of belief elements, 18% of states, and 21% of transitions as compared to the initial BM developed in an industrial setting.
Conclusion: UncerTolve can successfully evolve model elements of the initial BM, in addition to objective uncertainty measurements using real operational data. The evolved model can be used to generate additional test cases covering evolved model elements and objective uncertainty. These additional test cases can be used to test the current and future deployments of a CPS to ensure that it will handle uncertainty gracefully during its operations.
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, The Certus Centre (SFI) |
| Publication Type | Journal Article |
| Year of Publication | 2017 |
| Journal | Information and Software Technology (IST) |
| Volume | 87 |
| Pagination | 140-159 |
| Publisher | Elsevier |
| ISSN | 0950-5849 |
| Keywords | Belief Model, Belief Test Ready Model, Model Evolution, Model-based Testing, Uncertainty |
| DOI | 10.1016/j.infsof.2017.03.003 |
Search and similarity based selection of use case scenarios: An empirical study
Empirical Software Engineering (2017): 1-78.Status: Published
Search and similarity based selection of use case scenarios: An empirical study
Use case modeling is a well-known requirements specification method and has been widely applied in practice. Use case scenarios of use case models are input elements for requirements inspection and analysis, requirements-based testing, and other downstream activities. It is, however, a practical challenge to inspect all use case scenarios that can be obtained from any non-trivial use case model, as such an inspection activity is often performed manually by domain experts. Therefore, it is needed to propose an automated solution for selecting a subset of use case scenarios with the ultimate aim of enabling cost-effective requirements (use case) inspection, analysis, and other relevant activities. Our solution is built on a natural language based, restricted use case modeling methodology (named as RUCM), in the sense that requirements specifications are specified as RUCM use case models. Use case scenarios can be automatically derived from RUCM use case models with the already established Zen-RUCM framework. In this paper, we propose a search-based and similarity-based approach called S3RCUM, through an empirical study, to select most diverse use case scenarios to enable cost-effective use case inspections. The empirical study was designed to evaluate the performance of three search algorithms together with eight similarity functions, through one real-world case study and six case studies from literature. Results show that (1+1) Evolutionary Algorithm together with Needleman-Wunsch similarity function significantly outperformed the other 31 combinations of the search algorithms and similarity functions. The combination managed to select 50% of all the generated RUCM use case scenarios for all the case studies to detect all the seeded defects.
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines , MBT4CPS: Model-Based Testing For Cyber-Physical Systems , U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, MBE-CR: An Innovative Approach for Longstanding Development and Maintenance of the Automated Cancer Registry System |
| Publication Type | Journal Article |
| Year of Publication | 2017 |
| Journal | Empirical Software Engineering |
| Pagination | 1-78 |
| Date Published | 04/2017 |
| Publisher | Springer |
Miscellaneous
Uncertainty Testing Framework V.3
None, 2017.Status: Published
Uncertainty Testing Framework V.3
This is a public deliverable for U-Test project.
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies |
| Publication Type | Miscellaneous |
| Year of Publication | 2017 |
| Publisher | None |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty Testing Framework V.2
None, 2017.Status: Published
Uncertainty Testing Framework V.2
This is a public deliverable for the U-Test project.
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies |
| Publication Type | Miscellaneous |
| Year of Publication | 2017 |
| Publisher | None |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty Modeling Framework Version 2
None, 2017.Status: Published
Uncertainty Modeling Framework Version 2
This a U-Test public deliverable.
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies |
| Publication Type | Miscellaneous |
| Year of Publication | 2017 |
| Publisher | None |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
MPM4CPS: Multi-Paradigm Modelling for Cyber-Physical Systems
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
To date there is neither unifying theory, systematic design methods, nor techniques and tools for Cyber-Physical Systems (CPS). Individual engineering disciplines offer only partial solutions. Multi-paradigm Modelling (MPM) proposes to model every part and aspect of a system explicitly, at the most appropriate level of abstraction, using the most appropriate modelling formalism. The MPM4CPS action aims to promote the sharing of foundations, techniques, and tools to academia and industry. By bringing together and disseminating knowledge and experiments on CPS and MPM solutions across disciplines this goal can be achieved.
Funding source:
EU COST Action
Management Committee Members:
Tao Yue, Shaukat Ali
Publications for MPM4CPS: Multi-Paradigm Modelling for Cyber-Physical Systems
Talks, contributed
U-TCsGM: Generating and Minimizing Uncertainty-Based Test Cases for Cyber-Physical Systems (Tool Demo)
In MPM4CPS WG meetings in Malaga, Spain, 24-25 November 2016, 2016.Status: Published
U-TCsGM: Generating and Minimizing Uncertainty-Based Test Cases for Cyber-Physical Systems (Tool Demo)
In this tool demo, we will present the implementation of our recent research work on generating and minimizing executable test cases from the test models of a Cyber-Physical System tagged with subjective uncertainty. The algorithms are founded on uncertainty theory and NSGA-II—the most commonly used multi-objective search algorithm. We will demonstrate the complete process starting from creating a test model with uncertainty, generating test cases, minimizing test cases, and finally executing the minimized test cases using a real CPS case study.
| Afilliation | Software Engineering, Software Engineering |
| Project(s) | MPM4CPS: Multi-Paradigm Modelling for Cyber-Physical Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2016 |
| Location of Talk | MPM4CPS WG meetings in Malaga, Spain, 24-25 November 2016 |
| Keywords | Model-based Testing, Search-Based Testing, Uncertainty |
Model-Driven Testing of Cyber-Physical Systems with the Explicit Consideration of Uncertainty
In MPM4CPS WG meetings in Malaga, Spain, 24-25 November 2016, 2016.Status: Published
Model-Driven Testing of Cyber-Physical Systems with the Explicit Consideration of Uncertainty
It is a well-recognized fact that Cyber-Physical Systems (CPSs) face both known and unknown uncertainty during their operation. This demands the development of testing techniques that must take into account known uncertainty both in the CPS and its environment with the final goal of discovering unknown uncertainty against which the CPS can be tested. Eventually, the implementation of the CPS can be improved to shield it against the newly discovered uncertainty. In this presentation, we will present some of the results that we have achieved in an EU Horizon2020 project (U-Test) in this regard. We, first, demonstrate the modeling of test ready models of CPSs together with subjective uncertainty using the Uncertainty Modeling Framework. Second, we present two uncertainty-based test case generation and four test case minimization techniques relying on the test ready models founded on Uncertainty theory and multi-objective search. Third, we present the evaluation of the test case generation and minimization techniques that was conducted to select the best strategy to be used in the practice to test a real CPS. Fourth, we present the results of testing a real CPS with the selected best strategy.
| Afilliation | Software Engineering |
| Project(s) | MPM4CPS: Multi-Paradigm Modelling for Cyber-Physical Systems, The Certus Centre (SFI), U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies |
| Publication Type | Talks, contributed |
| Year of Publication | 2016 |
| Location of Talk | MPM4CPS WG meetings in Malaga, Spain, 24-25 November 2016 |
| Keywords | Cyber-Physical System, Model-based Testing, Search-Based Testing, Uncertainty |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Integrating Uncertainty Modelling with Use Case Modelling to Discover Unknowns
In MPM4CPS WG meetings in Malaga, Spain, 24-25 November 2016, 2016.Status: Published
Integrating Uncertainty Modelling with Use Case Modelling to Discover Unknowns
Use case modeling is a commonly used means for specifying requirements in practice. In the past, we have developed a use case modeling solution (with tool support), named as Restricted Use Case Modeling (RUCM), for the purpose of reducing inherent ambiguities of textual requirements and enabling automated analyses and generations. However, such use case models still contain uncertainties due to various reasons such as insufficient domain expertise and disagreement among stakeholders. We therefore, integrate RUCM with Uncertainty Modeling to provide requirements engineers an integrated platform (named as U-RUCM) for explicitly specifying uncertainties as part of use case models, such that both ambiguities and uncertainties in use case models can be reduced. U-RUCM was devised in the context of the EU Horizon 2020 U-Test project (http://www.u-test.eu/) and has been evaluated with two industrial case studies of two industrial partners of the U-Test consortium.
| Afilliation | Software Engineering, Software Engineering |
| Project(s) | MPM4CPS: Multi-Paradigm Modelling for Cyber-Physical Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2016 |
| Location of Talk | MPM4CPS WG meetings in Malaga, Spain, 24-25 November 2016 |
| Keywords | Modeling, RUCM, Uncertainty |
Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
The goal of the Zen-Configurator project is to increase the efficiency and effectiveness, and thereby reduce the cost, of configuring large-scale Cyber Physical System (CPS) product lines. To achieve this goal, we maximally automate error-prone and costly manual configuration activities and optimally assist the interactive configuration process. On one hand, the project relies on advanced technologies of constraint solving/evaluation, optimization using search algorithms, and propose state-of-art algorithms to enable automated configuration activities. On the other hand, the project grounds itself to address real challenges faced by industry and propose a practical and applicable solution and apply it in at least one application domain.
Funding source:
Research Council of Norway
All partners:
Simula Research Laboratory
Project leaders:
Tao Yue (PI), Shaukat Ali (Co-PI)
Publications for Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines
Journal Article
Recommending Faulty Configurations for Interacting Systems Under Test Using Multi-Objective Search
ACM Transactions on Software Engineering and Methodology 30, no. 4 (2021): 1-36.Status: Published
Recommending Faulty Configurations for Interacting Systems Under Test Using Multi-Objective Search
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Volume | 30 |
| Issue | 4 |
| Pagination | 1-36 |
| Publisher | ACM |
| DOI | 10.1145/3464939 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
PhD Thesis
Improving Post-Deployment Configuration of Cyber- Physical Systems Using Machine Learning and Multi- Objective Search
In University of Oslo. Vol. PhD. Oslo, Norway: University of Oslo, 2021.Status: Published
Improving Post-Deployment Configuration of Cyber- Physical Systems Using Machine Learning and Multi- Objective Search
Today, Cyber-Physical Systems (CPSs) are increasingly becoming an essential part of our daily lives and can be found in various domains such as energy, communication, and logistics. To accommodate different needs of users and provide customizations, CPS producers often adopt Product Line Engineering (PLE) methodologies. Consequently, CPSs are developed by integrating multiple products within/across product lines (PLs) that communicate with each other through information networks. Several PLE methodologies exist in the literature, however, their suitability for CPS PLs needs to be evaluated because of unique characteristics of CPS PLs (e.g., variabilities corresponding to multiple domains (e.g., electronics, mechanics), complex configuration processes). Hence, we need to identify key requirements of CPS PLE and evaluate existing PLE methodologies to assess their capabilities of supporting CPS PLE. Furthermore, most of the existing studies address challenges related to the pre-deployment configuration (i.e., making configuration decisions at design time) of individual products. There is a need for studies focusing on the post-deployment configuration (i.e., making configuration decisions at runtime) of interacting products.
In this thesis, first, we conducted a systematic domain analysis and proposed a conceptual framework for CPS PLs, based on which we evaluated existing PLE methodologies. Then, we focused on the post-deployment configuration of CPSs and made another two contributions: we proposed 1) an approach to capture patterns of configurations in the form of configuration rules and, and 2) another approach for recommending configurations to improve the post-deployment configuration experience from the perspective of testers and end-users.
To conduct the domain analysis, we analyzed three real-world CPS case studies. Based on the knowledge collected from the domain analysis and a thorough literature review on PLE, we proposed a conceptual framework, in which we 1) clarify the context of CPS PLE by formalizing CPSs, PLE, and configuration process; 2) present classifications of Variation Point (VP), constraint, and view types in addition to other modeling requirements to support the domain engineering of CPS PLs; and 3) formalize various types of automation that can be enabled to support the application engineering of CPS PLs. The completeness of the framework was evaluated using three real-world case studies containing 2161 VPs, 3943 constraints, and 40 views, 11 configuration tools, and an extensive literature review. Furthermore, we also evaluated four representative variability modeling techniques (VMTs): Feature Model (FM), Cardinality- Based Feature Model (CBFM), Common Variability Language (CVL), and SimPL. With the selected VMTs, we modeled a case study to assess if they can capture variabilities of CPS PLs. Results show that using SimPL, CVL, CBFM, and FM, we can capture only 81%, 75%, 50%, and 15% of the total variabilities, respectively.
To capture the configuration patterns in the form of configuration rules, we proposed the Search-Based Rule Mining (SBRM+) approach. SBRM+ combines multi-objective search with machine learning to mine configuration rules in an incremental and iterative way. We evaluated
the performance of SBRM+ using multiple real-world and open-source case studies from the communication domain and compared its performance with Random Search Based Rule Mining (RBRM+). Results show that SBRM+ performed significantly better than RBRM+ in terms of fitness values, six quality indicators, and 17 Machine Learning Quality Measurements MLQMs. As compared to RBRM+, SBRM+ improved the quality of rules up to 28% in terms of MLQMs.
To improve the post-deployment configuration experience, we proposed the Search-Based Configuration Recommendation (SBCR) approach, which recommends faulty configurations for CPSs with interacting products under test, based on mined rules. These configurations can be used to test CPSs and create guidelines for end-users to improve the post-deployment configuration experience. We evaluated SBCR using the same case studies, for which we mined the rules using SBRM+. Results show that SBCR significantly outperformed Random Search- Based Configuration Recommendation (RBCR) in terms of six quality indicators and the percentage of faulty configurations. Overall, SBCR made up to 22% more accurate recommendations than RBCR.
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | PhD Thesis |
| Year of Publication | 2021 |
| Degree awarding institution | University of Oslo |
| Degree | PhD |
| Number of Pages | 254 |
| Date Published | 04/2021 |
| Publisher | University of Oslo |
| Place Published | Oslo, Norway |
| Thesis Type | Collection of papers |
Journal Article
A framework for automated multi‑stage and multi‑step product confguration of cyber‑physical systems
Software and Systems Modeling (SoSym) 19, no. 4 (2020): 1-55.Status: Published
A framework for automated multi‑stage and multi‑step product confguration of cyber‑physical systems
Product line engineering (PLE) has been employed to large-scale cyber-physical systems (CPSs) to provide customization based on users’ needs. A PLE methodology can be characterized by its support for capturing and managing the abstractions as commonalities and variabilities and the automation of the confguration process for efective selection and customization of reusable artifacts. The automation of a confguration process heavily relies on the captured abstractions and formally specifed constraints using a well-defned modeling methodology. Based on the results of our previous work and a thorough literature review, in this paper, we propose a conceptual framework to support multi-stage and multi-step automated product confguration of CPSs, including a comprehensive classifcation of constraints and a list of automated functionalities of a CPS confguration solution. Such a framework can serve as a guide for researchers and practitioners to evaluate an existing CPS PLE solution or devise a novel CPS PLE solution. To validate the framework, we conducted three real-world case studies. Results show that the framework fulflls all the requirements of the case studies in terms of capturing and managing variabilities and constraints. Results of the literature review indicate that the framework covers all the functionalities concerned by the literature, suggesting that the framework is complete for enabling the maximum automation of confguration in CPS PLE.
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | Journal Article |
| Year of Publication | 2020 |
| Journal | Software and Systems Modeling (SoSym) |
| Volume | 19 |
| Issue | 4 |
| Pagination | 1-55 |
| Date Published | 06/2020 |
| Publisher | Springer |
| Keywords | Automated configuration, Constraint classification, Cyber-Physical Systems, Multi-stage and multi-step configuration process, Product Line Engineering, Real-world case studies, Variability Modeling |
| DOI | 10.1007/s10270-020-00803-8 |
Journal Article
Using multi-objective search and machine learning to infer rules constraining product configurations
Automated Software Engineering (2019): 1-62.Status: Published
Using multi-objective search and machine learning to infer rules constraining product configurations
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines , Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, MBT4CPS: Model-Based Testing For Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | Automated Software Engineering |
| Pagination | 1-62 |
| Publisher | Springer |
| DOI | 10.1007/s10515-019-00266-2 |
Uncertainty-wise Test Case Generation and Minimization for CyberPhysical Systems
Journal of Systems and Software 153 (2019).Status: Published
Uncertainty-wise Test Case Generation and Minimization for CyberPhysical Systems
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | Journal of Systems and Software |
| Volume | 153 |
| Date Published | 07/2019 |
| Publisher | Elsevier |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Proceedings, refereed
Stability Analysis for Safety of Automotive Multi-Product Lines: A Search-Based Approach
In The Genetic and Evolutionary Computation Conference (GECCO). ACM, 2019.Status: Published
Stability Analysis for Safety of Automotive Multi-Product Lines: A Search-Based Approach
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2019 |
| Conference Name | The Genetic and Evolutionary Computation Conference (GECCO) |
| Pagination | 1241-1249 |
| Publisher | ACM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Journal Article
Empirical Research in Software Engineering - a Literature Survey
Journal of Computer Science and Technology 33, no. 5 (2018): 876-899.Status: Published
Empirical Research in Software Engineering - a Literature Survey
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | Journal Article |
| Year of Publication | 2018 |
| Journal | Journal of Computer Science and Technology |
| Volume | 33 |
| Issue | 5 |
| Pagination | 876-899 |
| Date Published | 09/2018 |
| Publisher | Springer |
| DOI | 10.1007/s11390-018-1864-x |
Proceedings, refereed
Model- Based Personalized Visualization System for Monitoring Evolving Industrial Cyber-Physical System
In The 25th Asia-Pacific Software Engineering Conference (APSEC 2018) . IEEE, 2018.Status: Published
Model- Based Personalized Visualization System for Monitoring Evolving Industrial Cyber-Physical System
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines , MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Department of Engineering Complex Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2018 |
| Conference Name | The 25th Asia-Pacific Software Engineering Conference (APSEC 2018) |
| Publisher | IEEE |
Tool Support for Restricted Use Case Specification: Findings from a Controlled Experiment
In The 25th Asia-Pacific Software Engineering Conference (APSEC 2018) . IEEE, 2018.Status: Published
Tool Support for Restricted Use Case Specification: Findings from a Controlled Experiment
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines , Department of Engineering Complex Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2018 |
| Conference Name | The 25th Asia-Pacific Software Engineering Conference (APSEC 2018) |
| Publisher | IEEE |
Automatic Support of the Generation and Maintenance of Assurance Cases
In Symposium on Dependable Software Engineering: Theories, Tools and Applications. Cham: Springer International Publishing, 2018.Status: Published
Automatic Support of the Generation and Maintenance of Assurance Cases
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2018 |
| Conference Name | Symposium on Dependable Software Engineering: Theories, Tools and Applications |
| Pagination | 11-28 |
| Date Published | 08/2018 |
| Publisher | Springer International Publishing |
| Place Published | Cham |
| DOI | 10.1007/978-3-319-99933-3_2 |
MBT4CPS: Model-Based Testing For Cyber-Physical Systems
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
The complexity of Cyber Physical Systems presents unprecedented challenges for testing both nominal functionality and its associated extra-functional properties in unexpected situations. Thus, we need basic research to handle difficulties imposed by testing. Given the breadth and depth of the topic, as a starting point, we emphasize exclusively on dealing with the following two types of uncertain and risky situations. First, we focus on testing security features together in unpredictable and risky situations. Second, we concentrate on testing self-healing features, i.e., the ability of a CPS to recover from faults itself in risk and unpredictable situations. The successful completion of the project will produce distinct testing methods to test systems. Once such well-tested systems start operating in real life, these will be secure and safe.
Publications for the MBT4CPS project can be found in Simula's publication database.
Funding source:
Research Council of Norway
All partners:
Simula Research Laboratory
Project leaders:
Shaukat Ali (PI), Tao Yue (Co-PI)
Publications for MBT4CPS: Model-Based Testing For Cyber-Physical Systems
Journal Article
Testing Self-Healing Cyber-Physical Systems under Uncertainty with Reinforcement Learning: An Empirical Study
Empirical Software Engineering 26, no. 3 (2021): 52.Status: Published
Testing Self-Healing Cyber-Physical Systems under Uncertainty with Reinforcement Learning: An Empirical Study
| Afilliation | Software Engineering |
| Project(s) | MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Department of Engineering Complex Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2021 |
| Journal | Empirical Software Engineering |
| Volume | 26 |
| Issue | 3 |
| Pagination | 52 |
| Publisher | Springer |
| DOI | 10.1007/s10664-021-09941-z |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Journal Article
Quality Indicators in Search-based Software Engineering
ACM Transactions on Software Engineering and Methodology 29, no. 2 (2020): 1-29.Status: Published
Quality Indicators in Search-based Software Engineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, MBT4CPS: Model-Based Testing For Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2020 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Volume | 29 |
| Issue | 2 |
| Pagination | 1 - 29 |
| Date Published | May-04-2020 |
| Publisher | ACM |
| Place Published | New York, NY, USA |
| ISSN | 1049-331X |
| URL | https://dl.acm.org/doi/10.1145/3375636 |
| DOI | 10.1145/3375636 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Journal Article
Using multi-objective search and machine learning to infer rules constraining product configurations
Automated Software Engineering (2019): 1-62.Status: Published
Using multi-objective search and machine learning to infer rules constraining product configurations
| Afilliation | Software Engineering |
| Project(s) | Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines , Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, MBT4CPS: Model-Based Testing For Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | Automated Software Engineering |
| Pagination | 1-62 |
| Publisher | Springer |
| DOI | 10.1007/s10515-019-00266-2 |
Uncertainty-wise Test Case Generation and Minimization for CyberPhysical Systems
Journal of Systems and Software 153 (2019).Status: Published
Uncertainty-wise Test Case Generation and Minimization for CyberPhysical Systems
| Afilliation | Software Engineering |
| Project(s) | U-Test: Testing Cyber-Physical Systems under Uncertainty: Systematic, Extensible, and Configurable Model-based and Search-based Testing Methodologies, MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Zen-Configurator: Interactive and Optimal Configuration of Cyber Physical System Product Lines |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | Journal of Systems and Software |
| Volume | 153 |
| Date Published | 07/2019 |
| Publisher | Elsevier |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Testing Self-Healing Cyber-Physical Systems under Uncertainty: A Fragility-Oriented Approach
Software Quality Journal 27, no. 2 (2019): 615-649.Status: Published
Testing Self-Healing Cyber-Physical Systems under Uncertainty: A Fragility-Oriented Approach
| Afilliation | Software Engineering |
| Project(s) | MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Department of Engineering Complex Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2019 |
| Journal | Software Quality Journal |
| Volume | 27 |
| Issue | 2 |
| Pagination | 615–649 |
| Date Published | 03/2019 |
| Publisher | Springer |
| URL | https://link.springer.com/article/10.1007/s11219-018-9437-3 |
Proceedings, refereed
Towards a Framework for the Analysis of Multi-Product Lines in the Automotive Domain
In Proceedings of the 13th International Workshop on Variability Modelling of Software-Intensive Systems. New York, NY, USA: ACM, 2019.Status: Published
Towards a Framework for the Analysis of Multi-Product Lines in the Automotive Domain
| Afilliation | Software Engineering |
| Project(s) | MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Department of Engineering Complex Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2019 |
| Conference Name | Proceedings of the 13th International Workshop on Variability Modelling of Software-Intensive Systems |
| Publisher | ACM |
| Place Published | New York, NY, USA |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Edited books
Editorial to the Theme Issue on Model-based Testing
Software & Systems Modeling: Springer, 2018.Status: Published
Editorial to the Theme Issue on Model-based Testing
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, MBT4CPS: Model-Based Testing For Cyber-Physical Systems |
| Publication Type | Edited books |
| Year of Publication | 2018 |
| Publisher | Springer |
| Place Published | Software & Systems Modeling |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
First International Workshop on Verification and Validation of Internet of Things
IEEE, 2018.Status: Published
First International Workshop on Verification and Validation of Internet of Things
| Afilliation | Software Engineering |
| Project(s) | MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Department of Engineering Complex Software Systems |
| Publication Type | Edited books |
| Year of Publication | 2018 |
| Publisher | IEEE |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Journal Article
Modeling Foundations for Executable Model-Based Testing of Self-Healing Cyber-Physical Systems
Software and Systems Modeling (2018): 1-31.Status: Published
Modeling Foundations for Executable Model-Based Testing of Self-Healing Cyber-Physical Systems
Self-healing Cyber-Physical Systems (SH-CPSs) detect and recover from faults by themselves at runtime. Testing such systems is challenging due to the complex implementation of self-healing behaviors and their interaction with the physical environment, both of which are uncertain. To this end, we propose an executable model-based approach to test self-healing behaviors under environmental uncertainties. The approach consists of a Modeling Framework of SH-CPSs (MoSH) and an accompanying Test Model Executor (TM-Executor). MoSH provides a set of modeling constructs and a methodology to specify executable test models, which capture expected system behaviors and environmental uncertainties. TM-Executor executes the test models together with the systems under test, to dynamically test their self-healing behaviors under uncertainties. We demonstrated the successful application of MoSH to specify 11 self-healing behaviors and 17 uncertainties for three SH-CPSs. The time spent by TM-Executor to perform testing activities was in the order of milliseconds, though the time spent was strongly correlated with the complexity of test models.
| Afilliation | Software Engineering |
| Project(s) | MBT4CPS: Model-Based Testing For Cyber-Physical Systems , Department of Engineering Complex Software Systems |
| Publication Type | Journal Article |
| Year of Publication | 2018 |
| Journal | Software and Systems Modeling |
| Pagination | 1-31 |
| Date Published | 11/2018 |
| Publisher | Springer |
| Place Published | Berlin Heidelberg |
| ISSN | 1619-1374 |
| DOI | 10.1007/s10270-018-00703-y |
Employing Multi-Objective Search to Enhance Reactive Test Case Generation and Prioritization for Testing Industrial Cyber Physical Systems
IEEE Transactions on Industrial Informatics (TII) 14, no. 3 (2018): 1055-1066.Status: Published
Employing Multi-Objective Search to Enhance Reactive Test Case Generation and Prioritization for Testing Industrial Cyber Physical Systems
The test case generation and prioritization of industrial Cyber-Physical Systems (CPSs) face critical challenges and simulation-based testing is one of the most commonly used techniques for testing these complex systems. However, simulation models of industrial CPSs are usually very complex and executing the simulations becomes computationally expensive, which often make it infeasible to execute all the test cases. To address these challenges, this paper proposes a multi-objective test generation and prioritization approach for testing industrial CPSs by defining a fitness function with four objectives and designing different crossover and mutation operators. We empirically evaluated our fitness function and designed operators along with five multi-objective search algorithms (e.g., Non-dominated Sorting Genetic Algorithm (NSGA-II)) using four case studies. The evaluation results demonstrated that NSGA-II achieved significantly better performance than the other algorithms and managed to improve Random Search for on average 43.80% for each objective and 49.25% for the quality indicator Hypervolume (HV).
| Afilliation | Software Engineering |
| Project(s) | The Certus Centre (SFI), MBT4CPS: Model-Based Testing For Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2018 |
| Journal | IEEE Transactions on Industrial Informatics (TII) |
| Volume | 14 |
| Issue | 3 |
| Pagination | 1055-1066 |
| Publisher | IEEE |
| URL | http://ieeexplore.ieee.org/abstract/document/8241845/ |
| DOI | 10.1109/TII.2017.2788019 |
CERCIRAS: Connecting Education and Research Communities for an Innovative Resource Aware Society
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Parallel computing platforms have revolutionised the hardware landscape by providing high-performance, low-energy, and specialised (viz. heterogeneous) processing capabilities to a variety of application domains, including mobile, embedded, data-centre and high-performance computing. However, to leverage their potential, system designers must strike a difficult balance in the apportionment of resources to the application components, striving to avoid under- or over-provisions against worst-case utilisation profiles. The entanglement of hardware components in the emerging platforms and the complex behaviour of parallel applications raise conflicting resource requirements, more so in smart, (self-)adaptive and autonomous systems. This scenario presents the hard challenge of understanding and controlling, statically and dynamically, the trade-offs in the usage of system resources (time, space, energy, and data), also from the perspective of the development and maintenance efforts.
Making resource-usage trade-offs at specification, design, implementation, and run time requires profound awareness of the local and global impact caused by parallel threads of applications on individual resources. Such awareness is crucial for academic researchers and industrial practitioners across all European and COST member countries and, therefore, a strategic priority. Reaching this goal requires acting at two levels: (1) networking otherwise fragmented research efforts towards more holistic views of the problem and the solution; (2) leveraging appropriate educational and technology assets to improve the understanding and management of resources by the academia and industry of underperforming economies, in order to promote cooperation inside Europe and achieve economical and societal benefits.
Funding Source
- COST Action, EU
Publications for CERCIRAS: Connecting Education and Research Communities for an Innovative Resource Aware Society
Talks, contributed
Building Complex Software Systems in Classical and Quantum Computing Domains
In Connecting Education and Research Communities for an Innovative Resource Aware Society, Meeting Denmark, 2022.Status: Published
Building Complex Software Systems in Classical and Quantum Computing Domains
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, CERCIRAS: Connecting Education and Research Communities for an Innovative Resource Aware Society |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Connecting Education and Research Communities for an Innovative Resource Aware Society, Meeting Denmark |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Digital Twin-Enabled Operation Time Analyses
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
This project focuses on developing novel methods to build digital twins of cyber-physical systems to support advanced analyses (e.g., anomaly detection) at the operation time. To this end, we do research in learning digital twins automatically from historical and live data streams with advanced machine learning techniques. Moreover, to develop analyses, we employ various machine learning techniques such as Generative adversarial networks, transfer learning, and curriculum learning.
Funding source
- Internal strategic project. Funding is approximately 1 Million NOK per year.
Publications for Digital Twin-Enabled Operation Time Analyses
Journal Article
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
ACM Transactions on Software Engineering and Methodology (2023).Status: Accepted
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
Anomaly detection is critical to ensure the security of cyber-physical systems (CPS). However, due to the increasing complexity of attacks and CPS themselves, anomaly detection in CPS is becoming more and more challenging. In our previous work, we proposed a digital twin-based anomaly detection method, called ATTAIN, which takes advantage of both historical and real-time data of CPS. However, such data vary significantly in terms of difficulty. Therefore, similar to human learning processes, deep learning models (e.g., ATTAIN) can benefit from an easy-to-difficult curriculum. To this end, in this paper, we present a novel approach, named
digitaL twin-based Anomaly deTecTion wIth Curriculum lEarning (LATTICE), which extends ATTAIN by introducing curriculum learning to optimize its learning paradigm. LATTICE attributes each sample with a difficulty score, before being fed into a training scheduler. The training scheduler samples batches of training
data based on these difficulty scores such that learning from easy to difficult data can be performed. To evaluate LATTICE, we use five publicly available datasets collected from five real-world CPS testbeds. We compare LATTICE with ATTAIN and two other state-of-the-art anomaly detectors. Evaluation results show that LATTICE outperforms the three baselines and ATTAIN by 0.906%-2.367% in terms of the F1 score. LATTICE also, on average, reduces the training time of ATTAIN by 4.2% on the five datasets and is on par with the baselines in terms of detection delay time
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Publisher | ACM |
| Place Published | TOSEM |
Proceedings, refereed
Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators
Digital twins are increasingly developed to support the development, operation, and maintenance of cyber-physical systems such as industrial elevators. However, industrial elevators continuously evolve due to changes in physical installations, introducing new software features, updating existing ones, and making changes due to regulations (e.g., enforcing restricted elevator capacity due to COVID-19), etc. Thus, digital twin functionalities (often built on neural network-based models) need to evolve themselves constantly to be synchronized with the industrial elevators. Such an evolution is preferred to be automated, as manual evolution is time-consuming and error-prone. Moreover, collecting sufficient data to re-train neural network models of digital twins could be expensive or even infeasible. To this end, we propose unceRtaInty-aware tranSfer lEarning enriched Digital Twins LATTICE, a transfer learning based approach capable of transferring knowledge about the waiting time prediction capability of a digital twin of an industrial elevator across different scenarios. LATTICE also leverages uncertainty quantification to further improve its effectiveness. To evaluate LATTICE, we conducted experiments with 10 versions of an elevator dispatching software from Orona, Spain, which are deployed in a Software in the Loop (SiL) environment. Experiment results show that LATTICE, on average, improves the Mean Squared Error by 13.131% and the utilization of uncertainty quantification further improves it by 2.71%.
| Afilliation | Software Engineering |
| Project(s) | Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Publisher | ACM |
| DOI | 10.1145/3540250.3558957 |
Talks, contributed
Enhancing the Dependability of Cyber-physical Systems with AI-enabled Digital Twin
In Simula Research Laboratory, Norway, 2022.Status: Published
Enhancing the Dependability of Cyber-physical Systems with AI-enabled Digital Twin
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Simula Research Laboratory, Norway |
| Type of Talk | Presenting to guest lecturer Lionel C. Briand |
Uncertainty-aware transfer learning to evolve digital twins for industrial elevators
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Uncertainty-aware transfer learning to evolve digital twins for industrial elevators
Digital twins are increasingly developed to support the development, operation, and maintenance of cyber-physical systems such as industrial elevators. However, industrial elevators continuously evolve due to changes in physical installations, introducing new software features, updating existing ones, and making changes due to regulations (e.g., enforcing restricted elevator capacity due to COVID-19), etc. Thus, digital twin functionalities (often built on neural network-based models) need to evolve themselves constantly to be synchronized with the industrial elevators. Such an evolution is preferred to be automated, as manual evolution is time-consuming and error-prone. Moreover, collecting sufficient data to re-train neural network models of digital twins could be expensive or even infeasible. To this end, we propose unceRtaInty-aware tranSfer lEarning enriched Digital Twins LATTICE, a transfer learning based approach capable of transferring knowledge about the waiting time prediction capability of a digital twin of an industrial elevator across different scenarios. LATTICE also leverages uncertainty quantification to further improve its effectiveness. To evaluate LATTICE, we conducted experiments with 10 versions of an elevator dispatching software from Orona, Spain, which are deployed in a Software in the Loop (SiL) environment. Experiment results show that LATTICE, on average, improves the Mean Squared Error by 13.131% and the utilization of uncertainty quantification further improves it by 2.71%.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Publisher | ACM |
| Type of Talk | Conference paper |
| DOI | 10.1145/3540250.3558957 |
Enhancing the Dependability of Cyber-physical Systems with AI-enabled Digital Twin
In NORA Annual Conference 2022, 2022.Status: Published
Enhancing the Dependability of Cyber-physical Systems with AI-enabled Digital Twin
Cyber-physical Systems (CPS) have played an essential role in Industry 4.0 [1]. Since then, CPS are evolving to be increasingly heterogeneous, integrated, intelligent, operating in dynamic and everchanging environment. This exposes CPS to broader threats, which cannot be sufficiently tackled with traditional techniques. Our work focuses on exploring the potential of applying Digital Twin (DT) to improve dependability of CPS. The key idea is to build a DT as a virtual representation of a CPS, and develop DT functionalities with ML/AI algorithms to ensure the dependability of CPS operating in dynamic, uncertain and constantly-evolving environment. This research topic started in 2020 in the Engineering Complex Software Systems Department at Simula. As the first step, we chose Anomaly Detection as our main targeted research area, which is a sub-domain of the CPS dependability. Our current work consists of three phases: 1) designing and building a DTbased model, 2) enhancing it with Curriculum Learning (CL) [2], and 3) improving it with Transfer Learning [3]. • First, we have proposed a general DT-based model for anomaly detection. In this work, we built a Timed Automaton Machine (TAM) as the digital representation of the CPS, and implemented a Generative Adversarial Network (GAN) to detect anomalies. We evaluated this method (named ATTAIN) with three public datasets and achieved state-of-art results. • Second, we proposed LATTICE by extending ATTAIN by introducing CL to optimize its learning paradigm. CL is inspired by human learning process, which indicates that deep learning methods can benefit from a easy-to-difficult curriculum. We evaluated LATTICE with five public datasets and results show improvements over ATTAIN. • Currently, we are exploring to use transfer learning to further improve LATTICE. This is motivated because we found that most existing methods (including ours) are CPS-agnostic and become obsolete when new scenarios emerge. Therefore, we plan to improve predictive performance and reduce prediction uncertainty by transferring knowledge from these obsolete models to new models. We will evaluate this work on real elevator data from Orona—world leader in building industrial elevators.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | NORA Annual Conference 2022 |
Technical reports
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
Simula Research Laboratory, 2022.Status: Published
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses |
| Publication Type | Technical reports |
| Year of Publication | 2022 |
| Publisher | Simula Research Laboratory |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Proceedings, refereed
Anomaly Detection with Digital Twin in Cyber-Physical Systems
In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2021.Status: Published
Anomaly Detection with Digital Twin in Cyber-Physical Systems
Cyber-Physical Systems (CPSs) are susceptible to various anomalies during their operations. Thus, it is important to detect such anomalies. Detecting such anomalies is challenging since it is uncertain when and where anomalies can happen. To this end, we present a novel approach called Anomaly deTection with digiTAl twIN (ATTAIN), which continuously and automatically builds a digital twin with live data obtained from a CPS for anomaly detection. ATTAIN builds a Timed Automaton Machine (TAM) as the digital representation of the CPS, and implements a Generative Adversarial Network (GAN) to detect anomalies. GAN uses a GCN-LSTM-based module as a generator, which can capture temporal and spatial characteristics of the input data and learn to produce realistic unlabeled fake samples. TAM labels these fake samples, which are then fed into a discriminator along with real labeled samples. After training, the discriminator is capable of distinguishing anomalous data from normal data with a high F1 score. To evaluate our approach, we used three publicly available datasets collected from three CPS testbeds. Evaluation results show that ATTAIN improved the performance of two state-of-art anomaly detection methods by 2.413\%, 8.487\% and 5.438\% on average on the three datasets, respectively. Moreover, ATTAIN achieved on average 8.39\% increase in the anomaly detection capability with digital twins as compared with an approach of not using digital twins.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2021 |
| Conference Name | 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST) |
| Publisher | IEEE |
| DOI | 10.1109/ICST49551.2021.00031 |
Talks, contributed
Anomaly Detection with Digital Twin in Cyber-Physical Systems
In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 2021.Status: Published
Anomaly Detection with Digital Twin in Cyber-Physical Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses |
| Publication Type | Talks, contributed |
| Year of Publication | 2021 |
| Location of Talk | IEEE International Conference on Software Testing, Verification and Validation (ICST) |
| Publisher | IEEE |
| Type of Talk | Conference Paper |
Gemini Center on Quantum Computing
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Founded in 2020 by NTNU, University of Oslo, and SINTEF, our goal is to make Norway “quantum ready”. By 2021 also Simula is an associate partner. The background for this centre is the emergence of quantum computers, which many experts regard as one of the most important disruptive technologies in the near future. There is a huge investment in this type of technology from governments, research programs, companies and venture capitalists. The total spending on quantum initiatives worldwide in 2020 is estimated to be 22 billion USD. It is vital for Norway to start to build the necessary expertise in order to become “quantum ready”. This centre gathers a strong interdisciplinary, cross-institutional team of researchers from the University of Oslo, NTNU and SINTEF to promote quantum technology, establish contact with internationally leading expert groups, and work together with industrial partners. Highlighting potential use cases and joining forces to build the necessary expertise.
The centre gathers expertise in the emerging technology of quantum computing and focuses on algorithms/software and application areas (rather than hardware). Quantum computers work in a fundamentally different way than classical computers. To utilize them requires that one understands the underlying quantum-physical principles and can use these to rethink how algorithms are designed (quantum software). A multidisciplinary approach is indispensable.
Nordic-Estonian Quantum Computing e-Infrastructure Quest
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
NordiQuEst is a new collaborative effort between four Nordic countries (Norway, Sweden, Finland, and Denmark) and Estonia to build a dedicated Nordic-Estonian quantum computing (QC) ecosystem that integrates various quantum computers and emulators make them accessible to the Nordic-Estonian region to accelerate QC research, development, and education.
Partners
- Chalmers tekniska högskola AB, Sweden
- CSC - IT Center for Science Ltd. Finland
- DTU – Technical University of Denmark, Denmark
- ETAIS - Estonian Scientific Computing Infrastructure - University of Tartu, Estonia
- SINTEF, Norway
- SRL - Simula Research Laboratory AS, Norway
- VTT - VTT Technical Research Centre of Finland Ltd, Finland
Funding Source
- NeIC, NordForsk. Total funding of the Project: 19.2 Million NOK
Publications for Nordic-Estonian Quantum Computing e-Infrastructure Quest
Talks, contributed
Time for new Simula!
In Towards a Norwegian Quantum Computing Strategy, Simula Research Laboratory, Norway, 2022.Status: Published
Time for new Simula!
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Nordic-Estonian Quantum Computing e-Infrastructure Quest, Quantum Software Engineering Project |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Towards a Norwegian Quantum Computing Strategy, Simula Research Laboratory, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Talks, invited
Quantum Software Testing Tutorial
In NordIQuEst-ENCCS online HPC-QC workshop, 2022.Status: Published
Quantum Software Testing Tutorial
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project, Nordic-Estonian Quantum Computing e-Infrastructure Quest |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | NordIQuEst-ENCCS online HPC-QC workshop |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Cyber-Physical Systems of Systems (CPSoS) exhibit unpredictable behaviours due to their coevolution, employed machine learning techniques, collective behaviours, and operation in uncertain environments and unreliable communication networks. Consequently, testing these systems is highly challenging. The Co-tester project aims to address this challenge by developing novel collective-adaptive testing strategies. These strategies will be empowered with advanced artificial intelligence technologies (e.g., machine learning and evolutionary computation techniques), supported with novel strategies for discovering uncertain and unknown behaviours of CPSoS, its constituted Cyber-Physical Systems, operating environment, and networks for extensive testing of CPSoS.
Funding Source
-
Norwegian Research Council's FRIPRO scheme. Total funding 12 Million NOK.
Publications for Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty
Journal Article
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
ACM Transactions on Software Engineering and Methodology (2023).Status: Accepted
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
Anomaly detection is critical to ensure the security of cyber-physical systems (CPS). However, due to the increasing complexity of attacks and CPS themselves, anomaly detection in CPS is becoming more and more challenging. In our previous work, we proposed a digital twin-based anomaly detection method, called ATTAIN, which takes advantage of both historical and real-time data of CPS. However, such data vary significantly in terms of difficulty. Therefore, similar to human learning processes, deep learning models (e.g., ATTAIN) can benefit from an easy-to-difficult curriculum. To this end, in this paper, we present a novel approach, named
digitaL twin-based Anomaly deTecTion wIth Curriculum lEarning (LATTICE), which extends ATTAIN by introducing curriculum learning to optimize its learning paradigm. LATTICE attributes each sample with a difficulty score, before being fed into a training scheduler. The training scheduler samples batches of training
data based on these difficulty scores such that learning from easy to difficult data can be performed. To evaluate LATTICE, we use five publicly available datasets collected from five real-world CPS testbeds. We compare LATTICE with ATTAIN and two other state-of-the-art anomaly detectors. Evaluation results show that LATTICE outperforms the three baselines and ATTAIN by 0.906%-2.367% in terms of the F1 score. LATTICE also, on average, reduces the training time of ATTAIN by 4.2% on the five datasets and is on par with the baselines in terms of detection delay time
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Publisher | ACM |
| Place Published | TOSEM |
Master's thesis
Digital Twin for UAV Anomaly Detection
In The University of Oslo. The University of Oslo, 2022.Status: Published
Digital Twin for UAV Anomaly Detection
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty |
| Publication Type | Master's thesis |
| Year of Publication | 2022 |
| Degree awarding institution | The University of Oslo |
| Publisher | The University of Oslo |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Proceedings, refereed
Enhancing the realism of autonomous driving simulation with real-time co-simulation
In 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22) . IEEE/ACM, 2022.Status: Published
Enhancing the realism of autonomous driving simulation with real-time co-simulation
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22) |
| Publisher | IEEE/ACM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators
Digital twins are increasingly developed to support the development, operation, and maintenance of cyber-physical systems such as industrial elevators. However, industrial elevators continuously evolve due to changes in physical installations, introducing new software features, updating existing ones, and making changes due to regulations (e.g., enforcing restricted elevator capacity due to COVID-19), etc. Thus, digital twin functionalities (often built on neural network-based models) need to evolve themselves constantly to be synchronized with the industrial elevators. Such an evolution is preferred to be automated, as manual evolution is time-consuming and error-prone. Moreover, collecting sufficient data to re-train neural network models of digital twins could be expensive or even infeasible. To this end, we propose unceRtaInty-aware tranSfer lEarning enriched Digital Twins LATTICE, a transfer learning based approach capable of transferring knowledge about the waiting time prediction capability of a digital twin of an industrial elevator across different scenarios. LATTICE also leverages uncertainty quantification to further improve its effectiveness. To evaluate LATTICE, we conducted experiments with 10 versions of an elevator dispatching software from Orona, Spain, which are deployed in a Software in the Loop (SiL) environment. Experiment results show that LATTICE, on average, improves the Mean Squared Error by 13.131% and the utilization of uncertainty quantification further improves it by 2.71%.
| Afilliation | Software Engineering |
| Project(s) | Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Publisher | ACM |
| DOI | 10.1145/3540250.3558957 |
Talks, contributed
Enhancing the realism of autonomous driving simulation with real-time co-simulation
In 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22). IEEE/ACM, 2022.Status: Published
Enhancing the realism of autonomous driving simulation with real-time co-simulation
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22) |
| Publisher | IEEE/ACM |
Testing Self-Healing Cyber-Physical Systems under Uncertainty with Reinforcement Learning: An Empirical Study
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022), 2022.Status: Published
Testing Self-Healing Cyber-Physical Systems under Uncertainty with Reinforcement Learning: An Empirical Study
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022) |
| Type of Talk | Conference paper talk |
Learning Digital Twin Models
In Model-Driven Engineering of Digital Twins Seminar at Dagstuhl, Germany. Schloss Dagstuhl, Germany, 2022.Status: Published
Learning Digital Twin Models
Given that operational cyber-physical systems (CPS) produce continuous data, a complementary approach to model-based engineering is to learn digital twins models with machine learning techniques and providing functionalities such as predictions and anomaly detection.
This talk will start with presenting an opinion on the next generation of digital twins (Quantum Digital Twins), where some aspects of digital twins will be implemented as quantum software and executed on quantum computers, e.g., for simulating the physical environment that can be realistically simulated with quantum-mechanical principles.
Followed by this opinion, the talk will present some recent works on learning digital twins from historical data and continuous updates of digital twins with continuous data from operational CPS. Various machine learning techniques were applied, such as generative adversarial networks, curriculum learning, and transfer learning to learn digital twins. The digital twins were built for use cases from the transportation domain and water distribution/treatment plants. These digital twins were focused on anomaly detection and waiting time predictions.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Model-Driven Engineering of Digital Twins Seminar at Dagstuhl, Germany |
| Publisher | Schloss Dagstuhl, Germany |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Talks, invited
Modeling robustness behavior using aspect-oriented modeling to support robustness testing of industrial systems
In ACM / IEEE 25th International Conference on Model Driven Engineering Languages and Systems (MODELS), 2022.Status: Published
Modeling robustness behavior using aspect-oriented modeling to support robustness testing of industrial systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | ACM / IEEE 25th International Conference on Model Driven Engineering Languages and Systems (MODELS) |
| Type of Talk | 10-year SOSYM Most Influential Paper Award |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
AI-enabled Digital Twins for Cyber-Physical Systems
In EDT Community: Engineering Digital Twins – Seminar Series, 2022.Status: Published
AI-enabled Digital Twins for Cyber-Physical Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | EDT Community: Engineering Digital Twins – Seminar Series |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Engineering Project
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum software engineering (QSE) enables the cost-effective and scalable development of quantum software-enabled applications. However, QSE is a relatively new area of research, yet very critical for building the promised revolutionary quantum computing applications. Towards this end, this project focuses on developing novel quantum software requirements engineering, modelling, development, testing, and debugging approaches.
Funding Source
- This is a strategic initiative at Simula supporting eight person-years to build solid foundations of the QSE area. The overall funding is approx. 10 Million NOK.
Publications for Quantum Software Engineering Project
Proceedings, refereed
Generating Failing Test Suites for Quantum Programs with Search (hot off the press track at GECCO 2022)
In GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 2022.Status: Published
Generating Failing Test Suites for Quantum Programs with Search (hot off the press track at GECCO 2022)
The inherent complexity of quantum programs, due to features such as superposition and entanglement, makes their testing particularly challenging. To tackle these challenges, we present a search-based approach, called Quantum Search-Based Testing (QuSBT), for automatically generating test suites of a given size that possibly expose failures of the quantum program under test. QuSBT encodes a test suite as a search individual, and tries to maximize the objective function that counts the number of failing tests in the test suite. Due to non-deterministic nature of quantum programs, the approach repeats the execution of each test multiple times, and uses suitable statistical tests to assess if a test passes or fails. QuSBT employs a genetic algorithm to perform the search. Experiments on 30 faulty quantum programs show that QuSBT is statistically better than random search, and is able to efficiently generate maximal failing test suites.
This is an extended abstract of the paper [1]: X. Wang, P. Arcaini, T. Yue, and S. Ali "Generating Failing Test Suites for Quantum Programs With Search", 13th International Symposium on Search-Based Software Engineering (SSBSE 2021).
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Quantum Software Engineering Project, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference |
| Pagination | 47-48 |
| Publisher | ACM |
| URL | https://dl.acm.org/doi/10.1145/3520304.3534067 |
| DOI | 10.1145/3520304.3534067 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search
In GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 2022.Status: Published
Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search
Mutation testing is often used for designing new tests, and involves changing a program in minor ways, which results in mutated versions of the program, i.e., mutants. An effective test suite should find faults (or kill mutants) with a minimum number of test cases, to save resources required for executing test cases. In this paper, in the context of mutation testing for quantum programs, we present a multi-objective and search-based approach (MutTG) to generate the minimum number of test cases killing as many mutants as possible. MutTG tries to estimate the likelihood that a mutant is equivalent, and uses this as a discount factor in the fitness definition to avoid keeping on trying to kill mutants that cannot be killed. We employed NSGA-II as the multi-objective search algorithm. Then, we compared MutTG with another version of the approach that does not use the discount factor in its fitness definition, and with random search (RS), over a set of open-source quantum programs and their mutants of varying complexity. Results show that the discount factor does indeed help in guiding the test generation, as the approach with the discount factor performs better than the one without it.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Quantum Software Engineering Project, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference |
| Pagination | 1345-1353 |
| Publisher | ACM |
| URL | https://dl.acm.org/doi/abs/10.1145/3512290.3528869 |
| DOI | 10.1145/3512290.3528869 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
QuSBT: Search-Based Testing of Quantum Programs
In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2022.Status: Published
QuSBT: Search-Based Testing of Quantum Programs
Generating a test suite for a quantum program such that it has the maximum number of failing tests is an optimization problem. For such optimization, search-based testing has shown promising results in the context of classical programs. To this end, we present a test generation tool for quantum programs based on a genetic algorithm, called QuSBT (Search-based Testing of Quantum Programs). QuSBT automates the testing of quantum programs, with the aim of finding a test suite having the maximum number of failing test cases. QuSBT utilizes IBM’s Qiskit as the simulation framework for quantum programs. We present the tool architecture in addition to the implemented methodology (i.e., the encoding of the search individual, the definition of the fitness function expressing the search problem, and the test assessment w.r.t. two types of failures). Finally, we report results of the experiments in which we tested a set of faulty quantum programs with QuSBT to assess its effectiveness.
Repository (code and experimental results): https://github.com/Simula-COMPLEX/qusbt-tool
Video: https://youtu.be/3apRCtluAn4
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) |
| Publisher | IEEE |
| URL | https://ieeexplore.ieee.org/document/9793826 |
| DOI | 10. 1145/3510454.3516839 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Investigating Quantum Cause-Effect Graphs
In 2022 IEEE/ACM 3rd International Workshop on Quantum Software Engineering (Q-SE). IEEE, 2022.Status: Published
Investigating Quantum Cause-Effect Graphs
Cause-effect graphs have shown promising results in identifying relations among causes and effects of classical software systems, followed by designing effective test cases from them. Towards this end, we investigate the use of cause-effect graphs for quantum programs. Classical cause-effect graphs apply classical logic (e.g., AND, OR) to express these relations, which might not be practical for describing similar relations in quantum programs due to quantum superposition and entanglement. Thus, we propose an extension of cause-effect graphs, where quantum logic inspired functions (e.g., Hadamard) and their generalizations are defined and applied. Moreover, we present a metamodel describing various forms of cause-effect graphs. Finally, we demonstrate a possible method for generating test cases from a quantum cause-effect graph applied to a Bell state quantum program. Lastly, the design and utility of the resulting testing method is discussed, along with future prospects for general quantum cause-effect graphs.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Quantum Software Engineering Project |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 2022 IEEE/ACM 3rd International Workshop on Quantum Software Engineering (Q-SE) |
| Pagination | 8-15 |
| Publisher | IEEE |
Talks, contributed
Time for new Simula!
In Towards a Norwegian Quantum Computing Strategy, Simula Research Laboratory, Norway, 2022.Status: Published
Time for new Simula!
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Nordic-Estonian Quantum Computing e-Infrastructure Quest, Quantum Software Engineering Project |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Towards a Norwegian Quantum Computing Strategy, Simula Research Laboratory, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Talks, invited
Quantum Software Engineering: What, Why, and Where are we now?
In Quantum Leap and Mathematics, Soria Moria, Oslo, Norway, 2022.Status: Published
Quantum Software Engineering: What, Why, and Where are we now?
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | Quantum Leap and Mathematics, Soria Moria, Oslo, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Dependable and Noise-Aware Quantum Software Engineering
In CREST Center, The University of Adelaide, Australia, 2022.Status: Published
Dependable and Noise-Aware Quantum Software Engineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | CREST Center, The University of Adelaide, Australia |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Testing Work in ComplexSE
In Simula Research Laboratory, Norway, 2022.Status: Published
Quantum Software Testing Work in ComplexSE
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | Simula Research Laboratory, Norway |
Quantum Software Testing Tutorial
In NordIQuEst-ENCCS online HPC-QC workshop, 2022.Status: Published
Quantum Software Testing Tutorial
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project, Nordic-Estonian Quantum Computing e-Infrastructure Quest |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | NordIQuEst-ENCCS online HPC-QC workshop |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Proceedings, refereed
Quito: a Coverage-Guided Test Generator for Quantum Programs
In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021.Status: Published
Quito: a Coverage-Guided Test Generator for Quantum Programs
Automation in quantum software testing is essential to support systematic and cost-effective testing. Towards this direction, we present a quantum software testing tool called Quito that can automatically generate test suites covering three coverage criteria defined on inputs and outputs of a quantum program coded in Qiskit, i.e., input coverage, output coverage, and input-output coverage. Quito also implements two types of test oracles based on program specifications, i.e., checking whether a quantum program produced a wrong output or checking a probabilistic test oracle with statistical test. We describe the architecture and methodology of the tool. We also validated the tool with one quantum program and one faulty version of it. Results indicate that Quito can generate test suites and perform test assessments that detect faults, and produce test results with a good time performance.
Quito’s code: https://github.com/Simula-COMPLEX/quito
Quito’s video: https://youtu.be/kuI9QaCo8A8Artifact Available: https://doi.org/10.5281/zenodo.5288665
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2021 |
| Conference Name | 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) |
| Publisher | IEEE |
| URL | https://ieeexplore.ieee.org/document/9678798 |
| DOI |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
The primary objective of the Co-evolver project is to explore and exploit the coevolution design of a self-adaptive Cyber-physical Systems (CPSs) to a given level of maturity before deployment and enable the self-evolution of its coevolution strategy during operation, by drawing on theories and technologies from model-based engineering, evolutionary computation, and machine learning. The key scientific outcomes are 1) a multi-paradigm modeling framework for developing executable coevolution design models, 2) novel (co-)evolutionary algorithms and advanced applied studies on uncertainty-related theories and machine learning techniques to enable the continuous exploration and exploitation of coevolution designs, and 3) a comprehensive platform for evolving coevolution design models. The secondary objective is to apply the outcomes to at least one self-CPS application domain, opening a new stream of research in the domain of uncertainty-aware coevolution designs of self-CPSs.
Funding source:
Research Council of Norway
Project leaders:
Simula Research Laboratory
Publications for Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems
Journal Article
Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their Collisions
IEEE Transactions on Software Engineering 49, no. 1 (2022): 384-402.Status: Published
Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their Collisions
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | IEEE Transactions on Software Engineering |
| Volume | 49 |
| Issue | 1 |
| Pagination | 384-402 |
| Publisher | IEEE |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
On the Preferences of Quality Indicators for Multi-Objective Search Algorithms in Search-Based Software Engineering
The Empirical Software Engineering Journal (EMSE) 27, no. 6 (2022): 144.Status: Published
On the Preferences of Quality Indicators for Multi-Objective Search Algorithms in Search-Based Software Engineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | The Empirical Software Engineering Journal (EMSE) |
| Volume | 27 |
| Issue | 6 |
| Pagination | 144 |
| Publisher | Springer |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty-Aware Prediction Validator in Deep Learning Models for Cyber-Physical System Data
ACM Transactions on Software Engineering and Methodology (TOSEM) 31 (2022): 1-31.Status: Published
Uncertainty-Aware Prediction Validator in Deep Learning Models for Cyber-Physical System Data
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | ACM Transactions on Software Engineering and Methodology (TOSEM) |
| Volume | 31 |
| Number | 4 |
| Pagination | 1-31 |
| Publisher | ACM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Master's thesis
Learning To Model a Driving Simulator
In The University of Oslo. The University of Oslo, 2022.Status: Published
Learning To Model a Driving Simulator
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Master's thesis |
| Year of Publication | 2022 |
| Degree awarding institution | The University of Oslo |
| Number | NBN:no-98158 |
| Publisher | The University of Oslo |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Proceedings, refereed
Enhancing the realism of autonomous driving simulation with real-time co-simulation
In 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22) . IEEE/ACM, 2022.Status: Published
Enhancing the realism of autonomous driving simulation with real-time co-simulation
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22) |
| Publisher | IEEE/ACM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Are Elevator Software Robust Against Uncertainties? Results and Experiences from an Industrial Case Study
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Are Elevator Software Robust Against Uncertainties? Results and Experiences from an Industrial Case Study
| Afilliation | Software Engineering |
| Project(s) | Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Pagination | 1331-1342 |
| Date Published | 11/2022 |
| Publisher | ACM |
Talks, contributed
Uncertainty in Deep Learning
In Simula Research Laboratory, Norway, 2022.Status: Published
Uncertainty in Deep Learning
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Simula Research Laboratory, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Generating and Optimizing of Test Scenarios for Autonomous Driving Systems
In Simula Research Laboratory, Norway, 2022.Status: Published
Generating and Optimizing of Test Scenarios for Autonomous Driving Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Simula Research Laboratory, Norway |
Enhancing the realism of autonomous driving simulation with real-time co-simulation
In 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22). IEEE/ACM, 2022.Status: Published
Enhancing the realism of autonomous driving simulation with real-time co-simulation
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22) |
| Publisher | IEEE/ACM |
Testing Self-Healing Cyber-Physical Systems under Uncertainty with Reinforcement Learning: An Empirical Study
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022), 2022.Status: Published
Testing Self-Healing Cyber-Physical Systems under Uncertainty with Reinforcement Learning: An Empirical Study
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022) |
| Type of Talk | Conference paper talk |
Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
It is well-known that Quantum Computing (QC) has the potential to solve complex problems in various domains and bring breakthroughs in science and technology. Nowadays, quantum applications span over algorithms addressing optimization problems such as radiotherapy optimization, machine learning techniques, e.g., for detecting objects from images, as well as modeling and simulations, e.g., for handling uncertainties when predicting the future. The development of QC is also driven by the urgent need to solve ever-complex and large-scale problems, which current (super)computers cannot solve. QC comes right on time to bring revolutionary computation power to handle such complexity. Testing such applications, however, is a big challenge due to their radically different characteristics from their classical counterparts. This includes superposition, entanglement, and probabilistic nature of qubits. The overall ambition is to develop fundamentally new methods for automated and systematic testing of quantum programs, based on a rigorous theoretical foundation with the ultimate goal of supporting future ubiquitous services and data related to QC applications to guarantee their dependability. Also, to allow for testing complex quantum programs with a minimal amount of QC resources, we will develop novel quantum optimization algorithms. The cost-effectiveness of our methods will be demonstrated by testing quantum programs written in quantum high-level programming languages (e.g., Q#).
Funding source:
IKTPLUSS
Research Council of Norway
Partners:
The University of Malaga, Spain
The University of Maryland, USA
Durham University, UK
Publications for Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs
Edited books
Quantum Computing - Introduction to the special theme
Vol. 128. ERCIM, 2022.Status: Published
Quantum Computing - Introduction to the special theme
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Edited books |
| Year of Publication | 2022 |
| Volume | 128 |
| Publisher | ERCIM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Journal Article
Quantum Software Testing: Challenges, Early Achievements, and Opportunities
the European Research Consortium for Informatics and Mathematics News (2022).Status: Published
Quantum Software Testing: Challenges, Early Achievements, and Opportunities
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | the European Research Consortium for Informatics and Mathematics News |
| Publisher | ERCIM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
When Software Engineering Meets Quantum Computing
Communications of ACM 65, no. 4 (2022): 84-88.Status: Published
When Software Engineering Meets Quantum Computing
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | Communications of ACM |
| Volume | 65 |
| Issue | 4 |
| Pagination | 84-88 |
| Publisher | ACM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Proceedings, refereed
Generating Failing Test Suites for Quantum Programs with Search (hot off the press track at GECCO 2022)
In GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 2022.Status: Published
Generating Failing Test Suites for Quantum Programs with Search (hot off the press track at GECCO 2022)
The inherent complexity of quantum programs, due to features such as superposition and entanglement, makes their testing particularly challenging. To tackle these challenges, we present a search-based approach, called Quantum Search-Based Testing (QuSBT), for automatically generating test suites of a given size that possibly expose failures of the quantum program under test. QuSBT encodes a test suite as a search individual, and tries to maximize the objective function that counts the number of failing tests in the test suite. Due to non-deterministic nature of quantum programs, the approach repeats the execution of each test multiple times, and uses suitable statistical tests to assess if a test passes or fails. QuSBT employs a genetic algorithm to perform the search. Experiments on 30 faulty quantum programs show that QuSBT is statistically better than random search, and is able to efficiently generate maximal failing test suites.
This is an extended abstract of the paper [1]: X. Wang, P. Arcaini, T. Yue, and S. Ali "Generating Failing Test Suites for Quantum Programs With Search", 13th International Symposium on Search-Based Software Engineering (SSBSE 2021).
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Quantum Software Engineering Project, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference |
| Pagination | 47-48 |
| Publisher | ACM |
| URL | https://dl.acm.org/doi/10.1145/3520304.3534067 |
| DOI | 10.1145/3520304.3534067 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search
In GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 2022.Status: Published
Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search
Mutation testing is often used for designing new tests, and involves changing a program in minor ways, which results in mutated versions of the program, i.e., mutants. An effective test suite should find faults (or kill mutants) with a minimum number of test cases, to save resources required for executing test cases. In this paper, in the context of mutation testing for quantum programs, we present a multi-objective and search-based approach (MutTG) to generate the minimum number of test cases killing as many mutants as possible. MutTG tries to estimate the likelihood that a mutant is equivalent, and uses this as a discount factor in the fitness definition to avoid keeping on trying to kill mutants that cannot be killed. We employed NSGA-II as the multi-objective search algorithm. Then, we compared MutTG with another version of the approach that does not use the discount factor in its fitness definition, and with random search (RS), over a set of open-source quantum programs and their mutants of varying complexity. Results show that the discount factor does indeed help in guiding the test generation, as the approach with the discount factor performs better than the one without it.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Quantum Software Engineering Project, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference |
| Pagination | 1345-1353 |
| Publisher | ACM |
| URL | https://dl.acm.org/doi/abs/10.1145/3512290.3528869 |
| DOI | 10.1145/3512290.3528869 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
QuSBT: Search-Based Testing of Quantum Programs
In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2022.Status: Published
QuSBT: Search-Based Testing of Quantum Programs
Generating a test suite for a quantum program such that it has the maximum number of failing tests is an optimization problem. For such optimization, search-based testing has shown promising results in the context of classical programs. To this end, we present a test generation tool for quantum programs based on a genetic algorithm, called QuSBT (Search-based Testing of Quantum Programs). QuSBT automates the testing of quantum programs, with the aim of finding a test suite having the maximum number of failing test cases. QuSBT utilizes IBM’s Qiskit as the simulation framework for quantum programs. We present the tool architecture in addition to the implemented methodology (i.e., the encoding of the search individual, the definition of the fitness function expressing the search problem, and the test assessment w.r.t. two types of failures). Finally, we report results of the experiments in which we tested a set of faulty quantum programs with QuSBT to assess its effectiveness.
Repository (code and experimental results): https://github.com/Simula-COMPLEX/qusbt-tool
Video: https://youtu.be/3apRCtluAn4
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) |
| Publisher | IEEE |
| URL | https://ieeexplore.ieee.org/document/9793826 |
| DOI | 10. 1145/3510454.3516839 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Talks, contributed
Time for new Simula!
In Towards a Norwegian Quantum Computing Strategy, Simula Research Laboratory, Norway, 2022.Status: Published
Time for new Simula!
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Nordic-Estonian Quantum Computing e-Infrastructure Quest, Quantum Software Engineering Project |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Towards a Norwegian Quantum Computing Strategy, Simula Research Laboratory, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Engineering
In Simula Research Laboratory, Norway, 2022.Status: Published
Quantum Software Engineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Simula Research Laboratory, Norway |
| Type of Talk | Seminar with Lionel Briand |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Analysis, Evolution and Reengineering
In ICSA-LITE, 2022.Status: Published
Quantum Software Analysis, Evolution and Reengineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | ICSA-LITE |
Talks, invited
Quantum Software Engineering: What, Why, and Where are we now?
In Quantum Leap and Mathematics, Soria Moria, Oslo, Norway, 2022.Status: Published
Quantum Software Engineering: What, Why, and Where are we now?
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | Quantum Leap and Mathematics, Soria Moria, Oslo, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
The ADEPTNESS project seeks to implement and investigate a streamlined and automatic workflow that makes methods and tools to be seamlessly used during design phases as well as in operation. We will explore the generation and reuse of test cases and oracles from initial phases of the development to the system in operation and back to the laboratory for reproduction. Integrated into this workflow, unforeseen situations will also be detected in operation to enhance development models for increasing resilience. We will consider several aspects of uncertainties (such as uncertainties in the environment, uncertainty produced due to timing aspects of CPSoS, uncertainty in networks, etc.). Additionally, automatic and synchronized deployment techniques will be investigated to improve the agility of the whole workflow that covers the design-operation continuum.
Coordinator:
Mondragon Goi Eskola Politeknikoa Jose Maria Arizmendiarrieta S Coop
Funding:
Horizon 2020
Publications for Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems
Journal Article
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
ACM Transactions on Software Engineering and Methodology (2023).Status: Accepted
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
Anomaly detection is critical to ensure the security of cyber-physical systems (CPS). However, due to the increasing complexity of attacks and CPS themselves, anomaly detection in CPS is becoming more and more challenging. In our previous work, we proposed a digital twin-based anomaly detection method, called ATTAIN, which takes advantage of both historical and real-time data of CPS. However, such data vary significantly in terms of difficulty. Therefore, similar to human learning processes, deep learning models (e.g., ATTAIN) can benefit from an easy-to-difficult curriculum. To this end, in this paper, we present a novel approach, named
digitaL twin-based Anomaly deTecTion wIth Curriculum lEarning (LATTICE), which extends ATTAIN by introducing curriculum learning to optimize its learning paradigm. LATTICE attributes each sample with a difficulty score, before being fed into a training scheduler. The training scheduler samples batches of training
data based on these difficulty scores such that learning from easy to difficult data can be performed. To evaluate LATTICE, we use five publicly available datasets collected from five real-world CPS testbeds. We compare LATTICE with ATTAIN and two other state-of-the-art anomaly detectors. Evaluation results show that LATTICE outperforms the three baselines and ATTAIN by 0.906%-2.367% in terms of the F1 score. LATTICE also, on average, reduces the training time of ATTAIN by 4.2% on the five datasets and is on par with the baselines in terms of detection delay time
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Publisher | ACM |
| Place Published | TOSEM |
Journal Article
Uncertainty-aware Robustness Assessment of Industrial Elevator Systems
ACM Transactions on Software Engineering and Methodology (2022).Status: Published
Uncertainty-aware Robustness Assessment of Industrial Elevator Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Publisher | ACM |
| URL | https://doi.org/10.1145/3576041 |
| DOI | 10.1145/3576041 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Proceedings, refereed
Are Elevator Software Robust Against Uncertainties? Results and Experiences from an Industrial Case Study
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Are Elevator Software Robust Against Uncertainties? Results and Experiences from an Industrial Case Study
| Afilliation | Software Engineering |
| Project(s) | Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Pagination | 1331-1342 |
| Date Published | 11/2022 |
| Publisher | ACM |
Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators
Digital twins are increasingly developed to support the development, operation, and maintenance of cyber-physical systems such as industrial elevators. However, industrial elevators continuously evolve due to changes in physical installations, introducing new software features, updating existing ones, and making changes due to regulations (e.g., enforcing restricted elevator capacity due to COVID-19), etc. Thus, digital twin functionalities (often built on neural network-based models) need to evolve themselves constantly to be synchronized with the industrial elevators. Such an evolution is preferred to be automated, as manual evolution is time-consuming and error-prone. Moreover, collecting sufficient data to re-train neural network models of digital twins could be expensive or even infeasible. To this end, we propose unceRtaInty-aware tranSfer lEarning enriched Digital Twins LATTICE, a transfer learning based approach capable of transferring knowledge about the waiting time prediction capability of a digital twin of an industrial elevator across different scenarios. LATTICE also leverages uncertainty quantification to further improve its effectiveness. To evaluate LATTICE, we conducted experiments with 10 versions of an elevator dispatching software from Orona, Spain, which are deployed in a Software in the Loop (SiL) environment. Experiment results show that LATTICE, on average, improves the Mean Squared Error by 13.131% and the utilization of uncertainty quantification further improves it by 2.71%.
| Afilliation | Software Engineering |
| Project(s) | Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Publisher | ACM |
| DOI | 10.1145/3540250.3558957 |
Automating Test Oracle Generation in DevOps for Industrial Elevators
In 29th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, 2022.Status: Published
Automating Test Oracle Generation in DevOps for Industrial Elevators
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 29th IEEE International Conference on Software Analysis, Evolution and Reengineering |
| Pagination | 284-288 |
| Publisher | IEEE |
| DOI | 10.1109/SANER53432.2022.00044 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Talks, contributed
Automating Test Oracle Generation in DevOps for Industrial Elevators
In 19TH IEEE INTERNATIONAL CONFERENCE ON SOFTWARE ARCHITECTURE (ICSA 2022). IEEE, 2022.Status: Published
Automating Test Oracle Generation in DevOps for Industrial Elevators
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | 19TH IEEE INTERNATIONAL CONFERENCE ON SOFTWARE ARCHITECTURE (ICSA 2022) |
| Publisher | IEEE |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Digital Twin for Elevators Use Case
In ETSI Event, Berlin, Germany, 2022.Status: Published
Digital Twin for Elevators Use Case
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | ETSI Event, Berlin, Germany |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Enhancing the Dependability of Cyber-physical Systems with AI-enabled Digital Twin
In Simula Research Laboratory, Norway, 2022.Status: Published
Enhancing the Dependability of Cyber-physical Systems with AI-enabled Digital Twin
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Simula Research Laboratory, Norway |
| Type of Talk | Presenting to guest lecturer Lionel C. Briand |
Uncertainty-aware transfer learning to evolve digital twins for industrial elevators
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Uncertainty-aware transfer learning to evolve digital twins for industrial elevators
Digital twins are increasingly developed to support the development, operation, and maintenance of cyber-physical systems such as industrial elevators. However, industrial elevators continuously evolve due to changes in physical installations, introducing new software features, updating existing ones, and making changes due to regulations (e.g., enforcing restricted elevator capacity due to COVID-19), etc. Thus, digital twin functionalities (often built on neural network-based models) need to evolve themselves constantly to be synchronized with the industrial elevators. Such an evolution is preferred to be automated, as manual evolution is time-consuming and error-prone. Moreover, collecting sufficient data to re-train neural network models of digital twins could be expensive or even infeasible. To this end, we propose unceRtaInty-aware tranSfer lEarning enriched Digital Twins LATTICE, a transfer learning based approach capable of transferring knowledge about the waiting time prediction capability of a digital twin of an industrial elevator across different scenarios. LATTICE also leverages uncertainty quantification to further improve its effectiveness. To evaluate LATTICE, we conducted experiments with 10 versions of an elevator dispatching software from Orona, Spain, which are deployed in a Software in the Loop (SiL) environment. Experiment results show that LATTICE, on average, improves the Mean Squared Error by 13.131% and the utilization of uncertainty quantification further improves it by 2.71%.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Publisher | ACM |
| Type of Talk | Conference paper |
| DOI | 10.1145/3540250.3558957 |
Enhancing the Dependability of Cyber-physical Systems with AI-enabled Digital Twin
In NORA Annual Conference 2022, 2022.Status: Published
Enhancing the Dependability of Cyber-physical Systems with AI-enabled Digital Twin
Cyber-physical Systems (CPS) have played an essential role in Industry 4.0 [1]. Since then, CPS are evolving to be increasingly heterogeneous, integrated, intelligent, operating in dynamic and everchanging environment. This exposes CPS to broader threats, which cannot be sufficiently tackled with traditional techniques. Our work focuses on exploring the potential of applying Digital Twin (DT) to improve dependability of CPS. The key idea is to build a DT as a virtual representation of a CPS, and develop DT functionalities with ML/AI algorithms to ensure the dependability of CPS operating in dynamic, uncertain and constantly-evolving environment. This research topic started in 2020 in the Engineering Complex Software Systems Department at Simula. As the first step, we chose Anomaly Detection as our main targeted research area, which is a sub-domain of the CPS dependability. Our current work consists of three phases: 1) designing and building a DTbased model, 2) enhancing it with Curriculum Learning (CL) [2], and 3) improving it with Transfer Learning [3]. • First, we have proposed a general DT-based model for anomaly detection. In this work, we built a Timed Automaton Machine (TAM) as the digital representation of the CPS, and implemented a Generative Adversarial Network (GAN) to detect anomalies. We evaluated this method (named ATTAIN) with three public datasets and achieved state-of-art results. • Second, we proposed LATTICE by extending ATTAIN by introducing CL to optimize its learning paradigm. CL is inspired by human learning process, which indicates that deep learning methods can benefit from a easy-to-difficult curriculum. We evaluated LATTICE with five public datasets and results show improvements over ATTAIN. • Currently, we are exploring to use transfer learning to further improve LATTICE. This is motivated because we found that most existing methods (including ours) are CPS-agnostic and become obsolete when new scenarios emerge. Therefore, we plan to improve predictive performance and reduce prediction uncertainty by transferring knowledge from these obsolete models to new models. We will evaluate this work on real elevator data from Orona—world leader in building industrial elevators.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | NORA Annual Conference 2022 |
AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
The innovation planned is a state-of-the-art test infrastructure including new testing techniques utilizing algorithms from machine learning and evolutionary computation (i.e., two sub-areas of AI) to support cost-effective and systematic testing of the Cancer Registration Support System (CaReSS) to significantly improve its quality, and quality of data and statistics it produces, by dealing with the continuous evolution and unpredictable behavior of machine learning algorithms. This will positively affect all its end users, including researchers, patients, doctors, and government officials. The deployment of the new testing infrastructure at the Cancer Registry of Norway (CRN) will contribute to the further digitalization at the CRN, leading to significant improvements in the current testing practice at CRN. With minimal effort, the testing infrastructure has the potential to be deployed to other health registries in Norway and cancer registries in the world.
Funding source
-
The Research Council of Norway
Partners
- Simula Research Laboratory
- Cancer Registry of Norway
Project leaders
- Simula Research Laboratory: Shaukat Ali and Tao Yue
- Cancer Registry of Norway: Jan F. Nygård
Publications for AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System
Journal Article
Multi-Objective Search-Based Software Microbenchmark Prioritization
arXiv (2022).Status: Submitted
Multi-Objective Search-Based Software Microbenchmark Prioritization
Ensuring that software performance does not degrade after a code change is paramount. A potential solution, particularly for libraries and frameworks, is regularly executing software microbenchmarks, a performance testing technique similar to (functional) unit tests. This often becomes infeasible due to the extensive runtimes of microbenchmark suites, however. To address that challenge, research has investigated regression testing techniques, such as test case prioritization (TCP), which reorder the execution within a microbenchmark suite to detect larger performance changes sooner. Such techniques are either designed for unit tests and perform sub-par on microbenchmarks or require complex performance models, reducing their potential application drastically. In this paper, we propose a search-based technique based on multi-objective evolutionary algorithms (MOEAs) to improve the current state of microbenchmark prioritization. The technique utilizes three objectives, i.e., coverage to maximize, coverage overlap to minimize, and historical performance change detection to maximize. We find that our technique improves over the best coverage-based, greedy baselines in terms of average percentage of fault-detection on performance (APFD-P) and Top-3 effectiveness by 26 percentage points (pp) and 43 pp (for Additional) and 17 pp and 32 pp (for Total) to 0.77 and 0.24, respectively. Employing the Indicator-Based Evolutionary Algorithm (IBEA) as MOEA leads to the best effectiveness among six MOEAs. Finally, the technique's runtime overhead is acceptable at 19% of the overall benchmark suite runtime, if we consider the enormous runtimes often spanning multiple hours. The added overhead compared to the greedy baselines is miniscule at 1%.These results mark a step forward for universally applicable performance regression testing techniques.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | arXiv |
| Publisher | arXiv |
| Keywords | JMH, Multi-objective optimization, Performance Testing, Regression testing, Search-Based Software Engineering, software microbenchmarking, Test Case Prioritization |
| URL | https://arxiv.org/abs/2211.13525 |
| DOI | 10.48550/ARXIV.2211.13525 |
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
Using Microbenchmark Suites to Detect Application Performance Changes
IEEE Transactions on Cloud Computing (2022): 1-18.Status: Published
Using Microbenchmark Suites to Detect Application Performance Changes
Software performance changes are costly and often hard to detect pre-release. Similar to software testing frameworks, either application benchmarks or microbenchmarks can be integrated into quality assurance pipelines to detect performance changes before releasing a new application version. Unfortunately, extensive benchmarking studies usually take several hours which is problematic when examining dozens of daily code changes in detail; hence, trade-offs have to be made. Optimized microbenchmark suites, which only include a small subset of the full suite, are a potential solution for this problem, given that they still reliably detect the majority of the application performance changes such as an increased request latency. It is, however, unclear whether microbenchmarks and application benchmarks detect the same performance problems and one can be a proxy for the other. In this paper, we explore whether microbenchmark suites can detect the same application performance changes as an application benchmark. For this, we run extensive benchmark experiments with both the complete and the optimized microbenchmark suites of two time-series database systems, i.e., InfluxDB and VictoriaMetrics, and compare their results to the results of corresponding application benchmarks. We do this for 70 and 110 commits, respectively. Our results show that it is not trivial to detect application performance changes using an optimized microbenchmark suite. The detection (i) is only possible if the optimized microbenchmark suite covers all application-relevant code sections, (ii) is prone to false alarms, and (iii) cannot precisely quantify the impact on application performance. For certain software projects, an optimized microbenchmark suite can, thus, provide fast performance feedback to developers (e.g., as part of a local build process), help estimating the impact of code changes on application performance, and support a detailed analysis while a daily application benchmark detects major performance problems. Thus, although a regular application benchmark cannot be substituted for both studied systems, our results motivate further studies to validate and optimize microbenchmark suites.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | IEEE Transactions on Cloud Computing |
| Pagination | 1-18 |
| Date Published | 10/2022 |
| Publisher | IEEE |
| ISSN | 2168-7161 |
| Keywords | Benchmarking, Microbenchmarks, Performance Change Detection, Performance Testing, Regression Detection |
| URL | https://ieeexplore.ieee.org/abstract/document/9931939 |
| DOI | 10.1109/TCC.2022.3217947 |
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
Talks, contributed
Predicting Unstable Software Benchmarks using Static Source Code Features
In International Conference on Software Engineering (ICSE), 2022.Status: Published
Predicting Unstable Software Benchmarks using Static Source Code Features
Software benchmarks are only as good as the performance measurements they yield. Unstable benchmarks show high variability among repeated measurements, which causes uncertainty about the “true” performance of the measured software unit and complicates reliable change assessment. Conversely, if multiple repeated measurements have low variability, i.e., the distribution of the measurement results is narrow, a benchmark is considered stable. However, if a benchmark is stable or unstable only becomes evident after it has been executed, and its results are available.
In this paper, we introduce a machine-learning-based approach to predict a benchmark’s stability without executing it. Our approach statically extracts 58 source code features, for both benchmark code and code called by a benchmark. It parses the abstract syntax trees (ASTs) of all functions, counts the occurrences of each feature for each function, computes the reachable functions from a benchmark with static call graphs (CGs), sums up the feature occurrences for each benchmark, and feeds the features into a binary classifier. Inspired by previous software performance research, the employed features act as proxies for performance variability (and consequently benchmark stability) and are related to: (1) meta information, e.g., lines of code (LOC); (2) programming language elements, e.g., conditionals or loops; (3) potentially performance-impacting standard library calls, e.g., file and network input/output (I/O).
To assess our approach’s effectiveness, we perform a large-scale experiment on 4,461 Go performance benchmarks coming from 230 open-source software (OSS) projects.
First, we assess the prediction performance of our machine learning models using 11 binary classification algorithms. We find that Random Forest performs best with good prediction performance from 0.79 to 0.90, and 0.43 to 0.68, in terms of Area Under the Curve (AUC) and Matthews Correlation Coefficient (MCC), respectively.
Second, we carry out four sensitivity analyses to investigate the impact on the prediction performance, if the model is trained (1) with different variability thresholds that consider a benchmark as stable or unstable, (2) on benchmark executions with a varying number of measurement repetitions, (3) after applying specific pre-processing steps to remove co-linear and multi-colinear features as well as perform class-rebalancing on the training set, and (4) for different variability measures used as the dependent variable. We find that our model performs best when trained on larger thresholds (10%) and executions from more repetitions (30). While feature pre-processing does not have an impact across all studied algorithms, removing co-linear and multi-co-linear improves Random Forrest’s prediction performance by 0.023 MCC and 0.005 AUC. The model shows varying prediction performance, depending on which variability measures is used as dependent variable. However, it performs well across all three studied measures.
Third, we perform feature importance analyses for individual features and feature categories. We find that 7 features related to meta-information, slice usage, nested loops, and synchronization application programming interfaces (APIs) are individually important for good predictions; and that the combination of all features of the called source code is paramount for our model, while the combination of features of the benchmark itself is less important.
Our results show that although benchmark stability is affected by more than just the source code, we can effectively utilize machine learning models to predict whether a benchmark will be stable or not ahead of execution. This enables spending precious testing time on reliable benchmarks, supporting developers to identify unstable benchmarks during development, allowing unstable benchmarks to be repeated more often, estimating stability in scenarios where repeated benchmark execution is infeasible or impossible, and warning developers if new benchmarks or existing benchmarks executed in new environments will be unstable.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | International Conference on Software Engineering (ICSE) |
| Type of Talk | Journal-First Paper Presentation |
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
Talks, invited
Deliberate Microbenchmarking of Software Systems
In Cancer Registry of Norway, 2021.Status: Published
Deliberate Microbenchmarking of Software Systems
In this talk, I will summarize the findings of my PhD thesis:
Software performance faults have severe consequences for users, developers, and companies. One way to unveil performance faults before they manifest in production is performance testing, which ought to be done on every new version of the software, ideally on every commit. However, performance testing faces multiple challenges that inhibit it from being applied early in the development process, on every new commit, and in an automated fashion.
In this dissertation, we investigate three challenges of software microbenchmarks, a performance testing technique on unit granularity which is predominantly used for libraries and frameworks. The studied challenges affect the quality aspects (1) runtime, (2) result variability, and (3) performance change detection of microbenchmark executions. The objective is to understand the extent of these challenges in real-world software and to find solutions to address these.
To investigate the challenges’ extent, we perform a series of experiments and analyses. We execute benchmarks in bare-metal as well as multiple cloud environments and conduct a large-scale mining study on benchmark configurations. The results show that all three challenges are common: (1) benchmark suite runtimes are often longer than 3 hours; (2) result variability can be extensive, in some cases up to 100%; and (3) benchmarks often only reliably detect large performance changes of 60% or more.
To address the challenges, we devise targeted solutions as well as adapt well-known techniques from other domains for software microbenchmarks: (1) a solution that dynamically stops benchmark executions based on statistics to reduce runtime while maintaining low result variability; (2) a solution to identify unstable benchmarks that does not require execution, based on statically-computable source code features and machine learning algorithms; (3) traditional test case prioritization (TCP) techniques to execute benchmarks earlier that detect larger performance changes; and (4) specific execution strategies to detect small performance changes reliably even when executed in unreliable cloud environments.
We experimentally evaluate the solutions and techniques on real-world benchmarks and find that they effectively deal with the three challenges. (1) Dynamic reconfiguration enables to drastically reduce runtime by between 48.4% and 86.0% without changing the results of 78.8% to 87.6% of the benchmarks, depending on the project and statistic used. (2) The instability prediction model allows to effectively identify unstable benchmarks when relying on random forest classifiers, having a prediction performance between 0.79 and 0.90 area under the receiver operating characteristic curve (AUC). (3) TCP applied to benchmarks is effective and efficient, with APFD-P values for the best technique ranging from 0.54 to 0.71 and a computational overhead of 11%. (4) Batch testing, i.e., executing the benchmarks of two versions on the same instances interleaved and repeated as well as repeated across instances, enables to reliably detect performance changes of 10% or less, even when using unreliable cloud infrastructure as execution environment.
Overall, this dissertation shows that real-world software microbenchmarks are considerably affected by all three challenges (1) runtime, (2) result variability, and (3) performance change detection; however, deliberate planning and execution strategies effectively reduce their impact.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System |
| Publication Type | Talks, invited |
| Year of Publication | 2021 |
| Location of Talk | Cancer Registry of Norway |
| Type of Talk | Seminar Talk |
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
Variability of Microbenchmark Results and How to Deal with It
In Chalmers, Software Engineering Division, University of Gothenburg, Sweden, 2021.Status: Published
Variability of Microbenchmark Results and How to Deal with It
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System |
| Publication Type | Talks, invited |
| Year of Publication | 2021 |
| Location of Talk | Chalmers, Software Engineering Division, University of Gothenburg, Sweden |
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Welfare Technology is an emerging concept that refers to the use of new technology solutions within the fields of nursing, health, and social care. Preliminary studies have shown that Welfare Technology Solutions (WTSs) provide significant gains both in terms of increased quality of life and reduced levels of health service consumption by citizens.
The objective of this innovation is to improve the quality of the current and future WTSs of the City of Oslo (CoO), enhance their safety, privacy and security, and improve the testing practice at CoO, via automated, systematic, and cost-effective testing, for enhancing the overall quality of welfare services provided by the WTSs to end-users. The proposed testing solution is expected to reduce testing time and improve the overall efficiency of developing WTSs, while it will become accessible to other municipalities and counties in Norway in the future.
Funding
- The Research Council of Norway
Partners
- Simula Research Laboratory
- City of Oslo
Publications for WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo
Proceedings, refereed
Towards Requirements Engineering for Digital Twins of Cyber-Physical Systems
In 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation. LNCS, 2022.Status: Published
Towards Requirements Engineering for Digital Twins of Cyber-Physical Systems
| Afilliation | Software Engineering |
| Project(s) | WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo, Department of Engineering Complex Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation |
| Publisher | LNCS |
Talk, keynote
WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo
In Simula KA23, Oslo, Norway, 2022.Status: Published
WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo |
| Publication Type | Talk, keynote |
| Year of Publication | 2022 |
| Location of Talk | Simula KA23, Oslo, Norway |
Talks, contributed
Towards Requirements Engineering for Digital Twins of Cyber-Physical Systems
In In 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation. LNCS, 2022.Status: Published
Towards Requirements Engineering for Digital Twins of Cyber-Physical Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | In 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation |
| Publisher | LNCS |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Department of Engineering Complex Software Systems
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
There is an increased reliance on our daily life on Complex Software Systems such as smart Cyber-Physical Systems and Internet of Things. Such systems are used in many safety/mission-critical such as healthcare, autonomous cars, boats and underwater vehicles, maritime, energy, and smart roads, grids, buildings, and cities. Such systems are progressively turning into complex, heterogeneous, open, networked and “smart” systems that are composed of agents, sensors, actuators, information networks, and dedicated middleware and infrastructures. Given these complexities and high interaction with the physical environment, new, smart and efficient Software Engineering (SE) paradigms are needed to design, develop, test, and maintain such systems to ensure their dependability (e.g., safety) as well as other relevant quality attributes such as security and privacy.
Such systems are typically built by integrating various physical units, with known and uncertain assumptions on their physical operating environments, future deployments, middleware, and infrastructures. However, as the first dimension of the complexity and challenges, during a real operation of such a system, there is no guarantee that these (often flawed) assumptions will hold, which makes such a system vulnerable to unforeseen loopholes concerning one or more of the quality attributes. Therefore, a novel SE paradigm is expected to manage and handle uncertainty information (e.g., uncertain assumptions) throughout complex system development lifecycles in an intelligent manner. The second dimension is due to the increased deployment of Artificial Intelligence (AI) components for all kinds of reasoning and recognition tasks in modern complex systems. Current system development tools and techniques are not sufficient to cope with this complexity and challenge, and therefore a radical shift moving toward a novel SE paradigm is urgently expected. The third dimension concerns with the multi-disciplinary nature of developing, maintaining, and operating complex software systems, which concerns not only technical aspects but also ethical principles and address societal concerns especially when developing (AI-enabled) smart systems.
The aim of the Department of Engineering Complex Software Systems (ComplexSE) is to develop novel SE paradigms to address the challenges mentioned above to facilitate engineering of modern complex software systems. The ComplexSE department establishes itself with several well-established SE sub-disciplines including Model-Based Engineering, Model-Based Testing, Search-based Software Engineering, and Empirical Software Engineering.
Publications for Department of Engineering Complex Software Systems
Journal Article
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
ACM Transactions on Software Engineering and Methodology (2023).Status: Accepted
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
Anomaly detection is critical to ensure the security of cyber-physical systems (CPS). However, due to the increasing complexity of attacks and CPS themselves, anomaly detection in CPS is becoming more and more challenging. In our previous work, we proposed a digital twin-based anomaly detection method, called ATTAIN, which takes advantage of both historical and real-time data of CPS. However, such data vary significantly in terms of difficulty. Therefore, similar to human learning processes, deep learning models (e.g., ATTAIN) can benefit from an easy-to-difficult curriculum. To this end, in this paper, we present a novel approach, named
digitaL twin-based Anomaly deTecTion wIth Curriculum lEarning (LATTICE), which extends ATTAIN by introducing curriculum learning to optimize its learning paradigm. LATTICE attributes each sample with a difficulty score, before being fed into a training scheduler. The training scheduler samples batches of training
data based on these difficulty scores such that learning from easy to difficult data can be performed. To evaluate LATTICE, we use five publicly available datasets collected from five real-world CPS testbeds. We compare LATTICE with ATTAIN and two other state-of-the-art anomaly detectors. Evaluation results show that LATTICE outperforms the three baselines and ATTAIN by 0.906%-2.367% in terms of the F1 score. LATTICE also, on average, reduces the training time of ATTAIN by 4.2% on the five datasets and is on par with the baselines in terms of detection delay time
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Publisher | ACM |
| Place Published | TOSEM |
Book Chapter
Human-in-the-Loop Enhanced COVID-19 Detection in Transfer Learning-Based CNN Models
In Computational Intelligence for COVID-19 and Future Pandemics, 71-87. Singapore: Springer, 2022.Status: Published
Human-in-the-Loop Enhanced COVID-19 Detection in Transfer Learning-Based CNN Models
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems |
| Publication Type | Book Chapter |
| Year of Publication | 2022 |
| Book Title | Computational Intelligence for COVID-19 and Future Pandemics |
| Pagination | 71-87 |
| Publisher | Springer |
| Place Published | Singapore |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Edited books
Quantum Computing - Introduction to the special theme
Vol. 128. ERCIM, 2022.Status: Published
Quantum Computing - Introduction to the special theme
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Edited books |
| Year of Publication | 2022 |
| Volume | 128 |
| Publisher | ERCIM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Journal Article
Multi-Objective Search-Based Software Microbenchmark Prioritization
arXiv (2022).Status: Submitted
Multi-Objective Search-Based Software Microbenchmark Prioritization
Ensuring that software performance does not degrade after a code change is paramount. A potential solution, particularly for libraries and frameworks, is regularly executing software microbenchmarks, a performance testing technique similar to (functional) unit tests. This often becomes infeasible due to the extensive runtimes of microbenchmark suites, however. To address that challenge, research has investigated regression testing techniques, such as test case prioritization (TCP), which reorder the execution within a microbenchmark suite to detect larger performance changes sooner. Such techniques are either designed for unit tests and perform sub-par on microbenchmarks or require complex performance models, reducing their potential application drastically. In this paper, we propose a search-based technique based on multi-objective evolutionary algorithms (MOEAs) to improve the current state of microbenchmark prioritization. The technique utilizes three objectives, i.e., coverage to maximize, coverage overlap to minimize, and historical performance change detection to maximize. We find that our technique improves over the best coverage-based, greedy baselines in terms of average percentage of fault-detection on performance (APFD-P) and Top-3 effectiveness by 26 percentage points (pp) and 43 pp (for Additional) and 17 pp and 32 pp (for Total) to 0.77 and 0.24, respectively. Employing the Indicator-Based Evolutionary Algorithm (IBEA) as MOEA leads to the best effectiveness among six MOEAs. Finally, the technique's runtime overhead is acceptable at 19% of the overall benchmark suite runtime, if we consider the enormous runtimes often spanning multiple hours. The added overhead compared to the greedy baselines is miniscule at 1%.These results mark a step forward for universally applicable performance regression testing techniques.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | arXiv |
| Publisher | arXiv |
| Keywords | JMH, Multi-objective optimization, Performance Testing, Regression testing, Search-Based Software Engineering, software microbenchmarking, Test Case Prioritization |
| URL | https://arxiv.org/abs/2211.13525 |
| DOI | 10.48550/ARXIV.2211.13525 |
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
Uncertainty-aware Robustness Assessment of Industrial Elevator Systems
ACM Transactions on Software Engineering and Methodology (2022).Status: Published
Uncertainty-aware Robustness Assessment of Industrial Elevator Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Publisher | ACM |
| URL | https://doi.org/10.1145/3576041 |
| DOI | 10.1145/3576041 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Using Microbenchmark Suites to Detect Application Performance Changes
IEEE Transactions on Cloud Computing (2022): 1-18.Status: Published
Using Microbenchmark Suites to Detect Application Performance Changes
Software performance changes are costly and often hard to detect pre-release. Similar to software testing frameworks, either application benchmarks or microbenchmarks can be integrated into quality assurance pipelines to detect performance changes before releasing a new application version. Unfortunately, extensive benchmarking studies usually take several hours which is problematic when examining dozens of daily code changes in detail; hence, trade-offs have to be made. Optimized microbenchmark suites, which only include a small subset of the full suite, are a potential solution for this problem, given that they still reliably detect the majority of the application performance changes such as an increased request latency. It is, however, unclear whether microbenchmarks and application benchmarks detect the same performance problems and one can be a proxy for the other. In this paper, we explore whether microbenchmark suites can detect the same application performance changes as an application benchmark. For this, we run extensive benchmark experiments with both the complete and the optimized microbenchmark suites of two time-series database systems, i.e., InfluxDB and VictoriaMetrics, and compare their results to the results of corresponding application benchmarks. We do this for 70 and 110 commits, respectively. Our results show that it is not trivial to detect application performance changes using an optimized microbenchmark suite. The detection (i) is only possible if the optimized microbenchmark suite covers all application-relevant code sections, (ii) is prone to false alarms, and (iii) cannot precisely quantify the impact on application performance. For certain software projects, an optimized microbenchmark suite can, thus, provide fast performance feedback to developers (e.g., as part of a local build process), help estimating the impact of code changes on application performance, and support a detailed analysis while a daily application benchmark detects major performance problems. Thus, although a regular application benchmark cannot be substituted for both studied systems, our results motivate further studies to validate and optimize microbenchmark suites.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | IEEE Transactions on Cloud Computing |
| Pagination | 1-18 |
| Date Published | 10/2022 |
| Publisher | IEEE |
| ISSN | 2168-7161 |
| Keywords | Benchmarking, Microbenchmarks, Performance Change Detection, Performance Testing, Regression Detection |
| URL | https://ieeexplore.ieee.org/abstract/document/9931939 |
| DOI | 10.1109/TCC.2022.3217947 |
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their Collisions
IEEE Transactions on Software Engineering 49, no. 1 (2022): 384-402.Status: Published
Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their Collisions
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | IEEE Transactions on Software Engineering |
| Volume | 49 |
| Issue | 1 |
| Pagination | 384-402 |
| Publisher | IEEE |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
On the Preferences of Quality Indicators for Multi-Objective Search Algorithms in Search-Based Software Engineering
The Empirical Software Engineering Journal (EMSE) 27, no. 6 (2022): 144.Status: Published
On the Preferences of Quality Indicators for Multi-Objective Search Algorithms in Search-Based Software Engineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | The Empirical Software Engineering Journal (EMSE) |
| Volume | 27 |
| Issue | 6 |
| Pagination | 144 |
| Publisher | Springer |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty-Aware Prediction Validator in Deep Learning Models for Cyber-Physical System Data
ACM Transactions on Software Engineering and Methodology (TOSEM) 31 (2022): 1-31.Status: Published
Uncertainty-Aware Prediction Validator in Deep Learning Models for Cyber-Physical System Data
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | ACM Transactions on Software Engineering and Methodology (TOSEM) |
| Volume | 31 |
| Number | 4 |
| Pagination | 1-31 |
| Publisher | ACM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Testing: Challenges, Early Achievements, and Opportunities
the European Research Consortium for Informatics and Mathematics News (2022).Status: Published
Quantum Software Testing: Challenges, Early Achievements, and Opportunities
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | the European Research Consortium for Informatics and Mathematics News |
| Publisher | ERCIM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Publications
Journal Article
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
ACM Transactions on Software Engineering and Methodology (2023).Status: Accepted
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
Anomaly detection is critical to ensure the security of cyber-physical systems (CPS). However, due to the increasing complexity of attacks and CPS themselves, anomaly detection in CPS is becoming more and more challenging. In our previous work, we proposed a digital twin-based anomaly detection method, called ATTAIN, which takes advantage of both historical and real-time data of CPS. However, such data vary significantly in terms of difficulty. Therefore, similar to human learning processes, deep learning models (e.g., ATTAIN) can benefit from an easy-to-difficult curriculum. To this end, in this paper, we present a novel approach, named
digitaL twin-based Anomaly deTecTion wIth Curriculum lEarning (LATTICE), which extends ATTAIN by introducing curriculum learning to optimize its learning paradigm. LATTICE attributes each sample with a difficulty score, before being fed into a training scheduler. The training scheduler samples batches of training
data based on these difficulty scores such that learning from easy to difficult data can be performed. To evaluate LATTICE, we use five publicly available datasets collected from five real-world CPS testbeds. We compare LATTICE with ATTAIN and two other state-of-the-art anomaly detectors. Evaluation results show that LATTICE outperforms the three baselines and ATTAIN by 0.906%-2.367% in terms of the F1 score. LATTICE also, on average, reduces the training time of ATTAIN by 4.2% on the five datasets and is on par with the baselines in terms of detection delay time
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Journal Article |
| Year of Publication | 2023 |
| Journal | ACM Transactions on Software Engineering and Methodology |
| Publisher | ACM |
| Place Published | TOSEM |
Talks, contributed
Towards Requirements Engineering for Digital Twins of Cyber-Physical Systems
In In 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation. LNCS, 2022.Status: Published
Towards Requirements Engineering for Digital Twins of Cyber-Physical Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | In 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation |
| Publisher | LNCS |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Automating Test Oracle Generation in DevOps for Industrial Elevators
In 19TH IEEE INTERNATIONAL CONFERENCE ON SOFTWARE ARCHITECTURE (ICSA 2022). IEEE, 2022.Status: Published
Automating Test Oracle Generation in DevOps for Industrial Elevators
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | 19TH IEEE INTERNATIONAL CONFERENCE ON SOFTWARE ARCHITECTURE (ICSA 2022) |
| Publisher | IEEE |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Building Complex Software Systems in Classical and Quantum Computing Domains
In Connecting Education and Research Communities for an Innovative Resource Aware Society, Meeting Denmark, 2022.Status: Published
Building Complex Software Systems in Classical and Quantum Computing Domains
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, CERCIRAS: Connecting Education and Research Communities for an Innovative Resource Aware Society |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Connecting Education and Research Communities for an Innovative Resource Aware Society, Meeting Denmark |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Digital Twin for Elevators Use Case
In ETSI Event, Berlin, Germany, 2022.Status: Published
Digital Twin for Elevators Use Case
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | ETSI Event, Berlin, Germany |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Learning Digital Twin Models
In Model-Driven Engineering of Digital Twins Seminar at Dagstuhl, Germany. Schloss Dagstuhl, Germany, 2022.Status: Published
Learning Digital Twin Models
Given that operational cyber-physical systems (CPS) produce continuous data, a complementary approach to model-based engineering is to learn digital twins models with machine learning techniques and providing functionalities such as predictions and anomaly detection.
This talk will start with presenting an opinion on the next generation of digital twins (Quantum Digital Twins), where some aspects of digital twins will be implemented as quantum software and executed on quantum computers, e.g., for simulating the physical environment that can be realistically simulated with quantum-mechanical principles.
Followed by this opinion, the talk will present some recent works on learning digital twins from historical data and continuous updates of digital twins with continuous data from operational CPS. Various machine learning techniques were applied, such as generative adversarial networks, curriculum learning, and transfer learning to learn digital twins. The digital twins were built for use cases from the transportation domain and water distribution/treatment plants. These digital twins were focused on anomaly detection and waiting time predictions.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Model-Driven Engineering of Digital Twins Seminar at Dagstuhl, Germany |
| Publisher | Schloss Dagstuhl, Germany |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Engineering
In Simula Research Laboratory, Norway, 2022.Status: Published
Quantum Software Engineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Simula Research Laboratory, Norway |
| Type of Talk | Seminar with Lionel Briand |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Time for new Simula!
In Towards a Norwegian Quantum Computing Strategy, Simula Research Laboratory, Norway, 2022.Status: Published
Time for new Simula!
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Nordic-Estonian Quantum Computing e-Infrastructure Quest, Quantum Software Engineering Project |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Towards a Norwegian Quantum Computing Strategy, Simula Research Laboratory, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty in Deep Learning
In Simula Research Laboratory, Norway, 2022.Status: Published
Uncertainty in Deep Learning
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Talks, contributed |
| Year of Publication | 2022 |
| Location of Talk | Simula Research Laboratory, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Talks, invited
AI-enabled Digital Twins for Cyber-Physical Systems
In EDT Community: Engineering Digital Twins – Seminar Series, 2022.Status: Published
AI-enabled Digital Twins for Cyber-Physical Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | EDT Community: Engineering Digital Twins – Seminar Series |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Assuring the Quality of Quantum Programs with Automated Testing
In University of Lisbon, Portugal, 2022.Status: Published
Assuring the Quality of Quantum Programs with Automated Testing
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | University of Lisbon, Portugal |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Dependable and Noise-Aware Quantum Software Engineering
In CREST Center, The University of Adelaide, Australia, 2022.Status: Published
Dependable and Noise-Aware Quantum Software Engineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | CREST Center, The University of Adelaide, Australia |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Digital Twins for Cyber-Physical Systems: From AI to Quantum Computing
In COEMS Forsterk Seminar, 2022.Status: Published
Digital Twins for Cyber-Physical Systems: From AI to Quantum Computing
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | COEMS Forsterk Seminar |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Modeling robustness behavior using aspect-oriented modeling to support robustness testing of industrial systems
In ACM / IEEE 25th International Conference on Model Driven Engineering Languages and Systems (MODELS), 2022.Status: Published
Modeling robustness behavior using aspect-oriented modeling to support robustness testing of industrial systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | ACM / IEEE 25th International Conference on Model Driven Engineering Languages and Systems (MODELS) |
| Type of Talk | 10-year SOSYM Most Influential Paper Award |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Engineering: What, Why, and Where are we now?
In Quantum Leap and Mathematics, Soria Moria, Oslo, Norway, 2022.Status: Published
Quantum Software Engineering: What, Why, and Where are we now?
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | Quantum Leap and Mathematics, Soria Moria, Oslo, Norway |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Testing Tutorial
In NordIQuEst-ENCCS online HPC-QC workshop, 2022.Status: Published
Quantum Software Testing Tutorial
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project, Nordic-Estonian Quantum Computing e-Infrastructure Quest |
| Publication Type | Talks, invited |
| Year of Publication | 2022 |
| Location of Talk | NordIQuEst-ENCCS online HPC-QC workshop |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Proceedings, refereed
Are Elevator Software Robust Against Uncertainties? Results and Experiences from an Industrial Case Study
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Are Elevator Software Robust Against Uncertainties? Results and Experiences from an Industrial Case Study
| Afilliation | Software Engineering |
| Project(s) | Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Pagination | 1331-1342 |
| Date Published | 11/2022 |
| Publisher | ACM |
Automating Test Oracle Generation in DevOps for Industrial Elevators
In 29th IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, 2022.Status: Published
Automating Test Oracle Generation in DevOps for Industrial Elevators
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 29th IEEE International Conference on Software Analysis, Evolution and Reengineering |
| Pagination | 284-288 |
| Publisher | IEEE |
| DOI | 10.1109/SANER53432.2022.00044 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Enhancing the realism of autonomous driving simulation with real-time co-simulation
In 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22) . IEEE/ACM, 2022.Status: Published
Enhancing the realism of autonomous driving simulation with real-time co-simulation
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems, Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 4th International Workshop on Multi-Paradigm Modelling for Cyber-Physical Systems (MPM4CPS'22) |
| Publisher | IEEE/ACM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Generating Failing Test Suites for Quantum Programs with Search (hot off the press track at GECCO 2022)
In GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 2022.Status: Published
Generating Failing Test Suites for Quantum Programs with Search (hot off the press track at GECCO 2022)
The inherent complexity of quantum programs, due to features such as superposition and entanglement, makes their testing particularly challenging. To tackle these challenges, we present a search-based approach, called Quantum Search-Based Testing (QuSBT), for automatically generating test suites of a given size that possibly expose failures of the quantum program under test. QuSBT encodes a test suite as a search individual, and tries to maximize the objective function that counts the number of failing tests in the test suite. Due to non-deterministic nature of quantum programs, the approach repeats the execution of each test multiple times, and uses suitable statistical tests to assess if a test passes or fails. QuSBT employs a genetic algorithm to perform the search. Experiments on 30 faulty quantum programs show that QuSBT is statistically better than random search, and is able to efficiently generate maximal failing test suites.
This is an extended abstract of the paper [1]: X. Wang, P. Arcaini, T. Yue, and S. Ali "Generating Failing Test Suites for Quantum Programs With Search", 13th International Symposium on Search-Based Software Engineering (SSBSE 2021).
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Quantum Software Engineering Project, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference |
| Pagination | 47-48 |
| Publisher | ACM |
| URL | https://dl.acm.org/doi/10.1145/3520304.3534067 |
| DOI | 10.1145/3520304.3534067 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Investigating Quantum Cause-Effect Graphs
In 2022 IEEE/ACM 3rd International Workshop on Quantum Software Engineering (Q-SE). IEEE, 2022.Status: Published
Investigating Quantum Cause-Effect Graphs
Cause-effect graphs have shown promising results in identifying relations among causes and effects of classical software systems, followed by designing effective test cases from them. Towards this end, we investigate the use of cause-effect graphs for quantum programs. Classical cause-effect graphs apply classical logic (e.g., AND, OR) to express these relations, which might not be practical for describing similar relations in quantum programs due to quantum superposition and entanglement. Thus, we propose an extension of cause-effect graphs, where quantum logic inspired functions (e.g., Hadamard) and their generalizations are defined and applied. Moreover, we present a metamodel describing various forms of cause-effect graphs. Finally, we demonstrate a possible method for generating test cases from a quantum cause-effect graph applied to a Bell state quantum program. Lastly, the design and utility of the resulting testing method is discussed, along with future prospects for general quantum cause-effect graphs.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Quantum Software Engineering Project |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 2022 IEEE/ACM 3rd International Workshop on Quantum Software Engineering (Q-SE) |
| Pagination | 8-15 |
| Publisher | IEEE |
Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search
In GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference. ACM, 2022.Status: Published
Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search
Mutation testing is often used for designing new tests, and involves changing a program in minor ways, which results in mutated versions of the program, i.e., mutants. An effective test suite should find faults (or kill mutants) with a minimum number of test cases, to save resources required for executing test cases. In this paper, in the context of mutation testing for quantum programs, we present a multi-objective and search-based approach (MutTG) to generate the minimum number of test cases killing as many mutants as possible. MutTG tries to estimate the likelihood that a mutant is equivalent, and uses this as a discount factor in the fitness definition to avoid keeping on trying to kill mutants that cannot be killed. We employed NSGA-II as the multi-objective search algorithm. Then, we compared MutTG with another version of the approach that does not use the discount factor in its fitness definition, and with random search (RS), over a set of open-source quantum programs and their mutants of varying complexity. Results show that the discount factor does indeed help in guiding the test generation, as the approach with the discount factor performs better than the one without it.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Quantum Software Engineering Project, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | GECCO '22: Proceedings of the Genetic and Evolutionary Computation Conference |
| Pagination | 1345-1353 |
| Publisher | ACM |
| URL | https://dl.acm.org/doi/abs/10.1145/3512290.3528869 |
| DOI | 10.1145/3512290.3528869 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
QuSBT: Search-Based Testing of Quantum Programs
In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2022.Status: Published
QuSBT: Search-Based Testing of Quantum Programs
Generating a test suite for a quantum program such that it has the maximum number of failing tests is an optimization problem. For such optimization, search-based testing has shown promising results in the context of classical programs. To this end, we present a test generation tool for quantum programs based on a genetic algorithm, called QuSBT (Search-based Testing of Quantum Programs). QuSBT automates the testing of quantum programs, with the aim of finding a test suite having the maximum number of failing test cases. QuSBT utilizes IBM’s Qiskit as the simulation framework for quantum programs. We present the tool architecture in addition to the implemented methodology (i.e., the encoding of the search individual, the definition of the fitness function expressing the search problem, and the test assessment w.r.t. two types of failures). Finally, we report results of the experiments in which we tested a set of faulty quantum programs with QuSBT to assess its effectiveness.
Repository (code and experimental results): https://github.com/Simula-COMPLEX/qusbt-tool
Video: https://youtu.be/3apRCtluAn4
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs, Quantum Software Engineering Project |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) |
| Publisher | IEEE |
| URL | https://ieeexplore.ieee.org/document/9793826 |
| DOI | 10. 1145/3510454.3516839 |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Towards Requirements Engineering for Digital Twins of Cyber-Physical Systems
In 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation. LNCS, 2022.Status: Published
Towards Requirements Engineering for Digital Twins of Cyber-Physical Systems
| Afilliation | Software Engineering |
| Project(s) | WTT4Oslo: Improving Quality of IoT-based Welfare Technology Solutions in the City of Oslo, Department of Engineering Complex Software Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | 11th International Symposium On Leveraging Applications of Formal Methods, Verification and Validation |
| Publisher | LNCS |
Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators
In The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE). ACM, 2022.Status: Published
Uncertainty-Aware Transfer Learning to Evolve Digital Twins for Industrial Elevators
Digital twins are increasingly developed to support the development, operation, and maintenance of cyber-physical systems such as industrial elevators. However, industrial elevators continuously evolve due to changes in physical installations, introducing new software features, updating existing ones, and making changes due to regulations (e.g., enforcing restricted elevator capacity due to COVID-19), etc. Thus, digital twin functionalities (often built on neural network-based models) need to evolve themselves constantly to be synchronized with the industrial elevators. Such an evolution is preferred to be automated, as manual evolution is time-consuming and error-prone. Moreover, collecting sufficient data to re-train neural network models of digital twins could be expensive or even infeasible. To this end, we propose unceRtaInty-aware tranSfer lEarning enriched Digital Twins LATTICE, a transfer learning based approach capable of transferring knowledge about the waiting time prediction capability of a digital twin of an industrial elevator across different scenarios. LATTICE also leverages uncertainty quantification to further improve its effectiveness. To evaluate LATTICE, we conducted experiments with 10 versions of an elevator dispatching software from Orona, Spain, which are deployed in a Software in the Loop (SiL) environment. Experiment results show that LATTICE, on average, improves the Mean Squared Error by 13.131% and the utilization of uncertainty quantification further improves it by 2.71%.
| Afilliation | Software Engineering |
| Project(s) | Co-tester: Collective-Adaptive Testing of Coevolving Autonomous Cyber-Physical Systems of Systems under Uncertainty, Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Proceedings, refereed |
| Year of Publication | 2022 |
| Conference Name | The ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) |
| Publisher | ACM |
| DOI | 10.1145/3540250.3558957 |
Technical reports
Classifying Uncertainties in an Industrial Elevator with the Cynefin Framework
Simula Research Laboratory, 2022.Status: Published
Classifying Uncertainties in an Industrial Elevator with the Cynefin Framework
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Technical reports |
| Year of Publication | 2022 |
| Publisher | Simula Research Laboratory |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
Simula Research Laboratory, 2022.Status: Published
Digital Twin-based Anomaly Detection with Curriculum Learning in Cyber-physical Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Digital Twin-Enabled Operation Time Analyses |
| Publication Type | Technical reports |
| Year of Publication | 2022 |
| Publisher | Simula Research Laboratory |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty-aware Robustness Assessment of Industrial Elevator Systems
Simula Research Laboratory, 2022.Status: Published
Uncertainty-aware Robustness Assessment of Industrial Elevator Systems
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Design-Operation Continuum Methods for Testing and Deployment under Unforeseen Conditions for Cyber-Physical Systems of Systems |
| Publication Type | Technical reports |
| Year of Publication | 2022 |
| Publisher | Simula Research Laboratory |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Journal Article
Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their Collisions
IEEE Transactions on Software Engineering 49, no. 1 (2022): 384-402.Status: Published
Learning Configurations of Operating Environment of Autonomous Vehicles to Maximize their Collisions
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | IEEE Transactions on Software Engineering |
| Volume | 49 |
| Issue | 1 |
| Pagination | 384-402 |
| Publisher | IEEE |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Multi-Objective Search-Based Software Microbenchmark Prioritization
arXiv (2022).Status: Submitted
Multi-Objective Search-Based Software Microbenchmark Prioritization
Ensuring that software performance does not degrade after a code change is paramount. A potential solution, particularly for libraries and frameworks, is regularly executing software microbenchmarks, a performance testing technique similar to (functional) unit tests. This often becomes infeasible due to the extensive runtimes of microbenchmark suites, however. To address that challenge, research has investigated regression testing techniques, such as test case prioritization (TCP), which reorder the execution within a microbenchmark suite to detect larger performance changes sooner. Such techniques are either designed for unit tests and perform sub-par on microbenchmarks or require complex performance models, reducing their potential application drastically. In this paper, we propose a search-based technique based on multi-objective evolutionary algorithms (MOEAs) to improve the current state of microbenchmark prioritization. The technique utilizes three objectives, i.e., coverage to maximize, coverage overlap to minimize, and historical performance change detection to maximize. We find that our technique improves over the best coverage-based, greedy baselines in terms of average percentage of fault-detection on performance (APFD-P) and Top-3 effectiveness by 26 percentage points (pp) and 43 pp (for Additional) and 17 pp and 32 pp (for Total) to 0.77 and 0.24, respectively. Employing the Indicator-Based Evolutionary Algorithm (IBEA) as MOEA leads to the best effectiveness among six MOEAs. Finally, the technique's runtime overhead is acceptable at 19% of the overall benchmark suite runtime, if we consider the enormous runtimes often spanning multiple hours. The added overhead compared to the greedy baselines is miniscule at 1%.These results mark a step forward for universally applicable performance regression testing techniques.
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | arXiv |
| Publisher | arXiv |
| Keywords | JMH, Multi-objective optimization, Performance Testing, Regression testing, Search-Based Software Engineering, software microbenchmarking, Test Case Prioritization |
| URL | https://arxiv.org/abs/2211.13525 |
| DOI | 10.48550/ARXIV.2211.13525 |
I am a Postdoctoral Fellow working on the innovation project "AIT4CR: AI-Powered Testing Infrastructure for Cancer Registry System", in collaboration with the Cancer Registry of Norway.
Prior to Simula, I was a PhD student at University of Zurich, where I received my doctoral degree in April 2021, defending my dissertation "Deliberate Microbenchmarking of Software Systems".
Originally, I am from Vienna, Austria.
On the Preferences of Quality Indicators for Multi-Objective Search Algorithms in Search-Based Software Engineering
The Empirical Software Engineering Journal (EMSE) 27, no. 6 (2022): 144.Status: Published
On the Preferences of Quality Indicators for Multi-Objective Search Algorithms in Search-Based Software Engineering
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | The Empirical Software Engineering Journal (EMSE) |
| Volume | 27 |
| Issue | 6 |
| Pagination | 144 |
| Publisher | Springer |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Quantum Software Testing: Challenges, Early Achievements, and Opportunities
the European Research Consortium for Informatics and Mathematics News (2022).Status: Published
Quantum Software Testing: Challenges, Early Achievements, and Opportunities
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Enabling Future Dependable Ubiquitous Services and Data with Novel Testing Methods for Quantum Programs |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | the European Research Consortium for Informatics and Mathematics News |
| Publisher | ERCIM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664
Uncertainty-Aware Prediction Validator in Deep Learning Models for Cyber-Physical System Data
ACM Transactions on Software Engineering and Methodology (TOSEM) 31 (2022): 1-31.Status: Published
Uncertainty-Aware Prediction Validator in Deep Learning Models for Cyber-Physical System Data
| Afilliation | Software Engineering |
| Project(s) | Department of Engineering Complex Software Systems, Co-Evolver: Uncertainty-Aware Coevolution Design of Self-Adaptive Cyber-Physical Systems |
| Publication Type | Journal Article |
| Year of Publication | 2022 |
| Journal | ACM Transactions on Software Engineering and Methodology (TOSEM) |
| Volume | 31 |
| Number | 4 |
| Pagination | 1-31 |
| Publisher | ACM |
Consider publishing in the Empirical Software Engineering journal:
https://www.springer.com/journal/10664