Context Empirical research in software engineering contributes towards developing scientific knowledge in this field, which in turn is relevant to inform decision-making in industry. A number of empirical studies have been carried out to date in software engineering, and the need for guidelines for conducting and evaluating such research has been stressed.

Objective: The main goal of this mapping study is to identify and summarize the body of knowledge on research guidelines, assessment instruments and knowledge organization systems on how to conduct and evaluate empirical research in software engineering.

Method: A systematic mapping study employing manual search and snowballing techniques was carried out to identify the suitable papers. To build up the catalog, we extracted and categorized information provided by the identified papers.

Results: The mapping study comprises a list of 341 methodological papers, classified according to research methods, research phases covered, and type of instrument provided. Later, we derived a brief explanatory review of the instruments provided for each of the research methods.

Conclusion: We provide: an aggregated body of knowledge on the state of the art relating to guidelines, assessment instruments and knowledge organization systems for carrying out empirical software engineering research; an exemplary usage scenario that can be used to guide those carrying out such studies is also provided. Finally, we discuss the catalog’s implications for research practice and the needs for further research.

Context: Electroencephalograms (EEG) have been used in a few cases in the context of software engineering (SE). EEGs allow capturing emotions and cognitive functioning. Such human factors have already shown to be important to understand software engineering tasks. Therefore, it is essential to gain experience in the community to utilize EEG as a research tool. Objective: To report experiences of using EEG in the context of a software engineering education (review of master theses proposals). We provide our reflections and lessons learned of (1) how to plan an EEG study, (2) how to conduct and execute (e.g., tools), (3) how to analyze. Method: We carried out an experiment using an EEG headset to measure the participants' attention rate. The experiment task includes reviewing three master thesis project plans. Results: We describe how we evolved our understanding of experimentation practices to collect and analyze psychological and cognitive data. We also provide a set of lessons learned regarding the application of EEG technology for research. Conclusions: We believe that that EEG could benefit software engineering research to collect cognitive information under certain conditions. The lessons learned reported here should be used as inputs for future experiments in software engineering, where human aspects are of interest.

The importance of achieving high quality in research practice has been highlighted in different disciplines. At the same time, citations are utilized to measure the impact of academic researchers and institutions. One open question is whether the quality in the reporting of research is related to scientific impact, which would be desired. In this exploratory study we aim to: (1) Investigate how consistently a scoring rubric for rigor and relevance has been used to assess research quality of software engineering studies; (2) Explore the relationship between rigor, relevance and citation count. Through backward snowball sampling we identified 718 primary studies assessed through the scoring rubric. We utilized cluster analysis and conditional inference tree to explore the relationship between quality in the reporting of research (represented by rigor and relevance) and scientiometrics (represented by normalized citations). The results show that only rigor is related to studies’ normalized citations. Besides that, confounding factors are likely to influence the number of citations. The results also suggest that the scoring rubric is not applied the same way by all studies, and one of the likely reasons is because it was found to be too abstract and in need to be further refined. Our findings could be used as a basis to further understand the relation between the quality in the reporting of research and scientific impact, and foster new discussions on how to fairly acknowledge studies for performing well with respect to the emphasized research quality. Furthermore, we highlighted the need to further improve the scoring rubric.

Context: Collaboration depends on communication and upon having a similar understanding of the notions that are being discussed, and a similar appraisal of their value. Existing work seems to show that the collaboration between industry and academia is hampered by a difference in values. In particular, academic work focuses more on generalizing on the basis of existing evidence, while industry prefers to particularize conclusions to individual cases. This has lead to the conclusion that industry values scientific evidence less than academia does.
Objective: This paper seeks to re-evaluate that conclusion, and investigate if industry and academia share a definition of scientific evidence. If evidence can be found of competing views, we propose a more finely grained model of empirical evidence and its role in building software engineering knowledge. Moreover, we seek to determine if a more nuanced look the notion of scientific evidence has an influence on how academics and industry practitioners perceive that notion.
Method: We have developed a model of key concepts related to under- standing empirical evidence in software engineering. An initial validation has been conducted, consisting of a survey of master students, to deter- mine if competing views of evidence exist at that level. The model will be validated by further literature study and semi- structured interviews with industry practitioners.
Results: We propose a model of empirical evidence in software engineering, and an initial validation of that model by means of a survey. The results of the survey indicate that conflicting opinions already exist in the student body regarding the notion of evidence, how trustworthy different sources of evidence and knowledge are, and which sources of evidence and types of evidence are more appropriate in various situations. Conclusion: Rather than a difference in how industry and academia value scientific evidence, we see evidence of misunderstanding, of different notions of what constitutes scientific evidence and what strength of evidence is required to achieve specific goals. We propose a model of empirical evidence, to provide a better understanding of what is required in various situations and a better platform for communication between industry and academia.