If software developers’ estimates of how much effort they would need to complete tasks were strongly correlated with their actual use of efforts, we could safely select among the developers with the lowest effort estimates and predict that they will be among the most skilled ones. Unfortunately, this is not necessarily the case. As documented in studies in many domains, those with lower skill tend to know less about how little they know and, for this reason, give over-optimistic estimates of their own performance.

This effect is termed the Dunning-Kruger effect. Does this mean that effort estimates are useless as predictors of software development skill?

To find out more on this we requested 104 software developers to estimate four larger and five smaller software development tasks and measured their programming skills. We assessed the connection between lower estimates and higher skill through rank correlations and hit rates, where the hit rate measures the frequency of selecting the most skilled out of two developers when selecting the one with the lower effort estimate on the same task.

The results were as predicted by the Dunning-Kruger effect for the larger tasks. The developers in the lowest skill quartile had on average lower estimates than those in the highest skill quartile. To predict relative programming skill based on these estimates would be very inaccurate. The correlations between task estimates and skill were between -0.15 and 0.01 and the hit rates between 44% and 50%.

The developers’ effort estimates on the smaller tasks were, however, much better connected with their measured programming skills. The rank correlations were between 0.29 and 0.50 and the hit rates between 62% and 68%. Clustering the estimates of all the smaller tasks into the same factor gave a rank correlation of 0.45 between this factor and the measured skill and a hit rate of 70%. While not very strong, we show that the estimates on these tasks is better connected with measured skill than those in use in software development contexts, i.e., company-assesses skill category, length of experience, self-assessed skill and confidence in knowing how to solve the tasks. These indicators had rank correlations between 0.14 and 0.29, and hit rates between 57% and 65%.

A possible explanation of why lower skill was connected with higher effort estimates for the smaller, but not for the larger tasks, is that lower skill in solving a task can both lead to lower skill in identifying the complexity and the simplicity of a task. In the situation with the simpler tasks, we propose, those with better skill were also better in identifying how easily the tasks could be solved at the time of estimation. This explanation suggests that we may be able to design estimation tasks even better at separating those with high and low task completion skill than in our study, when focusing on hidden simplicity, i.e., simplicity that the more skilled are more likely to identify than the less skilled ones.

Context: Electroencephalograms (EEG) have been used in a few cases in the context of software engineering (SE). EEGs allow capturing emotions and cognitive functioning. Such human factors have already shown to be important to understand software engineering tasks. Therefore, it is essential to gain experience in the community to utilize EEG as a research tool. Objective: To report experiences of using EEG in the context of a software engineering education (review of master theses proposals). We provide our reflections and lessons learned of (1) how to plan an EEG study, (2) how to conduct and execute (e.g., tools), (3) how to analyze. Method: We carried out an experiment using an EEG headset to measure the participants' attention rate. The experiment task includes reviewing three master thesis project plans. Results: We describe how we evolved our understanding of experimentation practices to collect and analyze psychological and cognitive data. We also provide a set of lessons learned regarding the application of EEG technology for research. Conclusions: We believe that that EEG could benefit software engineering research to collect cognitive information under certain conditions. The lessons learned reported here should be used as inputs for future experiments in software engineering, where human aspects are of interest.