Evaluation of cost estimates should be fair and give incentives for accuracy. These goals, we argue, are challenged by a lack of precision in what is meant by a cost estimate and the use of evaluation measures that do not reward the most accurate cost estimates. To improve the situation, we suggest the use of probabilistic cost estimates and propose guidelines on how to evaluate such estimates. The guidelines emphasize the importance of a match between the type of cost estimate provided by the estimators and the chosen cost evaluation measure, and the need for an evaluation of both the calibration and the informativeness of probabilistic cost estimates. The feasibility of the guidelines is exemplified in an analysis of a set of 69 large Norwegian governmental projects. The evaluation indicated that the projects had quite accurate and unbiased P50 estimates and that the prediction intervals were reasonably well calibrated. It also showed that the cost prediction intervals were non-informative with respect to differences in cost uncertainty and, consequently, not useful to identify projects with higher cost uncertainty. The results demonstrate the usefulness of applying the proposed cost estimation evaluation guidelines.

Context: There are many claims, but not much evidence, about advantages connected with the use of relative instead of absolute estimates.

Objective: We test the claims that people are better at giving relative estimates, that producing relative estimates is faster, and that, relative effort to a large extent is developer-independent.

Method: We conducted two experiments. The first experiment addresses the two first claims and analyses the estimates from 102 professional software developers randomly allocated to providing relative or absolute estimates of software development effort and other quantities. The second experiment addresses the third claim and analyses the actual efforts from 20 professional software developers completing the same five programming tasks.

Results: We found, for most of the estimation tasks, only smaller differences between relative and absolute estimates, which is consistent with a similarity in underlying estimation processes. The time spent completing the estimation work was higher or the same for those using relative estimation. We found a strong developer-dependency in relative use of effort, suggesting that relative estimates are far from developer-independent.

Conclusions: The results give no support to the claims that relative estimation is better, faster or that relative effort to a large extent is developer-independent.