Evaluation of cost estimates should be fair and give incentives for accuracy. These goals, we argue, are challenged by a lack of precision in what is meant by a cost estimate and the use of evaluation measures that do not reward the most accurate cost estimates. To improve the situation, we suggest the use of probabilistic cost estimates and propose guidelines on how to evaluate such estimates. The guidelines emphasize the importance of a match between the type of cost estimate provided by the estimators and the chosen cost evaluation measure, and the need for an evaluation of both the calibration and the informativeness of probabilistic cost estimates. The feasibility of the guidelines is exemplified in an analysis of a set of 69 large Norwegian governmental projects. The evaluation indicated that the projects had quite accurate and unbiased P50 estimates and that the prediction intervals were reasonably well calibrated. It also showed that the cost prediction intervals were non-informative with respect to differences in cost uncertainty and, consequently, not useful to identify projects with higher cost uncertainty. The results demonstrate the usefulness of applying the proposed cost estimation evaluation guidelines.

Purpose: This paper examines to what extent resource managers are likely to make normatively correct decisions in complex, but realistic, software development team resource management situations.

Research design: Three scenarios were designed, all of them with a software development project consisting of a higher and a lower productivity team. The resource managers were asked to decide which team to increase (or decrease) the productivity by 10% in order to minimize the total effort or the total duration of the software project. Ninety-nine software professionals, most of them with substantial resource management experience, were randomly allocated one of the scenarios.

Findings: Most of the resource managers perceived their scenario as realistic and occurring in practice. The decisions tended, however, to be different from the normatively correct ones. In particular, when the scenario structure resembled that used to document a time-saving bias, the great majority of the managers made non-normative decisions. The findings suggest that the resource managers made decisions based on simple heuristics, often leading to non-normative decisions. When asked about the decisions they would make in practice, more of the resource managers gave normatively correct responses. Our findings suggest a cost-saving potential from more awareness of how to make team resource management decisions.

Originality: The study may be the first to document non-normative team resource decisions, including those related to the time-saving bias, in the context of project management.