IT investments in the public sector are large, and it is essential that they lead to benefits for the organizations themselves and for the wider society. While there is evidence suggesting a positive connection between the existence of benefits management practices and benefits realization, less is known about how to implement such practices effectively. The paper aims to provide insights into when benefits are most likely to be realized, and how benefits management practices and roles should be implemented in order to have a positive effect on the projects’ success in terms of realizing benefits. The authors collected data relating to ten Norwegian public IT projects. For each project, they collected data on benefits management from project documents, by interviewing the project owners and benefits owners, and follow-up surveys. The benefits internal to the organization were those with the highest degree of realization, while the societal benefits were those with the lowest degree. Projects assessed to have more specific, measurable, accountable, and realistically planned benefits were more successful in realizing benefits. Benefits owners were most effective when they were able to attract attention towards the benefits to be realized, had a strong mandate, and had domain expertise.

Context: To ensure the success of software projects, it is essential to select skilled developers and to use suitable work contracts. Objective: This study tests two hypotheses: (i) the use of work-sample testing (trialsourcing) improves the selection of skilled software developers; and (ii) the use of contracts based on hourly payment leads to better software project outcomes than fixed-price contracts. Method: Fifty-seven software freelancers with relevant experience and good evaluation scores from previous clients were invited to complete a two-hour long trialsourcing task to qualify for a software development project. Thirty-six developers completed the trialsourcing task with acceptable performance, and, based on a stratified allocation process, were asked to give a proposal based on an hourly payment or a fixed-price contract. Eight hourly payment-based and eight fixed-priced proposals were accepted. The process and product characteristics of the completion of these 16 projects were collected and analysed. Results and Conclusion: Only partial support for our hypotheses was observed. While the use of trialsourcing may have prevented the selection of developers with insufficient skills, the performance on the trialsourcing task of the selected developers did not predict performance on the project. The use of hourly payments led to lower costs than fixed-price contracts, but not to improved processes or products. We plan to follow up these, to us unexpected, results with research on how to design more skill-predictive trialsourcing tasks, and when and why different project contexts give different contract consequences.

There are large skill differences among software developers, and clients and managers will benefit from being able to identify those with better skill. This study examines the relations between low effort estimates, and other commonly used skill indicators, and measured programming skill. One hundred and four professional software developers were recruited. After skill-related information was collected, they were asked to estimate the effort for four larger and five smaller programming tasks. Finally, they completed a programming skill test. The lowest and most over-optimistic effort estimates for the larger tasks were given by those with the lowest programming skill, which is in accordance with the well-known Dunning-Kruger effect. For the smaller tasks, however, those with the lowest programming skill had the highest and most over-pessimistic estimates. The other programming skill indicators, such as length of experience, company assessed skill and self-assessed skill, were only moderately correlated with measured skill and not particularly useful in guiding developer skill identification. A practical implication is that for larger and more complex tasks, the use of low effort estimates and commonly used skill indicators as selection criteria leads to a substantial risk of selecting among the least skilled developers.

Context: Probabilistic effort estimates inform about the uncertainty and may give useful input to plans, budgets and investment analyses. Objective & Method: This paper introduces, motivates and illustrates two principles on how to evaluate the accuracy and other performance criteria of probabilistic effort estimates in software development contexts. Results: The first principle emphasizes a consistency between the estimation error measure and the loss function of the chosen type of probabilistic single point effort estimates. The second principle points at the importance of not just measuring calibration, but also informativeness of estimated prediction intervals and distributions. The relevance of the evaluation principles is illustrated by a performance evaluation of estimates from twenty-eight software professionals using two different uncertainty assessment methods to estimate the effort of the same thirty software maintenance tasks.

CONTEXT: The role of expert judgement is essential in our quest to improve software project planning and execution. However, its accuracy is dependent on many factors, not least the avoidance of judgement biases, such as the anchoring bias, arising from being influenced by initial information, even when it's misleading or irrelevant. This strong effect is widely documented. OBJECTIVE: We aimed to replicate this anchoring bias using professionals and, novel in a software engineering context, explore de-biasing interventions through increasing knowledge and awareness of judgement biases. METHOD: We ran two series of experiments in company settings with a total of 410 software developers. Some developers took part in a workshop to heighten their awareness of a range of cognitive biases, including anchoring. Later, the anchoring bias was induced by presenting low or high productivity values, followed by the participants' estimates of their own project productivity. Our hypothesis was that the workshop would lead to reduced bias, i.e., work as a de-biasing intervention. RESULTS: The anchors had a large effect (robust Cohen's d=1.19) in influencing estimates. This was substantially reduced in those participants who attended the workshop (robust Cohen's d=0.72). The reduced bias related mainly to the high anchor. The de-biasing intervention also led to a threefold reduction in estimate variance. CONCLUSIONS: The impact of anchors upon judgement was substantial. Learning about judgement biases does appear capable of mitigating, although not removing, the anchoring bias. The positive effect of de-biasing through learning about biases suggests that it has value.

The study examines connections between the outcome of software development projects and contract type, project size, benefits management practices, and agile practices. The collected information relates to 146 software projects—half from 2007-2012 and the other half from 2015-2017. Binary logistic models built to explain successful and problematic software development projects revealed that the use of time and materials contracts, as opposed to fixed price contracts, was associated with fourteen times higher likelihood of project success. Other practices with a strong connection to project outcome included frequent delivery to production and benefit management during project execution. We found that the explanatory strength of a practice sometimes differs substantially in models of successful and in models of problematic projects, as well as over time.

This paper reports two empirical studies on how the use of different contract types affects the outcomes of software projects. The first study evaluates the effect of contract type on project failure using information from a large international dataset of small-scale, outsourced software projects and tasks. The second study proposes and tests how the use of contracts is connected with project outcome using information about Norwegian software projects with a public client. Both studies find that the use of fixed price contracts is connected with a higher risk of project failure compared to time and materials types of contracts. The results from the second study suggest that different project outcomes with different contract types is explained by differences in how the provider is selected, how the client is involved in the project, the use of agile practices and the use of benefit management during project execution.

Context. Each year the public sector invests large amounts of money in the development and modifications of their software systems. These investments are not always successful and many public sector software projects fail to deliver the expected benefits. Goal. This study aims at reducing the waste of resources on failed software projects through better understanding of the success factors and challenges. Method.Thirty-five completed software projects in 11 organizations in the public sector of Norway were analyzed. For each project, representatives from the project owners, project management and the user organization were interviewed. Results. Small and large software projects reported different challenges, especially related to project priority. Taking advantage of agile practices such as flexible scope and frequent delivery increased the success rate of the projects. Projects with time and material contracts and involved clients during execution were more successful than other projects. The respondents experienced that extensive involvement and good competence of the client, high priority of the project, good dialogue between client and provider and appliance of agile practices were main success factors. Main challenges were related to technical issues, project planning and management, transition of the product to the user organization, involvement and competence of the client, and benefit management. Conclusions. Success factors tend to focus on human factors, e.g., involvement, competence and collaboration. Challenges focus on human factors as well as issues of technical nature. Both aspects need to be addressed to enable successful and avoid failed software projects. Competence, client involvement and benefit management are among factors that the public sector should focus on for realizing client benefits.

Numerous research studies and consultancy reports make claims about how often, or rather how seldom, software projects are successful, why so many of them fail, and how to succeed more often. These studies and reports have reported very much the same success and failure factors and the same advices since the 1960s. If we already know how to make a successful software project, why is the proportion of failed software projects about the same as earlier? Are software professionals ignorant of the published knowledge or are there other reasons? One reason may be that previous studies have had very little focus on the most important success dimension, i.e., client benefits, contain very little practical advice on how to succeed, and have not managed to include the context-dependency and complexity of the connections between process choices and outcome. In this keynote I present recent attempts to better define and operationalize project success and better analyse and describe the context-dependent and probabilistic network of connections between essential choices made early and the outcome of software projects. The hope is that this may inspire other researchers to emphasise research on software project management that results in more context-aware results of practical use for software professionals.

Context: A large waste of resources in software development projects currently results from being unable to produce client benefits.

Objective: The main objective is to better understand the characteristics of successful software projects and contribute to software projects that are more likely to produce the planned client benefits.

Method: We asked 63 Norwegian software professionals, representing both the client and the provider role, to report information about their last completed project. In a follow-up survey with 64 Norwegian software professionals, we addressed selected findings from the first survey.

Results: The analysis of the project information showed the following: i) The project management triangle criteria of being on time, on budget, and having the specified functionality are poor correlates of the essential success dimension client benefits. ii) Benefit management planning before the project started and benefit management activities during project execution were connected with success in delivering client benefits. iii) Fixed-price projects and projects in which the selection of providers had a strong focus on low price were less successful in delivering project benefits than other projects. iv) Agile projects were in general more successful than other projects, but agile projects without flexible scope to reflect changed user needs and learning, or without frequent delivery to the client, had less than average success in delivering client benefits.

Conclusions: The software projects that were successful in delivering client benefits differed from the less successful ones in several ways. In particular, they applied benefit management practices during project execution, they avoided fixed-price contracts, they had less focus on low price in the selection of providers, and they applied the core agile practices - frequent delivery to the client and scope flexibility.

Abstract—Confirmation bias is the human tendency to search for, collect, interpret, analyse, or recall information in a way that confirms one’s prior beliefs or preferences. In this paper, we review previous research and demonstrate confirmation bias and its effect in two software engineering contexts. The first study documents that managers bias their interpretation of randomly generated project data towards confirmation of their preferred contract type. The second study reports from an analysis of the results of 35 published comparisons of regression and analogy-based cost estimation models. Twenty of these comparisons evaluate the performance of a self-developed analogy-based estimation model relative to a regression-based model and may be biased towards finding evidence confirming a better accuracy of their own model. A statistical meta-analysis of all 35 comparisons showed that the analogy-based models were more accurate than the regression-based models, whereas removing the 20 comparisons where the researchers evaluated their own models gave the opposite result. Our results support the idea that it is important to consider the presence and degree of confirmation bias when collecting information about topics where there are prior preferences, beliefs, or vested interests to avoid misleading conclusions.

The meaning of an effort or cost estimate should be understood and communicated consistently and clearly to avoid planning and budgeting mistakes. Results from two studies, one of 42 software companies and one of 423 individual software developers, suggest that this is far from being the case. In both studies we found a large variety in what was meant by an effort estimate and that the meaning was frequently not communicated. To improve the planning and budgeting of software projects we recommend that the meaning of effort estimates is understood and communicated using a probability-based terminology.

The presented study aims at a better understanding of when and why software projects fail. The analysis is based on a data set of 785,325 small-scale outsourced software projects. A binary logistic regression model relying solely on information known at the time of a project's start-up correctly predicted 74% of the project failures and 67% of the non-failures. The model-predicted failure probability corresponded well with the actual frequencies of failures for most levels of failure risk. The model suggests that the factors connected to the strongest reduction in the risk of failure are related to previous collaboration between the client and the provider and a low failure rate of previous projects completed by the provider. We found the characteristics of the client to be almost as important as those of the provider in explaining project failures and that the risk of project failure increased with an increased client emphasis on low price and with an increased project size. The identified relationships seem to be stable across the studied project size categories, which indicate that the results may potentially be applicable to larger projects than the small-scale projects dominating this data set.

The software engineering discipline contains numerous myths and over-simplifications. Some of them may be harmless, but others may lead to inefficient practices and contribute to a fashion- and myth-based software engineering discipline. In this presentation I give examples of software engineering myths and over-simplifications, discuss how they are created and spread and illustrate how it is possible to base important software engineering decision and practice on available evidence from research, practice and own empirical studies. A move towards evidence-based software engineering requires that software professionals become more critical towards claims, know how to formulate answerable questions, collect and evaluate evidence and use evidence to guide important decisions. This requires training and, not least, a change in mindset. Results on how to do this are presented.