Excerpts from “Software Estimation: Demystifying the Black Art”
Steve McConnell’s book “Software Estimation: Demystifying the Black Art” is pure gold. In this post, I list some memorable excerpts and my notes from the book. Many of them are direct citations from the book, © Steve McConnell and Microsoft Press.
An estimate is not one number: it’s a range or a probability distribution.
“When you are asked to provide an estimate, determine whether you are supposed to be estimating or figuring out how to hit a target.”
A single number is likely a target, not an estimate.
“The primary purpose of software estimation is not to predict a project’s outcome; it is to determine whether a project’s targets are realistic enough to allow the project to be controlled to meet them.”
“A good estimate provides a clear enough view of the project reality to allow the project leadership to make good decisions about how to control the project to hit its targets.”
Don’t provide “percentage confident” estimates (esp. 90%) unless you have a quantitatively derived basis for doing so.
“We believe wide ranges make us appear ignorant or incompetent. The opposite is usually the case.”
Much better to overestimate than to underestimate (linear vs. non-linear cost).
“About 40% of all software errors have been found to be caused by stress; those errors could have been avoided by scheduling appropriately or by placing less stress on the developers (Glass 1994).”
“Recognize a mismatch between a project’s business target and project’s estimate as valuable risk information and insist on taking corrective action.”
Predictability (CMM) is usually more valuable for business than development time, cost, and flexibility (Agile).
The accuracy of an estimate depends on the level of refinement of the software’s definition. it’s impossible to spend more time to produce a better estimate of a project that is still vaguely defined.
Decide what will you not do early in the project.
“How much” and “how uncertain/probable” are two different estimation skills.
Don’t make commitments early in the project, before having a refined definition.
Plan increase of requirements (for example, by 40–50%).
Don’t forget about implied requirements, supporting (glue) requirements, and non-functional requirements when doing estimates.
Include all activities in estimates (code review, operations/support of the existing systems, bug fixing, mentoring/rump up of new team members, learning of new tools, etc.), not just coding and testing.
On projects lasting longer than a few weeks, account for sick days and vacations in estimates.
Engineer estimates are optimistic by 20–30%.
The more control knobs the worse (applies to anything).
Don’t try to make off-the-cuff estimates. Even a 15-minute estimate will be more accurate.
Match precision and accuracy of estimates.
Invest time to estimate the size of the software to be built. It translates (super-linearly) to project effort and time.
“Use estimation tools to compute the impact of diseconomies of scale.” (Accountable for the super-linear growth.)
“Good requirements capability makes a factor of 2 difference in the effort for the entire project.”
Small and medium-sized projects can succeed largely thanks to strong individuals. In large projects, the management (especially risk management), the maturity of processes and how well people coalesce in a team is as much important.
“The best estimation techniques for small projects tend to be “bottom-up” based on estimates created by people who will actually do the work.”
“When choosing estimation techniques, consider what you want to estimate, the size of the project, the project stage (early — middle — late), the development style (sequential — iterative), and the accuracy you need (early in the project low accuracy may be OK due to Cone of Uncertainty).”
“Count (requirements, features, tickets, bugs, classes, methods, LOC, etc.) if possible. Compute when you can’t count. Use judgment alone only as a last resort.”
Use historical data for productivity assumptions. Productivity cannot easily be varied within an organization from project to project.
Collect project’s and performance data as soon as possible.
Even if you don’t have historical data from past projects, you can collect data from the current project and use it to estimate the remainder of the project.
“Increased experience in the activity being estimated doesn’t lead to increased accuracy in the estimates for the activity (Jørgensen 2002).”
“Structured” judgment is much more accurate than “intuitive” judgment.
Separating large tasks into smaller ones improves the accuracy of estimates. Decompose estimates into tasks that will require no more than 2 days of effort.
Estimate in ranges: worst case, most likely case, best case for a task.
Compute expected case using the PERT formula: expectedCase = (bestCase + 4 * mostLikelyCase + worstCase) / 6.
The pessimistic version of the formula is expectedCase = (bestCase + 3 * mostLikelyCase + 2 * worstCase) / 6.
“Even experts occasionally forget to consider everything they should. Studies of forecasting in a variety of disciplines have found that simple checklists help improve accuracy.”
Compare your actual results to your estimates to refine your estimating ability.
Compose an estimate of a large task or a project of estimates of subtasks to benefit from the Law of Large Numbers: errors on the high and low sides cancel each other.
Use a generic software project work breakdown structure (WBS) to avoid forgetting some activities.
“Use simple standard deviation formula:sd = (sumOfWorstCaseEstimates — sumOfBestCaseEstimates) / 6
for estimates aggregating 10 tasks or fewer.”
Focus on making task estimates more accurate, especially their expected cases: they should be unbiased, in other words, you overrun them as often as underrun.
“An estimate should be focused on accuracy, not conservatism.” Once you move your focus away from accuracy, bias can creep in many different factors and the final estimate will be too far from accurate.
“The best estimation response to uncertainty is not to bias the estimate but to ensure that estimate accurately expresses any underlying uncertainty.” (By widening best/worst cases and/or increasing the variance of the expected case.)
“The uncertainty of estimate will flow through to the project plans and commitments. Because the focus of plans and commitments is performance, not accuracy, it’s appropriate to adjust your commitments in a conservative direction, based on the uncertainty of the underlying estimate.”
If you have at least 20 features to estimate in LOC or effort, use fuzzy logic:
- Classify features as Very Small, Small, Medium, Large and Very Large (with at least factor of 2 difference between categories);
- Estimate past features to get average LOC per feature size;
- Apply to the classified set of features to estimate.
If you develop software consisting mostly of standard components (report, dialog, business rule, web page, etc.) they can be used for classification instead of feature size. Numbers of standard components can be estimated themselves using min/mostLikely/max numbers and the PERT formula.
“Use story points to obtain an early estimate of an iterative project’s effort and schedule that is based on data from the same project.” However, great accuracy is discipline is required to ensure that story points’ numerical categories in the scale actually work like numbers, not like verbal categories such as small, medium, and large.
“In software, a quick “No” answer has great value. T-shirt sizing allows for early-in-the-project decisions to rule out features so that you don’t need to carry those features further into the Cone of Uncertainty.”
Make estimates separately with colleagues and then compare and discuss results to produce a consensus estimate.
“Use multiple estimation techniques, and look for convergence or spread among the results.”
Don’t debate the output of an estimate, take it as a given. “Project stakeholders might not like the output, but the appropriate corrective action is to adjust the inputs (for example, reduce the project’s scope) and to recalculate the outputs, not just to change the output to a different answer.”
“Focus on estimating size first. Then compute effort, schedule, and features (fitting within certain schedule) from the size estimate.”
“As soon as specific people who will be working on the project are known and assigned tasks (or groups of tasks), it’s time to switch from algorithmic to bottom-up estimation approaches when individuals estimate the tasks they are going to work on.”
Present your estimates in ranges, that allows you to tighten up the estimate as you move further in the project.
“Once the project has been completed, review the project’s estimation history to determine whether the project’s estimates anticipated the project’s eventual outcome.”
“Effort per line of code is more a function of project size and kind of software than a programming language. (What you get for each line of code will vary dramatically, depending on the programming language.)”
Basic Schedule Equation: ScheduleInMonths = 3.0 * StaffMonths ^ (1/3). Assuming adequate staffing. Works for medium and large projects only.
Informal Comparison to Past Projects Formula:EstimatedSchedule = PastSchedule * (EstimatedEffort / PastEffort) ^ (1/3). 1/2 for small projects.
Nominal schedule cannot be shortened more than 25% regardless of how much more people work on the project, or existing people working harder or “smarter”.
“For an extended schedule to reduce effort, you must actually reduce the team size. If you simply allocate the same people to the same project fractionally instead of reducing the number of people of the team, you’ll likely make matters worse instead of better.”
There are optimal team sizes for projects beyond which both schedule and effort increase.
“Use schedule estimation to ensure your plans are plausible. Use detailed planning to produce a final schedule.”
The bigger the project the more effort should be allocated to requirements work, architecture, planning, design, and testing, compared to construction (coding).
“Does your calibration data account for vacation, holidays, sick days, training time, customer support, production systems support, and so on?”
“Capers Jones reports that, on average, technical workers apply about 6 focused project time per day or 132 hours per month.” Remark: personally, I don’t find it really possible to sustainably do creative software work (design, coding, writing) for more than 4 hours per day and more than 5 hours of creative + reactive work (code reviews, reading and participating deep technical discussions) in total.
“Defect production is a function of effort and project size.”
A typical project produces about 50 defects per 1000 lines of code (depending on the programming language used). However, the industry-average ranges of defect production vary by more than a factor of 10.
“Use your project’s total Risk Exposure (RE) as the starting point for buffer planning. Review the details of your project’s specific risks to understand whether you should ultimately plan for a buffer that is larger or smaller than the total RE.”
“Allow for 1% to 4% increase in requirements per month (Jones 1998).”
“To go from one-company, one-site development to multi-company, multiple-city development, allow for a 25% increase in effort (Boehm at al 2000).”
“For first-time development with new language and tools compared to comparable development with familiar language and tools, allow for a 20% to 40% increase in effort (Boehm at al 2000).”
Document and communicate the assumptions embedded in your estimate, such as:
— Which features are required
— Which features are not required
— How elaborate certain features need to be
— Major unknowns
— Major influences and sensitivities of the estimate
— The accuracy of the estimate
— What the estimate can be used for
“Be sure you understand whether you’re presenting uncertainty in an estimate or uncertainty that affects your ability to meet a commitment.”
“Project stakeholders might think that presenting an estimate as a wide range makes the estimates useless. What’s really happening is that presentation of the estimate as a wide range accurately conveys the fact that the estimate is useless.”
From IEEE-CS/ACM Software Engineering Code of Ethics:
Software engineers shall ensure that their products and related modifications meet the highest professional standards possible. […] Software engineers shall ensure realistic quantitative estimates of cost, scheduling, personnel, quality and outcomes on any project on which they work and provide an uncertainty assessment of these estimates.
“Including uncertainty isn’t just a nicety, it’s part of professional’s ethical responsibility.”
“The fact that an external requirement exists doesn’t necessarily mean it’s possible to meet that requirement. It does mean that you need to make it perfectly clear to the executives that you understand the requirement and that you take it seriously.”
“You can negotiate the commitment, but don’t negotiate the estimate.”
“Treat estimation discussions as problem solving, not negotiation. Recognize that all project stakeholders are on the same side of the table.”
Use the Principled Negotiation method:
— Separate the people from the problem. Attack the problem, not the people.
— Focus on interests, not positions.
— Invent options for mutual gain.
— Insist on using objective criteria.
You possess technical knowledge, hence it’s your responsibility to generate creative solutions and planning options. Propose a full range of possibilities and tradeoffs.
Build a feature-oriented Cone of Uncertainty. Define some features as “definitely in”, some as “definitely out”, and some as “possible”.
“As you foster an atmosphere of collaborative problem solving, don’t make any commitments based on off-the-cuff estimates.”
