If you think you have never heard of the “planning fallacy”, you’re wrong. Have you ever undertaken a home improvement project? Consider for example painting a bedroom. You likely came up with a task list that includes removing the furniture, preparing the walls, caulking the cracks, painting the ceiling, then the walls, then the trim…etc. You may have even attempted to lay out a schedule on how long each task would take. Now think about how much elapsed time you thought the entire project would take vs. how long it actually took. Were you overly optimistic? Did you complete the project ahead of schedule? If you are like most of us, you were overly optimistic in estimating the completion date and likely did not consider past projects and how long they actually took. This is an example of the planning fallacy and it can have a significant impact on new product development (NPD) effectiveness.
Of course, in the do-it-yourself world of home improvement, there is not much riding on an accurate schedule estimate. That of course is not true for most industrial project managers, and careers can be helped or harmed based on meeting project schedules. Understanding what the planning fallacy is, how it manifests itself, and what you can do to minimize the impact is an extremely important consideration for senior management as well as project managers and the focus of this article.
There is considerable research on the planning fallacy (1,2) and I draw on that research as well as my own experience in managing an R&D organization at a mid-sized analytical instrument manufacturer. For anyone who has ever managed a project, you will instantly recognize the planning fallacy. For those new to project management, you should take the time to learn how it will impact your success.
A formal definition as described in the references is as follows: “The tendency to hold a confident belief that one’s own project will proceed as planned, even while knowing that the vast majority of similar projects have run late.” In other words, the research supports the fact that we consistently believe we will finish individual tasks sooner then we actually do. What is so fascinating is how all of us can rationalize holding two contradictory positions. We know that we have always been overly optimistic in our past projections, but believe this time will be different. We believe in our own capabilities, that we have learned from past mistakes, and will be able to more readily overcome obstacles this time. In my experience, engineers by their very nature tend to be overly optimistic. A “can-do” organization culture and group-think can compound the problem. Sometimes the skeptic voice can be labeled as the dreaded “negative thinker”. In most cases, we should base our estimates not only on the actual project tasks at hand, but also on past experiences with similar tasks. The research suggests we rarely do that.
There are other insights from the research. For instance:
1) While most people are overly optimistic in their predictions, deadlines are still a powerful motivator, and optimistic deadlines may serve to motivate.
2) Group discussion can actually lead to more optimistic estimates compared to individual predictions.
3) The relative “cost” of over or underestimation of task lengths influences behavior.
4) When one person is asked to create an estimate for a second person, they are more likely to take in account past experiences and develop a more realistic time estimate, compared to the second person’s projection.
Based on this research and my own experience there are several implications for management and project managers.
First, senior management and project managers have to recognize the mere existence of the planning fallacy and how it influences behavior. For instance, given that most estimates are optimistic to begin with, if management constantly takes already optimistic deadlines and forces them earlier, the new deadlines may become completely unrealistic and de-motivating. In the future, these same employees will pad estimates even further to game the system. Human nature being what it is, the task size will take on this bloated estimate, and you may miss an opportunity to speed up the project. Tasks expanding to the deadline is known as Parkinson’s Law.
This relates nicely to the concept of critical chain scheduling (3). In this methodology, task sizes are estimated at a 50% confidence level: half the time the task will take longer and half the time it will take less time. In other words, you drive out the padding and you maintain aggressive deadlines. You compensate by adding a project buffer to the end of the project to protect the critical chain (4) and feeding buffers for tasks that feed the critical chain. For this method to work, however, you have to change your thinking. First, you have to focus the resources to start the tasks when required, especially those on the critical chain, and ensure those resources are not suddenly diverted to other tasks and projects. Second you need to reward early completion of tasks, and not penalize late delivery.
Second, in coming up with task size estimates to begin with, it’s better getting multiple opinions for those tasks on the critical chain. Since meeting as a group has shown to produce even more optimistic projections, a project manager should independently ask several senior employees how long they believe tasks will take. In the context of coming up with the 50% confidence level for the task size, you could ask for a range of times from best case, most likely, and worst case.
A third aspect is looking back at how long similar tasks took, if the data exists. For the tasks on the critical chain, it may be worth the manager’s time doing this. One problem will always be the fact that in many instances, some key project tasks have never been done before, so there is always some level of uncertainty. I believe in most organizations, over time, experienced project managers will have a sense for the right amount of project buffer necessary for any individual project. There should be other important factors considered above and beyond the development tasks themselves. For instance, if the technology is new to the company, engineers may need time to become proficient. Another factor may be distributed teams, especially if this is the first time that has been attempted in the organization. Those other factors need to be considered as well in determining the size of project buffers.
A final thought relates to projects with high levels of complexity and/or unforeseeable uncertainty, or “unk unks”. Critical chain and other standard project management techniques work well on projects with standard variation and foreseeable uncertainty. Projects with high levels of complexity and “unk unks” will require other project management techniques such as iterate-and-learn and/or parallel trials to reduce uncertainty. Understanding the impact of the planning fallacy on the task size estimates, the target dates for completing key milestones, and the impact on the overall schedule in these projects will be critically important. Many complex projects and those with “unk unks” have schedules that are unrealistic from the very beginning that lead to much organization consternation and disappointment when the original completion date is missed, or functionality has to be scaled back to the meet the deadline.
Have you ever heard of the planning fallacy? Do you have examples of how it manifested itself in your projects? How have you dealt with it?
(1) Buehler, R., Griffin, D. and Ross, M. 1994. Exploring the “Planning Fallacy”: Why People Underestimate Their Task Completion Times. Journal of Personality and Social Psychology 67(3):366-381.
(2) Buehler, R., Messervey, D. and Griffin, D. 2005. Collaborative Planning and Prediction: Does Group Discussion Affect Optimistic Biases in Time Estimation? Organizational Behavior and Human Decision Processes 97:47-63.
(3) See for instance: Eliyahu M. Goldratt, Critical Chain. (Great Barrington, MA: The North River Press, 1997) and also Critical Chain Scheduling and Buffer Management: Getting Out From Between Parkinson’s Rock and Murphy’s Hard Places. Francis S. Patrick. 1998. http://www.focusedperformance.com.
(4) The critical chain is the same as the critical path, but modified to remove resource conflicts.
New Product Visions is a consulting company that helps organizations improve the effectiveness of their new product development processes. We specialize in small to mid-sized companies that manufacture highly engineered products. Contact us today about how we might help you!
Specialties: NPD consultants, new product development consulting, developing new products, new product development seminars, small business consulting, new product development expert, product development process, new product development strategies, integrating NPD for mergers & acquisitions, organizing for innovation, management role in NPD, project risk analysis