In your day-to-day life, do you typically know everything up front, or do you learn things along the way that tweak your course?
That's a question concerning any path in life - any series of steps, be it growing up, dating, raising kids, riding a bike, creating the atomic bomb, or developing software.
If you're like me, there's a bit of both. I have a plan, and I may have even put significant thought into it. But when it comes to walking it out, there's some bobbing and weaving - discovering details that cause me to make adjustments along the way. In fact, most times those details couldn't be anticipated, and yet they can significantly impact my course of action, or even change my final destination altogether!
Turns out, contrary to popular belief, the art of developing software behaves no differently.
Accuracy is a function of time
While you're eating breakfast on any given work day, can you predict the exact time you'll break for lunch? Even with days of prior experience under your belt, I bet you can't. While being fairly certain lunch will happen around noon, you probably cannot anticipate the interruptions during the day, the new priorities that change your plans, or the fires that could make you miss it entirely.
However, after the first hour at the office, you've seen the lay of the land, and I imagine your estimate of lunch time becomes more accurate. In fact, as you pass midmorning, you may be able to predict lunch time with decent accuracy.
In an endeavor with a large number of variables, your ability to predict outcomes - your accuracy increases as more of those variables are fixed, which generally happens over time.
In the beginning of a software project, I can estimate how long it will take given the information I have in hand. Of course, that estimate is only as accurate as my limited information. (Yes, the information - even a specification - IS limited when compared to what will be learned throughout the project.) My estimate is a ballpark estimate.
Anyone who demands an accurate project timeline up front (e.g. "within plus or minus 10%") must understand: Even tasking the development team to spend a month exercise mapping out every detail produces only a mildly better estimate, and adheres to the law of diminishing returns. There is an accuracy asymptote which cannot be exceeded without actually starting the project.
Smoke and mirrors
On the flip side, anyone who guarantees a non-trivial project's timeline up front within +/- 10% is only attempting to secure a project win in exchange for the inevitable renegotiation of that guarantee later on. Promises now, damage control later.
All initial estimations are ballpark at best. Only when the project is underway do timeline estimates (and thus actual development cost estimates) become more accurate. The adage, "the devil's in the details" is no more true than during a software project, when the once black and white requirements and precise priorities become greyscale and fluid as the actual intentions of the users and business requirements transition from fuzzy outlines into full focus.
As I said earlier, you can spend weeks or even months digging into every detail in an effort to make your timeline estimation more accurate. But at what cost?
Accuracy versus execution
Take a month, and estimate, estimate, estimate. Get as accurate as you can. What did you trade off? For starters, a month of actual work. Secondly, a month's worth of project execution that will yield a more accurate estimation of all the remaining work. Why? Because the initial user stories have already uncovered details, UX has created wireframes, and the development team developed a cadence that demonstrates how much work they can absorb and execute in a given sprint.
In that one month of execution, the timeline estimation is additionally honed, and you're now one month closer to reaching that completion date.
No third way.
In a software development project, as in life, there is much learned along the way. Is that feedback valuable during the process? Most people would agree it is.
There are two options: You can estimate, estimate, estimate and get a mildly more accurate project timeline (which becomes less accurate two weeks into the project thanks to the details and feedback uncovered). Or, you can execute, execute, execute, integrate the feedback along the way, and hone the estimation as the project runs its course.
That's it. There is no third option that produces an accurate timeline up front. Again, as with life and dating and raising kids, it just doesn't work that way.