process

Rocks and Stones

How to develop software without wasting time planning or ignoring important parts of the project

What are you working on today? What will you work on tomorrow? Next week? Next month?

I would answer these questions with:

  • today I am working on A and B
  • tomorrow I will probably work on C
  • I don't know what I will work next week, but I would love to find some time to work on X
  • next month? I have no idea

At any time during the day, I can only tell you the 2-3 things that I am currently working on, or plan to work in the nearest future. Everything else is up in the air, and is pretty random. Maybe I will work on feature X next week, but more likely I will work on something else. Maybe I will finish X this week, or maybe something more important will interrupt me, and I will get to X next month. Maybe next year. Maybe never - that's ok too.

Am I a bad developer? Am I bad at planning? Why can't I be more like a plumber or a construction worker and estimate the list of tasks for the next two weeks (and forget planning the next two months, that's impossible!) - is planning even possible?

Some people say no, and hell no, estimation is impossible.

Software is not construction

First, let me address the question of estimation in software development versus other trades. If you work on constructing a house, and this is your first construction job - your estimates are probably going to be waaaay off. If this is your 10th or 100th house, your estimates are probably going to be pretty spot on, if there are no unforeseen circumstances, like a weather disaster (you never know with Climate Emergency), or a construction permit delay.

Thus a construction worker with more experience can estimate the upcoming construction job, since it largely similar to the jobs they have already finished.

The software development is different. Yes, some tasks might be repetitive and can be estimated roughly: if you ask me to set up a continuous integration service for a small GitHub repository, I will estimate it at 10 minutes, since I can copy one of my existing Example CI configs and be done.

By the way, writing blog posts to publicly document common development tasks is an excellent way to become more productive yourself and make every developer around you a little more productive too, as I described in Multiple Benefits post.

But if you do the same task again and again, the software development allows you to perform a trick that is impossible or at least is very hard to pull off in the building construction trade: you can automate it or invent a completely new way to skip the task altogether. You can automate large tasks, or even 10 second ones - it is worth it. This is the creative part of the software development, where you literally can build a world-changing system in your garage, or bedroom, or college dorm, or library room. Do you want to make billions? Then do not let your development team do repetitive tasks! Accept that everything they do is potentially something brand new; something no one has ever done; something that hopefully is a super useful invention. And as a second side of this coin, accept that accurate estimates for tasks no one on your team has done before are thus impossible.

If you cannot live with such ambiguity, leave the software development world and join a construction crew. This is good honest work with tangible results and estimates matching reality (aside from very large projects known for their cost overruns).

Rocks and stones

Rocks and stones, an artist rendering

When we as a team plan a sprint for the next two weeks we naturally consider somewhat larger tasks - features and fixing important bugs. No one wants to spend the spring planning meeting time (which is super expensive: 8 engineers for a 1 hour meeting is a full day of focused work!) discussing how to properly prioritize a two line change to the user documentation:

- I think it is equally important as updating the year in the footer, I say 1 story point

- No, it is slightly more important, but I agree that it is 1 story point

- All in favor ...

Scheduling such large feature "rocks", as I call them, is like placing them in the two-week sprint "bucket". The sprint planning is done when N rocks are in the bucket. Will we finish them? No one knows, since the size of the rock is like Heisenberg quantum uncertainty - you only know the size of the task once it collapses is finished.

By the way, when we estimate the size of each rock, we often ignore the fact that it depends on who is doing it. The developers are not identical replacements, thus the estimate should depend on the person assigned, yet we often estimate first, then assign! At least I have never seen such sprint-planning feature.

So now every developer grabs a "rock" and works on it. What about all other, smaller stuff that our users might need? A small documentation update to clarify its meaning (like this one or this one)? When do we schedule it and how do we prioritize such tasks? How do we effectively argue that this part of documentation is unclear and would slow down our users?

A software product (an application, a CLI tool, a website) potentially has lots and lots of small things that require tiny fixes, content rewrites, error message improvements, etc. We cannot join such updates into larger "rocks" - then our updates would be infrequent and arbitrary. So when and how do we iterate over software, its documentation, its supporting docs like examples, blog posts, webinars, and workshops? Without addressing user pain points, or reacting to the customer complaints our software will lack polish, will forever be "almost good". We as a company would appear unresponsive and slow-moving. Just like building a wall from large rocks only, such approach naturally breaks down after stacking two or three levels of rocks. New England is full of such walls without mortar; they used to mark property boundary, and there was plenty of rocks strewn across the region after the last Ice Age.

🌎 🔥 Speaking of Ice Age - have you considered how the year 2020 is going to be the coldest year in the foreseeable future due to Climate Emergency? Yup, our planet is heating up, and can get into a super scary runaway hothouse scenario unless we stop using all fossil fuels soon.

Stone wall near Boston

A natural solution to prioritizing and scheduling small tasks (I call them "stones") is to let the developers themselves find a good time to fit them into the workday. A small documentation update might take 10 minutes - just enough to fill a time gap between other tasks or meetings. Or as something to code during a boring sprint-planning meeting!

Now a million dollar question:

If we let developers to scope, prioritize, and schedule small tasks, why not let them scope, prioritize, and schedule large tasks too?

After all - the developers working on project X might be the right people to judge the difficulty of every task, their relative priorities, and how to effectively complete as many as possible. I do not say "chaos, do whatever you want". I am saying - try to imitate a mason laying a wall who uses rocks and stones of different sizes depending on the current area in order to fill it efficiently without gaps. If someone can fill the day fully with tasks of various sizes that matter to the users - that's enough for me.

Masterfully built stone arc

Planning vs celebration

We put such emphasis on sprint planning, asking about rough or accurate time estimates, trying to fill every developer's bucket. Letting developers fill their days themselves:

  • would probably fill their days more efficiently
  • would more accurately represent the distribution of task sizes across the project; I strongly believe the task sizes follow Power Law - there are lots of smaller tasks and a few big ones
  • respect the developers' expertise and judgement
  • free time!

So what do we do instead of the sprint planning meeting? We celebrate what we have actually accomplished. Literally. We look back at all the issues closed during the sprint, all pull requests, all blog posts, all public presentations - anything that we have done, and we celebrate them, and we discuss:

  • What was difficult? How could we make it better next time?
  • What was simple and effective? How can we replicate the success next time?

And the most important, ultimate question:

  • Now that this particular task is done, what can we do next to build on top of it?

For example:

We have completed a feature, what do we need to do to release it with the biggest impact?

We have fixed a bug, how do we deliver the update to the users blocked by the bug?

We have finished a part of a larger project, what parts have we just unblocked?

We have created a useful internal tool to perform utility task X, how can we spread the word to get more programming street cred in the open source community?

We have made an example explaining our software, or conducted a webinar, or recorded a conference talk. Where are all the places in our documentation we should include it?

I often see "what is your definition of 'done'?" type of discussions; yet with sprint planning rush, we often skip the actual "useful to the users" part of 'done', instead preferring to be done when the code gets merged and the JIRA story moves to the "Done" column. Switching the focus to the celebration puts the focus back on the user; a complete story is a trigger for an avalanche of more stories to be put back into the pile.

FAQ

Let me try answering common questions I hear when I object to the typical Agile sprint-planning process. If you have more questions, please let me know, I am on Twitter @bahmutov.

If the developers pick their tasks, how do we make sure they actually perform my super important task that is critical to the survival of the company? Easy. You (the person who considers the task X to be of utmost importance) have to argue / explain / convince others that the task is very important. If you cannot, the demise of the project / team / company will prove you right. And if the company goes on - well, maybe the task was not as critical. If you truly believe the task is a _must_, maybe you could even do it yourself? Or convince the company to hire a person to do it?
Without time estimates the department X and department Y cannot synchronize the delivery of two parts of the project Often, I think such synchronization (tasks X and Y should be performed by certain deadline) is a mistake, and instead a sequence would work better: once the task X is finished, then the task Y can start. For example, writing the marketing release materials should often start _after_ the software feature is finished, instead of "predicting" the outcome.

On the other hand, sometimes teams do need to really deliver the software parts at the same time: compatible backend API and the frontend for example should be developed in parallel. But I doubt estimating the code is as useful here as putting focus on iterative development where the teams communicate and try to bite off small chunks of the backend and the frontend work to be delivered.

How would new hires or developers with little experience pick their tasks? Great question. A new hire definitely would need help understanding the company mission, user needs, and project scope. I suggest the new engineers pair with more experienced members of the team to learn the project better before transitioning to semi-autonomous and fully independent work.
What is the role of the product owner in this scenario? The product owner represents the customer. They should know the customer's problem really well and come up with some approximation to the solution, and then help engineers define the tasks to implement the solution. The product owner should provide the input on behalf of the customer, and label tasks in the proposed solution as "important", "less important", "nice to have" - whatever the labeling system you might pick. The product owner can also collect all the extra information to describe the impact of each task better: how many users would benefit from the bug fix X? How many people said they would sign up for a paid plan, if only it had feature Y?
What is the role of the project manager in this scenario? Unblocker. The project manager should make sure that every member of the team is as productive as they can be; and if there is someone experiencing a difficulty, unblock them. Is someone sitting waiting for CI to finish because the CI is now a bottleneck? Get a budget for bigger CI. Is someone picking up tasks that require using a new shiny framework A? Get them an [egghead.io](https://egghead.io/) subscription to learn the framework A. Is someone complaining about code organization? Make sure people on the team have a meeting about code organization to figure out what is going on.
What is the role of the scrum master? Not sure. If the product owner is the one running around collecting all additional insights and adding them to the tasks, so that the developers make better choices, and the project manager unblocks the engineers to be their most productive, and there is no sprint planning - then the scrum master slowly dissolves in the morning air until nothing remains.
What is the role of the team lead? The team lead works on the solution, overall architecture, task definitions, putting systems together, etc. They should be responsible for ensuring the tasks are possible to accomplish, and solve the customer's problems. The team lead should also notice potential problems creeping into the software while the engineers focus on the individual tasks.
What is the role of the management? Everyone from C-level executives down to the product owner and the team lead have an absolutely essential and supremely important role: aligning everyone in the company and across every team in the same direction to solve the customer's problems. If everyone on my team understands what problem we are solving, and how we go about it - then the team members will pick the tasks to work on with clear understanding of how it all fits together.

The manager should also listen to the engineers to determine if someone is carrying too many responsibilities. The best solution in that case is to hire a junior engineer to help at first and then take over some of the tasks.

What stops engineers from only picking small "stones" to work on? Picking the task to work on is a question of motivation and balance. A motivated engineer would pick tasks that are important (see "the role of the management") to increase their importance to the company. Sometimes they might bite more than they can chew, failing to finish the task. That is ok - projects fail left and right. We can still derive other benefits even from a failed task. But by letting people pick their own tasks we can achieve a balance - tasks large and small will be picked up.

Finally, it is up to the project manager and the team lead to observe the efficiency and the motivation of the team. If someone keeps picking small stones to work on, and you believe they are capable of more - talk to them. If someone constantly tries to hoist rocks, yet never finishing - talk to them too!

What happens to the daily standup in this scenario? Honestly, I don't think the daily standup is any different in this case. If you do have a daily standup to communicate as a team, sure. If you do not have it - no need to start one. All we have done was to eliminate the sprint-planning session and expand the sprint retrospective into a "celebration" meeting.
What about sprint velocity? How do we know if we are productive? You replace estimating the distance to be traveled with looking back at the odometer. You can see how productive your team was over the last week / month / year by actually looking at the things accomplished. Look at the items from the completed list discussed during the "sprint celebration" and count the numbers month to month. If the numbers are going down - you are less productive as a team. Maybe you lost some team members? If the numbers are going up - congratulations. Check out how I looked back at my own work.
How do we use Jira or GitHub to facilitate the "no estimates" approach? Every issue or task should be described in full details. Any kind of user feedback, client impact, etc - all this information should be reflected in the task via labels, tags, and other meta information. If you are a product owner and think the feature X would be important to the customers, label the task to implement feature X "important". By collecting all information and attaching it to the tasks, you will allow the engineers to intelligently and deliberately pick a story to work on.

As soon as a task is picked up, the engineer should assign it, so that no one else works on it. The rest proceeds as usual.

Will the engineers be constantly interrupted by something else thrown into the list of tasks? The list of tasks will constantly be growing, changing, and be pruned. Every engineer is free to ignore this list (within limits, a bug in production probably brings all hands on deck to fix it as soon as possible), until they are done with the previous task. An engineer can pick a personal subset of tasks to work on every day, or every couple of hours - it depends on those rocks and stones and how quickly they are finished. But there is no need to artificially pick N rocks and M stones to work on for the next two weeks because "that's our sprint, and we promised to deliver it". Use your best judgement to pick the next task or two, finish them, and pick the next ones. It is that easy. At the end of the sprint, review your accomplishments, celebrate and build on top of them.
But I need to plan the next quarter, I need an accurate roadmap It is funny how we spend more time updating and obsessing over the roadmap (things that we cannot sell), yet review the changelog less often (things that we _do have_ and _can sell_ 💰💰💰). It is fine to have a roadmap, maybe even rough time frames (next month, next quarter, next year), but I would concentrate more on delivering concrete benefits to the user every day rather than obsessing over the spring of next year.

See more