Agile - A Lightly Informed View

(Note: I’m making a short diversion from my series on virtual environments. I’m working on a post about using Ruby in a virtual environment, and I hope to publish that very soon. In the meantime, I hope you find this post on development methodology interesting.)

Many development teams are practicing or experimenting with Agile¹ development methodologies, typically some variant of Scrum. Many other development teams wish they were following Agile practices.

What is the reason so many folks are attracted to Scrum? I have some ideas.

These ideas are informed by reading a handful of books and numerous articles on Agile methodologies, listening carefully to talks by Agile practitioners, taking two training classes on Scrum, and by running or participating on several projects that used Scrum. I’m not an expert, although I can make some helpful suggestions. Given that background, I consider myself lightly informed on Agile and Scrum.

Perceived benefits

So why are Agile and Scrum methodologies of such interest to so many?

I think that when a developer considers using Scrum, they get excited because they perceive these benefits:

• Test-driven development (instead of build now, test later).

• Intentional evolution of design (instead of design everything first, then build later)

• Delivering working software early and often (instead of build pieces in isolation now, integrate later)

• No scope changes, no context switches, no big interrupts during a sprint (as opposed to weekly or semi-weekly emergencies, shifts in focus, scope changes)

• Committing only to a clear and constrained deliverable in the very near term (as opposed to a nebulously defined deliverable on a target date months or a year away)

• Working with a small, highly collaborative team (as opposed to a large team with diverse goals and priorities)

• Retrospectives and regular demos mean process feedback and process improvement is built in to the methodology (as opposed to a fixed process delivered from on high)

• High availability of the Product Owner (as opposed to just at the beginning for the kick-off and at the end for the acceptance test)

• Through frequent interactions, the Product Owner might gain direct understanding of what developers do and why it takes them so bloody long to deliver anything.

• Working at a sustainable pace (as opposed to the Death March tradition, augmented with Last Minute Emergency Scope Changes)

When a development manager looks at Scrum, they get excited because they perceive these benefits:

• The whole team helps develop the plan and estimates (not just the dev manager)

• The Product Owner owns the long term plans and is actively engaged in scoping the near term work and adjusting scope to accommodate changing business needs and priorities (rather than dumping most of the details into the dev manager’s lap)

• The team self-reports status (rather than the dev manager dunning staff for status updates)

• The team self organizes (enough said)

• Low error bars on estimates for near term work, higher error bars on estimates for long term plans (rather than having to produce and maintain a high fidelity plan for a year long effort)

• And, perhaps, no wrestling with MS Project and Gantt charts (which is painful for many development managers)

When the Product Owner looks at Scrum, they get excited because they perceive these benefits:

• The developers are going to be Agile (which must mean the developers are going to be much faster!!)

• There will be frequent demos (Product Owners rarely have time to review detailed specs or read status reports; a demo is worth tens of thousands of words)

• Fine grained control over the scope of work and scope of releases (no more hoping the team will deliver what the Product Owner thought they asked for)

• Better estimates and planning (enough said)

• Always knowing what is going on (no nebulous Elaboration or Construction phase, no end of cycle surprises about scope and quality)

Capturing the benefits

An article, even a book, could be written about each of those bullet points above—and probably has been.

However, I think the truth is that few teams attempting Scrum are actually getting all those benefits. Probably many teams are doing daily stand-up meetings and most of those have some form of Scrum board. My sense is that few teams do retrospectives in the Sprint Review meeting and also make adjustments based on the retrospective feedback. Probably many teams do not have a consistently and highly involved Product Owner. And probably many teams are too big—such that the sprints lack a cohesive focus. And probably few teams are doing iterative development (in terms of iterating on design and implementation, not just on process).

Here are some things that I think prevent a Scrum project from capturing all the benefits of Agile methodologies:

• Team is too big.

  The team should be 5 or less (not counting the Product Owner). An experienced organization might—might—be able to succeed with teams as large as 7 (at least that I heard that in the training courses I took). A larger team will tend to work on unrelated user stories which dilutes the focus of the team, reduces collaboration, and—worse—can make stand-up meetings, planning meetings, and sprint reviews tedious and of lower value.

• An uninvolved Product Owner

  The Product Owner needs to be interacting with the team throughout each sprint, and needs to be actively managing the backlog. The potential agility of the team (to switch focus at sprint boundaries) is completely wasted if the Product Owner is only interacting with the team rarely or just at the beginning and end of the project.

• Not doing test-driven development

  Each sprint needs to produce high quality code and needs to deliver a complete chunk of work. The team may be asked to switch focus to some other feature in the next sprint, rather than continue to polish what they were working on last sprint, so it is important to have a stable, consistent, and working set of software at the end of each sprint.

  Test-driven development ensures high-quality code up front, and is constantly (with each build) demonstrating what is implemented so far. For a team that practices test-driven development, there should be no end-of-sprint mad rush to find and fix bugs, and there should be no concerns about delivering partially working software at the end of the sprint.

  Test-driven development ratchets in features and quality, and leverages the power of automation and all the previously written tests to ensure that the software quality remains high and does not regress as new features and user stories are pursued in later sprints. There is arguably no other software practice that delivers so much value for such a tiny marginal cost.

• Not doing retrospectives or not taking retrospectives seriously

  During each sprint review the team should do a retrospective. When the retrospective turns up issues affecting team function or software quality, adjustments should be made to mitigate those issues for the next sprint.

  Without this regular introspection and regular attention to execution, the team can easily (and surprisingly quickly) devolve into old habits or behaviors that fail to capture full benefits from the Agile methodology.

• Sprints are too long

  Sprints should probably be two weeks. If you feel you need three week sprints, then you are probably trying to do too many user stories in a sprint, or trying to take on user stories that are too big (and should be broken up). Shorter sprints keep the process agile, keep the team focused, and keep the scope achievable (and that last, I think, leads to more accurate estimates of velocity).

  Also, shorter sprints and smaller stories also tend to force the developers team to revisit and extend the same region of software across multiple sprints. This iterative re-viewing of the code can lead to improved design and implementation.

• Trying to do too much in a sprint

  Doing too much in a sprint tempts developers to panic and take short cuts (like not doing test-driven development). Perhaps as bad, the team won’t finish all the stories and will eventually get used to not completing stories and will start expecting a story to take multiple sprints to complete.

I’m sure there are other “smells” that point to a team not getting the hoped-for benefits of Agile methods. But that’s enough for now.

Let me know of other important perceived benefits or other important smells to look out for.