Scrumerati

Scrum

Abstract

Scrum is an agile project management technique with high transparency, adaptive control, reasonably accurate release forecasting, and high productivity. It helps organize efforts involving teams and creativity (including software engineering, product design, venture capital management, charities, civic organizations, marketing communications, etc.). Because Scrum exposes managers and team members to greater visibility and accountability than traditional waterfall approaches, and because it requires different management structures, some organizations encounter resistance implementing it. Scrum focuses on management, so it can be combined with field-specific agile techniques, such as Extreme Programming.

Problem

The most common project management technique today is called "waterfall", reflecting that development flows from phase to phase, ending in the release of a completed project. In waterfall management, all the requirements are gathered up-front. Requirements gathering can take months. Product managers consider the existing marketplace, hypothesize future needs when the product will be released, estimate the development time needed to implement each requirement and analyze dependencies between development activities, putting together a Gantt chart that forecasts a release date. The development team develops the product, then the QA department tests the product, then the system is released as a beta, then bug reports and feature requests come in, then developers change the system, and out pops a completed product. Right on time. In theory.

However, once development starts, the Gantt chart typically loses credibility rapidly. Developers working under optimistic assumptions discover implementation difficulties—the creative process is not so easy to predict. Product requirements thought to be important at project initiation become irrelevant, superseded by competitive forces, new market developments, or better market understanding. The product managers add new requirements to the list. Finally, when the customers view the beta, surprises ensue. Typically, customers provide new requirements no one imagined, and ignore features all assumed were essential. The release goes out late. In an ongoing effort to "fix the situation" with the next product release, management often demands more meetings and more advance planning, taking more time before development starts.

Historically, waterfall forecasts are notoriously wrong; no one is shocked by projects forecast to release in 18 months that took 36 months. With late releases, organizations often sacrifice quality for speed. Companies can achieve bankruptcy this way; developers can lose their jobs and careers. In some situations, people have likely died.[6]

Solution

To solve this problem, reconsider some assumptions. If requirements obtained upfront are likely to radically change, a system that adapts rapidly to change would likely work better. If developer work estimates are reliably bad, find a way to map developer work estimates into actual developer time. If expensive advance planning results in marginally better or worse results than no planning at all, then plan over shorter horizons rather than longer.

Scrum is a software management and planning technique that balances these goals:

• Stakeholders can re-prioritize developer efforts at least monthly,
• Outsiders can experiment with full implementations of some features first hand at least monthly,
• Developers can measure their productivity empirically,
• Stakeholders can forecast release dates based on feature set, or tune the feature set to meet a release date,
• Developers can focus on their work uninterrupted, while simultaneously communicating beneficially in highly controlled, structured meetings, and
• Developers can refine their processes and work activities through feedback loops and self-organization.

Scrum was originally developed by Jeff Sutherland [1] and Ken Schwaber[2]

People

A Scrum Development Team comprises a Product Owner, between three and seven Developers, and a ScrumMaster. The Team is jointly responsible for the product, not only functionality, but quality, releasability, documentation, etc.

Team members are called Pigs; outsiders are called Chickens. The terminology arises from a joke: A chicken proposes to a pig that they form a restaurant together called "Ham and Eggs." The pig demurs, stating "I don't think it's a good idea. You'd just be involved, but I'd be committed."

The Product Owner in Scrum articulates stakeholder needs in the form of Product Backlog Items or User Stories, and assembles those Product Backlog Items into a prioritized list called a Product Backlog. There is only one Product Owner per Team (but several teams can have a single Product Owner), ideally there is only one primary Product Owner per Product (which may involve multiple Teams). Having only one Product Owner provides a "single wringable neck" for upper management, and a single authoritative source of prioritization and clarification for the Team. The traditional equivalent of the Product Owner is the "Product Manager."

Developers in Scrum include software engineers, user-interface designers, quality assurance engineers, database engineers, operations "shepherds", and other people who commit to a significant amount of work for the team. Ideally, development efforts are self-contained within the Team, so outsiders are not required to produce a releasable product. Furthermore, Teams and Developers are encouraged to be self-contained and cross-functional. If Developers don't know a skill, but that skill is needed by the team, they are encouraged to spend the time to learn and apply it.

The ScrumMaster facilities self-organization, supports the Team and Product Owner, enforces the process, encourages activities that improve the quality and likelihood of developing a releasable product, and serves as a sacrificial lamb to interruptions that would otherwise defocus developers. The ScrumMaster protects Developers from direct appeals from the Product Owner or other stakeholders. This supports Developer productivity: when Developers are interrupted, their productivity can drop dramatically. The ScrumMaster also works on behalf of developers to remove impediments, especially when those impediments arise from outside the team. The ScrumMaster is responsible for pro-actively providing transparency into the Team's progress and historic productivity information to outside stakeholders. The ScrumMaster encourages the team to improve its practices and make its work more feasibly releasable.

Process

Developers attempt to produce a Feasibly Releasable Product in a fixed time-period, called a Sprint. Each Sprint contains a Sprint Planning Meeting, Sprint Work, a Sprint Review (or Sprint Demo) and a Sprint Retrospective. Sprints occur in succession; in most cases there is no delay between the end of a Sprint and the beginning of the next.

Sprint Cycle

>Using the same duration for successive Sprints provides for easier planning and more reliable release forecasting. The maximum Sprint duration is a month; other common choices are four-weeks, three-weeks and two-weeks. In my experience, new practitioners of Scrum often start with longer Sprints. As they gain facility with Scrum and incorporate more efficient quality automation, they can reduce the duration.

All Scrum meetings are Time Boxed, meaning that they start exactly on time, and they last no more than a pre-established duration. A good meeting leader (in Scrum, almost always the ScrumMaster) monitors a meeting's quality: when its cost begins exceeding its value or veers from its goal, the leader should immediately discuss that observation with the team, and help redirect conversation or end the meeting. Meeting leaders may jot down "topics for later" to keep a meeting on track, inviting optional ad hoc discussion on those topics after the official meeting ends. If a meeting leader is not present, another member starts and runs the meeting. If members arrive late, they are penalized (a fine donated to charity is common). By enforcing start times vigorously, the process recognizes the enormous cost of idle developers, not only in squandered salary but also in defocused productivity.

Sprint Planning Meeting

In the first half of a Sprint Planning Meeting, the Product Owner presents the highest priority Product Backlog Items. For each item, the Product Owner describes it to Team member satisfaction, and obtains a jointly established effort estimate from the team. Many teams use Planning Poker to obtain this estimate[3]. New practitioners typically estimate Product Backlog Item work effort in units of Ideal Developer Days, but sophisticates use Story Points. When more than a Sprint's worth of effort has been estimated in total, the team selects a subset of high-priority backlog items it believes it can complete in the Sprint. This subset is called the Sprint Backlog; it is comprised of Sprint Backlog Items. With the Product Owner's consent, the Team can choose to skip a high-priority item. This can help focus the Sprint on a single theme, or defer items that need further evaluation.

As an Arab or Israeli user, I want the system to display text right-to-left.

1. Code unit-tests and right-to-left language support in client application
2. Create Arabic and Hebrew localization strings
3. Store Arabic and Hebrew strings in properties files
4. Code unit-tests first, then code web server transmitting localization strings to client
   
5. Design acceptable user-experience for Arabic and Hebrew language and users
6. Code unit-tests first, then code web server providing designed user-experience
   
7. Code integration test confirming system operates end-to-end in Arabic or Hebrew when Saudi Arabia and Israel locales are selected, respectively. System passes
   

Sprint Backlog Item with Tasks

In the second half of a Sprint Planning Meeting, the team breaks each Sprint Backlog Item into development Tasks, as in the example above. For each Task, one or more Developers volunteer, and those volunteers agree on the estimated work hours for the Task. Group estimation processes are rarely used for Tasks. There is no need to make sure the estimated hours in the Tasks sum to the estimated days for the Sprint Backlog Item. When the second half is complete, the Sprint Planning Meeting is over and developers begin their Sprint Work.

Sprint Planning Meetings should be time-boxed to 8 hours, typically 4 hours for the first half, 4 hours for the second. New teams or new backlogs may require all of this time. The Product Owner and ScrumMaster can make these meetings much shorter with Backlog Grooming.

Sprint Work

In Sprint Work, the Team starts tackling the work, beginning the highest priority Task in the Sprint Backlog, and moving down the list. A productive subset of Developers focuses on the topmost Sprint Backlog Item, taking on individual tasks. Another subset focuses on the next highest Sprint Backlog Item they can productively address (there may be dependencies or skill set issues), etc., until all Developers have something to do. If a Developer finishes all his or her Tasks, that Developer looks at the remaining work to see where he or she can help others, or move on to another Sprint Backlog Item. If no other Sprint Backlog work seems suitable, the Developer considers whether he or she can improve overall product quality, automate build processes, or otherwise improve the productivity of the team. As a last resort, a Developer might pull the next highest priority Product Backlog Item outside the Sprint Backlog, and attempt to complete it.

Each working day, the Team attends a Daily Standup Meeting (sometimes called a Daily Scrum or Scrum Meeting). Ideally, this meeting occurs at the same time every day and lasts no more than 15 minutes. Each Pig answers three questions: What did you do since the last Daily Scrum? What do expect to do before the next Daily Scrum? What impediments are currently blocking your progress? When all Pigs have answered, the meeting is done. Pigs can ask questions during the meeting, but, by virtue of the time-limit, answers must be kept very brief.

Sprint Burndown Charts

Sprint Burndown Chart

Sprint Burndown Charts help a Team track progress during a Sprint. The vertical axis is the total remaining estimated work hours for the Sprint, the horizontal access is work days. Every day, developers update the current Sprint's task list with the remaining estimated hours. If developers encounter difficulties their estimates can increase or if they add new tasks to a Sprint Backlog Item (both are allowed by Scrum), a task's estimated hours can go up. The ScrumMaster sums the remaining estimated hours at the end of the day and puts it on the graph.

In an ideal world, the Sprint Burndown Chart shows a straight line from the total estimated hours on the first Sprint day, converging to the X axis on the last day of the Sprint. It never happens; if it does, you should question whether the ScrumMaster is "cooking the data" to make it look perfect. Developers often discover estimated hours that were forgotten. When they are added to the estimates, the burndown line may diverge for a day or two.

Triage is a common activity mid-Sprint, for some Teams. If a Team discovers it cannot complete all the items in a Sprint, it revisits the Sprint Backlog with the Product Owner, makes sure that it can do enough backlog items to deliver a Feasibly Releasable Product, and moves low-priority items out of the Sprint (or leaves them in, but all agree they will be tackled only after everything else is done). Teams can also refactor Sprint Backlog Items to make them address part of a feature, preserving releasability, and add another Product Backlog Item to complete the job in a later Sprint.

Sprint Review

At the end of the Sprint, the Team hosts a scheduled Sprint Review Meeting or Sprint Demo. All stakeholders are invited. For each item in the Sprint Backlog, the team demonstrates, to the satisfaction of the Product Owner, that the Sprint Backlog Item was completed properly in the running product. In the best case, a third party installs the product following instructions written by the team, and the third party attempts to demonstrate each Sprint Backlog Item, thus showing that the product is releasable. In the worst case, the team asserts that it is done, with no validation. A responsible Product Owner in this situation should reject the Sprint Backlog Item, causing it to remain in the Product Backlog, to be later picked up in another Sprint. If all or most Sprint Backlog Items cannot be third-party validated, the Product Owner may consider sidelining or reconstituting the team.

Following the Sprint Review Meeting, the iterative cycle begins again with the next Sprint Planning Meeting. Note that during the Sprint, no one is allowed to change the Sprint Backlog without the unanimous consent of the entire team. However, Product Backlog Items outside the Sprint Backlog can be re-prioritized, rephrased or eliminated, and new Product Backlog Items can be added. This allows product development to adapt to new understandings of markets, product ideas, better assessment of Team skills, thoughtful assessment of risk, etc.

Sprint Retrospective

After the Sprint Review, the Team hosts a time-boxed 4-hour Sprint Retrospective to discuss and refine the process. This first involves taking pride in what was accomplished during the Sprint: What did we do right? Scrum is much harder than it may first appear, especially in dealing honestly and transparently with stakeholders and management.

The meeting also talks about improvements: What can fix, add or remove from our process that would achieve our goals better? Typical retrospective topics include revising the Done Criteria for Sprint Backlog Items, Tasks and Feasibly Releasable Products, finding more efficient ways to groom the Product Backlog, deciding how to split up Product Backlog Items better among multiple teams, considering new seating arrangements, deciding who can communicate a release forecast, etc.

Teams often spend 30 minutes to an hour listing possible improvements, then prioritize the list, then spend the rest of the time discussing specifics about how the highest priority items will be implemented in the next Sprint. Organizational impediments may block some improvements; if so, the ScrumMaster should work with managers to remove those impediments.

Release Forecasting

Velocity is a Team's historical average productivity in estimated-programmer-days per Sprint. The Sprint Backlog Items used to compute this number must have been completed in the Sprint where they are counted. If the Team estimates Product Backlog Items in advance, the team's Velocity can be used to forecast when any subset of those items could be completed.

Product Burndown Chart

A Product Burndown Chart displays a Team's Velocity, its progress in estimating future items, and the addition of new Product Backlog Items as the Product Owner and stakeholders gain experience with each Sprint's Feasibly Releasable Product.

The figure shows 100 estimated programmer days at the onset of Sprint 1. There were Product Backlog Items that had not yet been estimated. They were estimated during Sprint 1, and appear in the Sprint 2 bar, minus the estimates of Sprint Backlog Items completed in Sprint 1. From Sprint 2 to Sprint 3, you see the pure effect of the team's Velocity in completing about 22 estimated days of work. From Sprint 3 to 4, we see that the team burned down about 44 estimated days of work, but about 15 estimated days in new Product Backlog Items were added after the Product Owner showed the Sprint 2 feasible release to stakeholders. From Sprint 4 to 5, the team burned down 40 estimated days of work, but the Product Owner added 15 estimated days. At this point, we can project a possible release product at the beginning of Sprint 7.

Product Owners can consider how dropping different features from the Product Backlog might affect the release date. Of course, this must be done with thoughtful consideration of dependencies that might exist in the ordering of Backlog Items.

Challenges

To waterfall managers, Scrum is an alien process. To adopt Scrum is to admit that years of reliance on defined process planning—with its immediately-out-of-date Gantt charts and never-reached project deadlines—was wishful thinking all along.

Scrum places release forecast responsibility on the shoulders of Product Managers, who tune the release date based on the feature set they demand, and on empirical productivity metrics about the team. Using a waterfall process, upper-managers could blame their optimistic developers for a late release. But let's state the truth, it was usually worse: Most of the time developers had no chance to set a realistic schedule, rather a manager created an unrealistically aggressive schedule, perhaps in optimism or perhaps in an attempt to motivate productivity.

For developers, Scrum can be a challenging process. The term "Sprint" is very descriptive. I've had more than one developer state that "Scrum is intense." Scrum and other agile methods make it hard to hide sloth: developers have to report to the team daily, they have to release a working product monthly. Typically I've found that insecure people, rather than challenging the process itself, seek ways to limit transparency by modifying Scrum. They might suggest that Daily Standup Meetings occur only when needed, rather than daily, might find ways to avoid investigating the releasability of a sprint product, or might avoid test frameworks that can expose regressions.

Scrum challenges our job descriptions. Engineers become members of a highly-communicative, multifunctional team, and the team succeeds or fails together. The team is highly motivated to create automated unit, regression and integration tests. The team must determine "what is done", and hold itself to that standard. Does this mean the team is entirely responsible for QA? The answer to this question is different depending on the organization, but I can say, in Scrum, the team becomes gradually more and more involved in preserving and improving product quality. New projects have more flexibility; the team takes on QA tasks from its inception.

Group managers and directors have less-defined roles. They become change advocates, auditors, teachers. Sometimes they must get their hands dirty getting teams started, or making up for deficiencies that emerge. They must diplomatically push people in the right direction, making the process more efficient, keeping people happy and preserving their own integrity. To do decent performance reviews or allocate developers to teams, they must attend a lot of Daily Standup Meetings—if raises are misaligned with performance, it will be clear to the whole team.

With transparency comes unvarnished honesty and vulnerability. At one point, I was managing a Scrum-based project, but working for a waterfall manager. Every day was exhilarating, because my team led the way for others, and was scary, because I worried I would annoy people too much with my bluntness and get fired.

An obvious challenge: How do you scale this to a larger engineering group? Most all-Scrum departments use “Scrum-of-Scrums,” where teams aggregate all their product backlog items together into a giant list, and plan many backlog items together [4]. Sometimes they create and manage priorities around "Epics". I have never seen either approach used to manage multiple products. My company, Citrix Online, developed something different, which we call Enterprise Scrum. It handles multiple products, it is efficient, and it makes impediments and velocity visible at the highest levels of the company. You can find out more at my blog [5].

Summary

Scrum is an adaptive software management process that provides more visibility, more control, more accurate forecasting, and more productivity than standard waterfall processes.

A Scrum Development Team has three participant types: one Product Owner, one ScrumMaster and 3-7 Developers. Development proceeds iteratively, with fixed-duration development periods, called Sprints, lasting no more than a month. Each Sprint contains a Sprint Planning Meeting to determine what will be tackled in the Sprint, Sprint Work to complete development tasks, a Sprint Review Meeting showing off running code that could be suitable for release, and a Sprint Retrospective to tune the process.

Scrum includes three feedback-loops: Daily Standup Meetings help the team adapt rapidly and resolve problems together, Sprint Burndown Charts allow the team to review its progress and perform Triage to produce a working product in a short period. Sprints allows marketers and stakeholders to change development priorities on a monthly basis.

Product Burndown Charts provide the ability to forecast releases empirically, incorporating both developer productivity metrics and requirement addition trends into the forecasting process. They make planning and resource allocation easier. They provide a fourth feedback-loop, showing how product management choices affect release dates.

Jeff Sutherland assesses a company's agile maturity by asking CEOs these two questions: What are your teams' Sprint velocity? What organizational impediments have your teams identified, and what are you doing about them? To answer these two questions thoughtfully: your company must divide into Scrum Development Teams; each team must perform all the functions listed here; the teams must report velocity and impediments; upper-management must roll these data up for the CEO; and the company must be able to analyze and unblock team impediments. Simple questions are not always easy to answer.

As a software engineering manager, I've not found anything better than Scrum. I've worked in large and small companies, as everything from developer to CEO. In managing developer groups, I've long looked for a methodology I could apply with some rigor to deliver products on-time with a predictable feature set. I've used Rational Unified Process (RUP), Unified Modeling Language (UML), defined-process planning with Microsoft Project and Gantt Charts. None of these satisfied, though I have found some value in each.

In applying Scrum, I've seen greater productivity levels than I expected, and refreshing honesty and adaptability. It makes my work much more satisfying: I feel closer to customer needs, I think team members work at nearly peak performance. While we are typically exhausted at the end of a day, we also go home evenings and weekends and enjoy them.

References

1. Jeff Sutherland, Scrum Blog, http://scrum.jeffsutherland.com
2. Ken Schwaber, Agile Project Management with Scrum, Microsoft Press (2004), Amazon
3. Mike Cohn, Agile Estimating and Planning, Prentice-Hall (2008)
   Amazon
4. Mike Cohn
   Advice on Conducting the Scrum of Scrums Meeting
5. Dan Greening's Scrumerati Blog
   http://scrumerati.com
6. Anthony Finkelstein, Report of the Inquiry Into The London Ambulance Service (February 1993), International Workshop on Software Specification and Design Case Study
   http://www.cs.ucl.ac.uk/staff/A.Finkelstein/las/lascase0.9.pdf

Postscript

At the time this was written, Citrix Online was my employer, and supported my explorations and advocacy of agile management and marketing techniques. Many of its development teams are geographically distributed; we used Citrix Online's GoToMeeting product for our Daily Standup Meetings. Ron Lussier assembled the Scrum Cycle figure shown above.

This article was originally posted to Google Knol on October 30, 2008. It quickly became a top-pick, as well as a “Knol for Dummies” post. At the time of this post, it had over 12,000 page views. Google Knol is being decommissioned, so I have transferred this article to Scrumerati.com. If you are interested in Scrum and agile methods, particularly relating to enterprises or product management, I invite you to subscribe to Scrumerati.com by entering your email on the right side, or by adding its feed to your RSS reader.

Release Duration and Enterprise Agility: Global Webcast on 7 Feb 2012

Short release durations help companies test market theories to maximize profit. When managers measure release duration and advocate its reduction, they drive changes that improve the long-term productivity of the company.

Join me on February 7, 2012 at 11:00am EST, for a Scrum.org Global Webcast on release duration.  Register now at Scrum.org Global Webcast: Release Duration and Enterprise Agility.

We'll discuss how to obtain a release duration history from financial records, and how a finance-based definition can help you assess your ability to innovate profitably. We'll talk about various forms of technical debt and how they affect release duration.

 

Release Duration over a Company Lifetime

We'll show how measuring and restricting release duration helped improve code quality and processes in a mid-size software company. Both as a startup and mid-size subsidiary, the company's technical debt insidiously increased while it used traditional project management practices, even in the first months of its life, accumulating to a level where some projects took over 35 months to realize revenues. After focusing on reducing release duration and adopting Scrum and other agile processes, the company's engineering and product management groups eliminated much technical debt and drove release duration lower than it had ever been. The company's productivity, competitiveness and market share all improved during the same period.

Proposed: Agile Abandonment Disorder and its Root Causes

Jeff Sutherland, Scott Downey and I have found over the past few years that, more often than we wished, companies sometimes abandon agile methods. In our observations, it hasn't been because agile was performing badly; to the contrary, for the first time these companies had measurably improved their performance.

This year, Scott and I decided to tackle this issue as a research effort. We have looked at six companies, so far, but we hope to study more, with your help.

If you have observed agile abandonment, we'd like to talk to you. Your comments and company names will remain anonymous. My contact information is here.

If you've been working in agile coaching or training for a long time, and you've not experienced agile abandonment, or if you've found ways to counteract it, we'd also like to hear from you.

Scott and I proposed this topic for Agile 2012. Here is our submission:

Agile 2012 Proposal

Agile adoption can lead to dramatic improvements in market adoption, engineering efficiency improvements and engineer empowerment. But many companies have adopted agile practices, and then, despite seeing significant benefits, have abandoned those practices, allowed waterfall practices to return, and saw productivity decline. Why? If agile really helps companies succeed, how can we ensure its retention?

This session will summarize a series of root cause analyses we performed with agile leaders, using a “Root Cause Mapping” approach that helps groups of people identify contributing causes and share responsibility. Companies will remain anonymous in the presentation.

Most of us believe that agile adoption requires initial courage and effort, but agile retention also seems to require sustained courage and effort. Loss of a key leader, addition of a new leader, growth of the company beyond a certain scale, company acquisition, insufficiently refactoring mid-level manager roles, and the inability of product management to exploit agility all seem to correlate with agile abandonment.

Today, we have analyzed six companies that have abandoned agile methods. In discussing this issue with others, we have discovered abandonment is more common than we thought. We hope that this session proposal will inspire others who have experienced this problem to contact us with more examples. Email to dan+abandonment@greening.org.

If feedback is positive on this session proposal, we plan to incorporate additional companies, further analyze the phenomenon, consult with colleagues on mitigation strategies and assemble a set of best practices for protecting agile gains.

Help us out

If you think this topic merits discussion, consider lending your voice in favor of the topic. Registered users can provide public comments on the proposal at http://submit2012.agilealliance.org/node/14003.

Here's to persistent agility!

Could Your Company Lose Agility? Adaptive Portfolio Management can Help

Abstract

Agile methods create conflict. So to retain agility, we must actively promote it even after agile has taken hold. How can we equip a company with the cultural and process tools to sustain agility? Your choices may depend on your perspective. Social psychologists teach organizations to reinvent themselves, but this approach may be too slow for a large company. Change agents study, develop and evangelize specific processes, a more promising approach. "Adaptive portfolio management" is a specific process under development that may help sustain agile-thinking. It maximizes value by rapidly adapting to market and company changes. In using adaptive portfolio management, company leaders exercise agile fundamentals. They can become agile advocates for the whole organization, making agility more stable for developers on the ground.

In the service of productivity, agile methods create conflict. Consider continuous improvement, a canon of agility: It demands that people expose perceived problems and analyze their causes. Try it in any relationship, and you'll create conflict. Agile techniques increase transparency, increase accountability, weaken mid-level managers and deemphasize heroism: just a few more things that make some people mad. Some may start to see agile as the problem. “It was so much easier the old way!”

And so, agile culture requires active support even after it seems to take hold. Many experienced coaches have seen organizations that successfully implemented Scrum, and then stumbled. Some conclude this arises from software development's nature as a complex system. Scrum seeks to lightly control the chaos of software development—allowing for innovation while ensuring productivity; its feedback-driven control requires intellectual energy to sustain.

How can we equip a company with the cultural and process tools to sustain agility? The 2011 Scrum Coaching Retreat brought 70 Scrum coaches together in Boulder, Colorado Dec 7-9 to focus on this. We formed groups to analyse different elements of the problem.

Social psychologists and change agents

How we help a company sustain agile practices depends on our perspective. Some coaches are social psychologists—fostering agile culture and teaching organizations to reinvent themselves; others are change agents—studying, developing and evangelizing specific processes. I'm a change agent by nature.

I love analyzing and designing whole-company processes. The Agile Organization Design group seemed promising: it might have proposed ways to design organizations to support agile principles. However, social psychologists in the group challenged the presumption that we might design organizations at all. They sought strategies to enable an agile organization to design itself. One suggested that we "Fire all the managers." Another wanted the organization to "self organize". Self-organization takes a long time for Scrum teams, and much longer for large companies. As a result, leaders largely design companies by fiat. The change agents among us wanted to help those leaders, documenting organization designs that had worked for others, providing templates they might use as a starting point. The group made no progress; our goals were too different. I switched groups.

Adaptive Portfolio Management

The Adaptive Portfolio Management group provided a refreshing harmony, with likeminded folks seeking the best ways to prioritize, staff, interrupt and cancel projects. The group established a definition:

Adaptive Portfolio Management is the discipline of regularly changing the priority of business initiatives and resource commitments to maximize total portfolio return. This is implemented by:

1. Creating and managing an inventory of strategic initiatives
2. Prioritizing initiatives based on business drivers
3. Allocating budget to initiatives to maximize value
4. Periodically re-evaluating allocations based on changing needs
5. Transparently communicating across the organization

and a credo (really a manifesto, mimicking the format of the Agile Manifesto):

We value:

1. Optimizing the holistic portfolio over Protecting departmental budgets
2. Courage to change over Following a plan
3. Ability to evaluate the portfolio on a frequent basis over Infrequent (i.e. annual) planning cycles
4. Responding to emerging opportunities over Sticking to the plan
5. Maximizing value over Managing cost
6. Collaborating on decisions over Centralized authority

I was excited that 12 experienced people agreed on fundamental principles and worked so productively together. I was relieved that a portfolio management technique we developed at Citrix Online fulfilled the group's principles [gree2010]. We tried out an adaptive portfolio management game, developed by group member Jade Meskill [mesk2011]. In playing this game, competing teams largely developed the same risk-adjusted profit-maximizing strategies to order portfolio backlogs. We thought alike! Many of the graphics developed by the group would be suitable to describe Enterprise Scrum [scrp2011].

While adaptive portfolio management helps maximize a company's profitability, it can also help sustain a company's agile thinking and processes. If key decisions on project funding and prioritization are based on agile principles, those principles become infused in the corporate culture. Executives often own major product decisions. If those executives are using adaptive portfolio management, the rest of the company is likely to follow their lead and support other agile activities.

My colleagues and I are studying and developing better approaches for company-wide agile methods. We help companies become agile and stay that way. Contact me if the topic area interests you.

References

[gree2010]

Daniel Greening, “Enterprise Scrum: Scaling Scrum to the Enterprise Level,” 2010 43rd Hawaii International Conference on System Sciences (HICSS), Koloa, Kauai, Hawaii January 5-8, ISBN: 978-0-7695-3869-3 (10 pages)

[mesk2011

Jade Meskill. Agile Portfolio Management Game. Description and downloadable game materials. Blog entry. 9 Dec 2011.

[ries2011]

Eric Ries. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Crown Business, 2011.

[scrm2011]

Main Page, Scrum Coaching Retreat 2011, Boulder Colorado.

[scrp2011]

Adaptive Portfolio Management, Scrum Coaching Retreat 2011, Boulder Colorado.

Review: Liz Keogh, Learning and Perverse Incentives

Liz Keogh, Learning and Perverse Incentives, QCon London 2011

This 50 minute talk discusses perverse incentives: situations where incentivizing individual behavior causes an organization to become dysfunctional.  When we attempt to optimize an organization, but fail to use systems thinking (i.e., when we are optimizing from an internal perspective), we can create inappropriate results.

Different feedback mechanisms drive behavior in both positive and negative ways. Some examples: Reinforcing forces, balancing forces, feed forward, delayed feedback (and resulting oscillations). A positive feed-forward example: "Chance favours the prepared mind" (Louis Pasteur). If you prepare yourself properly, good things happen in the future. A negative feed-forward example: over-preparing for avoidance of a negative outcome. If many people worry about limited resources, people can hoard those resources and cause a systemic shortage.  Local optimization caused systemic failure.

We often place a strong emphasis on risk avoidance, but this can kill innovation. Risk is a natural part of any creative process, such as developing software, but some creative attempts will fail. When we take on risk, we learn. If you are in a creative field, you should never take on risk free projects. Some organizations make punishments so great for project failure, and rewards for doing new things successfully so minimal, that innovation stops. One perverse incentive is if someone punishes you for failing to achieve an estimate, you pad your estimates, and then your velocity goes up.  (See Liz's "Estimation Anti-Patterns" on her blog.) Another perverse incentive: requiring architectural review on after people on a team innovate and fail: too frequent reviews will kill innovation.

When we put on our Evil Hat, we say "How can game this system?"  That perspective is great for finding perverse incentives.  KPI's—Key Performance Indicators—are a potential source of gaming and arbitrage (exploiting that true value is different from the KPI). The quicker we're getting feedback from a metric, probably the further away we are getting from measuring true improvement. People tend to game the system for the metric, not the improvement, especially when KPIs are rewarded.

Your brain is brighter than you are. You will game the system, even if you think you aren't consciously wearing the Evil Hat. Rarely do people game the system deliberately, but there are some situations, particularly with middle managers, where people deliberately game the system for their personal benefit. (See Snakes in Suits below.)

How to improve:  Get away from solving a problem, and create a system where problems are solved. Evangelize and embrace frequent Root Cause Analysis, with 5 whys, to get to root causes.  Keep metrics close to the improvement you are trying to measure (see The Goal below.)  Create a learning environment. Even in standups, “This is what I learned today; This is what I hope to learn tomorrow; Can anyone help me find out... ?” could be useful.  Fit the process to reality, not reality to the process.  Try focusing on what's going right as well as what is going wrong. Then we focus on learning: what should improve, what should we keep? Thank people for doing a good job, and think “How can I anchor good work?”

Book List:

• Peter Senge, The Fifth Discipline: The Art & Practice of the Learning Organization
• Tom DeMarco and Timothy Lister, Walking with Bears
• Donald Reinertsen, Managing the Design Factory
• Edward de Bono, Six Thinking Hats
• Paul Babiak and Robert D. Hare, Snakes in Suits: When Psychopaths go to Work
• Eliot Goldratt, The Goal

Is Hiring More the Right Answer?

In my current company, I sometimes here people say “We need more people to handle this workload.” Every group in the company considered in isolation could use more people, no matter what group they are in; at least that's what most people think. But smart folks sometimes say, “We have too many people.”

Here's what the research tells us:  Until you reach about 7 people in a group, adding another person will likely make the group go faster [putn2011]. Specifically, starting from a 1 person group, at first, each time you add a person the value-produced per group goes up, However, something weird happens as you approach 7 individual contributors: The group productivity drops. I.e. An 8 person group produces less than a 7 person group. Why? Because the communication, coordination and mistake costs have overwhelmed the advantage of additional people.

To add one more complexity, the zero-sum game of corporate hiring means if your group hires more, my group must hire less. Smart software executives typically seek an optimal ratio of user experience designers, engineers, program managers, user experience managers, engineering managers, scrummaster, product managers, researchers, etc. to maximize corporate profitability. If your company has the right ratio, when a group is overwhelmed, you are actually over-stressing the whole system, producing technical debt, and should slow down.  

In my current company, user experience is one of our most important brand values.

Recently, a colleague asked me to consider hiring more user experience designers for a product where software engineers develop new features very slowly. Given that insight, you might suspect the software carries a lot of technical debt, and you would be right. If we hire more user experience designers, it would likely generate more feature requests, then more pressure on the engineering team to produce features faster, then more shortcuts in coding, and then even more technical debt.

Technical debt leads to systemic slowdowns in productivity. Its accumulation is insidious and can be disasterous for a company. In the face of a highly competitive marketplace, technical debt scares me.

Here are some sophisticated alternatives to consider when a user experience group is overwhelmed:  

• The ratio of user experience people to developers really might be too low. In this case, reduce the rate you hire engineers to increase the rate you hire user experience designers.
• Engineers might be producing features too fast; you should get more engineers to work on reducing technical debt to make the future brighter.
• Your user experience group might not be saying "No" to unimportant assignments. In some cases, product owners or engineers can do a reasonable job with a little guidance. In other cases, the project isn't profitable enough to bother doing.

You can apply this sort of thinking to any situation where you think you need more people. Will more people make your group more productive?  Even if the group is more productive, will the company be more productive? Take a step back and think about the relative ratios of different types of contributors. Maybe the company would benefit most from adding more people to a different group.

When someone asks you to state your team's top impediment, here is what they are asking: “What is the biggest thing keeping my existing team from contributing most to corporate productivity?”  Hopefully this post explains why “Not enough people” isn't usually a good answer.  Sometimes, in fact, the impediment is too many.

[putn2011]

Doug Putnam, “Team Size Can Be the Key to a Successful Software Project,” Quantitative Software Management, IT Almanac, 2006.

Agile 2010 Impressions

Agile 2010 was held in Orlando near the Disney Epcot resort August 9 to 13. I focus on agile enterprises and attended many enterprise-focused talks. If you are interested in the developer focused view, Martin Fowler provides his thoughts here. My impressions follow.

Portfolio management is being implemented in conjunction with quarterly “sprints” in other places beside Citrix Online, including Motley Fool (Max Keeler), Tektronix Communications (Brian Miller) and Progressive Medical (Ben Blanquera). The four of us are starting to share ideas and experiences.

Value and Product Management is a hot topic. More than 50% of the time, speakers strongly argued that NPV and ROI was an inappropriate metric (this was largely from the folks dealing with team-level Scrum), but an intriguing minority asserted that Product Managers who don't assess priorities using NPV and ROI are irresponsible. Deep thinkers, like Luke Hohmann, acknowledge that NPV and ROI are important at the portfolio level. One of the most interesting presentations in this area was the presentation “Extending Agile - to finance (really?),” where NPV is consciously part of every discussion, even at the story level. The CFO is part of the product stakeholder team and sits with engineers weekly. The company has a big chart showing how value and expense get aggregated into corporate financial statements.

Forrester is seeing that Agile engineering processes are transforming the enterprise outside engineering. This was a subtle undercurrent of many other presentations. Of course, we are seeing this phenomenon in Citrix Online, where APO is teaching marketing, customer support and human resources departments about agile processes. A year ago, agile advocates in Citrix Online, especially me, were bitterly disliked by many people both inside and outside engineering. Now we are highly regarded and sometimes sought after.

CMMI+Scrum continues to intrigue, as it demands that the organization find ways to measure and improve enterprise-wide processes, potentially complimenting Scrum's feedback loops and retrospectives at the team level. However, almost all CMMI books provide examples exclusively from waterfall. Many people talked about needing an enterprise capability model  like CMMI that better reflected Agile thinking. Finally, Kent Johnson made a great comment that put it together for me: “CMMI dictates engineering and process quality metrics and improvement, but the real goal is value.”

Like Martin Fowler, I was underwhelmed with the presentation quality, finding myself stupified by many talks whose descriptions initially seemed interesting, but ultimately just rehashed long-argued religious platitudes (“you can't measure value with NPV,” “you really should have teams co-located,” “it's all about team self-organization”). Yawn.

I bumped into Tom Poppendieck, gushed shamelessly about his brilliance and fame, and then complained that I had heard most of the Agile 2010 material before. Why did he still attend? He found that most of the good exchanges happened for him in the hallways during the presentations: that's where the bored people hang out, and many of the bored people are the advanced thinkers. Armed with this alternative, I sallied forth.

Advanced thinkers move beyond the platitudes to ask the tough questions, like “Why can't we measure value with NPV?” and work through the Five Whys. Their Why not? talks captured my attention. Two rare examples from Agile 2010:

• Scott Downey and Jeff Sutherland, the first to strongly argue for the highly-prescriptive Hyperproductive Scrum (Agile 2009), this year demonstrated an Excel spreadsheet that translated team productivity metrics into a letter to the team documenting its likely impediments. They called it RoboCoach. This generated interest from many and outrage from a few. Bravo!
• Ben Blanquera described how his CFO helped develop a hierarchy of revenue and expense to motivate every Product Backlog Item. The CFO sits with the team during Sprint Planning to explain the value of every story to the team, in profit. This session was lightly attended, but I think it had everything to do with the title and description. “Extending Agile - to finance (really?)” sounds to me like a talk about making the finance department agile, doesn't it? It wasn't.

With a perspective on Agile 2010, I remain your humble servant.

Agile 2010: Enterprise Scrum, Creating an Agile Company

I presented our work on Enterprise Scrum at Agile 2010 this year. The session was well-attended for a specialized talk like this one (really only suitable for software engineering teams larger than 30 people), with about 40 people in the audience.

Enterprise Scrum: Creating an Agile Company

Enterprise Scrum, a fractal extension of Scrum and XP, has organized all development at Citrix Online since Jan 2009. We estimate team months, run quarterly Sprints, reassign teams, meet in weekly stand-ups. We start or postpone whole projects that use Scrum or Scrum-of-Scrums. No other known companies yet use Enterprise Scrum. It provides extreme visibility and control for CXOs. It promotes agile thinking enterprise-wide, driving adoption outside engineering. It demands NPV justification and forces executives to prioritize decisions transparently. It makes us more profitable.

I learned at the conference that Tektronix Communications adopted a variant of Enterprise Scrum two months ago, after Keith Miller had read this paper:

Daniel Greening, “Enterprise Scrum: Scaling Scrum to the Enterprise Level,” 2010 43rd Hawaii International Conference on System Sciences (HICSS), Koloa, Kauai, Hawaii January 5-8, ISBN: 978-0-7695-3869-3 (10 pages)

I invited Keith to present a few slides on the Tektronix implementation during my Agile 2010 talk. I love to rope people in at the last minute. Keith did a great job, and added credibility by showing other organizations find utility in this approach. You can download the presentation slides, including both mine and Keith's sections, here:

http://dan.greening.org/talks/Agile_2010_Enterprise_Scrum.pdf

I am assembling people in a similar role as me (orchestrating agile processes for engineering groups from about 50 to 343 (7²+1 to 7³) people) to share ideas and experiences. One participation requirement is evidence that you have contributed significantly to the well-being of engineering groups outside your own company. This could include presenting at agile conferences, helping CSTs with scrum training or other indicators. Feel free to contact me if you think you fit this category, and would like to get involved.

Reviews

13 attendees responded to an audience survey for this talk. Agile 2010 does a great job of surveying attendees and sent the responses the last day of the conference (which is really fast).

Ratings

Attendees were asked to score each question from 1 to 5, where 1 = Very Poor, 2 = Poor, 3 = Satisfactory, 4 = Good, 5 = Excellent.

Question	Ave Score
How well did this presentation meet your expectations?	4.58
Would you recommend this session to a colleague?	4.67
Evaluate the speaker's presentation skills.	4.58
Evaluate the speaker's command of the topic.	4.75
Did the topic description match the presentation content?	4.75
Overall rating of the presentation.	4.67

Comments

• Dan is well-versed, presents well and kept audience engaged. Very valuable lessons and learned discussion – Wish this session had been 90 minutes instead of 60.
• Maybe some interactivity?
• Excellent data.
• Too complicate to understand.
• I had no idea we could monetize points.
• Nice concept to share with our execs and portfolio teams.

Enterprise Scrum:Scaling Scrum to the Executive Level

My paper on scaling Scrum to the executive level has been published by IEEE. It was selected as the best paper in the agile track at HICSS. Download it here:

Daniel Greening, “Enterprise Scrum: Scaling Scrum to the Enterprise Level,” 2010 43rd Hawaii International Conference on System Sciences (HICSS), Koloa, Kauai, Hawaii January 5-8, ISBN: 978-0-7695-3869-3 (10 pages)

Abstract:

Our company manages 25 software engineering teams across 6 products using a single top-down Enterprise Scrum. We know of no other company doing this, yet it provides extreme visibility and control at the CXO level. It promotes agile thinking enterprise-wide, driving non-engineering departments to adopt Scrum. We believe it is making us more profitable.

We estimate effort in team months, run quarterly Sprints, assign whole teams to projects, meet in weekly stand-ups. We start, postpone or cancel whole projects.

Within individual projects, we still use 1-4 week Sprints and all the trappings of the classic Scrum process, including, in some cases, Scrum-of-Scrums.

New challenges arise: Shared resource constraints suggest Kanban methods. Net Present Value can justify prioritization, but creates controversy. Moving teams between projects requires rapid programming environment setup. The process forces executives to justify decisions. We want simple improvement metrics, but they seem elusive.

We are actively adapting this process at Citrix Online. I would love feedback and suggestions related to either the process or the paper itself.

The Demise of NUMMI

A colleague of mine, Kris Niles, sent me this long (59 minute), but compelling audio from "This American Life", on the demise of the Fremont NUMMI auto plant. The GM Fremont plant was shutdown in 1982, restarted as GM/Toyota NUMMI in 1984, adopted the agile “Toyota Production System” through a massive education program (they flew all NUMMI plant personnel to Japan, where Toyota trained them), and almost immediately started producing high quality cars.

This is not a story of failing to change; it is a story of changing too slowly.  GM could not replicate the learning it obtained from NUMMI fast enough. Its first attempt to replicate NUMMI, by converting its Van Nuys plant to Toyota Production System (without retraining everyone), failed miserably. Resentful workers seeing that seniority was no longer valued, rebelled. Managers with cushy privileges and familiar control, pushed back.  GM closed Van Nuys and laid everyone off.

GM President Roger Smith finally started to seriously implement Toyota Production System in 1993, and gradually spread it throughout GM, but it took over a decade: too long to forestall bankruptcy. Some quotes from the piece: "Steering the Titanic with a canoe paddle."  "If you take 30 years to figure it out, you're going to get run over. And they got run over."

GM’s market share dwindled and ultimately it fell to bankruptcy in 2009. NUMMI was abandoned by GM to its partner Toyota. NUMMI was shutdown on April 1, just a couple of days ago, by Toyota.

This episode of This American Life describes cultural issues with eerie familiarity: resistance to agile methods from both individuals and managers. Folks that have comfortable, familiar roles in dysfunctional systems they don't want to give up.

We speak like GM when we postpone automated testing; when we allow ourselves to create new technical debt “because we need to ship features faster”; when we assert it is "impossible" to release new features to the outside world every Sprint; when we fail to sideline toxic employees; when we discourage a team from self-organizing to tackle problems; when we complain about having to interact more with lesser teammates to help them go faster; when we allow impediments to persist; when we fail to train everyone in agile methods "because they aren't managers"; and when we fail to “stop the line immediately” to address and fix process problems. If you are an agile company, you may think these problems are going away, but I bet you aren't making them go away fast enough.

If you are piloting the Titanic, don't steer it with a canoe paddle.

The link: http://www.thisamericanlife.org/radio-archives/episode/403/nummi