Scrumerati

Need Your Help: Agile Abandonment?

Several companies with a strong history of using agile methods, even "hyperproductive" variants with published metrics, have abandoned their commitment to agile, sometimes accompanied by dramatic staff defections. There seems to be a correlation (not necessarily causation) with departure of key staff members. I am especially curious about this because, years ago, this happened to an employer of mine. Whether due to management failures, or failures of the agile framework, I want to understand and improve this situation.

Open Space Session, Scrum Gathering Orlando 2010

I am currently researching this phenomenon, to explore the root causes in a future paper. I do not want to bias our research by hosting a public debate, instead preferring to have one-on-one email or phone discussions with parties involved.

I will share drafts of our work prior to publication with any participants.

If you are aware of a company abandoning a commitment to agile methods, whether you were directly involved, or are aware of others I could interview, please contact me. Dan Greening dan-abandonment@greening.org, cell +1 (415) 810-3693. If you wish to remain anonymous, whether in referring to others or discussing your own direct experience, I will honor that request.

Please refrain from publicly commenting on this post, and rather contact me directly, to avoid biasing research participants. (Crossing my fingers and hoping that works.)

Enterprise Scrum: Scrum Gathering

On 9 March 2010, I gave a talk on Enterprise Scrum at the 2010 US Scrum Gathering in Orlando, Florida. I am grateful that about 50 people showed up for my talk, from about 300 total Scrum Gathering attendees. People were intrigued by fractal thinking, by the blunt assertion that engineering teams rapidly burn money, and by the prioritization of work using forecasted Net Present Value. Enterprise Scrum can create very healthy product lines and companies. In a sense, Enterprise Scrum turns enterprises into internal venture capital funders.

Despite that large Scrum implementations can dramatically increase the productivity of 250 to 30,000 developers, the exotic world of agile methods confuses or scares most large company executives, if they even know of agile methods at all. It's too bad, because deploying Enterprise Scrum or other "large Scrum" variants could likely save a few dinosaurs from extinction.

I love sharing these ideas with others. My personal mission is to help society produce more value (happiness, etc.) with less work, for everyone. I would be happy to present Enterprise Scrum to your organization or executive team, if it resonates.

Here is the background information on the talk.

Speaker Biography

Dr Dan Greening is Citrix Online's Director of the Agile Program Office. This team provides Program Management, ScrumMastering, User Experience Testing and User Interface Design services to the company. He was the primary evangelist in converting Citrix Online to be a pure Scrum organization, and advocates for Lean, Theory of Constraints, and other continuous improvement frameworks. His paper on Enterprise Scrum was nominated for best paper in the 2010 Hawaii International Conference on System Science.

Greening is co-founder of three previous startup companies, two of which realized moderate liquidity.

He was Principal Investigator on 3 National Science Foundation SBIR awards.

He holds Ph.D. and Masters degrees in Computer Science from UCLA, and a Bachelors in Computer Engineering from University of Michigan. He is a Certified Scrum Professional. He published a paper on stochastic optimization and fractals in the proceedings of Santa Fe Institute's Complex Systems Summer School.

He has studied the emergent properties of large organizations for decades. In 1986, while a computer science graduate student, Greening was elected President of the University of California Student Association, representing 144,000 students at 9 campuses. Under his leadership, UCSA created self-organized teams that successfully championed state legislation for improved student health care, student safety, and financial accountability in the University.

Abstract

Enterprise Scrum is "Scrum Writ Large," a continuous improvement process modeled after Scrum, but scaled, using fractal thinking, to a much larger organization. It is not Scrum-of-Scrums. It was developed in late 2008 at Citrix Online, and has been iteratively improved since then.

Enterprise Scrum, as implemented in Citrix Online, has quarterly sprint cycles, weekly standup meetings, effort measured in "estimated team months," and prioritization partly based on Net Present Value. Planning is iterative, to avoid squandering developer time estimating effort until the last possible moment. Several resource allocation strategies have been explored, everything from total self-organization to traditional approaches. Through a series of retrospectives and adaptation, a hybrid is emerging.

Evidence for Enterprise Scrum's success is anecdotal, but strong: Executives know and plan against the engineering department velocity, remove high-level productivity impediments, and promote agile in their own departments. Because the value of key engineering projects, including technical-debt reduction, is understood and prioritized at the CXO level, not hidden in the priesthood of the Engineering Department, there is a healthy appreciation of the value engineers provide. We have seen situations where other departments voluntarily redirected funds to engineering, because engineering could prove its high profitability.

Topic

Scrum practitioners regularly face problems gaining advocates at the top levels of medium to large organizations. In part this results from Scrum's focus on individual teams and small features, while large organizations assess and prioritize major projects largely around profitability, mission synergy and cost.

To successfully promote agile and continuous improvement processes at the executive level, we must understand corporate values, incorporate them in our processes, and explain process utility in ways that executives understand. We must provide executives with tools that let them easily prioritize projects and, once they have started, identify systemic and project-level blockers that impede progress.

This talk will likely help agile practitioners working within large organizations, and help people considering investing in or initiating startup companies. The concepts are similar for both. If you have a large organization, and you are competing with startups, you may need an approach like Enterprise Scrum to remain viable.

Objectives

• how to easily drive agile understanding into the whole organization.
• how to apply fractal thinking to Scrum and other continuous improvement techniques.
• how to reduce apples and oranges comparisons (such as comparing a feature project against a technical debt reduction project) to Net Present Value.

Pushing Agility Upstream

Most agile teams live in a waterfall ecosystem. When powerful stakeholders—like business partners, customers, other departments or executives—demand future commitments, while interrupting contributors with unprioritized demands, smart Scrum teams raise a protective shield. They make the Product Owner manage stakeholder priorities, and make the ScrumMaster defend them against interference. If anyone provides a date to a stakeholder, it's going to be the Product Owner: the Single Wringable Neck.

But instead of just defending ourselves against an external onslaught of unprioritized need, we can go on the offense. We can evangelize agility upstream. Impossible requirements point out that stakeholders could themselves benefit from agility. If we teach them agility, they and we will both go faster. Together we can develop a strong, sustainable and profitable business ecosystem.

My buds and I have pushed agile upstream twice recently, and it's more fun than you might think.

Example 1: Marketing

Traditional marketing departments demand that staff respond to immediate needs: the "hair on fire" approach to planning. In the past, engineers working closely with marketing have accepted these interrupts, and some of our own past work reflects the ragged architecture and appearance you might expect. Expensive rework wasted engineering time. Bugs caused significant problems.

In 2007, our engineering department adopted Scrum. After that, marketers ran headlong into ScrumMasters defending engineering sprint boundaries. Marketing's first reaction was to find ways around Scrum restrictions, demanding their own dedicated teams, or time-boxing vaguely defined marketing related work that engineers would have to fulfill.

Engineering colleagues at first tried to satisfy these needs, but then realized marketing demands were distracting them from important work. I assumed we just needed to explain ourselves better. Scrum meant we wouldn't be able to satisfy marketing needs the same way. Marketing would have to prioritize needs earlier, based on our work estimates. They would have to allow us to say "No" to low priority work. Nick Kim (another evangelist ScrumMaster) and I started a conversation with key managers.

But as we talked, we realized that marketing suffered the same problems our we had experienced in the Bad Old Waterfall Days. Creative contributors in marketing looked overburdened and frazzled. They talked about trying so hard to meet requirements, but no one controlled the flow, so they were always late. We started to advocate that the marketing department adopt Scrum. Nick became their ScrumMaster for a few months. Over time, we helped motivate adoption of combination of Scrum and Kanban in much of marketing.

Marketing folks seem a lot happier now. Engineering and marketing are working together more harmoniously, and, I claim, producing better work.

Example 2: The Company

To meet market needs, executives want everything as soon as possible. Traditional companies hire whole product silos with dedicated engineers, or acquire whole companies, to address newly defined products. At first, it works: you have all the people you need for the current market demand profile. But gradually those historic assignments become less aligned with company value. Over time, your product silo produces less and less value per engineer. Engineers working within siloed Scrum teams might boast of high productivity based on code output, but tied to last year's markets, their ongoing work earns less money over time. This exposes the fallacy of measuring work in "function points," "lines-of-code," and other code-based metrics: a team can produce lots of function points, but make little money.

Startup companies can often produce value more rapidly than large companies. Unhindered by long-standing work responsibilities, desperate for liquidity and operating intimately with customers, startup engineers have more motivation to work on projects that bring money in fastest.

We are a big company with several products, so we have to create a process substitute for startup-motivation. Faced with unranked needs from executives, and concerns about engineering productivity, we created a single Enterprise Scrum, with a three-month sprint cycle, to organize all engineering work. We asked decision makers to prioritize work by dollar value. We concentrated our best efforts on the top-ranked projects. This structure has transformed Citrix Online engineering to become a more responsive group, with growingly predictable deliveries.

As engineering becomes more sophisticated in tackling external projects, friction between engineering and others can reveal learning opportunities. Urgent projects require a just-in-time mechanism to handle sudden market changes without disruption. Productivity problems around user-experience, software architecture and operations indicate the need for multi-quarter portfolio planning. Difficulties prioritizing "apples against oranges" may reveal a deeper problem: Sometimes people don't know what their projects are really worth; they should be trying to figure it out.

These efforts are making the whole company more agile. Our President knows our engineering velocity, and we manage priorities to that velocity. VPs and Directors intercede frequently to remove impediments in engineering. Things are a lot better.

Promote Agility, Develop Strength

Agile methods can help most organizations get better stuff done in less time. A key philosophy drives Scrum: If we are going to work on something, it should matter to someone outside our team. If you're a skilled agilist, these concepts are second nature. You could help many people around you, by carrying the agile message upstream.

Ideal stakeholders and Scrum practitioners assess value thoughtfully, respect focused work, discuss uncertainties, accept responsibility for problems, and help reveal embarrassing root causes. Unfortunately, when you start promoting Scrum to the unconvinced, expect some disappointments, pushback and politics. But if you persistently and diplomatically promote agility, carefully wending the shoals and backwaters of other departments, you will strengthen your company. If you do it with integrity, you should emerge with a great reputation. And that will help your career.