Management Summary
In 2016 I was asked to review a major Toyota program that had been running for several years and had failed to deliver any effective working functionality to its users during its lifetime. It was consuming a budget in the region of $10 million dollars a month while utilizing around 150 contract resources from a well known consulting company that were both onshore and offshore. Despite multiple reboots and retooling it consistently failed to meet any milestones or outcomes, and was mired in an endless cycle of planning to replan. Its project delivery methodology was a 180 page PowerPoint deck provided by the consultancy company.
This is the story of how a major program was saved by descaling it and applying Scrum, and how after multiple years we saw the teams finally deliver working software to the business in just 90 days. I think of this as my own Code Red program (reference described below), and it was my first major success in Toyota North America, and subsequently continuously lauded by senior executives as a major successful case study. It’s what kicked started the company wide adoption of Scrum and what led me to define Scrum The Toyota Way.
Acknowledgement
It is important to note that while I led this initiative, there was a team, and their contributions were equally as valuable as my own. It is also testament to the leadership of my first ever general manager who I learned much from in my early career, and who I’m proud to say came to learn new ideas from me in his later years. Alas, he is no longer with us. It was his decision making as a senior executive, based on my recommendations, that ultimately allowed me to save the program and see it succeed.
Note: This is in no way a criticism of Toyota, in fact it’s quite the contrary as they listened, adapted, and improved. Kaizen at its best. This is a personal reflection of what I was asked to solve. If anything it is a criticism of the consulting industry that I am part of. Consulting companies running rampant and out of control. As someone once said to me “the greater the chaos, the greater the reward”. Consulting companies thrive on your chaos, and often create it so they can be paid to manage it. They even create it without realizing they are doing so, and that’s even worse!
Key Takeaways
- Small is agile and fast. Big is rigid and slow. And expensive!
- The best source of the truth is the people doing the work. Talk to them often!
- Success is more likely if the customer or users are fully engaged. Talk to them often!
- Dependencies slow you down. Closely coupled teams are dependency ladened. Reduce them!
- Hierarchies slow you down. Reduce them!
- Context switching is a huge hidden waste of resource. Eliminate it!
- Email is not a document management system. Eliminate it!
- PowerPoint is not a status reporting tool. Eliminate it!
Remember: If you found this post useful and are not already a subscriber, sign up for the NEWSLETTER.
What Was Code Red?
In 2013 President Obama launched his signature healthcare plan nicknamed Obamacare with a website known as HealthCare.gov.
On October 17, more than two weeks after the site had officially launched, it still wasn’t working. Only 30 percent of people who tried to access the site succeeded, and even those who managed to connect were generally unable to actually sign up for a healthcare plan.
In the Time Magazine article it states “…the post-launch panic was so deep that President Obama actually considered scrapping the site altogether and starting fresh.”
The project was descaled and a small colocated cross functional team had the site working in a month, something the largest consulting companies had failed to do with 100s (maybe 1000s) of people working for several years. The website cost a staggering $1.7 billion dollars! The initial budget was $93.7 million (source: Harvard).
Assessing the Toyota Program
A series of group interviews was conducted using a reflective responses technique. The most important consideration when interviewing people in a group is to remove fear and bias. Fear of repercussions from management if “I say the wrong thing” or “tell on my leaders or colleagues”, and bias through the undue influence of those around the interviewees or in the same session.
The program management was asked to invite representatives at all levels of delivery on the program, business leadership through to developers and testers, and both onshore and offshore team members. It was specifically requested that no one should participate in a group where their direct line management was present.
To establish psychological safety groups of people were asked to attend 1 hour peer workshops with just myself and one colleague, and absolutely no managers present. In fact the managers attended their own session as peers. We did not conduct formal Q&A sessions and instead ran a simple and engaging workshop designed to enable those present to express themselves without fear of recrimination. Executives in charge of the program were interviewed individually.
A series of large Post-it note flip-chart sheets were affixed to walls in the interview rooms. Each was given a question/heading. The headings were hidden prior to the question being asked to prevent forward thinking and planning of responses. The people participating were then asked to take Post-it notes and write responses to the headings and post them on the large sheets beneath the headings. For the offshore teams we built a remote tool to help us. This was before Mural and Miro existed!
Participants were given a 2 minute time-box and only allowed to visit the wall once to affix their responses. They were asked not to read other peoples responses and acting as facilitators we ensured compliance. Participants were also asked not to confer during the process, and we often moved people further apart to any prevent bias, coercion, or simple copying. We were looking for patterns, not robotic responses. For offshore teams we instead on their webcams being turned on so we could monitor for manager interference from the vendor. Sorry, but it’s a thing! “don’t tell the client anything!”
If it was felt bias would be a problem in the group respondents were asked to hand their Post-it® notes to a facilitator who would then place them with others from the group under the appropriate heading on the wall. This way no one knew who had written which response, with perhaps the exception of handwriting experts! If there was any doubt the facilitator would rewrite the Post-it note. The offshore teams simply texted us their responses. Today I might use a tool like Mentimeter to collect anonymous responses.
In practice it was found that participants had little trepidation and were eager to participate and let us know their thoughts. It was a much needed release for them!
It was important for participants to write down the first things that came into their head when asked the question. This was to avoid people thinking what to write and trying and find ideal responses. A phrase I learned sometime ago was “if you’re sitting their wondering what to write, you should stop as you’re just having a meeting with yourself”.
The Questions Asked
Please note that what was asked of participants is of course contextual. What you would ask a manufacturer of water pumps will be very different to what you might ask a digital software developer. Here are the questions used in this instance. They are from a much larger pool and have many variations, so do not use this as a template. You need to design your own.
1. What does this program mean to you?
The idea behind this question was to elicit the understanding or support for the program. The hope was to see inspirational notes but in the main they were bullet point descriptions of the program, and often more negative comments.
2. What are the positives in this program?
A chance for participants to highlight the good things they were experiencing. The responses were sparse!
3. What are the things that could be improved in the program?
This revealed many improvement opportunities. Many problems if we are being honest here.
4. How do you get informed or become informed about important aspects of the program?
5. How do you inform others about important aspects of the program?
Questions 4 and 5 were designed to understand communications in the program. This revealed a huge reliance on asynchronous communication tools such as email and minutes of meetings. There was a huge meeting culture with a deep hierarchy of managers and decision makers. There were few direct conversations, and no immediate and empowered decisions being taken.
6. How do you measure success?
This question was designed to understand what metrics are being used to determine the success or otherwise of the program.
7. Who can disrupt your day and impact your plan?
8. Who can you disrupt either as a consequence of being disrupted yourself or by the actions you choose to take?
Question 7 and 8 were extremely revealing. This showed a very closely coupled workforce who were controlled by a very deep hierarchy of managers with people in directing roles. It revealed huge thrash causing context switching, together with red tape ladened bureaucratic processes, leading to extremely demotivated people, and significantly impacting the ability to delivery.
Most large programs tend to be heavily laden with dependencies. Internal and external vendors together with internal teams are all part of the value stream producing the outcomes. Making the value stream visible is critical in understanding the current system. See my article on The Agile Transformation Myth.
Happiness at Work
Each participant was asked to rank their happiness with their role on a scale of 1 to 5 with 1 being abject misery and death seemingly more attractive, and 5 being so happy they never wanted to leave the office. Yes, I really did ask it just like this.
Participants were also asked to rate their happiness with working at Toyota, working for their employer (they were mostly all contractors apart from the lead executives), and what was one thing that should be done tomorrow to increase their happiness.
The Findings
People and Process
What was discovered was a very demotivated workforce on this program that felt abandoned and neglected. They were mired in ridiculous process designed by people who make PowerPoint decks for a living, and who were simply unable to fix their own creation. The phrase “as useless as a chocolate teapot” comes to mind here.
Vision and Mission
There was a lack of enthusiasm for the program, with most people being unable or unwilling to describe the program in anything other than a purely mechanical fashion. A total lack of enthusiastic adjectives championing the virtues or the groundbreaking and revolutionary vision of this program. People were almost entirely apathetic, and none showed any real enthusiasm to make the program an amazing success. It was just the work they had to do. Key members of senior management were completely disengaged and disinterested in the program.
The Meeting Culture
There was a dreadful meeting culture where meetings were held about meetings. More than once the comment was “we set a date to decide a date”. People were mired in meetings, mainly disengaged and using meetings to hot desk in as they continued doing other work on their laptops, paying scant regard to the meeting topics.
Email & PowerPoint Epidemic
Email was endemic, with many conversations stretching on for months in a single email thread, and with email being used for traceability of requirements and documentation archival! The reliance on PowerPoint presentations for status reporting was out of control. Status reporting was not evidence based, and was, in the main, simply stating work done previous period and the planned work next period. There was no way to know when actual value would be delivered.
Siloed Teams
The teams were structured with largely homogenous skillsets. This guaranteed that no team could be independently successful, multiplying the delays and disruptions caused by excessive dependencies external to a team. The Team structures were so bogged down in a morass of external dependencies that neither the Business nor Leadership could have had any reasonable hope to resolve the serial conflicts that were guaranteed to arise from the functional isolation.
Deep Hierarchies
The layers of management created unnecessary bureaucracy and paralysis that were simply wasteful and provided no value to the program. The responses in the reflection exercises revealed so many interdependencies that almost everyone could, and more often than not, did interrupt and impact everyone else. There was an expression that no one was seemingly able to actually get anything done that they had planned. Planning work was pointless.
Managers Managing Managers Managing the People who do the Work.
It was quite staggering to find so many people in the interviews with the role of manager. The distribution of direct technical contributors actually designing, writing, or testing code versus leadership and support staff was completely unbalanced, and was likely due to the unnecessarily complex web of interdependencies. I personally ran a session with 15 people in the room, each with the title manager. I asked them what they did. They all replied “scheduling manager”. They were a team of managers whose only role was to schedule meetings for other people to attend, many of them managers as well.
Context Switching
Context switching is the cognitive waste that occurs when people are switched from task to task and fall out of flow (lose focus). When people work on concurrent tasks, switching constantly between them, their ability to focus and work effectively declines rapidly. As described by Gerald Weinberg in 1992, when a person works on two things at a time their effectiveness drops by 20%. This rises to 40% when we add a third task, and by five concurrent tasks we have lost around 75% effectiveness. This is one of the most significant causes of project delays.
In this program it was caused in part by malformed teams and a lack of prioritization. Managers were managing by squeaky wheel. Whoever was making the most noise was prioritized. Several participants, coming from all disciplines, expressed their frustration at the fact that no one person knew the entire program’s requirements. This led to a chaotic pursuit of the work based largely on rumor, rather than direct, clear, and confident leadership toward a specific goal.
Measuring Value and Progress
Form over function was an oft used phrase, which suggests people were focused on getting something built, rather than building the right thing, or building the thing right. The phrase “pixel pushing” was used by some. This refers to being asked to do cosmetic changes as opposed to focusing on the real requirements, probably at the behest of a manager.
Multiple sequential and dependent software releases were being worked on simultaneously leading to endless rework as they tripped over each other. Despite multiple attempts by some teams to release their work they were constantly blocked by other interdependent teams who were not ready for deployment, or were entirely unaware that a deployment was imminent. We noted evidence of this when multiple business divisions were expected to collaborate. Their lack of effective collaboration was exacerbated by the siloing effect.
It was not possible to get a clear expression of what ‘value’ had been delivered, and what ‘value’ was contained in future releases. There was no immediate understanding of how the value of the program was being measured, other than meeting milestones and creating more output. There was a total inability to demonstrate any working software to the customer.
Hitting dates or milestones seemed to be more important than actually being able to measure real value delivered. Few people even mentioned the customer. We specifically asked a couple of the groups, “do you have any friendly customers who would be willing to work with you as you provide small increments or value and for you to seek their feedback”? The responses were glassy eyed, “I think so, but I couldn’t tell you who they are”. The customer was absent and disconnected from those doing the actual work.
An Action Plan is Formed
After my recommendations were studied a decision was made to shut down the program in its current form, re-think its outcomes, and then re-launch after re-tooling. On my recommendation Scrum was selected as the operating model to guide the new teams in a disciplined way of working. As part of the “Scrum The Toyota Way” teaching we consider Scrum as a PDCA cycle with time-boxes with key roles to guide the process. Scrum is not some magical incantation cast by wizards with omniscient powers, nor a panacea. It is however, very effective if used as intended, a small lightweight approach to planning. Yes, Scrum is ostensively a planning approach.
Customer Engagement
The executive leadership concerned with the program were asked to reevaluate the outcomes desired by the customer for the system, and to ensure there was full support and commitment from all stakeholders to make those outcomes successful. The business was asked to become fully engaged and become participatory members of the program, and the right leaders did step forward. Toyota leaders are passionate about improvement and loyal to a fault.
Requirements Evaluation
A Product Owner was chosen to comb through the project and collect all requirements under consideration. Requirements were triaged and prioritized for orderly and rapid execution. This enabled focused and clear conversations with customers and stakeholders, who were asked to make trade-offs in the interest of balancing investment and time to market.
An assessment was made with the customer fully engaged as to the key features, functions, and capabilities desired. The Product Owner reviewed what had been delivered and validated by the customer, business, or stakeholders, and what, if anything, could ship into production to real users. Nothing could.
Team Members
To ensure rapid decision making and execution any roles not directly contributing to the outcome were eliminated and people redeployed to value creation elsewhere. We also significantly flattened the management hierarchy to reduce noise and increase the speed of decision making.
The program management were asked to evaluate the contractor team members they had working on the program and to select their best 15 people. The rest were rolled off the program and either moved to other initiatives or removed by the consulting partner to be used with other clients etc. The bloat on the program was palpable and the waste was truly horrific.
Teams needed to become lean cells capable of working independently and delivering value each week. Teams needed to be empowered and given the tools and access to systems to enable them to work without interruption. Team size was limited to prevent waste that is created by adding unnecessary communication pathways. As a team size (nodes) increases so does the number of possible pathways of communication (links). This leads to ineffective communication. How many meetings have you been in with more than 10 people? That’s why nothing ever gets accomplished.
Executive Action Team
Borrowing from the playbook designed at 3M when I led their first large implementation of Scrum, an Executive Action Team (EAT) was created, a technique pioneered by the co-creator of Scrum. The Executive Action Team was accountable for removing impediments that only they had the empowerment to act on. They met daily just as any team would do in their own version of a Daily Scrum or daily standup. They had a backlog of impediments and blockers that were raised by the teams and they self assigned and took rapid and concrete action to resolve or mitigate anything hindering progress of the program. They remained accountable for ensuring nothing blocked progress.
Executive Product Ownership
An Executive Sponsorship Group was created, who met at least once per week. From the same 3M playbook we adopted the Executive Meta Scrum (EMS). This group is tasked with ensuring we remain focused on the correct priorities, and to be informed of progress on those priorities. This group owned the scope of the program, and worked to determine what value should be delivered, and in what order. They set the priorities, and agreed any changes or deviations from those priorities. They were the ultimate Product Owner.
3M was the first true large scale implementation of Scrum at Scale and where the metascrum patterns for layered product ownership were devised and tested. The idea of a metascrum at executive level preexisted, but the adoption of multiple layers of “metascruming” throughout a scaled product owner organization had never been implemented before in this way, and no name existed for that organizational design element. When I used the term Product Owner Metascrum I had to convince the co-creator of Scrum to adopt it. He had argued it was only an executive pattern. When its usage was explained no alternative was proposed and the name stuck. The visuals of this pattern my colleague created at the time now form part of the Scrum at Scale guide and the rest is history.
The notion of agility suggests the ability to pivot rapidly, changing direction as you respond to external events and triggers or new information, and it is the executive metascrum that decided when and how the program would pivot. At 3M, the CEO would hold her EMS every 2 weeks. Yes, the CEO was doing backlog refinement every 2 weeks with her leadership team.
Eliminate Committees
The creation of these two decision making leadership teams removed the need for committees and steering groups to exist. The program outcomes were made clear, and all people on the program were delivering value through prioritizing and focusing on the clear outcomes. Non value added work was simply not undertaken.
Reporting and Metrics
Evidence based reporting was implemented with progress and status available real time. This provided trends and forecasts based on real facts together with historical data, and not arbitrary forecasts or milestones that were the norm at the time. Real-time simply meant that as work was started and completed it was recorded and visualized for all to see, a core Lean tenet. Scrum, and the Lean Thinking that underpins it, advocates that work is refined (broken down) into small batches. This simplification enables accurate measurement of progress and aids problem solving and decision making.
Outcomes
From the 15 members selected to remain on the program two Scrum teams were formed. Sometime later a third was added as need was identified. The Product Owner, as part of the overall team, worked with the team members and stakeholders to refine a prioritized backlog of work to be completed, with each feature or function request be evaluated for its value (the why).
A coach from the Toyota Agile Practice, something we had formed, was assigned to guide this new group. He was instrumental in forming a working metascrum between two major Toyota entities to foster collaboration between them, and to ensure alignment on backlog priorities. This also significantly improved collaboration between various dependent groups that the program still relied on. It also gave rise to a pattern we called a Bridge Team to “bridge the gap” between the traditional waterfall project management world and the emerging Agile world, something I’ll write about soon.
90 days after the program relaunched these new teams deployed a working system!
After years of trying with $100s of millions spent and with 100s of people involved, this group of people in two small teams succeeded. My Code Red had worked!
In the following months the teams went from strength to strength engaging directly with the external dealer network, and launching more and more functionality in the system, always listening to the customer feedback and adapting the requirements backlog in response.
This was the first major success for using agile thinking combined with the Toyota DNA in the Toyota Production System. Scrum The Toyota Way was born.
