Communications of the ACM
Home/Magazine Archive/May 2009 (Vol. 52, No. 5)/Reflections Today Prevent Failures Tomorrow/Full Text
Virtual extension
Reflections Today Prevent Failures Tomorrow
He that would perfect his work must first sharpen his tools.
Confucius
Raising the American Flag at Iwo Jima on Mount Suribachi is symbolic of the war effort of 1945. The bombing of Pearl Harbor, D-day, the evacuation of Saigon, and the toppling of Sadaam's statue are symbols that pervade our society as reminders of our Fathers commitment to protecting our country and the world. Historically, after veterans returned home, they considered their war experiences taboo subjects and reluctantly shared details of specific war missions. If, indeed, history repeats itself, we must diligently document the successes and failures of military missions. Realizing the criticality of lessons learned the military has devised a plan to capture immediate feedback after every completed mission. We extract these learnings to apply in software process improvement efforts.
Process improvement requires understanding and measuring existing processes, analyzing the data to discover cost, quality, or schedule improvement opportunities, and introducing process changes to capture those opportunities.10 Within the domain of software development, these efforts are often framed within large enterprise-level frameworks such as the SEI's IDEAL (initiating, diagnosing, establishing, acting, and learning) model.9 Similarly, Basili's Experience Factory model sets out the structure and process approaches for identifying, developing, refining, and adopting "clusters of competencies."2
Organizational-level process improvement efforts can be successful only if improvement ideas are generated and proposed.8 Haley suggests the reliance on ad hoc task teams to develop process improvement proposals (PIP) that may be maintained in a database for periodic review.7 Kellner recommends that PIP be identified from process performers, other projects, and the literature.8
The military's After Action Review (AAR) method, as a model for identifying process performer-generated PIP, may then be integrated into an organizational-level effort. In typical software projects today, such PIPs are identified through lessons learned meetings that may be held quarterly, by phases, or at the end of a project. In contrast, Army AARs are an integral part of a mission plan. We propose that this ingrained process improvement strategy serves as a valuable model for application within software development projects.
What is an AAR? The AAR process has been effectively utilized by the U.S. Army since 1981; it has been so embraced by soldiers and units that it spread voluntarily to other units.5 Army teams utilize the AAR process as a channel to foster process improvements through interactive team discussions.
AAR is a professional discussion of an event, focused on performance standards, that enables soldiers to discover for themselves what happened, why it happened, and how to sustain strengths and improve on weaknesses. It is a tool leaders and units can use to get maximum benefit from every mission or task. It does not grade success or failure... always weaknesses to improve and strengths to sustain.11
Team members and leaders gain a multi-dimensional view of specific individual, unit, process and leader strengths and weaknesses. Army units have adopted the AAR process1 as an integral piece of the unit and training cycle to aid in team development as well as true process innovation and evolution. The process is used everywhere from two-person buddy teams to an entire brigade of Army maneuver elements. It is truly a "culturalized" phenomena used to maintain focus on continuous process improvement.
What can we learn using an AAR? One company commander described an AAR conducted in preparation for Operation Iraqi Freedom. The unit moved from their staging area to the open desert along the Kuwait/Iraq border in the large combat columns they were planning for the move into Iraq:
"The AAR process flushed out a lot of false assumptions as far as what vehicles moved poorly in the sand, long distance communication shortfalls from the lead vehicle to the rear elements of the column. We capitalized by revamping our movement formations and repositioning vehicles within the columns. Without resolution, we would have been troubleshooting these simple issues in combat... detracting from our lethality."
The AAR taught this commander and his brigade what works, what doesn't work, and key risks of the upcoming operation.
The AAR is key to the war process improvement effort. Table 1 details the recommended sequence within an Army AAR as set out for use immediately following a combat training exercise.11 In parallel, we describe equivalent steps for an industrial software project. The steps that directly map across Army missions and software projects include the introduction of the scope, reviewing project or mission action objectives and intent, and summarizing the recent project or mission events. In addition, the AAR activities of discussing sustaining tasks (for example, what the team should keep doing, also called sustains) and improving tasks (for example, practices that must change, also called improves), reviewing specific tasks required to implement the AAR learnings, and summarizing the findings.
Two AAR steps differ notably in their implementation within the two domains. In addition to discussing key issues (for example, steps 4a and 4b); both will separately discuss any critical risk that has materialized, but the critical risks affecting the two domains obviously differ dramatically (for example, steps 7a and 7b). The Army officers will focus their attention on fratricidal events; thereby, immediately devising a plan to prevent such future acts of friendly fire, while still providing force protection. In contrast, the software project team will focus their discussion on any occurrence of project failure; thereby, devising a plan to ensure success in future projects or project phases, while mitigating risk.
Extracting Key Principles. In this study, past and present Army Commanders and Non-Commissioned Officers have captured key lessons for conducting effective AARs at the team (troop) level. Based upon a series of personal open-ended interviews with 12 Army officers, we have extracted key AAR lessons that can be applied in any project setting where process improvement is needed.
When To Conduct
Many software projects follow the accepted practice of periodic (perhaps quarterly) scheduled lessons learned sessions, as communicated by one of the interviewed captains (whose civilian position is with a software development firm). The firm relied on team members to either enter comments into a lessons learned database throughout the project lifecycle, or enter them at the conclusion of the quarter. The Quality Assurance team would then meet with subgroups of team members and discuss the lessons prior to presenting the final report to senior management.
Shifting to our recommended AAR-based approach entails actively planning, managing, and adapting the integration of PIP. Significant time lapses between key events and lesson learned meetings means key knowledge and improvement losses. First hand knowledge and fresh perspective is lost, and mistakes may be replicated during the lag time. So here we address the question: when should AARs be conducted in a software development project?
Continuously. In keeping with the Army's culture of continuous process improvement and illustrated by Figure 2, the AAR and the PIPs it generates should serve as the centerpiece of project operations. The project plan must allow for time to conduct AARs before, during and after project milestones. This is consistent with the AAR approach used by the interviewed Army officers, who discussed AARs as planned both during and immediately following significant events or actions. For example, one aviation officer described conducting informal AARs during in flight battle handovers and at an airfield during mission handovers.
A single AAR can have significant but isolated impact. A culture of continuous AAR's creates a learning organization, as aptly stated by one Lieutenant Colonel: "In learning organizations where people are open, AARs are happening continuously. Organizations as a whole really don't learn; but as individuals learn, as a result of going through the AAR process, the organization grows."
In Turmoil. One of the best times to conduct an AAR is in the height of project turmoil, when the problem is not clearly defined and the typical remedy is to allocate more resources.3 The AAR method recommends holding an AAR, to allow the project manager to "stop, look and listen". A seasoned patrol leader from Iraq stated that some of their most effective AARs were conducted in turmoil, in the vehicle or over the radio during a patrol. When human nature may be saying "act, and act quickly," they would instead quickly huddle during stops or key mission breaks to discuss sustains and improves before beginning the next phase of the patrol; thereby, immediately applying the lessons.
In Stability. When a project is going almost too well, or the project is running smoothly with few issues or deviations from expected norms, this is also a good trigger to hold an AAR. Doing so will capture the sustains in order to maintain positive momentum, and allow the project manager to disseminate these strengths to other project team members who could benefit from improved processes. The AAR could also identify potential risks or shortfalls which might not surface until later in the project; thus, emplacing controls to manage or mitigate the identified risks.
Don't Wait Until Everyone Is Available. Time and resource constraints may make it unrealistic to conduct an AAR with 100% of your project team. Instead of delaying the meeting, the military culture of continuous process improvement suggests performing an AAR using the team members that are available the so-called representative AAR. With partial participation, the team may learn less in a single representative AAR, but the cumulative learning effect of integrating and sharing the results of multiple representative AARs, conducted when the ideas and memories are fresh, is tremendous. One Captain illustrated this representative approach: "In full-scale combat, unplanned representative AARs are essential the urgency and changing conditions mean pulling together an entire section is impossible. Instead, when I get just a minute with a section sergeant or a small number of soldiers, we review sustains and improves. This enables me to maintain situational awareness down to the individual soldier level, and allows the section to work unconsciously towards process improvement."
In contrast, software development teams typically take action only during project turmoil, (such as, a test module fails). Therefore, the AAR method applies quite well to a geographically dispersed team that must continuously monitor their progress during turmoil and stability. By conducting representative AARs with partial participation (that is, at one location) and sharing this knowledge across the dispersed project team, the Project Manager can more effectively manage the success of the project.
The AAR must become entrenched in the software development process, as natural and expected as status reports and activity logs, beginning during the planning phase, throughout the execution of all lifecycle phases, and during project wrap-up. During the project lifecycle, the team can get a strong indication of what is going right and wrong, and the end-of-project AAR serves as a knowledge management tool for future engagements.1
Ensuring Success
How does the typical software project team benefit from such invaluable process improvement? In order to successfully incorporate an AAR process into a project methodology, the project team must adopt and take ownership of the process. Under the AAR methodology, the team owns the responsibility of self observation and note taking in order to sustain strengths and improve weaknesses. One notable change must be immediacy and accountability. Managing lessons learned throughout the project lifecycle allows the team to get a strong indication of the success of the project.
Leader Mindset. Leaders must remain open to self improvement and not hesitate to point out their own progress through the AAR process.5 As one army Sergeant noted, "team members and leaders must have a thick skin". A leader using the AAR as a medium to gather performance appraisal comments, negative or positive, for employee performance reviews is a gross misuse of the AAR. The AAR must remain an independent improvement-focused process which fosters free thought and observation, whether good or bad. It addresses project management and governance procedures; thus, allowing a subordinate freedom to point out a critical leader mistake. It is imperative that the subordinate feels safe making a point without repercussions, since in the Army, that simple leader mistake could cost lives on the battlefield.
Team Member Initiative. The AAR offers an interactive environment where all team members may participate regardless of tenure. The use of AARs within a project team promotes individual reflection and learning1 and fosters initiative "figuring out what needs to be done before being told what to do."6 One Company Commander highlighted that "it is common for an incoming junior level soldier to point out an inefficient process or a simple process improvement that the seasoned soldiers had overlooked."
Team Member Accountability. Holding team members accountable for communicating and adopting process improvements is key to the value of a well-executed and integrated AAR process. This lesson is most evident and critical in Army aviation, where both the risks and costs of failure are at their highest; here the AAR process is implemented at its most meticulous levels. For example, each attack pilot's flight hours and training are tracked, with the pilot classified as either green (ready) or red (deficient). Any pilot who has not reviewed all current AAR-generated messages or conducted updated training required by AAR-generated improves is placed in red status. A red status pilot cannot even sit in the cockpit until he is returned to green status.
Own The Process. Team members must understand that their input and ideas matter1 and they must be allowed creativity and voice in order to not only identify a shortcoming, but also to propose a workable solution. Since the project team holds the resident expertise and current knowledge of a project, the AAR can facilitate problem resolution.
One former company commander stated: "I required all unit members to bring three sustains and three improves to an AAR. The identified improves must also have proposed solutions, the AAR must move beyond problem identification into problem resolution." The commander further noted that the team was more apt to take ownership and implement the changes identified through the AARs since they were involved in the problem resolution process; thus, ensuring continued success of the project.
Going Beyond The Team
AARs are tremendous tools for identifying, sharing, adapting, and using PIPs within a team. Long term and systemic benefits can only occur when those learnings live beyond a single team or a single project.
Cross Team Boundaries. Organizational boundaries often adversely affect communication between groups.4 Since AAR lessons routinely span these boundaries, the AAR can aid in cross team communication. A Major described an effective method utilized in Iraq: "Once a month, soldiers of similar jobs from different units would meet and conduct a job specific AAR. For instance, a lead vehicle gunner from one patrol team would meet with lead vehicle gunners from other patrol teams. These AARs served to spread best practices across multiple teams."
Being separated by time and distance can prevent cross team communication. One Army Major's unit effectively addressed this problem by posting lessons learned: The tactics, techniques, and procedures (TTP) used to counter improvised explosive devices (IEDs), must constantly evolve to counter the evolving insurgent IED emplacement tactics. Each patrol was required to conduct an AAR upon returning to base, regardless of whether an IED was encountered or not. They placed their AAR comments, both sustains and improves, in a lessons learned room. Each outgoing patrol member was required to stop by the lessons learned room to read all current postings. Patrol leaders from other units also routinely visited the room to read current lessons and post their comments.
Software development project teams can easily adopt these practices by sharing knowledge with other organizational teams and learn best practices for addressing and resolving issues. These knowledge sharing sessions can be conducted via face-to-face meetings, video-conference, voice-conference, or electronic postings. By capturing and disseminating the lessons learned from each individual project, software developers can reduce errors and streamline development efforts; thus, reducing the time to complete the project.
Integrate With Knowledge Management. A single AAR provides an immediate short term benefit; however, a longer term benefit can only occur by integrating AAR learnings within a broader knowledge management effort. Two key methods were identified from the Army interviews that dealt with maintaining this knowledge. One Captain maintained a working draft of the unit Standing Operating Procedures on his computer. He would update the draft copy as AARs occurred. Following true tactical pauses, he would circulate the draft SOP to all major team players to ensure its accuracy, resulting in revisions and republication that would provide critical knowledge for all teams in combat.
Will We Repeat History?
As a Company Commander of a combat support company sitting on the Kuwait/Iraq border you know that it is your job to ensure every solider and piece of equipment is ready to roll into combat when the order is given to move forward. Eight months of intensive training does not ease the concerns/questions:
Is my company ready? Have we really focused on process improvement? Has our training really prepared us for the rigors we are about to face?
The AAR stands as the cornerstone to the evolution and constant improvement mindset seen in small Army units. While each leader is charged with leading their units through hard, realistic training for combat, the unit continues to benefit from the results of the AARs following each training event. Combat does not always end like the rehearsals, but through a self improvement and constant learning mindset, the unit can overcome any obstacle and change history.
Can Any Team Use AAR?
The military culture is unique; nowhere else are the consequences of team failure higher, and nowhere else is the team concept tested under such stressful conditions. Can the AAR be successful in a conference room as well as a battlefield? An AAR aids any project manager and team in identifying, evaluating, and adapting PIPs to overcome high learning curves, minimize training costs, while insuring problems are identified and resolved early in the lifecycle. It is expected that, while all teams can benefit from the AAR practices, full benefits can only be enhanced in teams with both high trust and high commitment.6 The AAR process improvement model sets out an organizational roadmap for initiating, planning, and implementing process improvement actions throughout the system development lifecycle. Therefore, the learnings of the AAR process lay the foundation to ensure success of software process improvement teams.
References
1. Baird L., Holland, P., Deacon, S. Learning from action: Imbedding more learning into the performance fast enough to make a difference. Organizational Dynamics, Spring 1999, 1922
2. Basili, V., Caldiera, G., McGarry, F., Pagerski, R., Page, G. and Waligora, S. The software engineering labortatoryAn operational software experience factory. Technical Report, 1992, ACM.
3. Brooks, F. The Mythical Man-Month, Addison Wesley Reading, PA, 1975.
4. Curtis, B., Krasner, H., Iscoe, N. A field study of the software design process for large systems. Comm. of the ACM 31, 11 (Nov. 1988), 1268.
5. Darling, M., Parry, C., Moore, J.Learning in the thick of it. Harvard Business Review, JulyAug. 2005, 8492.
6. FM 22-100. Army Leadership Be, Know, Do. Headquarters, Department of the Army, 31 Aug. 1999.
7. Haley, T. Software process improvement at Raytheon. IEEE Software, 3341.
8. Kellner, M., Briandt, L., and Over, J. A method for designing, defining, and evolving software processes. IEEE, 1996. 3748.
9. McFeeley, B. IDEAL: A User's Guide for Software Process Improvement (CMU/SEI-96-HB-001). Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, 1996.
10. Sommerville, I. Software Engineering, 7th edition. Pearson Education, 2004.
11. TC 25-20. A Leader's Guide to After Action Reviews, Headquarters, Department of the Army, Sept. 30, 1993.
©2009 ACM 0001-0782/09/0500 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2009 ACM, Inc.
No entries found