Communications of the ACM
Implementing Distributed Mission Training
The goal of Distributed Mission Training is to provide individuals with opportunities to train for real-world operational missions in an environment that is not constrained by security, cost, and safety restrictions. Even the most elaborate real-world field exercises place severe limitations on participants so they cannot really train the way they will operate. Unlike a training range, pilots in a virtual flight training environment can fly, attack, and defend themselves as they would in wartime. More importantly, they can then review their actions, discuss what they did well or poorly during training, decide how they would do it differently in the future, and then practice new plans in subsequent training events.
Here, we present a case analysis of a DMT implementation. An experimental training program named RoadRunner'98 was conducted at the U.S. Air Force Research Laboratory to evaluate the training effectiveness of DMT systems. RoadRunner'98 combined virtual (man-in-the-loop) training events on DMT platforms with computer-generated constructive models. The systems were integrated over a wide geographic area through secure networks carrying voice, data, and image information.
The objectives of RoadRunner'98 were to demonstrate the state-of-the-art DMT technologies, identify the strengths and shortfalls of these technologies, explore how to best use this new training environment to enhance team training and develop a research and development agenda for the future.
DMT Structure of Roadrunner'98
RoadRunner'98 consisted of four F-16, one A-10, four F-15, and AWACS command and control virtual simulators networked both locally and over a wide area as shown in Figure 1. In addition, limited fidelity platforms were linked as adversaries, and an observation capability was provided at the Pentagon. The composite force missions were carried out over a virtual gaming area. Each mission was executed over a secure, wide-area network using Distributed Interactive Simulation (DIS) communications protocols. In these missions, virtual platform players interacted with constructive forces including friendly and enemy fighters, helicopters, and ground vehicles, plus enemy surface-to-air threats.
All links in the network were commercial T-1 lines except for the connection from Kirtland Air Force Base, NM, to Tinker Air Force Base, OK, which was built around two conventional telephone lines, and the connection from Mesa to Ft. Dodge was an experimental satellite link. The bandwidth provided by the T-1 line was fully sufficient for the exercises conducted. However, phone line connections to Tinker provided significantly less bandwidth. Therefore, the data stream was filtered down to only voice communication and aircraft location. While this bandwidth was sufficient to support virtual AWACS participation, the connection suffered from reliability problems.
All training platforms were virtual. The virtual F-16 platforms were equipped with the Modular-Mobile Display for Advanced Research and Training (M2DART)a state-of-the-art, full-field view, rear-projection, dome-display system. The visual imagery was provided to two of the F-16 platforms from Lockheed-Martin SE2000+ computer image generators, which utilized polygonized terrain and feature representations augmented with cell texture maps. Imagery was provided to the other platforms from Silicon Graphics' Reality Monster image generators which used polygonized terrain representations augmented with aerial mapping photographs of the training area. Compared to the Reality Monsters, the SE2000+ image generators provide more vertical features that can be discerned at low altitude, while the Reality Monsters provided more scene realism at medium altitudes. The four virtual F-15 platforms were equipped with a single 48-inch CRT displaying each forward, out-the-window visual imagery. The AWACS controllers used a training simulator adapted for DMT exercises.
Constructive simulations provided many active entities, including AWACS and tanker aircraft, and supporting friendly combat forces. Constructive forces also included enemy fighter aircraft and a surface-to-air defense system that incorporate radar, missiles, and anti-aircraft artillery. The control consoles at the various sites provided both an interface for system operators to conduct a mission and an observation station for instructors to monitor team performance. The control console incorporated an operator's station, the test director's station, and an observer's station. The instructor monitored mission progress on a six-screen video display system. The test director could communicate with all participants and issue global operational commands. The post-mission analysis systems incorporated synchronized replay of a plan-view display together with videotapes recorded from each virtual training platform.
Training Scenarios
The training scenarios were based on team exercise objectives, technical feasibility, and similarity to F-16 and F-15 operational missions. The training scenarios were designed to simulate a small portion of an ongoing conflict that included both ground and air forces. In these scenarios, F-16s would typically fly into enemy controlled territory to attack a target such as an airfield defended by both surface and airborne threats with air cover from the F-15s. The exercises were conducted over a synthetic gaming area in Nevada. Pilots were free to fly anywhere within the gaming area, at any altitude, and at any speed. Pilots were also free to employ countermeasures against missiles as required. The mission scenarios required pilots to fly specified routes to and from the target areas, which were selected to insure pilots would encounter constructive forces. If attacked, however, pilots were free to deviate from planned routes. The role of the virtual F-15s was to achieve air superiority within a specified time interval. The role of the virtual F-16s was air-to-surface attack, and the task of the AWACS team was to provide radar coverage. The lead F-16 pilot was the mission commander whose tasks were to organize and instruct the different virtual teams on the mission plan, lead and coordinate their actions, and conduct post-mission discussions.
RoadRunner'98 and similar exercises provide information that will significantly affect the development of DMT systems and applications.
Three sets of pilots and AWACS crews with 3001,200 actual flying hours to their credit participated in the study. At the beginning of each mission-planning period, the mission leader contacted the other players to review the mission plan. At the scheduled time, pilots and controllers would get into their platforms and perform systems and communication checks. When all players had checked in, the test director commenced the training exercises. After each scenario was completed, teams were instructed to conduct their post-mission analyses in two phases: local (intrateam) and wide-area (interteam). During local analysis, teams would replay the mission tapes with the goal of understanding what happened during the mission and generating points for review by all participants. The teams prepared a list of discussion topics for the mission commander who prepared the interteam discussion agenda. All teams would then restart their tapes at the beginning of the mission and pause for discussion as directed by the mission commander. At the conclusion, the participants filled out a questionnaire identifying performance issues of DMT.
Results and Mission Performance
RoadRunner'98 was conducted over a period of five days, with each team performing two training missions per day. In order to evaluate the training effectiveness of DMT, similar missions were performed on the first and last days of training. This exercise structure enabled an assessment of team performance changes attributable to the DMT experience in the five days of training.
All missions were completed with no aborts or significant delays. The exercises established the reliability of the communication systems and the sufficiency of the bandwidth provided for the scenarios tested. Comparing team performance from the first mission to the last shows that accuracy in ordnance delivery increased by 30% and aircraft losses to enemy fighters decreased by 44%.
In addition to mission performance data, training instructors observing from the test director's console rated their team using a (15) teamwork effectiveness scale. Instructor ratings significantly improved for overall mission performance, communications and coordination, situation awareness, and crew coordination from the first day to the last day of training as shown in Figure 2.
Assessments of the quality of DMT were carried out by the trainees as well. Before the start of the RoadRunner'98 program, they were asked how well their current level of training on actual aircraft prepared them for many of the mission tasks. They completed a similar questionnaire at the end of the week regarding the effectiveness of DMT for the same tasks. Aircraft training was rated higher than DMT primarily for tasks involving visual detection and recognition of aircraft in their fields of view. DMT was rated as more effective than aircraft training for tasks infrequently practiced in the air due to resource, safety, and security considerations. In particular, DMT was rated as more effective for radar and communications intensive, beyond-visual-range tasks.
Pilots were also asked to critique the effectiveness of DMT after each mission. The weak points were the inability to determine another player's aspects at realistic ranges and the behaviors of some of the constructive models. The consistent strengths of DMT were the mission recording and replay systems, the opportunities for premission planning and post-mission analyses, the high level of immersion provided by the DMT environment compared to other simulators, and the instructor's ability to monitor all team players from the console.
Summary and Conclusions
RoadRunner'98 and similar exercises provide information that will significantly impact the development of DMT systems and applications. Don Norman [1], in his work on product development, describes how systems designs based on lists of desired features often fail when they are integrated into work environments. He recommends that designs should emerge from observing people at work along with frequent prototype evaluations by users and the design team.
Reviewing participant critiques from Road-Runner'98 demonstrates the value of Norman's advice. Everyone who has flown an F-16/M2DART simulator, other than fighter pilots, rates its fidelity and training potential as unlimited. While the pilots in RoadRunner'98 appreciated the scene quality, they also criticized simulators for failing to display other aircraft adequately. As configured during Road-Runner'98, the F-16/M2DART did not support the intended user's needs although non-pilots would never have identified this shortcoming. On the other hand, AFRL's mission replay system, which was a conglomeration of a DIS plan-view display and four videotapes, appears awkward and clumsy to any engineer. While pilots recognized its flaws, they rated its training value extremely high. Teams would frequently spend two hours reviewing a 45-minute mission and still be talking about it as they left the debriefing room. During RoadRunner'98, the replay system was neither graceful nor elegant, but it was highly effective in providing the pilots with the information they needed.
The objective of RoadRunner'98 was to evaluate the quality of virtual team training provided with state-of-the-art VR and networking technologies and to evaluate their potential for significantly enhanced training capabilities. Overall, the participants and training instructors rated DMT as very effective within specified domains, notably, multiaircraft air combat against multiple enemy fighters. The most significant opportunities for improvement in DMT technologies were in visual displays, constructive modeling, simulation of electronic interactions, and systems for distributed mission planning and post-mission analysis.
Another major application of DMT identified by the participants and instructors is to use these technologies to gain leadership skills and experience. The team leader plans the mission, makes decisions, informs the other players, manages the missions, and leads the post-mission analyses. Gaining this experience in actual aircraft or other equipment can be very expensive. The participating players in Road-Runner'98 used the exercise to provide leadership experience for younger individuals at greatly reduced cost compared to aircraft training. Overall, RoadRunner'98 demonstrated both the value and limitations of DMT and also illustrated the areas for improvement in future DMT systems.
©1999 ACM 0002-0782/99/0800 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 1999 ACM, Inc.