Communications of the ACM
Home/Magazine Archive/September 1999 (Vol. 42, No. 9)/The Effectiveness of Distributed Mission Training/Full Text
Communications of the ACM
The Effectiveness of Distributed Mission Training
The advances in computing and communication technologies have led to synthetic training environments with tremendous potential. Distributed Mission Training is such an environment. DMT has adaptively focused and integrated VR and networking technologies for team training in dynamic operational environments. Intra- and interteam communication, coordination, and decision making in such environments are central to mission-critical team performance. While these attributes are especially significant in military operations, they define effective teamwork in general, ranging from sports and entertainment to production and service environments.
The concept of DMT has been technologically demonstrated in defense operations. Equipped with advanced image generation technologies, high-resolution displays, and secure distributed networks, DMT systems connect a wide variety of local and geographically dispersed virtual training platforms for mission-critical team training. Further, DMT technologies provide extremely high levels of both physical and functional fidelity in team training. The resulting real-time virtual training networks include aircraft simulations and other simulated systems such as tanks and ships. In addition, these platforms can be linked to actual training equipment, such as aircraft, thus offering the potential for synthetic environments that support training at the individual, team, and joint service levels. As a result of these increased capabilities, combined with significant reductions in costs, the training and technology development communities are greatly expanding their use of synthetic virtual training environments.
However, there are still formidable challenges in the implementation of these systems. Indeed, training is substantially more than putting hardware and software systems to work together. It involves managing the participants' training experiences to give them a greater potential for accomplishing real-life missions than before. Indeed, training is a systemic phenomenon. The development and implementation of instructional systems [4] follows the general principles of systems engineering in that it involves mission analysis, identification of system inputs and outputs, and allocation of functions to various system components. Based on this concept, simulators are merely subsystems or components of the overall training system. Consequently, it is necessary to ask whether or not these components are functioning correctly and generating the proper outputs. This leads us to the key question in the design and implementation of DMT: Does the integration of DMT technologies into a training system materially improve the likelihood those who use DMT will successfully accomplish their missions?
This article examines this question. In particular, the article focuses on the challenges DMT should address, existing empirical and analytical data that gives cause for optimism, some design considerations for DMT systems, and the major cultural changes that must take place in the way organizations view training so that DMT can have the largest possible impact.
Team Training Poses A Naturalistic Paradigm
A naturalistic perspective on training is to view the process and its effectiveness from the way people use their experience to analyze, interact, and make decisions in field settings [7]. This is dramatically different from the traditional ways of analysis in the sense that emphasis is placed on how trainees are concerned about sizing up a situation and refreshing their situational awareness through feedback, rather than reacting to environmental stimuli and their demands. The contextual factors that affect the way real-world operations occur are central to naturalistic analysis, and are summarized in Table 1.
A central theme emerging from an analysis of training processes is that decision making stems from situational awareness and assessment, prioritization in dynamic task environments, and action/feedback structures in event management. Individuals operating collectively in mission-critical, task-oriented environments are, in essence, decision makers who together determine the final outcome of a mission. So, the naturalistic perspective requires an investigation of what makes them effective decision makers. Once established, these attributes can be used to specify which knowledge, skills, and processes must be learned in order to achieve targeted performance. Table 2 presents a summary of attributes of an effective decision maker operating within the context of a team and its goals.
Does the integration of DMT technologies into a training system materially improve the likelihood those who use DMT will successfully accomplish their missions?
The objective of any team-training enterprise is to develop individuals trained to a desirable level of expertise in these attributes. This translates to a training focus on the major areas shown in Table 3.
This analysis clearly describes what is required to accomplish mission-critical team performance. First, context-specific domain knowledge is crucial. Next, a set of cognitive processes and skills need to be addressed (see Table 3). Finally, the psychomotor skills required in the operational settings need to be emphasized. Focusing on these facets in a training program would accelerate the acquisition of proficiency, and the learning and organizing of domain knowledge that supports complex team maneuvers. These capabilities are crucial in achieving the ultimate objective of a desired level of expertise in both individual tasks and team missions. Team training based on a combination VR technologies and real-time concurrent connectivity among remote players addresses these facets, while at the same time it appears to overcome most of the practical limitations of time and space in a meaningful training environment. While concurrent connectivity is necessary in team-training scenarios, the role of VR as a training medium has been a subject of great discussion among researchers. We take the position that VR is indeed an effective medium given the following line of reasoning.
First, simulated training environments can significantly accelerate proficiency by exposing trainees to the kind of situations they are likely to encounter in the real world, but which could be hazardous or very expensive to practice in actual operational settings. For example, in training Air Force pilots in mission combat, shooting an enemy aircraft can never be practiced with real aircraft exactly as would be encountered in an actual situation. Second, simulations can be controlledthe characteristics of the training scenarios, situational cues, and decision outcomes can be provided as aids in the development of situational awareness, pattern recognition, and template building. The constructive models that play a crucial role in DMT systems are intended for this purpose. Finally, simulations are also an effective means to train reasoning skills, metacognitive skills, risk-assessment skills, and communication skills without the overhead and complexities of real-world training.
This analysis, while highlighting the needs of team training from a naturalistic perspective, also raises research questions regarding the training effectiveness of concurrently connected VR environments like DMT:
- On what basis do trainees perceive similarity in training situations? What triggers a template in human memory? How do people seek additional information, learn from other co-trainees, or exchange data?
- How can the virtual training components be integrated with real aircraft training components to provide comprehensive training effectiveness?
- In knowledge-rich training environments, how should the knowledge be organized into virtual training modules so it fosters ready access to information when necessary?
- How can the quality of training in virtual platforms be evaluated? For example, how do we know when someone becomes an expert?
These are open questions, yielding a rich research agenda.
Lessons from History
Although there have been a number of efforts involving large-scale simulation for analysis, development, and training [8], most of this early research focused on command and control systems. Recently, there has been a substantial research thrust on identifying team skills and developing techniques for training those skills [10, 11]. These works have identified several training strategies such as guided practice and cross-training that can be effectively exploited in DMT systems. However, very little of this research has been applied to designing training for the high-performance team skills typical of DMT.
The earliest virtual platform training programs involved flight simulators and concentrated mainly on training basic procedural and psychomotor skills. By the mid-1980s, however, technology advanced to the point it was possible to interconnect simulators and conduct team training. The first example of such training was the SIMNET project in which a number of tank simulators were interconnected to provide collective training [1]. Subsequently, several studies on team-training pilots on virtual platforms have been carried out. Principal among these were networked F-15 virtual platforms studies at McDonald Aircraft Company, Multiservice Distributed Training Testbed (MDT2) [5], and the RoadRunner'98 study of networked DMT systems (see the Crane article in this section). In all these studies, a full spectrum of simulated team missions was carried out. Teams performed their normal mission planning and post-mission analyses. A qualified instructor monitored the training scenarios and provided additional guidance to the teams.
The growth in computer and communication technologies provides unparalleled opportunities to create synthetic environments to revolutionize training.
Training effectiveness research data collected from these experiments examined both process and product measures. Process measures pertain to the frequency and quality of communication, mutual support, and decision making. Product measures pertain to numerical assessments of performance. In addition, each pilot underwent extensive interviews to determine how well they thought the training experience prepared them compared to their normal team training in their actual equipment. The results were quite positive. In all cases, the pilots showed considerable improvement on both the process and product measures from the first day of training to the last day. The probability of successfully completing a mission improved significantly during the training week. Perhaps even more encouraging were the interviews that showed participants were generally quite positive about the experience.
It is important to stress how seldom these teams are able to not only train together on a consistent basis but also be able to plan and analyze missions together. When training with actual equipment, the physical distances that separate individual players when the exercise is over make it difficult to have such sessions. As a result, even though each of the individual service teams was proficient at its part of the mission, they had very little opportunity to integrate their skills with the other members and work those skills in a realistic tactical environment. All these studies show it is possible to cross train using DMT with multiple virtual platforms in a single training package linked over distances to produce an effective composite training environment.
Naturalistic Design Considerations
The studies discussed here indicate that DMT offers the potential to be a significant training tool. Based on the data gathered thus far, and faced with the increasing pressure to reduce training costs and improve trainee skills, DMT appears to be a cost-effective training approach. Networkable, high-fidelity DMT systems encompassing a wide range of virtual training platforms are fast emerging, and are most likely to dominate the training landscape of the next millennium. However despite the current enthusiasm for DMT, a variety of significant challenges in achieving the desired training effectiveness still remain. We examine some of these issues from the point of view of human-centric and naturalistic systems design as follows.
Skill acquisition, retention, and transfer. Clearly the intent in developing DMT systems is to provide additional opportunities for individuals to acquire and maintain job-relevant skills. There is a great deal of basic research on skill acquisition and decay that can be used as a starting point for identifying those factors that influence skill acquisition and retention [9]. There is, however, very little data describing how complex skills are acquired and maintained in real-world work settings [12]. Further, almost no research involving a detailed examination of how continuation or field training experiences, as opposed to formal institutional or academic training, affects the development of a skilled team player. A well-structured plan of research is required to understand the development and retention of high-performance skills. The results of such research could then be used to define the training strategies and types of experiences necessary to maintain the training readiness of teams in general.
There is very little evidence regarding the transfer of skills from virtual platforms to the actual equipment [2]. Although significant improvements have been repeatedly claimed using both outcome and process measures, almost no empirical evidence is available regarding the degree to which the knowledge, attitudes, and skills learned or enhanced in virtual platforms transfer to actual mission performance. This absence of transfer data is not unique to DMT. Goldstein [6] observes that very few training programs include a systematic evaluation of their effectiveness. Such evaluations are usually difficult to conduct. Bolcovici [3] describes some of the factors and the problems involved in attempting transfer of training studies within the military.
Although transfer of training research is difficult to accomplish, it is essential to validate the benefits derived from DMT. Specific studies on transfer of training from DMT systems to real-world performance are needed. Without appropriate transfer of training data, we are left in a situation similar to that encountered by many college students: one may be capable of passing the exams within a class but not be able to apply the skills and knowledge acquired in that class in other courses. In addition, transfer of training experiments provide a means of determining which variables in a virtual training system have the greatest impact on the quality of training yielded by that system. Such information is invaluable in deciding between alternative system configurations and developing the most cost-effective training systems. Finally, critical to the development and interpretation of research involving skill acquisition, retention, and transfer is the need to develop measures of individual and team performance most appropriate for the complex high-performance skills characteristic of DMT environments. Without such measures, it is difficult to develop valid training metrics, validate fidelity requirements, or determine training needs.
Instructional features. Even if it were possible to create a synthetic environment that fully replicated the real world, it is not necessarily the case that such a design would represent an optimal training environment. Indeed, if the focus is on merely re-creating reality in a digital world, it is quite possible that a suboptimal training environment is created. Such a possibility would result from failures in considering the constraints and training possibilities of hardware and software tools, and the strategies and tactics for using those tools that would allow greater training efficiencies than are currently possible in the real world.
DMT represents a quantum leap in the complexity of simulation-based training. Indeed, it involves a shift from direct control of an individual learning psychomotor and procedural skills to indirect control of large numbers of individuals executing complex hierarchically nested sequences of psychomotor, procedural, cognitive, and team skills in fluid, rapidly changing environments. This new training environment, where the instructor may be much more likely to be involved in process and supervisory control of training activities rather than one-on-one instruction, demands a human-centered design approach that targets hardware and software designs to meet user needs. Table 4 presents a set of critical design questions that must be well considered in this process.
Vertical and horizontal connectivity. Training includes actual equipment, such as aircraft, operating in its natural environment (real training events), simulation trainers operating in a virtual environment (virtual training), and computer-based training (constructive systems). These components can be combined in a number of ways to create many different synthetic environments for training that range from a level of small group engagement to strategic training involving a joint task force. It is also possible to nest various levels of training within higher levels. Thus, a small group may be engaged in a set of tasks occurring as a direct result of the actions taken by the joint task force leader and the results of that engagement will directly influence their subsequent planning and decisions.
While it is possible to link a wide variety of live, virtual, and constructive training events both horizontally across a capability echelon and horizontally across echelons, it is essential we identify the training benefits for each level of participants in team training. The capability to create larger and larger scenarios with more participants does not necessarily increase the training value of DMT. It is possible to position individuals or teams in certain specialties or echelons serving as training aids for other specialties or echelons. If this is not considered during the design of training scenarios, we run the risk of alienating some participants and also reducing their specific task-critical training opportunities. Assuming there is a valid training reason for linking various echelons of players and classes of training in mission-specific aggregate training events, a number of unresolved technical issues (for example, aggregation and separation of team members and communication between constructive and manned systems) as well as training issues (for example, behavioral representation and scenario management) still need to be examined.
Physiological training issues. In real-life missions, team players are usually exposed to a wide variety of psychological and physiological stresses. While technology is providing us with an ever-increasing ability to replicate many of the psychological stresses, we are still severely limited in our ability to duplicate many of the physiological stressors encountered in practice. For example, consider the physiological environment in which the modern fighter pilot operates. This environment involves breathing oxygen, being cramped in a small space, experiencing a number of conflicting sensor cues, and enduring various degrees of G-stress. It is impossible to fully replicate these physiological stressors in an affordable simulator. Unfortunately, we still do not know the optimal mix of cognitive, procedural, psychomotor, and physiological training required to develop and maintain one's ability to successfully employ a fast, highly maneuverable fighter aircraft. Althought DMT may allow trainees to receive better training in certain higher-order, cognitive tasks, it may also provide poorer training in those tasks that are closely tied to one's ability to quickly recognize an opportunity and aggressively maneuver the aircraft to secure a tactical advantage. These are critical research issues that need significant attention.
Conclusion
The tremendous growth in computer and communication technologies provides unparalleled opportunities to create synthetic environments to revolutionize training. Today, we are witnessing the first attempts to integrate this technology within Air Force training systems as part of the DMT program. Initial demonstrations of DMT have been extremely encouraging. Although the DMT technologies are indeed quite impressive, they represent only a fraction of the big picture that emerges in the training landscape. Equally important are the soft technologies (human factors, education, and applied cognitive science) that are essential for the development, implementation, and assessment of training. These soft technologies are critical to the design and development of training delivery systems as well as training evaluation. While DMT has made significant technical progress, there still are a number of human-centered challenges that must be addressed in order to deliver effective and efficient training.
References
1. Alluisi, E.A. The development of technology for collective training: SIMNET, a case history. Human Factors 33 (1991), 343362.
2. Bell, H.H., and Waag, W.L. Evaluating the effectiveness of fight simulators for training combat skills: A review. The Int. J. Aviation Psychology 8 (1998), 223242.
3. Boldovici, J.A. Measuring transfer in military settings. Transfer of Training: Contemporary Research and Applications. J.S. Cormier and J.D. Hagman, Eds. Academic Press, New York, 1987.
4. Dick, W. and Carey, L. The Systems Design of Instruction. Harper Collins, Tallahassee, Fla., 1990.
5. Dwyer, D.J., Oser, R.L, and Fowlkes, J.E. A case study of distributed training and training performance. In Proceedings of the Human Factors and Ergonomics Society 39th Annual Meeting. Human Factors and Ergonomics Society, Santa Monica, Calif., (1995), 13161320.
6. Goldstein, I.L. Training in Organizations: Needs Assessment, Development, and Evaluation. Brooks/Cole, Monterey, Calif., 1986.
7. Klein, G.A., Orasanu, J., Calderwood, R., and Zsambok, C.E. (1993). Decision Making in Action: Models and Methods. Ablex, Norwood, NJ.
8. Parsons, H.M. Man-Machine System Experiments. Johns Hopkins University Press, Baltimore (1972).
9. Patrick, J. Training Research and Practice. Academic Press, San Diego, Calif., 1992.
10. Salas, E., and Cannon-Bowers, J.A. Methods, tools, and strategies for team training. Training for a Rapidly Changing Workplace: Applications of Psychological Research. M.A. Quinones and A. Eheenstein, Eds. APA Press, Washington, D.C., 1997, 249279.
11. Salas, E., Dickinson, T.L., Converse, S.A., and Tannenbaum, S.I. Toward an understanding of team performance and training. Teams: Their Training and Performance. R. Swezey and E. Salas, Eds. Ablex, Norwood, NJ, 1992, 329.
12. Welford, A.T. Fundamentals of Skill. Methuen & Co., London (1968).
Tables
Table 1. Contextual factors in real-world operations.
Table 2. Attributes of effective performers in team contexts.
©1999 ACM 0002-0782/99/0800 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 1999 ACM, Inc.
No entries found