Communications of the ACM
Home/Magazine Archive/March 2008 (Vol. 51, No. 3)/Requirements Engineering in New Product Development/Full Text
Communications of the ACM
Requirements Engineering in New Product Development
What happens when an organization develops a new product? Although a common activity, both academics and practicing product developers have a limited understanding of the process. The process is both technically and socially complex and interacts with the organizational context in unpredictable ways. A basic property of the process is uncertainty [5, 6, 8], which an organization must gradually reduce as it transforms an ambiguous product concept into specific product requirements and ultimately a tangible product that can be launched in the marketplace. What makes new product development (NPD) and the related process of requirements engineering (RE) difficult to manage is that the sources of uncertainty are largely hidden from the view of managers or analysts, embedded in task-related interactions within complex social networks of interdependent organizational roles.
Recommendations for managing NPD and RE uncertainty emphasize cycles of communication among organizational stakeholders [3, 8]. But how effectively do people in different roles communicate when they have conflicting goals and must compete with one another for scarce organizational resources? And how does the effectiveness of their interactions affect RE and time-to-market for new products? Academics have made little progress modeling and evaluating the effectiveness of such interactions, and product developers often must solve problems by trial and error, which is constrained by the sociopolitical climate of their organizations.
We have been studying and modeling the NPD process in a large multinational high-tech company (MHTC) for the past five years. We have spent hundreds of hours talking to key players involved in NPD and RE, both informally and through formal interviews for data collection. We developed diagnostic methods to help understand the complexity of requirements engineers' social network interactions and to identify significant sources of uncertainty affecting the RE and NPD processes. Here, we examine and report the effectiveness of the interactions and communication patterns between RE and five other organizational roles involved in the NPD process at MHTC.
Study Background
MHTC is a leading Canadian telecommunications firm. To improve its competitiveness, it adopted a strategy of introducing more new products into the market. To be more sensitive to market conditions, responsibility for initiating the NPD process was reassigned from engineering to a restructured marketing unit of the organization called market sponsors (MS).
The company is organized functionally: employees who perform similar functions are grouped into departments, such as operations, sales, marketing, information systems, and so on. Cross-functional communication and coordination difficulties are common in functionally designed organizations not only because each department has its own goals but also because each tends to speak its own specialized language. The collaboration required for NPD success made these communication and coordination problems much more apparent to management. For example, efforts to move new product ideas from marketing through the rest of the organization encountered major difficulties with the new NPD RE process. The problem was information incongruence: requirements engineers demanded a level of technical detail that market sponsors could not provide.
To address these communication and coordination problems, MHTC established another organizational role called feasibility analyst (FA). The task of the FA was to conduct a feasibility study in which the marketing department's new product requirements were mapped onto the firm's existing technical capabilities to determine whether or not the new product could be built and supported at a reasonable cost. Management used the FA's report to decide whether or not to commit company resources to develop the new product. Approved NPD projects were assigned a project manager (PM) whose task was to manage the product development process by setting up and coordinating teams from different functional groups, including an external software development (SD) vendor. Finally, all these roles collaborated with people in the operations (OP) department, which took responsibility for managing product operations following a new product launch.
Cross-functional communication and coordination difficulties are common in functionally designed organizations not only because each department has its own goals but also because each tends to speak its own specialized language.
Method
We investigated the effectiveness of social network interactions and communication patterns between requirement engineers and other organizational roles at MHTC using a semi-structured interview methodology adapted from the "echo" method of Bavelas [2, 4, 9]. Interviews began by asking participants to describe their work situation in general terms and to identify other individuals or groups in the organization with whom they interacted on the job. As a result, a diagram in the shape of a star was formed for each interview, with the participant in the middle surrounded by other nodes representing individuals or groups comprising the participant's immediate task-related social network. The participant was then asked to provide concrete examples of behaviors performed by other groups that were helpful from the participant's point of view, and examples of behaviors that were not helpful. By asking for behavioral examples, the method is designed to focus participants' attention on aspects of the work situation that are meaningful in relation to their own task objectives rather than from the perspective of an external observer.
Twenty-nine individuals were interviewed, representing the five roles identified here (approximately six per role): requirements engineering (RE), project managers (PM), market sponsors (MS), feasibility analysts (FA), and operations (OP). Interviews ranging from two to three hours were recorded and transcribed for coding. Responses were coded collaboratively by the research team members through an iterative process of coding, discussion, and recoding using the QSR N6 [7] qualitative data analysis software. The coding focused on identifying the major categories of helpful and unhelpful behaviors mentioned by interviewees for each role. From these behavior categories, we were able to identify specific patterns of interactions within each working relationship and to evaluate the effectiveness of each working relationship from the perspective of both parties involved.
Results
Strengths and Weaknesses of Each Role. Figure 1 summarizes the main categories of helpful and unhelpful behaviors of RE as identified by the other roles. In total, 201 helpful and unhelpful examples were identified. The percentage of comments in a category reflects its relative importance. Coordination was seen as the most important behavior and availability as the least important. Problem solving and knowledge were of equal importance, followed by communication, and job attitude.
The difference between the percentage of helpful and unhelpful behaviors in a category gives an indication of the relative effectiveness of RE's performance in that category. RE's major strength was knowledge, followed by communication, job attitude, and availability. Their major weakness was coordination. The percentages of helpful and unhelpful comments about problem solving were equal, indicating this was neither a particular weakness nor a strength.
Figure 2 summarizes the main categories of helpful and unhelpful behaviors of other roles as identified by RE. In total, 183 examples of behaviors were identified. The most important category was knowledge, followed by communication, coordination, and problem solving. The least important categories were job attitude and availability. The percentage of unhelpful behaviors was greater than that of helpful behaviors for all six categories, indicating that RE perceived other roles as relatively weak in all areas. The major weaknesses were in communication and coordination, followed by knowledge and problem solving.
One striking difference between the two figures is the asymmetry in perceptions between RE and other roles. Requirements engineering was perceived by others as strong in all categories except for a slight weakness in coordination, whereas RE viewed other roles as weak in all categories. One possible explanation for this disparity is that the other roles generate uncertainty for RE in the form of ambiguous requirements and view the RE as helpful since it manages this uncertainty. On the other hand, RE expects less uncertainty in the requirements of other roles, but these expectations are seldom met.
Effectiveness of Interactions. The ratio of the number of helpful to unhelpful behaviors provides an indication of the relative interaction effectiveness of a link in the network, when compared between links or to the organizational norm. In this case, the organizational norm (average) was estimated as 0.95, computed as the ratio of the total number of helpful to unhelpful behaviors for all roles. Figure 3 is a socio-technical network diagram based on RE's perception of other units and other units' perception of RE. The direction of each arrow indicates a flow of behaviors from one role to the perceiver role, and the thickness of arrows corresponds to the number of behaviors identified for each link. Ratios larger than 0.95 are displayed in green to indicate effectiveness above the organizational norm, and ratios less than 0.95 are displayed in red to indicate below-average effectiveness. Because software developers (SD) were outside vendors we were unable to interview them, thus there is only one arrow from RE's point of view.
There are a number of items worth noting. First, the range of interaction effectiveness ratios (from 0.13 to 1.67) reflects variability in the relationships between RE and other roles. Second, the least-effective relationship is between RE and market sponsors (MS). One reason is that the MS is often uncertain about new product requirements and is also the source of requirements change during the NPD process, creating additional work for the RE. Meanwhile, RE's demands for precise information for requirements documentation are viewed by MS as frustrating and bureaucratic, since new product requirements are still being conceived and full details are not yet available. Third, the difference between link effectiveness can be used as a measure of the degree of asymmetry in the relationship. For example, the difference between RE and MS is 0.08, which reflects a fairly symmetrical relationship. On the other hand, the difference between project managers (PM) and RE is 0.95, and the difference between feasibility analysts (FA) and RE is 0.92, indicating fairly asymmetrical perceptions within these relationships. One possible explanation for these asymmetries is that other roles in the organization represent uncertainty to RE, while those roles view RE as a knowledgeable problem solver and appreciate RE's efforts.
Dynamics of Interactions. To provide a more detailed picture of what happens as RE interacts with the other roles, specific examples of helpful and unhelpful behaviors are given here. Due to space constraints, only unhelpful behaviors are reported for links with below average (< 0.95) interaction-effectiveness ratios, and only examples of helpful behaviors are given for links with above-average ratios.
Requirements Engineering and Market Sponsors
- MS Unhelpful (RE's view): changing requirements halfway through the project, unrealistic expectations about delivery time, doesn't listen to technical explanations, not proactive, not prioritizing customer requirements, force-fits projects within a budget, lacks trust.
- RE Unhelpful (MS's view): overestimates costs, uses technical jargon, proposes expensive solutions, inflexible.
Requirements Engineering and Project Managers
- PM Unhelpful (RE's view): lacks SAP knowledge, lacks technical knowledge, calls unnecessary status meetings and reports, doesn't understand we work on multiple projects, expects immediate response to their inquiries.
- RE Helpful (PM's view): good teamwork, meets deliverables and deadlines, conscientious of what their client groups look for, good budget and resource management, flexible, access to experts, tracks projects, provides status reports, identifies problems and gives solutions, proactive, technical knowledge and experience, dedicated and available.
Requirements Engineering and Feasibility Analysts
- RE Helpful (FA's view): responds quickly, open to questions, knowledgeable in their area, suggests solutions, gives realistic expectations of a product, good relationship with operations (OP), accountable and dependable.
- FA Unhelpful (RE's view): offers solutions without knowledge and expertise, insufficient information about new product at meetings, lacks experience, creates patchwork solution, solves the wrong problem, lacks training, mixes up business and product requirements, documents give sales pitch instead of technical requirements, goes directly to OP and bypasses us.
Requirements Engineering and Operations
- RE Helpful (OP's view): looks at different components, such as networks and maintenance, provides good insight on business direction, good technical knowledge and system solutions, good relationship, knows our communication processes, helps us with technology issues, knows customer perspective and company perspective, clarifies processes and steps, helps write methods and procedures, involves us early in the project, gets input from subject-matter experts.
- OP Unhelpful (RE's view): changes product or technology without informing us, slow response on signoff, unavailable when needed, lacks big picture, engages MS without our consent, changes solutions and dates, gives FA solutions based on minimal knowledge, doesn't solve low-level problems before coming to us.
Requirements Engineering and Software Developers
- SD Helpful (RE's view): builds good code, educates others, good links to analysts, proactive, flexible, willing to adjust priorities, works as part of our team, integrates with our people, after-hours support, provides good information, good communication and relationship, complete and accurate estimates, clarifies when we don't understand the questions.
Our findings pave the way for the continuous improvement of social-network interactions by increasing the frequency of helpful behaviors and reducing the frequency of unhelpful behaviors.
Improving Social Network Interactions
The results provide insights into the behavioral characteristics and effectiveness of socio-technical networks involved in RE and new product development at MHTC. As such, the results establish a base from which to develop strategies for changing network interactions to improve RE and NPD performance. Since many MHTC's interaction difficulties were the outcome of its functionally designed organization structure, one potential way to improve interactions would be to restructure the organization. For example, changing to a product- or market-based design would group REs together with other interdependent roles. A matrix structure could also provide better cross-functional communication and coordination. However, it has been our experience that attempts to redesign large organizations face substantial opposition from major stakeholders. Thus, the recommendations proposed here emphasize more localized strategies that can be implemented within the existing organization structure.
Reduce the Frequency of Unhelpful Behaviors. We found unhelpful behaviors created more work for individuals and delayed NPD projects anywhere from one day to three months. To improve interaction effectiveness between two roles, either the sender role must reduce its unhelpful behaviors or the recipient role must increase its capability to handle the uncertainty associated with unhelpful behaviors [1].
The results indicate that market sponsors were a major source of uncertainty in NPD, engaging in behaviors like "changing requirements" that requirement engineers considered to be unhelpful. One way of reducing the frequency of this behavior would be to involve the RE during the early "ideation" stage of NPD. However, the cost accounting system at MHTC discouraged such early interactions since it required that an RE's time be charged against NPD project funding, which was only available after the project had been approved for feasibility analysis. At least two solutions are available to solve the problem. First, recognizing its importance to project success, higher management in the MS unit could allocate a fund for early RE-MS interactions. Second, REs could provide early project advice free of charge, with costs allocated as overhead to all projects.
Increase the Frequency of Helpful Behaviors. The results also indicate that REs played a major role in handling NPD uncertainty by sharing their knowledge, which other roles regarded as very helpful. A notable difference between experienced and inexperienced REs was that experienced REs had better knowledge of the organization, of MHTC technical systems, and of who to contact to obtain specific information. Sharing this knowledge with less experienced REs would therefore increase the latter's uncertainty-handling capability. One way to do so would be to set up a knowledge center staffed by experienced REs available to answer questions from REs with less experience or to connect them to another person able to provide the necessary information.
A Cycle of Continuous Improvement. The effectiveness of link interactions was measured in comparison to the overall norm of helpful and not helpful behaviors in the organization. The ratio of 0.95 means there was approximately one unhelpful behavior for every helpful behavior. Needless to say, an organization interested in improving its performance could increase its standards for evaluating interaction effectiveness. For example, if MHTC used a standard of three helpful behaviors for every two unhelpful behaviors (a ratio of 1.5), three of the four effective links in Figure 3 would be reclassified as ineffective. A cycle of continuous improvement can be established in which link-interaction effectiveness is improved, standards are raised, and the process repeated. Statistical techniques for tracking variation in system performance (for example, six sigma methods) could conceivably be applied at the levels of specific behavior categories, link interactions, or the overall socio-technical network.
Conclusion
This study used a socio-technical methodology to measure the effectiveness of interactions between requirements engineering and other groups involved in the NPD process at MHTC. We identified major strengths and weaknesses in the process and specific behaviors that helped and hindered NPD performance. We provided an overall assessment of interaction effectiveness from the point of view of each participant. Our findings pave the way for the continuous improvement of social-network interactions by increasing the frequency of helpful behaviors and reducing the frequency of unhelpful behaviors.
The methodology can be generalized and applied as a diagnostic tool to assess interaction effectiveness in other organizational settings. It enables managers and analysts to obtain a descriptive model of the behavioral characteristics of a complex socio-technical process, providing insights into the sources of task-related uncertainty and a base from which improvement strategies can be devised. The method focuses on social-network interactions among organizational roles and uses a generic style of questioning that does not presume the existence of particular procedures or practices. It is important to recognize that the method provides both a local context-specific view of behaviors, as well as a system-level perspective of the organizational process. By focusing on interaction networks the method reflects role expectations and other organizational constraints that influence task performance (such as division of work, performance measurement, power relationships, and so forth). It is also important to recognize that the method is both quantitative and qualitative in nature. Qualitative results provide rich context-specific information about individual jobs, while the quantitative analysis provides a systemic view of interactions and their effectiveness using a common baseline for comparison.
References
1. Ashby, W.R. An Introduction to Cybernetics. Chapman and Hall Ltd., London, 1956.
2. Bavelas, A. A method for investigating individual and group ideology. Sociometry 5 (1942), 371377.
3. Beise, C. IT project management and virtual teams. In Proceedings of the 2004 SIGMIS Conference on Computer Personnel Research: Careers, Culture, and Ethics in a Networked Environment. (Tucson, AZ, 2004), 129133.
4. Duimering, P.R., Purdy, L., Safayeni, F., and Scala, J. Applications of Ashby's Law of Requisite Variety to the practice of manufacturing. In Proceedings of the World Multiconference on Systemics, Cybernetics and Informatics 3 (Orlando, FL, 1998), 749756.
5. Holtzblatt, K. and Beyer, H.R. Requirements gathering: The human factor. Commun. ACM 38, 5 (May 1995), 3032.
6. Maidantchik, C., Montoni, M., and Santos, G. Learning organizational knowledge: An evolutionary proposal for requirements engineering. In Proceedings of the 14th International Conference on Software Engineering and Knowledge Engineering (Ischia, Italy, 2002), 151157.
7. QSR International Pty Ltd. QSR N6. 2002.
8. Robertson, S. Learning from other disciplines. IEEE Software 22, 3 (2005), 5456.
9. Safayeni, F., Yu, A., Purdy, L., and Lee, E. Assessing the potential of e-mail for engineers: Case study. Journal of Management in Engineering 8, 4 (1992), 346361.
Figures
Figure 1. Requirements engineering's helpful and unhelpful behaviors as perceived by other roles.
Figure 2. Other roles' helpful and unhelpful behaviors as perceived by requirements engineering.
Figure 3. Effectiveness of social network interactions between requirements engineering and other roles.
©2008 ACM 0001-0782/08/0300 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2008 ACM, Inc.
No entries found