Communications of the ACM
Communications of the ACM
Cooperative -Usability Practices
Usability engineering and testing blossomed in the U.S. in the mid-1980s. At the time, testing was viewed as another variation of software quality assurance testing. In Scandinavia the approach to usability has different rootsinvolving users informally in the design process rather than having them participate in a formal usability program [1, 3].
Usability activities are often applied to development processes as a means of bridging the gap between users and developers. Due to this middle position, which at times can be extremely vulnerable, usability groups need to cooperate and establish appropriate relations with both users and development teams. They need to cooperate with users to obtain knowledge about their work practices, and they need to cooperate with developers to make them apply this knowledge about work practice.
In this article we provide an overview of the practices of usability groups in the three U.S. and the three Danish companies featured in this section. Based on their practices, we discuss the usability groups' relations to developers and users and examine some of the differences between the groups and between the countries.
The Danish companies: Bang & Olufsen (B&O), Danfoss, and Kommunedata (KMD), were the first three Danish companies to have usability lab facilities. These three companies are participating in our research project, "Usability in Danish Industry," which is a three-year project that aims at developing the theoretical and practical foundations of usability work. As part of "Usability in Danish Industry," studies of usability practices have been conducted in 12 U.S. companies. Most of the studies have had a duration of several days and are based upon both observations of the work of usability specialists and interviews with a number of different people, including usability specialists and their managers. In this article we focus on the three U.S. companies featured in this section: IBM, American Institutes for Research (AIR), and Microsoft.
In a previous overview of usability labs, Nielsen [5] focuses on quantitative measurements of the physical characteristics of the lab: size of the floor space, number of rooms, number of cameras, existence of one-way mirrors, and number of staff members. Our overview focuses to a higher degree on social and organizational aspects such as the educational background of the usability staff and the organizational position of the usability group. Furthermore, we address forms under which usability groups cooperate with development teams and users.
We discuss six different aspects of usability practice and in the accompanying tables provide a profile of the nature of usability practice at the six companies. The tables provide some basis for comparing companies, but since we are not talking about a quantitative measurement with a fixed scale, there is no basis for an exact comparison. They are profiles that illustrate how each company is positioned in a spectrum of possibilities with respect to the aspect in question. For instance, in the case of usability group staffing at B&O, Table 1a indicates predominantly psychology, which is actually the educational basis for usability activities at B&O, but a head count of the usability staff would not result in psychology being the educational background of the majority of people in the usability group.
Educations in human factors and cognitive psychology seem to be the most dominant among usability specialists. A background in industrial engineering is often found among usability specialists, who then apply their knowledge to the building of prototypes or mockups of products. In cases where a group's responsibilities encompass not merely testing but also the design of a product, group members tend to have competencies in areas such as visual design and technical communications as well. Usability groups who wish to get involved early in the process of product development have also started to include anthropologists, who contribute by studying the working practices of users in the field.
People with experience in computer science and programming are often to be found in usability groups when usability managers find it beneficial to the group to be able to build its own prototypes for use in evaluation sessions or to show its suggestions to the development team.
In the software industry there seems to be a tendency toward separating designers and usability specialists into two distinct organizational units, and thus it is beyond the usability group's work tasks to conduct design activities.
The skills of the usability groups are dynamically evolving in at least two ways: people with one professional background start taking up approaches to usability originally belonging to other professions, in particular cases in which they belong to multi-professional teams. For instance, members of the usability group at Danfoss with engineering backgrounds have started conducting field studies even though none of them are anthropologists. They simply were inspired by working with anthropologists on several projects and have started to take up their principles in their own work. And usability groups from time to time are part of reorganizations that concern their connection to other organizational units. For instance, in one company (not listed in the tables) the designers are now separated from the usability specialists, but this only happened very recently. The visual designers, the interaction designers, and the usability engineers used to be part of the same organizational group but are now split up in order to develop their own professional focus. The usability group now includes only human factors and cognitive psychology skills, whereas it used to possess design skills as well.
In general, there are two alternatives for how to position the usability staff in an organizational structure: either it can be centralized in a single department or the members can be distributed among the development teams. Various others organizational forms exists between these two extremes.
The centralized solution seems to be the predominant one. The usability specialists belong to one organizational unit and refer to their own usability manager. Other project teams within the company or clients ask for assistance concerning usability activities, and one or more usability specialist then works together with the team during the project on a close basis. Thus, the usability specialists are assigned different tasks across the company's product lines and they work with several project teams. This is the case at KMD, Danfoss, and AIR. At B&O, the core group of usability staff is still centralized, but it differs from a traditional centralized solution in the sense that this company also involves people from other professional areas, such as designers, who come from other departments to work temporarily within the usability group.
Many large companies tend to practice both organizational forms at the same time in order to reduce the complexity of organizing and coordinating the work of thousands of employees. The companies are divided into self-contained divisions according to their product lines, each of the divisions having their own usability specialists assigned to them. This organizational position means that the usability group is distributed at the upper organizational level, as the specialists do not work from a centralized base for the whole company, but are spread over the various divisions. However, further down the organizational chart they often work from a centralized base in the sense that they are part of a central group in the division, referring to their own usability manager instead of being a member of a single development team.
There are two alternatives for positioning the usability staff in an organizational structure: centralize a single department or distribute members among the teams.
Between the two extreme options of having the usability staff either centralized or distributed, other forms exist. At IBM, a kind of matrix structure has recently been implemented. The matrix is a combination of a centralized and a distributed solution, in which the usability specialists still refer to their own usability manager but at the same time work very closely with specific development teams. By implementing this kind of matrix structure, IBM is aiming to capture a dual focus by having the team members belong to both a permanent, specialized department and an ad hoc group oriented toward usability. Thus they are attempting to maintain some of the advantages related to a centralized and a distributed usability group in the organizational structure.
There is no obvious solution to how to position usability groups in an organizational structure, and in an attempt to find the form best suited to the specific company, most companies have experimented with different forms for a number of years (see Table 1b).
Usability groups conduct a range of various activities, ranging from initially exploring the context in which their products are to be used to conducting post-release tests.
Field studies aim at exploring users' practices. Most usability groups are to some extent engaged in such field activities in the beginning, or at least have experiments running on integrating such fieldwork as part of their usability activities. In many cases, the field studies provide the platform for the activities they follow. Often conducted as a combination of interviews and pure observation, they take place at workplaces or at private homes, depending on where the products or systems are to be used.
If the usability group is not responsible for the design of a product, it is often asked to provide a heuristic evaluation or expert review on some early design proposals developed by other team members.
Some usability groups hold design sessions in which potential users, designers, and developers are invited to participate in a session aimed at designing parts of the products, primarily by means of storyboarding or rapid and low-fidelity paper prototyping.
Design and usability tests that take place in the usability lab and are considered by most people to be the traditional usability activities. Users are brought into the labs where by means of scenarios or use tasks they work with a prototype and test specific features related to the design or functionality of the product. Prior to running formal usability tests, many companies conduct explorative design tests, or so-called quick studies, which are early tests that take place in the lab by means of a prototype. The difference from the formal usability tests, which take place later in the development cycle, lies in the extent of the tests and the nature of the objects tested. In design tests it is the overall design and conceptual principles that are considered, whereas in formal usability tests specific features of the design or functionality are tested.
When a product has been put on the market, its value is sometimes tested in a benchmark assessment or a post-release product test. These kinds of post-release tests can take place either in the lab or in the field. A post-release test can also consist of a competitor evaluation. Here, a competitive product is tested against the company's own product in a so-called head-to-head test (see Table 2).
Some usability groups conduct nearly all of these usability activities to some degree, whereas others have specialized in conducting only a few of them. This greatly depends on the group's degree of involvement in design activities. However, all of them attempt to reap the benefits of combining early activities in the field with later activities in the lab [2, 4].
Test reports are one of the very common ways to bring usability information into the development process. Such reports are often rather standardized and document not only errors and problems encountered by the users during evaluations, but quite commonly contain suggestions for overcoming such errors. They tend to be very extensive, and thus take a long time to produce and involve a heavy workload for the author. At the same time, the development teams find the reports of limited use if they get them too late. In many cycles of product development, the turnaround time has to be very short, and the developers cannot incorporate the recommended changes if they are not available shortly after the test. A few usability groups have found a solution to this problem by simply writing up two kinds of reports with different formats and contents. The first one is delivered to the development team shortly after the test and provides only an executive summary with information useful for the developers. The second one is finished several days later and contains more detailed information that is kept in the group's archive.
In many cases the use of reports is supplemented with a meeting at which the developers and usability people discuss the report. Both parties state that meetings between development teams and usability groups enhance the collaboration between the respective parties by providing both with a better understanding of the work.
Some usability groups, like the one at Danfoss, use posters as a means of documenting findings from usability activities. The posters are often used as reminders of previous activities when they are brought into workshops and prototyping sessions later in the process.
Video recording is used by most usability groups to capture tests, evaluations, field studies, and so on. Since it is a very time-consuming task to edit a videotape showing the highlights of the events, video is only rarely used as a means of communication. Rough-cut videos of the users shot on location is in some cases used as a powerful way of showing developers aspects of use practices to which they normally do not have access. Final-cut videotapes are in some cases used to promote the usability work itself to top management or developers.
A few companies, for instance B&O, have serious doubts about whether design ideas can be communicated in text and pictures; instead, they strongly believe the best way to communicate is by passing prototypes of the product to the developers. The argument is that the experiential qualities of a product are impossible (or at least difficult) to describe and are best communicated using the same material as the final product (see Table 3).
Usability groups conduct different kinds of usability activities that take place both in the field and in the lab. Consequently, many different ways of involving users can be found, and users play different roles in the processes of product development, ranging from actively engaging in the design to being passively observed in their daily work context (see Table 4).
At some point, all of the usability groups have users act as testers in the lab. When the users act as testers, they conduct tasks on a prototype of the product and then answer questions or discuss issues related to the work on the prototype.
Some usability groups conduct design sessions in which users play active roles in shaping the future product. The users engage in discussions, evaluate proposals or come up with their own suggestions, thus in this case acting as designers. Very few usability groups involve users in this manner. However, most of the companies we're examining do practice this by inviting a large group of people to help in generating new ideas, in particular during the development of a completely new product.
If the usability activities consist of field studies, the users typically engage in these activities simply by performing their normal work tasks in their ordinary work environments. They are observed, or maybe interviewed, by members of the usability group, whose aim is to gain knowledge about the typical situations in which the product will be used. Of course, the variation in methodology differently emphasizes the role of the user, who can participate more or less actively in the studies. But in the field studies the users are in their own work environments in order for the usability specialists to understand the users' premises and conditions for using the product.
Traditionally, usability groups have conducted evaluations and tests of prototypes that have already been built at a rather late stage in the development process, but most of them are interested in getting involved at other stages as well.
To be involved at an early stage implies participating in design activities and hence collaborating in the building of prototypes. Of course, early involvement can also take place by means of conducting usability evaluations of very early design proposals, which is done in many companies. Some usability groups engage in a constant interaction between design and evaluation of the design, and they are thus involved throughout the development process.
If usability specialists are involved early, it is often due to the fact that the group is responsible for the design as well. If they are involved throughout the development process, it is often as a consequence of their organizational placement; for instance, if they are part of a research division, or if they are organized as a matrix that is designed to pay attention to user-centered design activities at all stages of the development process, as in the case of IBM (see Table 5).
Some of the differences in approach to usability can be ascribed to historical differences between the two countries, which will be discussed in the next sections. However, in both countries, we see an emerging pattern toward closer cooperation between the groups of people involved, and that the nature of cooperation depends on the aspects we've discussed.
Cooperation with Developers
The point in time at which usability activities take place in the process of product development obviously influences how closely, and in which forms, the usability group cooperates with the development team. Usability groups and users seem to prefer getting involved as early as possible in the product development process. It is not very productive to conduct evaluations and suggest solutions to problems if the problems could have been avoided in the first place by applying knowledge from users. If the usability group merely gets involved at a late stage, the collaboration does not tend to be very close. In this case, the process is fairly well defined: a prototype is designed and developed by the developers and is handed over to the usability group, which then evaluates it. The collaboration between the usability group and the development team often consists of the developers observing one or more usability tests from the observation room.
If the usability group is involved throughout the development process, the collaboration is closer, as the usability group performs more activities with the developers. They develop and evaluate the prototype together, each of them allowing the other access to its own professional knowledge. The usability groups with design skills seem to have greater success in getting involved early and, quite obviously, in having an impact on design issues.
As regards the organizational placement of usability groups, in general there seems to be a tendency for distributed groups to be closely involved at an early stage of product development due to their closer collaboration and contact with developers and other participants on the team. Centralized groups become isolated more easily, though they benefit from a better nurturing of their professional skills.
Informal aspects of the company often have an impact on the role of the usability group in product development. For instance, if the manager of the usability group and the manager of a development team know one another, the usability specialists will most likely be asked to participate at an early point in the development process. The personalities of the usability specialists themselves, like their ability to "sell" usability, also have a profound impact.
Cooperation with Users
All usability groups seem confident about having users participate as testers in the controlled lab environments, but observing users in their own environments is a more difficult issue to handle and a lot of experimentation with different kinds of field studies takes place.
Some usability groups allow the users to play a very active role, not only as testers in the lab, but also as designers of the product, or perhaps by using the product and being observed in a real use situation. All of the companies we mention here encourage strong user participation. It is no longer regarded as sufficient to have the users participate as testers in fairly controlled lab environments; instead, the knowledge obtained from conducting field studies and in particular from having the users participate actively in design sessions is now considered valuable, both to the usability specialists and to the design team. Active user participation can help usability groups in their struggle to get involved earlier in the development process. The usability group's knowledge of users' preferences is their contribution to (and can thus be their justification for) participating in early design activities in the development process. They can obtain this knowledge only in cooperation with users and are consequently interested in developing their relationship with the users.
The companies choose different ways to develop this user relationship. At Danfoss collaboration takes place with some of the local electricians, who are very often the professionals implementing the products. The usability staff at Danfoss often calls the electricians to ask for their opinions regarding a specific prototype, or to learn about the work domain in which the products are to be installed. At B&O a group of people are members of a steady user panel from which participants are selected for evaluations or tests. Because KMD was originally founded by its customers and users (the local and regional authorities in Denmark), the company has always had a very close relationship with the users. Thus the usability staff at KMD as a matter of course calls in typical end users from various local and regional authorities depending on the product in question.
Comparing the Two Countries
The U.S. usability groups were in general established long before the Danish groups and thus have more experience than the Danish ones. The Danish usability groups have been inspired by the U.S. tradition but have strongly been influenced by the Scandinavian tradition of system development, which has a long-standing practice of collaboration between users and developers, [1, 3]. The Danish usability groups believe that field studies are always worth the effort and that learning about users is enough motivation. The usability groups of Danish companies seem to have a higher degree of confidence in doing field activities than their U.S colleagues, which is probably due to a heritage from the Scandinavian system development perspective, where the users have traditionally been involved very informally in most system development processes. Usability groups in Denmark go out and explore what takes place in a very open manner, whereas U.S. usability groups are more concerned about which methods and perspectives to apply when going out into the field. But differences among the companies may also account for the variation of the extent of field studies; for instance, IBM and Microsoft are large retail software publishers with a much broader user population than, for instance, KMD. Though the U.S. groups tend to apply a more formal approach and to be more focused on lab activities, they are increasingly developing systems requirements with users at their workplace. Actually, many usability specialists experiment with various kinds of field studies and user participation in design activities as well as bringing users into the labs.
The major U.S. companies run large-scale usability tests, for instance, by having a coordinator or scheduler employed in the group. In a U.S. context this position saves the usability specialists having to spend time on scheduling the users. In Denmark, the usability specialists argue that it is beneficial if they themselves handle all contact and communication with users.
All usability groups seem to be striving for an earlier involvement in the process of product development. The staff, the skills of the group, and the group's organizational position are aspects influencing the degree to which the usability groups actually achieve this influential early position.
Regardless of organizational placement, many usability specialists find themselves in a difficult position in their companies. The acceptance and credibility of usability work cannot yet be taken for granted in many companies, and many usability specialists still have to make an effort convincing top management of the value of usability. It is not our intention to discuss the many constraints that are met daily by usability specialists. However, we would like to emphasize that although the methodologies and credibility of usability practices have greatly improved, we still need to reflect upon these practices, and, in particular, strive to improve our relationships with the people with whom we need to cooperate.
References
1. Bjerknes, G., Ehn, P. and Kyng, M., Eds. Computers and Democracy, A Scandinavian Challenge. Gower Publishing Company, Brookfield, VE, 1986.
2. Bødker, S. and Madsen, K.H. ContextAn active choice in usability work. ACM interact. 4, 1998, 1725.
3. Greenbaum, J. and Kyng, M. Eds. Design at Work: Cooperative Design of Computer Systems. Lawrence Erlbaum Associates, Hillsdale, NJ, 1991.
4. Nielsen, C. Testing in the Field. In Proceedings of APCHI98 conference. Shonan Village Centre, Japan (July 1517, 1998), pp. 285289.
5. Nielsen, J. Usability laboratories. Behav. Info. Tech. 13, 1, (1994).
Footnotes
This article was supported in part by the Danish National Center for IT Research, Grant No. 23.
©1999 ACM 0002-0782/99/0500 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 1999 ACM, Inc.
No entries found