Credit: Andrey Suslov
Automated cars are in our future—and starting to be in our present. In 2014, the Society of Automotive Engineers (SAE) published the first version of a taxonomy for degree of automation in vehicles from Level 0 (not automated) to Level 5 (fully automated, no human intervention necessary).8 Since then, this taxonomy has gained wide acceptance—to the point where everyone from the U.S. government (used by the NHTSA5) to auto manufacturers to the popular press are talking in terms of "skipping level 3" or "everyone wants a level 5 car."1 As technology gets developed and improved, having an accepted taxonomy helps ensure people can talk to each other and know they are talking about the same thing. It is time for one of our computing organizations (perhaps ACM?) to develop an analogous taxonomy for automated assistants. With Siri, Alexa, Cortana, and cohorts selling in the "tens of millions"2 and with more than 20 competitors on the market,7 having an easily understandable taxonomy will help practitioners and end users alike.
There is already a significant body of literature aimed at improving the design and use of automated assistants in both industry and academic arenas (with a variety of category names for these devices and systems, using some combination of "automated," "digital," "smart," "intelligent," "personal," "agent," and "assistant"), as the bibliographies of cited works show. Some recent work focused on task content, use cases, and features. The task content of human activity has been widely studied over a long period of time, but Trippas et al.9 note that "how intelligent assistants are used in a workplace setting is less studied and not very well understood." While not presenting a taxonomy of assistants, this type of task content analysis could be used as an aid in intelligent assistant design. Similarly, Mehrotra et al.4 studied interaction with a desktop-based digital assistant with an eye to "help guide development of future user support systems and improve evaluations of current assistants." Knote et al.3 evaluated 115 "smart personal assistants" by literature and website review to create a taxonomy based on cluster analysis of design characteristics such as communications mode, direction of interaction, adaptivity, and embodiment (virtual character, voice), and so forth—a technology and features-based taxonomy. A commercial study of 22 popular "intelligent ... or automated personal assistants"7 reported "Intelligent Agents can be classified based on their degree of perceived intelligence and capability such as simple reflex agents, model-based reflex agents, goal-based agents, utility-based agents and learning agents." While this is an arguably useful taxonomy, it also primarily addresses the technology used and not the actual use of the automated assistant. The website additionally presents editor and user ratings of ease of use, features, and performance that may be of value to end users.
The taxonomy suggested here focuses on the end-user view of "work output," and while this approach uses a somewhat subjective measurement scale, further work might incorporate objective data such as is suggested in the references. The U.S. Department of Labor has commissioned work that provides detailed analysis of the job content, knowledge, and skills required of human assistants at various levels that could be used in further refining this taxonomy.6 Furthermore, the current taxonomy does not address the time an assistant might spend in performing a task, another factor that might be used in expanding a classification scheme.
The work-output or "capability" taxonomy for assistants I have long used is based on observation of the skills and experience of human assistants over the past 40 years. Today's administrative or personal assistants (the human kind) perform a wide range of functions, albeit with highly varying levels of accuracy, knowledge, skill, enthusiasm, and initiative. The best are professionals with superior skills who genuinely earn their titles—they provide highly valuable (and valued) assistance to the people they work with, leveraging their abilities in pursuit of the goals of the organization that employs them. These people should (and mostly do) enjoy all the kudos, benefits, and satisfaction that comes from being a professional recognized for excellent work.
Experience working with assistants of all ranks and skills has led me to want to expand the ranks of the best, whether in the future they will be human, automated, or human-augmented-with-automation. Anyone who has worked with an assistant knows that if the assistant is not very good (for example, produces sloppy or inaccurate work, or takes longer than the expected or allotted time), the person or device is more often going to be a source of frustration and annoyance than assistance.
This assistant capability scale, while initially designed to rate human assistants, can readily form the basis for an intelligent automated assistant scale. As described in the examples below, it ranges from Level 1 (Entry Level Assistant) to Level 5 (Super Assistant). The key work-output characteristics of each level reflect an integration of skills and experience as follows:
Level 1: Work-output based on passively performing specifically assigned tasks;
Level 2: Work-output based on actively performing assigned tasks, developing related sub-tasks;
Level 3: Work-output based on using basic general knowledge and experience to understand specifically assigned duties and perform readily discerned tasks;
Level 4: Work-output based on using broad knowledge and experience, general and in the task area, to understand broadly assigned duties and perform implied tasks; and
Level 5: Work-output drawing on all available knowledge and experience from a variety of sources, general and in the task area, to infer useful duties, executed without supervision—just like you would have done them if you had the time, or even better!
The taxonomy suggested here focuses on the end-user view of "work output."
Adapting this to automated assistants, the levels could be linked to the human assistant scale: Level 1 "performs like an entry level assistant" to Level 5 "performs like a super assistant." Or, they could follow the SAE approach and be more descriptive of the level of automation versus human intervention required: Level 0 "no automation," Level 1 "requires significant human input, supervision and review," Level 2 "requires some human input, supervision and review," Level 3 "requires some human input and review," Level 4 as "requires minimal human input and some review" and Level 5 as "requires minimal human input and review." Admittedly these are subjective names, but examples help clarify, and they can still be related to the human assistant capability scale.
It may be easiest to understand the taxonomy by example. Several are presented here representing typical tasks for an administrative assistant. As the taxonomy should encompass a broad range of uses of intelligent assistants and other kinds of tasks, the listed examples should be taken only as illustrative of the work-output at each level of capability.
Level 1: An entry level assistant gets a call from your boss for a meeting and puts it on your calendar.
Level 2: An assistant gets the call, puts it on your calendar, and lets you know when your boss wants to see you.
Level 3: A good assistant finds out what the topic is and tells you that too.
Level 4: A really good assistant asks you what materials you need, assembles them, puts them in a folder, and gives them to you in time to review for the meeting.
Level 5: A super assistant finds most of the materials based on learning what the meeting is about, and calls your attention to what additional information you might also want to prepare.
Level 1: An entry level assistant sees that you put a conference call on your calendar.
Level 2: An assistant sees that you didn't note the phone number and password and asks you about them.
Level 3: A good assistant asks who the other attendees are and notes that on your calendar too.
Level 4: A really good assistant asks about attendees and topics, what information you might want distributed in advance, whether you want a reminder sent out, and so forth. [Overreach at this level, which should not happen at Level 5, would be calling the conference call organizer to request a change in agenda without first consulting with you.]
Level 5: A super assistant figures all this out based on your past behavior, on the title of the conference call, on the names of the other attendees, and just does it!
Level 1: An entry level assistant schedules a candidate interview and puts it on your calendar.
Level 2: An assistant puts the person's résumé in a folder and gives it to you before the interview.
Level 3: A good assistant gives you this information in enough time for you to review before the candidate comes in.
Level 4: A really good assistant looks up additional information about the candidate on the Web.
Level 5: A super assistant reads all this information and highlights interesting points for your attention.
Level 1: An entry level assistant puts an out-of-town meeting on your calendar.
Level 2: An assistant makes your travel arrangements as per your instructions.
Level 3: A good assistant looks up travel alternatives and brings them to you, and then makes your travel arrangements according to your instructions.
The key work-output characteristics of each level reflect an integration of skills and experience.
Level 4: A really good assistant validates your trip schedule, makes sure you can get from one place to another, arranges cars and pickups, and goes over it with you several days before your trip, in time to make changes if necessary. [Overreach at this level, which shouldn't happen at Level 5, would be booking dinner at an expensive restaurant and box seats at a Broadway play.]
Level 5: A super assistant does all this based on your past trips, adds maps of the areas you are visiting, maps showing buildings you are going to (including any security arrangements), any other complex instructions, information on local sites that might be of interest, and so forth.
Level 1: An entry level assistant is told, step-by-step, what to do to plan a meeting you are hosting for colleagues from multiple locations, and requires that you follow up to ensure these things are done in the way you want.
Level 2: An assistant is told the general outlines of the event and the tasks to be done in preparation, and is able to follow through with most, reporting back to you.
Level 3: A good assistant discusses the general outlines of the event, comes up with the tasks, and reports back to you on any issues.
Level 4: A really good assistant does that and comes up with suggestions on how to deal with the issues.
Level 5: A super assistant suggests to you what needs to be done to have a really great meeting—and then does it all!
The human assistant scale presented in this Viewpoint can be readily (although subjectively) applied to intelligent automated assistants to help developers (and perhaps the systems themselves) improve their capabilities. I have used this capability scale to help human assistants understand the kind of things they ought to be working on to improve their capabilities. It (or I) has not uniformly succeeded in that regard, although it should not be expected that assistants will necessarily perform at the same level for all types of tasks. The question is whether your assistant (human, automated, or human augmented with automation) is performing at Level 1 now, but can the assistant perform at Level 5 with some coaching. Perhaps Level 2 on some tasks and Level 4 on others? How do Alexa, Siri, and Google Assistant rate on various types of tasks—Level 1 on some, Level 5 on others? Will our future assistants be all digital, or will the super assistants of the future be the human ones who figure out how best to augment their skills with their own digital assistants? As for now, I am betting on the latter—at least until AI makes further advances into the realm of adding "the human touch." Either way, fairly soon everyone will want to skip Level 3 and have a Level 5 intelligent assistant.
1. Davies, A. Everyone wants a Level 5 self-driving car—Here's what that means. Wired Magazine (Aug. 26, 2016); http://bit.ly/31xX5Ej.
2. Kilgore, T. Amazon to 'double down' on Alexa as sales far exceeded expectations. MarketWatch; https://on.mktw.net/2StcOjA.
3. Knote, R. et. al. Classifying smart personal assistants: An empirical cluster analysis. In Proceedings of the 52nd Hawaii International Conference on System Sciences 2019, 2019; http://bit.ly/31HvYqt.
4. Mehrotra, R. et al. Hey Cortana! Exploring the use cases of a desktop based digital assistant. In Proceedings of the First International Workshop on Conversational Approaches to Information Retrieval (CAIR'17), Tokyo, Japan; http://bit.ly/2S78Zls
5. National Highway Traffic Safety Administration. Automated vehicles for safety; http://bit.ly/2ups21a.
6. O*NET OnLine. Summary Report for Executive Secretaries and Executive Administrative Assistants; http://bit.ly/2GZNtsv.
7. Predictive Analytics Today. Top 22 intelligent personal assistants or automated personal assistants; http://bit.ly/2UwiTi7.
8. SAE International. Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles J3016_201609; http://bit.ly/2vdMG4A
9. Trippas, J. et al. Learning about work tasks to inform intelligent assistant design. In Proceedings of the Conference on Human Information Interaction and Retrieval (CHIIR '19), (Mar. 10–14, 2019, Glasgow, Scotland, U.K.; http://bit.ly/385LKgX
The author wishes to thank the reviewers and the Associate Editor for their thoughtful comments that led to my refining this Viewpoint.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2020 ACM, Inc.
No entries found
