Communications of the ACM
Communications of the ACM
Measuring Success
How can we measure success in the design and evolution of personalized interactive services for e-business? The ability to design, implement, and maintain user interfaces and user navigation in personalized interactive services requires defining meaningful metrics and feedback techniques. Here, we discuss several important issues used to define useful metrics and feedback techniques for measuring the effectiveness of a personalization Web service.
The structure of the Web is rapidly evolving from a loose collection of Web sites into organized marketplaces. The phenomena of aggregation, portals, large enterprise sites, and business-to-business applications are resulting in centralized, virtual places, through which millions of visitors pass. With this development, it becomes possible to gather unprecedented amounts of data about individuals. Data sources capturing purchase histories, casual browsing habits, financial activities, credit histories, and demographics can be combined to construct highly detailed personal profiles.
Not only is it possible to collect vast amounts of data, it is vital for e-businesses to be able exploit the data effectively. In the Internet environment, products and services are constantly in danger of becoming commodities, shoppers can explore competing Web sites without leaving their chairs, and bots and agents make comparison shopping almost effortless. Data serves two important functions. First, it becomes possible to nurture loyalty by analyzing information learned about customers over many visits. Secondly, e-business intelligence, which aggregates data over many customers, allows managers to evaluate how effective their user interface is, and continually improve the site based on measurement feedback to keep visitors on the site longer. While the survival of an e-business requires the ability to perform both, the scope of this article is e-business intelligence.
But what about privacy? Gaining public trust is key to the success of the new marketplaces. This will be accomplished only by ensuring that strict privacy policies are enforced and by convincing the public that personalized services made possible by the gathering of visitor data are worth it in terms of improved services.
Good business practice dictates the use of effectiveness measurements to guide the design of all Web site features. For Web sites with personalized interactive content, the process must take the highly dynamic nature of the content into account.
E-business Intelligence
E-business intelligence is the analysis and use of information collected about visitors to an e-business Web site. Good business practice dictates the use of effectiveness measurements to guide the design of all Web site features. For Web sites with personalized interactive content, the process must take the highly dynamic nature of the content into account. Here, we outline a complete process for a design-measure-analyze feedback cycle.
To measure success, it is important to understand what success means. What types of visitors does an e-business want to attract, what messages need to be conveyed, what should the visitor be able to accomplish, and what does the e-business want the visitor to do? The goal of Web site content may be to educate, entertain, promote a product, provide a public service, or sell. Using the personalized interactive service approach advocated in articles by Karat and Kramer in this section, the goals are quite clear from the beginning, since the notion of personalization by task is basic to the methodology.
The metrics required to evaluate success follow directly from the goals. For example, a retailer whose goal is to sell products needs to measure revenue and conversion rate. For longer-range goals, such as increasing customer wallet-share and loyalty, measuring the activity of repeat visitors, and shopping cart composition are needed.
An interesting question is what metrics are best for evaluating the effectiveness of Web site design features? An important example of measuring effectiveness comes from the online ad banner industry. E-marketers rely on ad banners to direct visitors to their Web sites, and the ad banner companies set their prices based on clickthrough and look-to-buy metrics, both of which measure effectiveness. Clickthrough data measures the ratio of clicks to impressions, where an impression is simply the display of an ad banner on a Web page. A high clickthrough rate means visitors who see the ad click on it frequently, therefore, the ad is bringing many visitors to the site. Look-to-buy data compares ad banner impressions with sales transactions and revenue directly attributable to the ad banner. It is a better measure of ad banner effectiveness, since the quality of visitors coming from the ad banner is captured and return on investment more accurately measured.
Look-to-buy metrics are generalized in [1, 2] to measure the effectiveness of a variety of design and merchandizing features, promotions, and product displays within the Web site. Rather than viewing a Web site as pages and hyperlinks, the Web site is decomposed into a collection of component features, each with specific measurable goals. As an example, the accompanying figure illustrates a scatterplot diagram of look-to-buy rates of products from an e-retailer. Each rectangle represents a particular product offered in an online store. Color indicates the home department of the product, height represents the product's retail price, and width, the profit margin of the product. The X-axis of the diagram represents product impressions, and the Y-axis represents purchases. Products in the lower-right corner are over-promoted, since there are many lookers and relatively few buyers. Products in the upper-left corner may be under-promoted, since the few shoppers who see these products tend to buy them. These products may benefit from more aggressive display.
Look-to-buy metrics work well for dynamic, personalized content. In fact, ad banners fall into this categoryads typically are dynamically rotated and may also be personalized. With look-to-buy metrics, each personalized component on a page can be counted and its effectiveness evaluated. Generally, however, if the goal is something other than maximizing sales, the appropriate metric would be look-to-X, where X is the goal.
The ability to collect and combine customer data from multiple sources enables richer analysis. Clickstream data, which captures the sequence of Web pages seen by each visitor to a Web site, is the standard data source for tracking visitors browsing behavior. It is astounding to think that every browser click on the Web is captured and logged by a Web server somewhere and processed! However, voluminous as this data is, it is low level and contains limited information. Many useful metrics cannot be calculated with clickstream data alone. Integrating clickstream data with other sources considerably expands the quality of information. Here are several examples:
- Sales data must be integrated with clickstream data for measuring design effectiveness and return on investment.
- Ad banner impressions from ad banner companies must be integrated with clickstream data to evaluate marketing campaigns.
- Call center data integrated with clickstream data helps to understand customer behavior across several channels. Often, customers shop on the Web and purchase over the phone. Not only is this integration important for the proper attribution of sales, the customer representative can also make use of clickstream data to provide more personal service.
- Demographic data from market research companies, integrated with other profile and behavioral data, helps segment customers for marketing campaigns.
Newer technologies and services make large-scale collection and sharing of data possible. We mention several:
- CPexchange1 is an emerging open standard for the privacy-enabled exchange of customer information. Since an XML definition is part of the standard, transmission of profile data across and between enterprises is facilitated.
- The consolidation of Internet advertising services has resulted in centralized sources of clickthrough data. Internet advertisers are able to capture browsing behavior across multiple e-businesses, thus being able to form more in-depth profiles than any single e-business can.
- Agents and shopping bots, positioned between the browser and the Web, can capture a complete clickstream history for all Web sites visited by an individual.
Once goals, metrics, and data sources are identified, the Web site must be designed to collect and correlate data, extract information, and calculate metrics. Accurate tracking generally involves using cookies, which provide a solution to the important problem of correlating clickstream data with other profile data. However, there are security and privacy issues with cookies, and customers often disable them in their browsers.
Server-based tracking is not always accurate, because often Web pages are cached by proxies or on the client. Client-side tracking techniques, including the use of Java applets and Javascript, can observe and record user interactions that never reach the server. Finally, clear gif technology,2 which attaches cookies and scripts to invisible images, enables third-party services to aggregate large amounts of data. Clear gif technology is well-suited to dynamic content tracking, since the invisible images can be directly mapped to customized Web page components.
Using metrics to glean the overall effectiveness of the site can be viewed from the perspective of the Web owner and the perspective of the Web visitor. From the business perspective, metrics may suggest where improvements can be made with regard to design, layout, and navigation issues. Metrics can also be used to create visualizations that demonstrate visitor's behaviors in ways that may otherwise be missed. For example, the number of times a product appears on the site compared to the number of times people actually bought a product.
The merchandiser or marketer wants to know everything about individuals so they can market their wares more effectively and so they can win customer loyalty. With that desire they may be tempted to poke and prod visitors for answers. But that effort may backfire or provide answers that only minimally help in determining what users may do in future visits. With the bundles of data that they do have, which lack valid knowledge about the motivation of the visitors, Web managers can only make some assumptions about visitor buying behavior. However, the data collected provides a constant feedback of various aspects of what happened that dayso if they start promoting a new feature and few people select it they can make changes to the presentation that will be tested the very next day! This level of turnaround helps Web owners meet the considerable challenge of supporting a fun, well-designed, easy to navigate, dependable Web site.
In addition to revealing Web traffic, the metrics could be used to build loyalty programs for frequent visitors. There may also be some number the merchant could share with the visitor to make the shopping experience more fun: How many people are here with me now? What are the five most popular items? It is unclear what value this may add to the overall user experience, but it does address some of the social aspects of online shopping.
When considering metrics and building user profiles from the visitors' perspective, it is imperative to consider the entire user experience at the Web site. In additional to personalization features, the user experience includes the tasks, services provided, navigation, design, and the overall value the visitor gains by visiting the site. To the extent that metrics gathered can be interpreted to enhance the user experience in these areas the more satisfied the visitor would be with the site, which will encourage future returns to the site.
Competitive Web sites need to be edited regularly to ensure the content is effective. For highly dynamic and personalized sites, feedback may occur in real time, enabled by recommendation engines that learn. Growing customer profiles result in more accurate and helpful personalization. The more the e-business knows the customer, the better the service should be. Ultimately, loyalty results from the investment that a customer makes in educating the business about him or herself and that a business makes in learning about the customer.
Who Will Benefit in the Long Run?
Properly designed profiling has the potential to help make Web sites more effective and efficient for visitors, while at the same time making them more productive for owners. Visitors will be offered products or services they want more often. The burden of searching out desired products, services, or information, now carried by the visitor to Web sites, will shift to the site owner, who will have a better grasp of what to provide to maximize the visitor's experience. Is the visitor of the future better off than the owner, or do both win? Certainly, the visitor will be more often satisfied than is the case today. So, too, will the owner, having been more efficient in providing the product, service, or information to the visitor. The owner will have increased the value of the enterprise by improving the visitor's experience.
The promise of profiling is to help make the visitor and the site owner winners. If the cost for a visitor to establish a Web profile is too high then profiling may not become widespread, and effective personalization will suffer. If the benefits of having a Web profile outweigh the costs, then perhaps more visitors will make the effort to establish and maintain profiles. Even with visitor profiling, Web site owners will not be able to escape the task of designing a clear, usable, memorable, reliable Web site that considers all aspects of the visitor experience.
To the extent that a Web owner will know something about the visitor with little direct visitor feedback, the effectiveness of such knowledge can be diminished. A visitor may be acting in a role other than their own during a visit, such as when a visitor buys a gift for someone else, a niece perhaps, based on what the visitor knows of his or her niece's interests and tastes. The site, not knowing the visitor is assuming a different role, might inappropriately offer its wares based on the profile of the visitor. By allowing the visitor to monitor and improve his or her profile, both the visitor and the site owner are served. The visitor should be able to profile the interests of the niece. The site owner can then tailor the visit experience to serve the purposes of the role the visitor is playing during this particular visit.
Sophisticated profiling can go even farther. For example, a feature such as "Where am I? Help me find what I want!" may be useful for someone accessing a travel site via his or her personal handheld device. If the site were employing advanced tracking technology, it could know the visitor was actually walking around Seattle at the time of the visit, providing immediate, relevant information about the neighborhood early in the visit, without forcing the visitor to drill down through menu options.
Just how user profiling will provide enduring and empowering qualities to a visitor's interactions with a Web site is somewhat unclear. It may be considered as an added bonus for those who participate, or if designed and integrated well, it may be regarded as a necessary feature. The value of personalization will depend upon how well it serves the visitor.
References
1. Lee, J., Podlaseck, M., Schonberg, E., Hoch, R., and Gomory, S. Understanding merchandising effectiveness of online stores. Intern. J. E-Comm. and Business Media. (To appear).
2. Lee, J., Podlaseck, M., Schonberg, E., Hoch, R., and Gomory, S. Analysis and visualization of metrics for online merchandising. Lecture Notes in Computer Science: Advances in Web Usage Tracking and Profiling. Springer-Verlag, New York, N.Y. (To appear).
©2000 ACM 0002-0782/00/0800 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2000 ACM, Inc.
No entries found