Communications of the ACM
Communications of the ACM
Intranet Model and Metrics
Although many corporations store a great deal of information in their corporate intranets, few have a reliable means of measuring the effectiveness of their intranet portals to use this information to meet specific business needs. Turning information into knowledge capital that corporations can leverage quickly for competitive advantage requires a model and metrics that tractably support it. Most intranet portal measurements are based almost exclusively on usage statisticswith little or no thought given to design or user experienceand corporations apply them in a non-standardized manner, providing no meaningful insight into how well intranets help corporations achieve their strategic objectives.
As IT professionals and business decision makers seek ways to forge their information into knowledge capital that can be leveraged quickly for competitive advantage, they require a model and supporting metrics to do so. What has been missing is a comprehensive model and methodology to base measurements on logically related groups of metrics which, when measured periodically, provide actionable steps to optimize the efficiency and effectiveness of intranet portals.
As IT professionals and business decision makers seek ways to forge their information into knowledge capital that can be leveraged quickly for competitive advantage, they require a model and supporting metrics to do so.
Intranet vs. Internet
An essential fact when measuring effectiveness of corporate portals is to recognize and account how their purpose is similar to, but fundamentally distinct from, Internet portals. Intranets exist to fulfill different purposes for different constituencies than does the Internet. The key difference lies in the underlying mission of the portal itself: on the Internet, the portal sites' business model is based on attracting a portion of the advertising budgets of corporations that might otherwise advertise in other media (such as print, TV, or radio). Thus, the general purpose of the public portals is to attract large numbers of repeat visitors, to build online audiences with the inclination to buy what the portal advertisers are offering. These portals have essentially settled into a one-directional relationship with their users.
Inside the organization, the portal takes on an entirely different character. It takes its purpose from the overall mission of the organization: to add sufficient value for its customers to create a sustainable business model. It takes its features and functionality from the mandate to operate at world-class efficiency and effectiveness in order to remain competitive. Achieving this competitiveness requires a bi-directional model that can support the knowledge workers' increasingly sensitive needs for pertinent, helpful, timely content and interactive information management tools.
Today, a fundamental shortcoming in trying to increase value from corporate intranets is due to a lack of comprehensive and credible means by which to measure how effectively the portals meet the demands of their employees (knowledge workers) and other intended audiences in pursuit of carrying out business objectives. As Figure 1 illustrates, most approaches to metrics do not begin from a strategic management viewpoint that takes into account a prioritization of critical business requirements essential for value creation. Far more work in the IT community focuses on applying metrics to knowledge worker processes as they impact immediate costs and benefits, and this work fails to recognize longer-term payback as these processes relate to company competitiveness (for example, processes that sustain key business activities that support value creation). In order to write better software to design intranet portals and measure their performance, value must be understood and linked to critical business requirements with the proper balance of metrics that can be used to further derive better estimates of return on investment (ROI).
Few approaches to metrics begin from a strategic management viewpoint, which lets organizations prioritize critical business requirements essential for value creation. Designing better software for intranet portals and effectively measuring the portals' performance requires linking value to critical business requirements with the proper balance of metrics that help corporations derive more meaningful ROI estimates. In effect, this focus would close the gap of understanding between knowledge workers, IT professionals, and business decision makers.
Intranet Efficiency and Effectiveness Model
The intranet efficiency and effectiveness model (IEEM) provides a needed framework for the "family of measures" approach by graphically depicting multiple indicators that derive the unique contributions of IT at the process level. It further provides derivations for common units of analysis (time and money) by linking sets of metrics and conversion ratios to business processes as they relate to knowledge workers, IT managers, and business decision makers seeking to increase value.
Information technology does not just contribute data; it impacts many business processes that produce results required to sustain value. Analyzing processes within the IEEM can help corporations to better understand which processes are critical to sustaining productivity. One measure of intranet effectiveness is how well the company's portals support its business requirements. Research on the IEEM can help identify metrics and conversion ratios that corporations can apply to their portals to determine where they need to focus efforts on meeting strategic business requirements. By dividing an intranet into segments, the IEEM helps corporations define, apply, and refine a balanced baseline of metrics for measuring what is important, instead of what is available.
To better appreciate the purpose of portals and reasons for their occasional redesign, it is important to understand the domains supporting information management as they relate to finding and understanding information, the domains' constituents, and how segmenting these constituents facilitates highlighting and focusing on key business requirements. Three domains the IEEM identifies are:
- Front end, which addresses user-based factors, such as accessibility and site navigation;
- Back end, which addresses site-based factors, such as personalization, taxonomy, and information search; and
- People, processes, and technology, which addresses knowledge-worker-based factors, such as their vision, purpose, and products or servicein short, how well these factors support business requirements that promote productivity.
Constituents represent the data necessary in the front- and back-end domains to find information supporting the people, processes, and technology domain (see the sidebar "Requisites for Portal Design").
Intranet Composition
After identifying the requisites and their constituents, we used a series of diagram procedures to illustrate and analyze the composition of an intranet, and used this information to determine the IEEM and its fundamental baseline of metrics. To determine associations between metrics groups and critical business requirements, we used an affinity diagram to create a conceptual model separating the intranet into distinct segments that underlie each domain: content, business requirements, design, domain infrastructure, usage, and user experience. This provided us with greater resolution to map out the problem and solution space. On top of this diagram, we identified the various users and their roles within each segment. Next, we placed an interrelationship diagram on top of the affinity diagram to highlight pertinent metrics (such as relevance, ease of navigation, and user satisfaction surveys) and their logical relationships between related users and their roles.
We further classified these metrics into hard, soft, and derived forms [3], outlined them on the diagrams, and put them into a cause-and-effect tree table after consulting with a variety of business decision makers and knowledge workers. Lastly, using the resulting IEEM diagram and prioritization table, we put all of these factors into a prioritization matrix to illustrate levels of importance and to establish a metric baseline from which to begin measurement [1].
The resulting theoretical model illustrates the placement of the underlying and interdependent domains, segments, and constituents, as well as the consequential and logical metric groupings supporting business requirements. Figure 2 is a simplified version of this model. It shows where the segments fit into the three domains. The segments are green to reflect the mix of front-end (blue) and back-end (yellow) factors and the inner core of the IEEM is offset in purple (the darker shade of the people, process, and technology domain of derived metrics) to represent the efficacy of relevant metrics.
Figure 2 further separates efficiency from effectiveness. Efficiency measures are predominantly quantifiable (or hard) metrics, that is, numbers and durations of time or both. They comprise the usage and user experience segments. Effectiveness measures include efficiency measures but also take into account qualitative factors (soft and derived metrics). These measures make up the design, business requirements, domain infrastructure, and content segments. Analysts must distinguish and appreciate differences between metrics (hard, soft, and derived) in each segment to attain the best practice of them.
The IEEM includes an intranet's three corresponding audiences: corporation business decision makers, portal owners and managers, and users. Efforts to organize, prioritize, and apply metrics for measuring an intranet's effectiveness must consider all three audiences. For people within these audiences to benefit from enhancements suggested by IEEM analysis, business decision makers must know who and where these individuals are within the corporation as well as their roles in fulfilling business requirements.
The IEEM focuses on strategic fit, functional objectives, and the opportunity or necessity for making process improvements as the keys to success. In addition, the IEEM introduces a common theoretical framework for measuring all facets of intranet processes critical to assessing value. Its holistic approach, however, does not eliminate subjectivity altogether. The IEEM accounts for critical qualitative factors that other commonly used measuring techniques (which concentrate on usage statistics, such as traffic volume) often overlook. In short, to be more comprehensive, the model must account for some soft metrics that are neither strictly quantitative nor free of human interpretation or assumption. Nevertheless, when parties agree on a relatively well-defined set of performance metrics, a relatively unambiguous collaborative interpretation is possible [3].
Matrix for Metrics and Prioritization
Table 2 shows an actual baseline estimation determined by business decision makers and IT professionals of the top several metric areas, the segment they come from, and some of the business requirements they sustain. The prioritization of these metrics is based on the metric groups that have the most significant impact on overall value from a business management perspective. For example, business decision makers and IT workers might rate "relevance" (a soft metric from the "content" segment) as a first-priority metric because of its effect on new opportunities for business, increased creativity, and better decisions. A hard metric, "traffic volume" (from the "usage" segment) might earn a priority-two rating for its impact on reach, ROI, and user loyalty.
The distribution of metrics is of additional interest from a theoretical standpoint because at least one metric area originates from every segment (see the "Segment" column in Table 2). This supports our assertion that to achieve accurate and comprehensive effectiveness measurements, analysts should take more metrics from segments other than the usage segment. Nevertheless, it's best to limit the number of critical metrics areas to include only those that directly correlate to a business benefit (although the number can vary for each corporation, the IEEM baseline example here focuses on seven.) Otherwise, analysis can become overly complicated, threatening order, implementation, and credibility.
Many corporations focus on routing metrics, such as number of hits per page, top 10 search strings, most popular downloads, and number of referrals from other sources (banner advertising, search engines, and direct links). These metrics exist in far greater number than other metrics because they address the issues many organizations face today: Web site accessibility and visibility.
These usage-related metrics are also popular among the technically oriented workers who usually do the measuring because they require less time and are more mathematically straightforward than the more time-consuming soft metrics that measure user behavior and experience in the IEEM front end. In addition, routing metrics are relatively easy to understand at the business level, and the data is relatively easy to collect using Web server log files. In fact, most Web analytics packages provide many routing metrics as prepackaged reports, so deferring to these out-of-the-box tools is natural. Unfortunately, corporations often apply these tools to their intranets as they would to their Internet Web sites, but there are fundamental differences between the two and simple substitutions like this provide an inaccurate assessment of an intranet portal's performance. Simple statistics on plumbing alone won't provide business decision makers with all of the feedback necessary to track productivity improvements.
The audience most overlooked in ascertaining intranet performance is the user, even though many of the constituents necessary to sustain effective information finding are in the front-end domain, where the knowledge worker resides. Consequently, when selecting metrics with which to measure their intranet's effectiveness, corporations must carefully consider metrics from the design and user experience segments.
Metrics and focused surveys inform analyses of user behavior patterns within portals and help corporations refine subsequent metrics and surveys. For example, to ascertain why numerous visitors are abandoning a particularly important site, a corporation should examine user behavior through focused surveys, direct observation, and analysis of other metrics used at the site.
A combined metrics and survey program is also critical to the communication process and development of a feedback loop that helps IT learn which initiatives provide the best business value. For example, raw visitor metrics might indicate that a corporation should archive or discard an infrequently visited research page, when in fact a single recent access might have been the critical piece in securing a major new revenue stream for the organization. Surveys offer many benefits and can lead to obvious but overlooked additions such as providing an online employee manual equipped with a search engine, reducing the amount of time people spend looking for the manual and the information in it.
A combined metrics and survey program is also critical to the communication process and development of a feedback loop that helps IT learn which initiatives provide the best value.
To further elaborate on the interplay of surveys and metrics, Figure 3 dissects a metric area from Table 2 (unique users) to show the reasoning behind the application of metrics to IEEM domains and audiences. The headings describe the organizations, people, and processes involved. The matrix has been color coordinated to the IEEM to ensure thorough interpretation of the association between the model and metrics, including surveys required. All metric areas and specific metrics are colored to better represent what they support and where they belong in the IEEM as they are used with that metric priority or category.
In addition to being grounded in a theoretical framework, the IEEM methodology is practical in that it allows corporations to obtain estimates using many common units (time being the most useful) that are directly traceable to specific pages, links, and designs in a portal. Thus, it's possible to derive portal effectiveness in relatively practical ways. Moreover, this approach does not rely on a particular software program, so it can run in any network without additional hardware or software costs, other than the server space necessary to store log file queries.
Figure 3 is one partial example of more than a dozen actual examples of baseline sets of metrics and conversion ratios resulting from high-level analysis of intranet efficiency and effectiveness. The U.S. Navy, the Defense Information Systems Agency (DISA), and a large Seattle, WA-based software company are studying and refining these baselines to determine the best techniques for:
- Analyzing user behavior;
- Using periodic soft metrics (that is, short, focused electronic surveys) to confirm predominately hard metric results related to behavior;
- Automating capture of optimum processes for submission as business rules and best practices; and
- Creating a single reporting service for comparing portal performances consistently to gauge impact of process changes.1
Determining which complementary metrics can be grouped and which groups best indicate how well a portal supports a business requirement can lead to efficacy indications. Refining these groupings from all intranet segments (every organization is unique and therefore should work to refine their own metrics after periodic measurements) helps corporations improve critical business requirements, such as agility, disintermediation (reducing the number of points required by process occurrences), loyalty, opportunity, and reach.
Conclusion
The intranet is the most measurable medium ever, yet there has not been a successful demonstration of the methods or strategy necessary to successfully implement a model or measurement technique that can indicate the effectiveness of an intranet. To our knowledge, there is little in the literature that addresses this matter. We have shown that metrics can be prioritized, logically grouped, and then sub-grouped with known specific, measurable hard and soft metrics for each of the three intranet audiences by focusing on critical business requirements that drive value and productivity within each. Since successful Web analytics are more a matter of skill than of technology, portal managers and business decision makers might consider juxtaposing their current approaches to intranet measurement and ROI on the IEEM, or using the IEEM as a baseline for developing metrics that focus on the critical business requirements that derive competitive advantage and value. Understanding how seemingly intangible assets affect performance can mean the difference between growth and erosion of value.
References
1. Baskerville R. Investigating information systems with action research. Commun. AIS 2, 4 (Article 2), 1999.
2. Favaro, J. Modified from managing requirements for business value. IEEE Software 19, 2 (Mar. 2002), 1517.
3. Jacoby, G. A metric model for intranet portal business requirements. Ph.D. dissertation, Dept. of Computer Science, Naval Post Graduate School, 2003.
Footnotes
This work was supported in part by ARO under project 5NPGARO032 and by AFOSR under project F1ATA05192G001. A more extensive version of this research was presented at the 2004 International Multi-Conference in Computer Science and Engineering.
1Hard and soft metrics taken together with consideration given to their strengths and weaknesses allows an enterprise to make informed decisions on the investment in, or the ongoing value of its data warehouse and portal system. Achieving success through the use of any performance metric will depend as much on how well it is applied as it does on when it is used. Studies based on samples and averages over time can make for easier and more credible comparisons. Hence, continuous benchmarking should be instantiated to confirm and correct baseline measurements and conversion ratios through periodic (monthly, quarterly, annually) portal status reviews that measure progress against previous baseline results. For these reviews, portal owners should use the metrics to determine which roles and content are being underserved by the portal and which processes could correct this and better leverage the portal's capabilities.
Figures
Figure 1. Software engineering approach to metrics [
Figure 2. IEEM metric types and examples for all segments.
Figure 3. IEEM example of part of one metric breakdown for unique users.
©2007 ACM 0001-0782/07/0200 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2007 ACM, Inc.
No entries found