Communications of the ACM
The Student Productivity Paradox: Technology Mediated Learning in Schools
With increasing pressure on educational institutions to enhance learning outcomes and effectiveness, many school administrators and school boards are investing heavily in information and communications technologies (ICT). Of the 51 chief state school officers in the U.S., 48 ranked the "use of technology in instruction" as the second most important issue facing public education in the year 2000, and the most important issue expected to face public education in the year 2020.4 In addition, the K-12 market in the US will increase spending on ICT from $5.1 billion in 2005 to $6.0 billion in 2009.
ICT is becoming the way of the future-today's students are technology-sawy and heavily reliant on ICT for entertainment, socialization and many other aspects of their lives.5 Using ICT to improve education seems a natural extension of this trend.
Despite the wealth of studies on technology and education, questions about the acceptance of ICT among younger students remain unaddressed. In one much-cited research commentary, Alavi and Leidner1 call for increased research on technology-mediated learning (TML). They recommend that researchers explore "the explicit relationships among technology capabilities, instructional strategy, psychological processes, and contextual factors involved in learning."
Business organizations use TML to improve worker performance through better training, while lowering training costs. School administrators have introduced technology in an apparent attempt to achieve similar efficiencies and effectiveness. However, the performance parameters for business and education are very different. Furthermore, the assessment of performance in education may be impacted by a myriad of factors that are not immediately relevant to business or evident to those who manage or research businesses. Despite the increasing use of ICT in schools (for example Palm, Inc. has provided several schools across the U.S. with Palm Pilots to use in their classrooms),9 little attention has been paid to the impact of ICT on learning. Without an understanding of this relationship, educators cannot make informed decisions about school investments in ICT.
There is a need to take stock of ICT investments in schools to assess whether they have been effective, rather than the result of well-intentioned administrators jumping onto the ICT 'bandwagon.' The targets of these investments-today's students-have grown up surrounded by a range of sophisticated technologies. The 2003 US Census showed that 76.2% of households with school age children (6- to 17-years-old) had one or more computers and 67% of them were Internet users.12 Although those trends continue upwards, there is a paradoxical indifference to some ICTs among this generation.5 Clearly, the value of investment in ICT depends on creative or other productive outcomes from its use: to be truly successful, ICT investment must match passionate interests with powerful ideas.7 The question here is, if school students are provided with ICT, will it provide them with learning opportunities that are not otherwise available?
The question is a challenging one. Measurements of value and productivity arising from corporate ICT investments are elusive and frequently controversial. The idiosyncrasy of teaching and learning make evaluation of the impact of ICT in education even more challenging.1,13 Despite the widely recognized benefits of instructional ICT in improving educational outcomes and effectiveness in the corporate setting, the research community has yet to investigate thoroughly the use of such technologies in the school setting. A recent review of the research literature revealed an imbalance of opinion pieces over empirical research. The majority of these articles tout the positive impact of ICT on students' learning experiences. Although there is a substantial body of research on ICT use in education, few studies have directly assessed the impact of ICT on student achievement or students' experience of these technologies.8
We used the Technology Acceptance Model (TAM)2 to explore the students' acceptance and perceptions of the PDA's utility. TAM provides robust measurement of ICT outcomes and enabled us to assess students' perceptions of the PDA as a medium for learning. This gave us insight into the relationships between the PDA technology, the school context and the psychological processes as TML was introduced into the school setting.1 Understanding these issues is very important, and not only from the students' perspective. How students use the ICT provided will impact not only their own learning but also the success of the ICT program in the schools and, in turn, society at large.
Many educators believe that introduction of ICT to K-12 students will no doubt improve education11 and encourage further investments in ICT. Despite the ever-improving price-performance ratio, ICT investments represent a more significant proportion of a school budget than that of a corporation. The consensus found in the research published to date is that ICT improves education.6 However, teaching and learning are notoriously difficult to assess.3,13 Much prior research lacks concrete evidence of a causal relationship between ICT investment and learning outcomes. Like the 'productivity paradox' in the business world, the overall costs and benefits of ICT in the education setting are unclear and, to date, largely unmeasured.8
We addressed these issues in our study of a program to introduce PDAs at a charter school in the Southeast U.S. during the 20042005 academic year. The school administration had investigated the potential uses and benefits of PDAs and gained approval from their board to purchase and distribute PDAs across all eighth grade classes. The small screen and absence of a conventional keyboard made PDAs an unusual choice for classroom use. Although PDAs support a range of productive activities, their support for reading and writing is modest. We designed our study with these factors in mind, adapting methods that have been widely used to explore the effects and outcomes of PDAs and similar ICTs in a range of settings.
Since the school's plan effectively mandated the use of the PDAs, the ad ministration involved parents from the outset by sending home a description of the program and holding sessions to teach parents about the technology. The administration issued each student a PDA as a loaner and teachers adopted the PDAs as the medium for learning, replacing traditional textbooks and notebooks. Teachers in the four core subjects (math, language arts, science, and social studies) transformed materials into a digital format. Teachers used word processing, spreadsheet, presentation and other software10 to create instructional materials that students accessed in school and at home. Students completed assignments on the PDA and turned them in via infrared transfer to each teacher's computer. In addition to the parent briefing, training was provided for the teachers and students. The PDA provided a far narrower range of applications than those developed at the University of Michigan10 and elsewhere.
Students had full control of their PDAs during and after school, ensuring that there were no excuses about lack of access. Although use of the PDAs was mandated, it was unrestricted. Administrators were keen to avoid rejection by the students, so scheduling personal time, playing music, storing non-school related documents and other uses were allowed. PDA technology has matured, and students seem less concerned or constrained by the lack of power.10
Prior to adoption, we assessed students' and the parents' perceptions of the role of technology in this 'new way of learning' using surveys that included constructs from Oliver's expectation-confirmation theory (ECT).6 Fifty-four parents and 76 students responded. Both students and parents expressed high expectations about the positive impact of using the PDAs in the classroom. The responses-summarized in Table 1-are striking for their uniformly high values. It is particularly interesting to note that at 6.22-well above the 'non-committal' mid-point value of 4-students' expectation that ICT would expand their ability to learn was very strong.
Perhaps not surprisingly, parents' expectations about the positive effects of ICT on learning were even stronger than students' perceptions (see Figure 1, perception 1). This pre-implementation response shows strong parental support for the project. It appears that both students and parents accepted the administration's efforts toward a comprehensive implementation, whereby the PDA would completely replace textbooks, notebooks and other media associated with traditional instructional strategies in the four core subjects. But did this strong support give rise to a belief among the students that the ICT helped them learn?
In order to answer this question, we evaluated students' experience of the technology, rather than their expectations. At the end of the academic year, we assessed the students' experience of the PDA mediated 'classroom with no boundaries.' All 76 students responded: parents were not included in this second survey since they did not directly use the PDAs. We used constructs from the Technology Acceptance Model2 to assess the students' learning experience. Data from the two surveys provided complementary perspectives of students' expectations and experience and provided insight into the use and utility of PDAs for learning. We were surprised by our findings, some of which ran completely contrary to the expectations that arose from our pre-implementation measurements.
Even though they had no difficulty with the use of the technology or recognizing its capabilities, students were not convinced that it was the right tool for learning. Variations in the perceptions of usefulness and ease of use (see Table 2) contrast strikingly with the uniform support for ICT evident in Table 1.
Figure 2 shows that although students thought the PDAs were easy to use, they did not think the PDAs were useful. The line in Figure 2 indicates the 'non-committal' mid value. Although positive, the measure of students' attitudes towards technology is not as strong as we anticipated - and administrators hoped - it would be from our pre-implementation study. Most troublesome was the finding that the majority of the students said they did not intend to continue using the PDA in high school - this, despite the fact that their use of the PDA was not restricted to the classroom.
Clearly, the environment at the charter school we studied is not typical of all schools. The nature of 'productivity' in the school setting adds it own challenges that require further research.7,13 Nevertheless, the data generated by our adaptation of well established methods used in the business sector provide insight into students' experience and perceptions of PDA mediated learning.
These findings send an important, though perhaps unwelcome, message to school administrators. The data suggest that this ICT investment was prompted by the assumption that enabling and encouraging the students to improve their productivity would increase learning opportunities. Hind-sight-and our data-suggest that this view is rather naive. Although well supported and justified on the grounds of positive anticipation among administrators, students and the parents about the effects on learning, the students' experience was disappointing.
The low perceived usefulness of the PDAs suggests that there is a danger that just jumping onto the ICT bandwagon-even when the investment in ICT seems well-justified (see Table 1)-could be misguided. This suggests that bringing ICT into schools does not guarantee that students will improve academically. The initiative driving the project emphasized student effectiveness, though in a rather ill-defined way. All sides-students, parents, teachers and administrators-were enthusiastic about the technical capabilities that the PDAs offered. But this is only on of the four highly inter-related factors that are critical for the success of TML.1 By exploring the psychological processes and contextual factors surrounding the students' use of the PDA, our measures of usefulness and perceived utility2 highlight the significance of these dimensions.
At the end of the school year, all 76 students were given the opportunity to buy the PDA. We were especially surprised to find out that only one student opted to do so. Like other findings from our study, this raises more questions. Are administrators and government officials on the right track in their efforts "Toward a New Golden Age in American Education?"11 Our study strongly suggests that blind faith in the value of ICT-epitomized in the 'build it and they will come' approach to implementation in corporate settings - is inappropriate in education. Unlike corporate functions, such as production and logistics, the learning process is complex and not amenable to enterprise-wide standardization.
Before investing heavily in ICT, school administrators should more fully appreciate that the nature of 'productivity' in learning is elusive. Our study provides some well-reasoned empirical evidence that highlights the need for researchers to further investigate the specific needs and expectations of students, teachers, parents and administrators who adopt ICT for use in education. Our findings caution that the outcomes of investments in ICT are uncertain and may not always align with the intentions of school administrators.
References
1. Alavi, M., and Leidner, E.D. Research commentary: Technology-mediated learning-A call for greater depth and breadth of research. Information Systems Research, 12(1): 110, 2001.
2. Davis, F.D., Bagozzi, P.R., and Warshaw, P. R. User acceptance of computer technology: A comparison of two theoretical models. Management Science 35, 8 1989, 9821003.
3. Higgins, L. Laptop program to cost Michigan schools money" Knight Ridder Tribune Business News, Washington, 2003.
4. Morgan, A. D., Matrang, M., Peltier, G. L, and Hill, G. C., What issues will confront public education in the years 2000 and 2020? Predictions of chief state school officers. The Ciearing House. 71, 6, 339342,1998.
5. NetDay. Voices and Views from Today's Tech-Savvy Students; http://www.netday.org/downloads/National_Findings_Highlights.pdf.
6. Oliver, R. L. "Effect of Expectation and Disconfirmation on Postexposure Product Evaluations" Journai of Applied Psychoiogy 62(4) 480486,1977.
7. Papert, S. The Turtle's Long Slow Trip: Macro-Educological Perspectives on Microworids, Journai of Educational Computing Research 27(1) 727, 2002.
8. Peslak, A. R. The Educational productivity paradox: Studying the effects of increased IT expenditures in educational institutions. Communications of the ACM 48,10,111114, (Oct. 2005).
9. Pierson, D. In the classroom; Education is put in hands of teenagers; Youths in Palmdale are given Palm Pilots to help with assignments. Los Angeles Times: B2 (Oct. 16,2002), Los Angeles.
10. Soloway, E.; Norris, C.; Blumenfeld, B. F.; Krajcik, J. and Marx, R. Handheld Devices are Ready-at-Hand Communications of the ACM 44, 6,1520, June 2001.
11. U.S. Department of Education, Office of Educational Technology, 2004. Toward a New Golden Age in American Education: How the Internet, the Law and Today's Students Are Revolutionizing Expectations, Washington, D.C.
12. U.S. Census Bureau, (2005). Computer and Internet Use in the United States: 2003. Current Population Survey, Washington, D.C.
13. Van Melle, E., Cimellaro, L., and Shulha, L. A Dynamic framework to guide the implementation and evaluation of educational technologies" Education and Information Technologies, 8, 3, 267285, 2003.
©2008 ACM 0001-0782/08/0900 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2008 ACM, Inc.