Communications of the ACM
Home/Magazine Archive/May 2009 (Vol. 52, No. 5)/Software Developers' Views of End-Users and Project.../Full Text
Virtual extension
Software Developers' Views of End-Users and Project Success
For many years software development has been plagued by systems that, if they are completed at all, are limited in functionality, and are delivered over-budget and late.3 It has been suggested that an on-going "culture gap" is behind many of the problems associated with such systems,7,10 and that "poor communication" between end-users and developers has helped to foster negative opinions between the two groups.7,10 While few would argue that meeting customer/users requirements is the reason for developing a system in the first place, poor communication impacts requirements gathering thus systems do not ultimately meet their required functionality/requirements. We have developed a preliminary qualitative assessment of software developers' (project managers and practitioners, including programmers, database developer and systems analysts) understanding of what they believe end-users perceive contribute to successful software development (essentially critical success factors) and, ultimately, what they believe defines a successful final system. We interviewed developers because we felt that their views would help indicate any possible disconnects between the two groups. We did not interview or survey end-users because we do not have access to a nationwide database of end-users with experience in software development projects. Our intent is not to expand the understanding of user satisfaction; rather, we use the software project management literature to discover what users consider important to the development process, as well as their view on aspects of a 'successful' final system. We compare this literature with responses from software developers regarding their views of what they believe end-users look for in a 'successfully' developed system. In short, this is an exploratory effort comparing what software developers think their end-user's consider to be important in the development process and final system, and what end-users report they consider to be important (as suggested in the user satisfaction literature).
The Software Development Process
Themes in the literature, related to a successful development process, include user involvement during development, effective communication between users and the development team, users and their everyday, 'regular' work, and requirements management. Several studies discuss the importance of user participation in information systems development projects, which leads to greater user satisfaction with the system.2,4 Involvement, and subsequent satisfaction, can also lead to higher usage of the system and a greater likelihood that users will perceive the system as being useful. User involvement also encourages a sense of ownership which helps developers design a better system, as well as making that system more likely to be used.4
Communication between the development team and users, which is a component of user participation, is essential to support meeting users' business needs, ensure that the completed system will be used, that
- Requirements are met
- System maintenance necessary to meet requirements is minimized
- A positive relationship between developments and users is fostered and
- Developers and users clarify any assumptions that may lead to downstream problems, such as inadequate requirements and costly rework.10
Users must be sure that they refrain from terms and concepts that are routine within their business domain, but yet are foreign to members of the development team while developers must refrain from use of technical jargon.
Developers should understand that users have a specific role in the development process; i.e., to make a useful contribution through "valuable business knowledge and experience."10 While developers' have the professional responsibility to design and build the system, users need to address their day-to-day responsibilities as they simultaneously contribute to the development effort. Related to this contribution and to managing customer/user requirements is the need to establish initial system specifications, as well as to accommodate changes to specifications as dictated by a potentially changing business environment as the project progresses.10This also implies adequate communication between users and the development team regarding the nature of additional or modified requirements, as well as potential impact on the project's schedule and budget.
The Final System (Outcome)
The software development literature suggests that users' perception of a 'successful' final system is often addressed through user satisfaction, which serves as a surrogate measure of system effectiveness. Satisfaction is the most widely used success measure in the literature.1,2,8,9 Specifically, user information satisfaction is the "extent to which users believe the information system...meets their information requirements."8 It's not surprising that user satisfaction is reinforced by a system that meets the users' needs, which includes output that is relevant to user requirements.8 This would seem to be the fundamental purpose for undertaking the project in the first place, although a given system may provide functionality that is not necessarily needed/required by the users, but nonetheless requested.6
Armstrong et al.1 identified four major constructs that build on several prior studies related to measuring user satisfaction with computer-based systems: information quality, system usefulness, system usage and system complexity. Information quality includes sufficient, up-to-date and clear information, effective reports, accurate initial information, accurate data processing and availability of timely output. Systems that are perceived to be inconsistent or inaccurate can result in lower usage, which, of course, reduces the utilization of the available functionality.8 System usefulness includes increased productivity, a system that saves staff time, improves job performance,1 assists users in doing their work more efficiently,10 and adequate response time.6 In some cases, a system is used because it is the only feasible way available to get task(s) done.6 System usage includes a system that is easy to learn and easy to use, while delivering the necessary functionality.1,6 Usability is important because fully using a system depends on the relative ease of using it (via its interface).6 Ease of use can also contribute to higher productivity.5 System complexity covers the systems complexity level and cost to operate. However, this construct is somewhat beyond the scope of our work.
The literature also suggests that, to some extent, users may share the traditional managerial (organizational) view that it is desirable to complete a system that delivers functional needs on time and within budget.4 In addition, they also look for a reliable system.11
Discussion
The results of this study were derived from data collected through an on-line survey (see Methodology for details of our data collection), which examined software development success and its relationship to the process of developing software, as well as its final system/ project. This paper discusses qualitative data collected through asking software developers the following open-ended question:
"What aspects of software project development do you believe are important to your users' definition of project success, relating to the process and/or the final software product."
We collected responses from 53 project managers and 12 practitioners. All respondents reported having lived the longest, and done the majority of their software development work, in the United State. They represented 60 organizations and reported being currently employed in software development in Texas (6 respondents), Colorado (5), California, New York and Ohio (4 each), and Illinois, Massachusetts, Minnesota, North Carolina and Pennsylvania (3 each). Seventy-three percent of respondents (47) were male. (An unpublished 2001 Annual Report by the Bureau of Labor Statistics indicated that 73% of computer programmers and computer systems analysts/scientists were male.) One quarter of our respondents was between the ages of 35 and 44, and over one-half were between 45 and 54 years of age. (The breakdown was 2% were 2024 years of age, 8% were 2534, 25% were 3544, 56% were 4554 and 9% were 5564. One respondent didn't provide a response.) 45% reported having completed a four-year bachelor's degree and 42% had a graduate or doctorate degree. (The breakdown was 45% completed a bachelors degree, 40% a graduate degree, 6% a two-year associates degree, 5% some college-level course, but did not complete a degree program, 2% a doctorate, and 2% graduated from high school or had achieved high school equivalency.)
Respondents were asked to provide the approximate number of years (full-time equivalent) that they had worked in each of several listed job titles. The most frequently cited (with mean number of years) were Project Managers (85%; 8 years), Programming Analyst (74%; 7 years), Programmer (65%; 6 years) and Senior Manager (49%; 7 years). Others included team leader (40%; 5 years), systems analyst (37%; 8 years), business analyst (29%; 7 years), database developer (18%; 6 years), database administrator (12%; 6 years), change control officer (9%; 4 years), and other positions (15%; 7 years). Fewer than one-quarter of respondents (15 of 65), reported having a financial interest, other than earning a paycheck (such as ownership/shares in the company), in a development project(s) during their career. Fifteen developers reported having a financial interest in an average of 20% of the projects on which they had worked.
When conceptually appropriate, compound responses were split up into more refined answers, resulting in a total of 114 responses from project managers and 24 from practitioners. See Table 1 for breakdown of respondents and their responses, split into either related to the software development process, or to the final system/outcome.
Table 2 presents an analysis of the responses from project managers and practitioners related to their perception of users and the development process, ranked by descending frequency of response. Within the most frequently mentioned process component, Customer/users, we identified two related themes. The first theme was communication between users and team, where our respondents mentioned the importance of keeping the customer informed (perhaps through status reports). The second was user involvement in the development process. Respondents mentioned the importance of users feeling that they were involved in the decision making process, as this promotes "buy-in and a sense of ownership". Developers also commented on aspects of the process related to requirements management, which was the second most frequently mentioned component of the process. Specifically, the importance to users that they define "specific" and "realistic requirements and expectations" and understanding actual users' needs. See Table 2 for detailed breakdowns of respondents/responses.
A lone response related to the Development Team was "team buy in and commitment," which was provided by a project manager. This comment relates to buy-in and ownership. Although respondents did not directly mention the importance of system reliability to users, they did recognize the importance of "user acceptance testing" and "opportunity to state acceptance and testing criteria."
Table 3 presents an analysis of the responses from project managers and practitioners related to their perception of users and the final system/outcome, ranked by descending frequency of response. Requirements/Functionality/Performance, the highest overall ranked category by both groups, included responses related to meeting expected functionality and addressed intended business needs. (However, there was no specific mention of several aspects related to requirements, such as relevant, sufficient and up-to-date information.) Within the next highest ranked category were those responses (all provided by project managers) related to ease of use/usability and learning, and user friendly systems. We also found evidence that our respondents (specifically project managers) believed that users placed importance on completing a system that delivers functional needs on time/when needed, and within budget, delivering "value" in a "cost effective manner." Within Work and Process Efficiency, responses (all provided by project managers) related to completed systems make the completion of business processes easier, facilitating more efficiency in the end-users' job and good business decisions.
Conclusion and Lessons Learned
In general, we found strong agreement between developers' perspective of end-users' view of 'successful' project development (such as the process) and final system and users' perceptive as reflected in the software management literature. Effective requirements management, including flexibility to address dynamic business needs over the life of the development effort, was mentioned.10 And while about one in three developers provided general references to meeting requirements/functionality, we did not see many responses related to details of the system, including accurate data processing and sufficient, up-to-date and clear information, and effective and timely reports (information quality). There was some understanding of the importance of user participation in the development process,2,4 facilitated by effective communication between users and the development team.10 However, developers did not indicate they recognized the importance users place on their need to balance the daily demands of their regular work with the added commitment of contributing to the software development effort. (Only one respondent, a practitioner, mentioned the importance to users that their participation in the process "requires little effort from them".)
Related to the final system, developers also seemed to have a good understanding of the importance to users of meeting system requirements/functionality/performance objectives. In particular, this included system usefulness as reflected in comments related to the need for adequate system performance. Likewise, developers also seemed to have an appreciation of system usage; specifically, the importance to users of a system that is easy to learn and use. Developers also seemed to have an understanding of the importance of objectives related to the cost and schedule of the development effort. However, relatively few developers indicated that they thought users perceived quality and reliability (presumably indicated by stability and a lack of bugs) as important components of a successful project, although perhaps this requirements was assumed by developers to be important to users. Related to this, no developers mentioned the importance of the users having confidence in the system.
While participants of this study provided a wide variety of aspects related to the final system, we believe that there is a need to stress to developers particular system characteristics that are important to user in order for them to consider a given project to be successful. We don't mean to imply that developers do not understand the importance of any particular item. Rather, our intent is to reinforce with developers how end-users view the development process and a completed project. Such reminders can assist project development managers and teams in marketing their services, framing system requirements, designing, testing, and demonstrating early prototypes and the completed system. Further study should follow related to a more systematic examination of software developers' perceptions regarding end-users and project success and, ideally, an associated survey/discussion with some end-users of the system(s) that these same developers designed and built.
References
1. Armstrong, B., Fogarty, G., Dingsdag, D., and Dimbleby, J. Validation of a computer user satisfaction questionnaire to measure IS success in small business. Journal of Research and Practice in Information Technology 37, 1, (Feb. 2005) 2740.
2. Baroudi, J. J., and Orlikowski, W. J. A short-form measure of user information satisfaction: A psychometric evaluation and notes on use. Journal of Management Information Systems 4, 4, (Spring 1988) 4459.
3. Charette, R. N. Why software fails; We waste billions of dollars each year on entirely preventable mistakes. IEEE Spectrum 42, 9, (Sept. 1, 2005) 4249.
4. Dodd, J. L., and Carr, H. H. Systems development led by end-users: An assessment of end-user involvement in information systems development. Journal of Systems Management 45, 8. (Aug. 1994) 3440.
5. Doll, W. J. and Torkzadeh, G. The measurement of end-user computing satisfaction. MIS Quarterly 12, 2. (June 1988) 259274.
6. Goodwin, N. C. Functionality and usability. Comm. of The ACM 30, 3. (Mar. 1987) 229233.
7. Hornik, S., Chen, H. G., Klein, G., and Jiang, J. J. Communication skills of IS providers: An expectation gap analysis from three stakeholder perspectives. IEEE Transactions On Professional Communication 46, 1, (Mar. 2003) 1734.
8. Ives, B., Olson, M. H., and Baroudi, J. J. The measurement of user information satisfaction. Comm. of The ACM 26, 10, (Oct. 1983) 785793.
9. Jiang, J. J., Klein, G., and Discenza, R. Perception differences of software success: Provider and user views of system metrics. The Journal of Systems and Software 63, 1, (2002) 1727.
10. Shah, H.U., Dingley, S., and Golder, P. A. Bridging the culture gap between users and developers. Journal of Systems Management 45, 7, (Jul 1994) 1821.
11. Smidts, C., Huang, X., and Widmaier, J. C. Producing reliable software: An experiment. Journal of Systems and Software 61, 3, (Apr. 2002) 213224.
Tables
Table 1. Respondents' Understanding of Users' Perception of Project Success
Sidebar: Methodology
How the Study Was Done
Development of survey items:
The survey was developed through a review of aspects of successful software project development projects in the literature and interviews with software practitioners from the Philadelphia (PA) Metropolitan area. We purchase a national mailing database of software development project managers who are employed at organizations that development software (profit, non-profit, educational, military and government) in the United States. We randomly choose 2,000 organizations (from over 3,500 organizations) to mail letters to their project managers, asking for their participation, as well as their software development staff, and directing them to the URL of our on-line survey. Responses were collected automatically through a Microsoft Access database stored on our server. Specific demographic items were as follows:
- Your gender
- Your age range
- Your highest completed level of education
- Indicate the approximate number of years (full-time equivalent) that you have worked in each of the following jobs: [Business Analyst, Change Control Officer, Database Developer, Database Administrator, Network Engineer, Programmer, Programming Analyst, Project Manager/Leader, Senior Manager, Systems Analyst, Team Leader, Other (please specify job)]
- In your career, in what percentage of software development projects did you have a financial interest, other than earning a paycheck (i.e. ownership/shares in the company, etc.)?
- In which country have you done the majority of your software development-related work?
- In which country have you lived in the longest?
Data collection:
Respondents participated in a brief on-line survey of US-based software development organizations/departments between July and November 2003. Respondents were contacted through a mailing to US-based organizations that develop software (including commercial, government and academic). Each contacted organization was advised of their password and directed to our Web site. Respondents were asked if they primarily considered themselves to be project managers or practitioners.
©2009 ACM 0001-0782/09/0500 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2009 ACM, Inc.
No entries found