Computer-Intensive
Academic Programs

How to evaluate, plan, support, and implement (in that order) your campus technology investments

Steve Ehrmann is director of the TLT Group’s Flashlight Program (The TLT Group is the American Association for Higher Education’s Teaching, Learning and Technology affiliate). Here he points out that learning goals should guide planning for technology investments, and suggests a process to ensure that enthusiasm for new hardware and software does not overshadow real program goals for student outcomes.

Many educational programs and institutions have recently installed high-speed networks, created requirements for all students to own computers, or taken comparable steps toward mainstream computer access and use.

Much rides on these massive capital investments, so it’s not surprising that people at these institutions feel a certain anxiety about the following questions:

1. How can we tell whether all this time and money is educationally worthwhile?
2. Will our investment pay educational dividends fast enough, before our new technology becomes obsolete?

Both of these are evaluative questions, as well as questions about planning, implementation, and support — that’s obvious. What’s not so obvious is that the answers to these two questions are determined by factors that have surprisingly little to do with technology. I base this conclusion on the following three assertions about how the educational use of technology really works.

Assertion 1: “It’s the Activity, Steve!”
The results of technology use are determined by the activity for which the technology is used. Use the Web to support an excellent distance learning program (that includes good course materials, good recruitment, good training for staff, etc.) and, as a result, enrollment and graduations could well increase. Offer the same technology, but without an organized distance learning program, and there will probably be no improvement in educational access. (For more on improving educational outcomes by using technology, see “Implementing the Seven Principles: Technology as Lever”.

By the same token, research has shown that if, for example, a lecturer simply shifts from using the blackboard to using a computer in order to display the same words to students, one should not expect major improvement in learning outcomes.

Technology can enable change in educational activity, but technology availability almost never compels change in the activity. And if the activity doesn’t change, outcomes don’t change much either.

So when and how do activities change?

Assertion 2: “It’s Not (Usually) the Technology, Steve.”
Let’s imagine Old Siwash University, a commuter institution. Faculty and administrators are interested in improving student mastery of material, collaborative skills, and retention by fostering more student collaboration online. So they invest heavily in student access to computers and the network.

What factors will determine whether students will collaborate online productively as soon as the computers become available? Here’s a small subset of the factors that can hinder online collaboration:

• Few course assignments actually require collaboration.
• Some students believe that collaborative learning is a waste of time.
• Some students worry that collaboration will be seen as cheating.
• Some faculty members really do believe that collaboration is cheating.
• Some students don’t understand how to use the computer conferencing system.
• Some faculty members don’t understand how to manage large volumes of student email and become overwhelmed.
• Some students don’t learn how to connect to the network from off-campus.
• Some students do not have the right hardware or software.

What’s most striking about this list is how little of it is specific to technology. Most of the factors that determine whether an investment in technology pays off in better education have more to do with the activities than with the technology.

That may seem surprising. After all, many enthusiasts have called computers a “driver” of change. “Buy them and change will come,” they chant. But “empowering” is different from “driving.” “Driving” implies things will follow some predetermined path to some specific good (or bad) result. “Empowering” suggests that computers widen people’s range of choices. So it’s the options people actually choose that then “drive” the outcome. And their ultimate choices are affected by many factors.

That leads me to one last assertion.

Assertion 3: “If you want technology investments to pay off, you have to identify the barriers.”

• Many different barriers (often nontechnological) can reduce or prevent a return on investment in technology.
• Once it’s purchased, technology ages (and loses value) quickly.

It makes some sense to begin the process not by buying technology but by considering what educational goals you would like to achieve. Second, ask what barriers currently prevent that kind of educational excellence. Third, begin lowering those barriers. That’s key to making sure that, as the technology becomes available, it is used the right way, right away. Let’s take this little paragraph apart: What steps does your . . . program need to take, if you want your investments in technology to pay off?

Developing an Effective Plan
What might a sensible process look like? Let’s consider the example of an institution that is investing in university access to networked computers. (The same approach would work for other investments in upgrading technology, for example investing in a Web-course management system.)

Step 1. Determine Your Academic Goals, Key Activities, and the Appropriate Uses of Technology
Work with pioneers who know your institution or program. What educational goals and activities could and should be advanced if your [program] invests in universal computer access? We’re not talking just goals here (improved retention, better mastery of the subject materials) but about educational activities (more student interaction, internationalizing the curriculum) and the role technology could play in improving those activities.

Let’s call that set of three linked elements a triad:

1. The educational goal or outcome.
2. The activity that will help achieve that goal or outcome.
3. The technology used to carry out the activity.

So pull together a list of triads that at least some folks hope and expect will be advanced when networked computing has been made universally available....

[A real list of triads for a program would likely be quite long, but as one example Old Siwash might identify one triad like the following:

Goal: Students graduate with exceptionally good skills at working in teams/ organizations

Activity: Collaborative learning is extensive, even in course where most students commute

Technology: Support for online communication, collaborative problem solving]

Step 2. Baseline Data and Barriers to Entry
Keep in mind that Old Siwash has not yet made the new technology available. It’s the perfect time to begin a coordinated process of evaluation, planning, support, and implementation. Notice the rather nontraditional order of those four items. We begin with a study, designed to ask at least three questions of vital interest to the institution:

1. How extensive and successful is the triad today?
2. What’s hindering even better performance?
3. Where computing is already common, is it being used to advantage in this triad?

For example, the study of our triad (the one with the goal “Students graduate with exceptionally good skills at working in teams, organizations”) might discover that:

1. Only a small fraction of students report good experiences with collaborative learning in their coursework.
2. Many barriers inhibit this activity (the assignments are not well designed for collaborative learning, students think collaboration is a waste of time, etc.)
3. Even in courses where all students already have computer access, student study patterns are pretty much the same as in courses where few students have computers.

Design your study of the triad so that it can be replicated in future years. As we’ll see below, you’ll want to repeat the study periodically to see whether the triad is being advanced, whether the technology is playing a distinctly valuable role.

Step 3. Lower the Barriers and Take Another Reading
Because computer investments age so quickly, the lack of universal access to computers ought to be the last barrier to fall. That way, once those computers (which may have only a two- or three-year life span) are made available, they’ll be put to valuable use immediately.

So because Old Siwash has chosen to focus on collaborative learning, the next step is to begin lowering the barriers identified in the study. For example, Old Siwash would probably:

• Strengthen faculty development around issues of collaborative learning, including grading policies, the kinds of assignments that attract and motivate students to work together, facilitation skills, and techniques for managing high volumes of student communication.
• Make sure that the software is easy for novice users to understand by supporting user-friendly systems and developing a good training program.
• Make sure that students will be able to connect to the network from those places, and at those times, when they are most likely to need to collaborate (home, campus library, community center, etc.).

As this effort to lower barriers to collaboration reaches its peak, it is a reasonable time to replicate the initial study, just before universal access begins. Do the results show that the barriers are indeed coming down? Is collaborative learning beginning to increase? Has there been even better improvement in those courses where everyone already has the needed access to technology?

Step 4. Finish Initial Implementation and Take Another Reading
Assuming that the news from the most recent study is good, Old Siwash can now shift its implementation efforts into high gear. This is the moment when the first large groups of students get machines.

If the preparation has been done well, studies even just a few months in the first term of availability should show extensive use of the computers to support collaborative learning. If not, the study will indicate which remaining barriers should become the object of intensive effort.

Step 5. Diagnostic Evaluation and Cost Studies
There are at least two other types of evaluation that could also be of use as computer use becomes more widespread to carry out this triad.

First, as you’ve probably already noticed, the barriers never permanently go away. Nor are many of them visible to the naked eye. For example, a faculty member about to teach a course may not know her students’ attitudes about collaborative learning or about how good they are at it. She may not know whether all of them have the needed hardware and software nor whether they all have had the training that was supposedly available to them.

A good diagnostic survey instrument could help each faculty member quickly assess the class’s readiness to use computers for collaborative learning and, even more important, give the faculty member enough guidance that at least some of the problems can be fixed in time to still have a successful course.

A second type of study could also help Old Siwash improve the triad more quickly. I’ll call it a “cost study,” but that label is misleading. Higher learning is very labor-intensive, even when computers are in use. So the largest component of costs is typically determined by how faculty, staff and students spend their time. “Cost study” is also a misleading label because the goal is typically not to cut budgets. Instead, people doing the kind of study I’m describing are concerned about helping colleagues use available time, equipment, space and cash in the most productive and satisfying ways possible.... So the most important step in doing this kind of study is deciding its purpose and boundaries and then figuring out what types of people should be on the study design team.

Keep in mind that this is not just a study of the costs of the technology. Instead what’s vital is a better understanding and mastery of the costs of the triad. If they don’t understand these costs, programs and staff members could be broken, financially and in spirit as the program grows larger. Understanding costs, especially time costs, is crucial to maintaining a humane and productive academic community as technology use accelerates.

The Crux of the Gist
Computing investments age quickly. Because computers are tools, their value is mainly “pulled” by the activities for which they are used and the success of those activities. That pull comes mainly from non-technological factors such as the demand for that activity or other factors affecting the program’s ability to carry out the activity. Therefore one important function for evaluation is to diagnose, in advance, the nontechnological factors that will affect the use of imminent investments in technology. With that insight, the program should move quickly to lower barriers to the activity so that, as soon as the computer power becomes available, it is quickly put to productive, efficient use.

As computers become more widely available, the institution should also pay attention to helping its faculty and staff diagnose and fix barriers that are specific to individual courses and services. Finally, the institution ought to help prevent staff and budget burnout by analyzing and controlling the full costs of the triads in use.

Ehrmann, S. C. 2000. Computer-intensive academic programs: How to evaluate, plan, support, and implement (in that order) your campus technology investments. AAHE Bulletin (November): 8–12. Full text of the article can be found at http://www.aahe.org/bulletin.htm.