|
Chapter
1
The Context for Assessment
This chapter examines:
- The significance of assessment
- The external rationale for assessment
- Internal perspectives on assessment, including faculty concerns
- Standards and principles of good practice for assessment
1.1. Significance of the Assessment Movement
The assessment movement is part of a paradigm shift in discussions of educational quality. These discussions increasingly focus on the achievement of expanded learning outcomes, knowledge, skills, and professional values and attitudes and an understanding of how those outcomes are achieved.
First, assessment has broadened the scope for determining educational quality. Traditionally, institutional quality has been determined primarily on the basis of resources, reputation, and selectivity in admissions. Assessment today seeks evidence that the intended learning outcomes of higher education have been achieved. The philosophy behind this movement is that educational institutions should be in the business of increasing access to education and "developing talent," rather screening out students who do not fit a narrow set of entrance criteria (Astin, 1985). Much of what students gain from their undergraduate experience is a function of what they bring to it. Today, a comprehensive assessment program combines students' characteristics with the educational environment in a multifaceted analysis to understand and improve learning outcomes for all students.
Second, in accounting the focus on outcomes has expanded the definition of learning beyond traditional interpretations. Expectations of accounting graduates now include indications of professional competence and skills of lifelong learning (AAA Committee on the Future, 1986; AECC Objectives, 1990; Deppe, Sonderegger, Stice, Clark and Streuling, 1991). In addition, professional values and attitudes are now considered important. The AECC Objectives emphasize the motivation to continue learning throughout one's career, and the internalization of ethical standards and values of the profession. Expanded curricula found at Brigham Young University (BYU), University of Southern California (USC), Arizona State University (ASU), and others now emphasize information management, complex problem solving, decision making, critical thinking, communication of information, and teamwork, all skills that will encourage students and graduates to continue to learn throughout their lives.
Complex skills and professional orientation have generally been implicit in the curriculum. Their emergence as an explicit focus of instruction presents new challenges for curriculum development, instruction, and assessment.
Finally, assessment includes strategies to understand the educational environment and its influence on the outcomes observed. This environment includes program content and pedagogy, student-faculty contacts, academic support programs, and co-curricular involvement (such as internships and professional practice projects).
Data on student characteristics, the educational environment, and learning outcomes can be integrated through sophisticated analytical techniques such as multivariate analysis or, when such sophistication is not available, through the discussion of data from a variety of sources.
Because assessment of learning outcomes at the program level is relatively new to higher education, analytical methods and theories are still being developed and tested. For example, performance-based measurement techniques raise new questions of reliability and validity. Current trends also require development of new, more practical methods to obtain evidence about complex abilities and attitudes as well as strategies to integrate assessment information into the total quality improvement efforts of faculty at the classroom and program level.
1.2. The Case for Assessment: Public Policy and Professional Perspectives
The assessment movement in higher education can be traced to a combination of public policy mandates and institutional experiments in the late 1970s (Ewell, 1991b). As an instrument for public policy, assessment can serve the ever-increasing demand for accountability. State legislatures and accrediting agencies now routinely include assessment as a means to validate learning outcomes.
1.2.1 State Mandates and Assessment
Some form of assessment is now mandated in over 40 states, and over 90% of institutions of higher education now claim to have assessment programs (El-Khawas, 1990). What began in the 1980's as experiments by a few institutions (for example, Alverno College, Northeast Missouri State University, King's College, James Madison University, and University of Tennessee) and states (for example, Virginia, New Jersey, and Tennessee) has grown to a formal institutional requirement encouraged or mandated by state governments.
In general, state mandates have left the design of assessment programs to individual institutions to reflect their particular mission and student characteristics. Their intent was to prompt educational institutions to inform the public about program results. More recently, perhaps because some institutions were less than enthusiastic in their compliance, public officials in some states (for example, Missouri, South Carolina, Tennessee) have looked to common, state-wide assessment tools. The rhetoric of accountability has shifted also, from a focus on informing the public to "documenting a concrete return on investment" (Ewell, 1991a, p. 14).
1.2.2 Accreditation and Assessment
Accreditation, whether regional or national, is the process by which judgments of quality are made based on established and approved standards. Most accrediting standards now require some form of assessment based on the individual institution's mission. Accreditation bodies generally do not mandate what form assessment should take; rather, they seek evidence that results exist and are being used for program improvement.
Assessment as evaluation has long been an implicit and explicit part of accreditation, program review, and planning. It has always aimed at ensuring quality and gauging improvement over time; however, a focus on learning outcomes as part of the review process is fairly recent. The current AACSB accreditation standards state that "Each degree program should be systematically monitored to assess its effectiveness" [C.2.2]. Monitoring includes evaluation of faculty in their roles as teachers as well as scholars, with attention "to course development, effective teaching, and instructional innovations" [FD.3.b].
AACSB standards for accrediting programs of accounting call for a broad-based assessment process:
Key factors in the assessment of quality are the qualifications, development, and involvement of faculty; the design and effectiveness of the curriculum; the nature and effectiveness of the instructional processes utilized; the learning resources available to both faculty and students;...[and] the processes in place to plan, assess, and assure quality....(Preamble) The standards also emphasize that "The evaluation of the accounting faculty will rely heavily on assessments of teaching effectiveness..." [A-IN.2]. Teaching effectiveness, however, extends beyond the "relevance of course content or suitable instructional methods" to what students take from the course in the form of knowledge, skills, and professional orientation.
The standards correctly suggest an integral relationship between assessment and curricular or instructional change, but observers note that unfortunately the two processes are often perceived as unrelated. The disjunction is evident when institutional self-studies report assessment results in a separate section without linking them to changes implemented or planned (Ewell, 1992, p.14). Without this linkage, assessment becomes an administrative burden rather than a tool for continuous program improvement.
1.2.3 Program Review and Assessment
Program review involves a comprehensive evaluation of such factors as:
- The components and structure of the program
- The qualifications of faculty and students
- The role of research
- The level of financial and information/technology resources available to support the program
Assessment of learning outcomes is increasingly recognized as an important part of program review expected by many state boards and accreditation agencies. In its resource guide for program review, the Association of American Colleges observes that:
the goal of program review should be to increase the self-consciousness of faculty members and administrators about their educational practices so they can improve the quality of teaching and learning. (AAC, 1992, p. 1) Assessment fosters such self-consciousness. In turn, an emphasis on learning in program review suggests opportunities to incorporate outcomes assessment:
Corporate attention to the ways in which a major works as a total program, rather than as a set of discrete (and often disparate) courses, can establish a series of points at which faculty members both assess and support students' continuing progress in the field. (AAC, 1992, p. 27) Program review is thus broader than assessment. In the current era of accountability, a well designed program review will include assessment of entering student characteristics, the educational environment, and learning outcomes, as well as client satisfaction.
1.2.4 Expectations of the Profession
In a professional field such as accounting, employers, too, have a stake in the results of assessment programs. Employers want evidence that institutions prepare graduates "to become professional accountants, not to be professional accountants at the time of entry into the professional...and to lay the base on which lifelong learning can be built" (Objectives, p. 1). As noted in the Introduction, concerns of leaders in the profession prompted the formation of the AECC to implement changes that would benefit the profession. Assessment benefits the program and its graduates when evidence confirms competencies endorsed by the profession.
An assessment program may also benefit the profession by helping faculty address minority student enrollment and retention. The pool of minority graduates in accounting remains small in relation to the population as a whole, limiting firms' ability to diversify their staff. Minority students, particularly African Americans, express interest in accounting (Carpenter, Friar, and Lipe, 1993), but the programs may not be structured to heighten their interest and progress in the profession. In elementary accounting courses, minority students withdraw at a higher rate than white students. Those that persevere tend to receive lower-than-average grades (Carpenter and others, 1993). Studies to investigate and remedy causes of this phenomenon will help the program attract and retain a more diverse student population. They will also benefit the profession by providing a more diversified and better qualified pool of future employees.
1.3. The Quest for Excellence: Faculty Perspectives on Assessment
The decision to implement a formal assessment program is a choice for quality and continuous improvement that extends beyond individual faculty. Although external mandates provide the catalyst, it is faculty and administrative initiative and commitment to educational quality that will render assessment a meaningful aspect of program development.
1.3.1 Continuous Program Improvement
Continuous quality improvement (originally a goal of businesses operating in a competitive environment) has also become a theme in discussions of the future of higher education (for example, see Change [May/June 1993]; Educational Record [June, 1993], NACUBO Business Officer [December 1993] and numerous issues of the AAHE Bulletin). In the business community, quality is generally measured in terms of "bottom-line" results and client satisfaction. In the higher education context, results include both students' performance and the satisfaction of those whom the program serves, most notably students, alumni, and employers.
Assessment for improvement at the classroom and program levels has become more common in recent years. Many faculty use informal classroom evaluation to monitor students' progress and responses during instruction (Angelo, 1991; Angelo and Cross, 1993). Some departments use data available from their university's institutional research office to monitor trends in enrollment, student characteristics, and student or alumni satisfaction. Information to be used to help understand program outcomes can often be obtained from existing senior exit interviews or surveys, scores on standardized tests such as the CPA exam and GMAT, and current student, alumni, and employer surveys.
In addition to these traditional tools, new types of information are needed to evaluate expanded professionally relevant outcomes. Obtaining and using such information is a major focus of this guide. Faculty review of this information will suggest curricular and instructional changes for continuous improvement of learning outcomes.
1.3.2 Contributions of Assessment to Lifelong Learning
The expanded educational outcomes sought by the profession include an emphasis on learning-to-learn skills. Assessment helps faculty advance the development of these skills by:
- Articulating these goals and relevant performance criteria
- Encouraging faculty to include learning to learn as part of instruction and assessment in their courses
- Providing diagnostic and summary feedback on learning-to-learn capacities for use by students and faculty
In general, assessment focused on learning outcomes helps students learn to evaluate their own growth - a key learning-to-learn skill. Chapter 9 suggests ways to include learning to learn in assessment.
1.3.3 Faculty Concerns about Assessment
Faculty frequently express concerns about assessment. They wonder, legitimately, how the data will be used. They doubt that important outcomes can be quantified. They are uncertain whether they have the expertise, time, or inclination to become engaged in assessment. In a recent survey of accounting faculty, virtually all respondents in schools where assessment is currently "under discussion" rated instrument selection a major concern. About 75% rated "developing faculty motivation and participation" as a major problem (FSA Task Force on Assessment, in preparation).
Surveys and interviews of faculty and students also indicate that creating a positive environment for assessment is viewed as a major challenge facing those responsible. Curricular change is "labor intensive, requiring an inordinate time commitment" with few rewards. Faculty may "lack confidence in teaching new and unknown competencies." Both faculty and students fear that innovative programs will fail to prepare students for the content-heavy CPA exam (Pincus, Scott, Searfoss, and Clark, 1993).
Faculty may also be concerned that assessment will be used to force the acceptance of a single, rigid model of the accounting program. Administrators responsible for initiating assessment can point to the diverse range of curricular approaches and assessment methods in AECC grant-funded programs to counteract this concern. The Objectives should be understood as a broad set of principles rather than as the blueprint for a monolithic approach to accounting education. Assessment can, in fact, promote creativity by inspiring confidence in the choices made by faculty to improve learning.
Faculty may well resist an assessment program imposed (or perceived as imposed) from the "top down," particularly if the purpose and value of the program are not made clear. They will also resist a program whose results may be used to judge the effectiveness of individual faculty members. In an ideal environment, faculty are willing to undertake educational experiments to seek and respond to feedback from students and other stakeholders, and to engage in open discussion of educational matters. In turn, administrators must be fair and open about the uses of assessment data, and willing to reward faculty participation in curricular change and assessment.
Once an assessment program is underway, some concerns may be reduced. For example, only one-third of respondents in schools "currently operating a program to assess student learning and development" felt motivation was a major concern and instrument selection is viewed as problematic by faculty actively involved in assessment (FSA Task Force on Assessment, in preparation ).
A wise administrator will recognize and seek to address the concerns of both faculty and students. Faculty must have reason to believe that the benefits of assessment will outweigh the costs. They must see their investment in curriculum development and assessment as leading to tangible program improvement. Incentives and rewards for faculty participation in assessment must be developed to initiate and sustain a serious effort. Currently, the reward structure at many institutions is neutral or negative toward such participation, although nationally, a focus on curriculum, instruction, and assessment is increasingly recognized as essential for educational quality (e.g., Change, July/August 1993; Boyer, 1990).
Faculty concerns may also addressed by building trust through continuous, open communication about the program. Informal seminars may alleviate faculty concerns by:
- Creating a forum for open discussion of the issues
- Describing a continuum of assessment practices
- Exploring ways in which the faculty themselves currently use assessment in teaching and program review
Briefings on assessment projects should be included on the agenda at departmental meetings. Members of the assessment committee should also meet informally with faculty individually or in small groups to facilitate two-way communication about assessment.
Concerns may be further reduced by adopting guidelines for good practice such as those presented in Figure 1.1. These principles suggest issues for faculty to consider as they develop their own guidelines for assessment. The original document, available from AAHE, presents a brief explanation of each principle. Each is reflected in various ways throughout this guide, as indicated by the chapter references in parentheses in Figure 1.1.
FIGURE 1.1
PRINCIPLES OF GOOD PRACTICE FOR ASSESSING STUDENT LEARNING
AAHE Assessment Forum
(1992, pp. 2-3)
The assessment of student learning begins with educational values (Introduction, Chapters 4, 6, and 7).Assessment is most effective when it reflects an understanding of learning as multidimensional, integrated, and revealed in performance over time (Chapter 3 and 8).
Assessment works best when the programs it seeks to improve have clear, explicitly stated purposes (Chapters 5 and 6).
Assessment requires attention to outcomes but also and equally to the experiences that lead to those outcomes (Chapters 9 and 10).
Assessment works best when it is ongoing, not episodic (Chapters 3 and 8).
Assessment fosters wider improvement when representatives from across the educational community are involved (Chapters 2 and 9).
Assessment makes a difference when it begins with issues of use and illuminates questions that people really care about (Introduction, Chapter 4).
Assessment is most likely to lead to improvement when it is part of a larger set of conditions that promote change (Chapters 1, 2, and 4).
Through assessment, educators meet responsibilities to students and to the public (Chapter 1 and throughout this guide).
