|
Chapter
4
Establishing Purposes and Priorities of
Assessment
(Ewell, 1994, p. 21)Because assessment requires a considerable investment of time and fiscal resources, the faculty and responsible administrators should carefully consider why the department is initiating or developing an assessment program. Although administrative request (or even mandate) is in some cases the initial motivating force, it is the department’s responsibility to decide what specific purposes the assessment should serve, and what priorities should guide implementation.
Table 4.1 illustrates a worksheet to be completed by stakeholders and then used for discussion of assessment priorities.
4.1 Primary Purpose: Obtaining Information for Educational Decision Making
From a quality standpoint, the primary purpose of assessment is providing information to improve learning outcomes and client satisfaction. Feedback to individual students and faculty is important, but it is not primary purpose of program-level assessment.
Assessment answers a variety of questions about the academic program. This document focuses primarily on understanding what students learn and how their learning can be improved:
- How do our students’ skills improve as they advance in the program?
- How do our graduates measure up to employers’ expectations and needs?
- What effect does our program have on students with various characteristics?
- How can our program be improved?
- What changes will most benefit specific subgroups of students?
4.2 Secondary Purpose: Obtaining Information for Accountability
Assessment findings may also be used to inform key constituencies about program effectiveness, for example, program reviewers, accreditation representatives, or potential employers. The assessment program thus helps the department:
- Prepare for self-study for pending accreditation
- Strengthen its program review cycle
- Document program quality to attract better students
- Documents results to meet external demands for accountability
- Gain credibility and ensure public funding by demonstrating return on investment
- Enhance program profile to attract private funds
Having a good assessment program itself is an indicator of quality suggesting that the department has a philosophy of continuous improvement.
4.3 Establishing Assessment Priorities
Attempting a comprehensive assessment all at once is an overwhelming task. Instead, priorities should balance areas of immediate concern with longer-range information needs. The assessment committee’s review of program documents will reveal valued features or goals of the program that warrant careful evaluation. The committee can further clarify priorities through discussions among the faculty and with students, alumni, and potential employers. By selecting areas of interest to stakeholders, the committee increases the incentive to obtain and use assessment results.
Priorities may be determined by identifying:
- Program distinctiveness
- Significant recent changes (for example, curricular or pedagogical innovations)
- Student group whose rate of attrition is unexpectedly high or low
- Concerns frequently voiced by students, faculty, and/or employers
Selecting one or two high-priority areas enables the faculty to develop a balanced, multi-dimensional portfolio to address strategic interests of the department. For example, the faculty may set a high priority on the ability to adapt to new technologies. The assessment should therefore help faculty identify students’ current abilities, current curricular emphasis, and ways to strengthen technological adaptability. Section 4.5 illustrates the translation of assessment purposes and priorities into specific assessment objectives and methods.
Results of the assessment should be shared with students as well as faculty to foster open discussion of program strengths, weaknesses, and strategies for improvement. Reporting and use of results are discussed in Chapter 11.
4.4 Using Priorities to Establish a Timetable for Assessment
When determining priorities for assessment, the committee should consider:
- Faculty interest and motivation to obtain information
- The value of early success on a high-quality pilot project that provides useful information
- The need to build faculty experience with assessments
- The time and energy required to develop a useful assessment portfolio
- The rhythms of the academic year
The assessment committee may wish to consider a cyclic plan in which some objectives are assessed every year, others every few years. Assessment of high-priority areas should be the focus in the first year or two of the program, with additional priorities added when questions about these areas are resolved and their assessment becomes more routine ad/or less frequent. Periodic review of the assessment program, described in Chapter 12, will reveal areas where its scope can be reduced, for example, when a new program initiative has become fully institutionalized.
The assessment committee may wish to set goals for the program itself, for example, to obtain information on at least two significant learning outcomes of the accounting program, or to evaluate satisfaction with the program as reported by at least one significant constituent group each year.
4.5 Translating Purposes of Assessment into Assessment Objectives and Methods
Questions about learning outcomes can be addressed through studies of:
- Knowledge, skills and professional orientation acquired by students
- Trends in students’ mastery of curricular goals from year to year
- The influence of specific program components on learning outcomes
- Student, faculty, alumni, and employer perceptions of the program
- Employer perceptions of graduates’ strengths and weaknesses
The examples below illustrate multi-faceted approaches to achievement of specific assessment purposes.
Purpose #1: Improve the accounting program by increasing consistency of outcomes across all sections of the elementary course in accordance with AECC’s Position Statement on "The First Course in Accounting" (Position Statement No. 2).
Assessment Objective: Clarify current focus of the introductory course and analyze students’ achievements in light of AECC goals. Use findings to negotiate consensus on major objectives. Monitor progress after changes are implemented.
Possible Methods: Examine instructional materials and portfolios of student work selected from the first course to clarify current areas of emphasis. Use focus groups to obtain faculty and students’ perceptions and suggestions for change. After refining goals and implementing changes, track quality and consistency of learning outcomes, and performance in subsequent courses.Purpose #2: Improve program quality by strengthening students’ knowledge of technological innovations, skill in using current technology, and ability to learn new technologies quickly.
Assessment Objectives: Identify employer needs related to technology and their perceptions of graduates’ technological skills. Assess students’ technological skills.
Possible Methods: Conduct employer surveys and/or focus groups; distribute questionnaires to obtain students’ self-ratings on technological knowledge, skills, and confidence in learning new technologies. Plan changes and monitor impact.Purpose #3: Improve sequencing of curricular content.
Assessment Objective: Identify redundancies and gaps in the curriculum to improve sequencing while reducing unnecessary overlap in course content.
Possible Methods: Administer pre-tests to check prior learning/retention. Analyze predictive value of grades in prerequisite courses. Plan changes and track results.Purpose #4: Improve student performance and reduce attrition at the entry level, particularly among women and students of color.
Assessment Objective: Identify possible sources of attrition.
Possible Methods: Compare characteristics of students who withdraw or fail to those who succeed. Verify completion and performance in courses completed. Use focus groups to obtain students’ views of classroom/program climate and instructional methods. Consult campus instructional improvement staff to analyze sources of difficulty and develop solutions. Implement changes and track results.Purpose #5: Determine whether ethical issues should receive greater emphasis in the curriculum.
Assessment Objective: Determine current emphasis on ethics and values in the curriculum. Assess students’ knowledge and ability to apply ethical concepts in case discussions, and in their professional lives.
Possible Methods: Review ethics coverage in specific courses; conduct in-class simulations or pilot study using focus groups to determine students’ ethical decision-making skills and awareness of professional ethics and values (for example, see Dirsmith and Ketz, 1987; Ponemon, 1993). Consult with employers and alumni. If gaps or deficiencies exist, develop and implement changes and monitor results.4.6 The Broader Purpose of Assessment: Assessing Vitality and Excellence
Regardless of the pragmatic purpose for which assessment is designed, the process can also serve a broader purpose in terms of faculty and curricular vitality. De Mong, Lindgren, and Perry report that "The assessment program may increase teamwork among faculty causing them to view their contribution to the program as a whole and not in terms of an isolated course or area" (1994, p. 14). Acknowledging that assessment is time-consuming, they comment based on their experience in the McIntire School of Commerce at the University of Virginia that the assessment program has benefited the department in two ways:
First, it has forced us to define our expectations for our students. Once we had defined our expectations, we were able to focus our study and evaluation of the curriculum proposals on our common goals. Second, the assessment program has provided a true feedback loop for our accounting program. We were able to enjoy our successes and develop strategies to overcome our weaknesses. As our experience with assessment accumulates, we expect even greater rewards. (1994, p.26)
Assessment turns faculty attention to the curriculum, instruction, and students’ needs as learners and future professionals. It provides an empirical foundation for discussions of issues central to the educational process. Whatever purposes, objectives and methods are selected, the role of assessment in fostering a departmental culture centered on excellence should not be overlooked.
TABLE
4.1
PURPOSES: Monitor
Health of Program Guide
Planning & Improvement Efforts Attract
Better/Different Students Provide
Info Relevant to Policies/Decisions Attract
Resources Increase
Accountability
STAKEHOLDERS: Faculty [ ] [ ] [ ] [ ] [ ] [ ] Students [ ] [ ] [ ] [ ] [ ] [ ] Employers [ ] [ ] [ ] [ ] [ ] [ ] Alumni [ ] [ ] [ ] [ ] [ ] [ ] Potential
Donors [ ] [ ] [ ] [ ] [ ] [ ] Trustees [ ] [ ] [ ] [ ] [ ] [ ] Legislators [ ] [ ] [ ] [ ] [ ] [ ] Business
Community [ ] [ ] [ ] [ ] [ ] [ ] Accounting
Associations [ ] [ ] [ ] [ ] [ ] [ ] Other
Accounting Programs [ ] [ ] [ ] [ ] [ ] [ ]
DIRECTIONS:
Rate the importance of each stakeholder using the
scale below. 1 = VERY
IMPORTANT 2 =
MODERATELY IMPORTANT 3 =
UNIMPORTANT Adapted
from Southern Regional Education Board, "Benchmark
Criteria." Gailes Gaines, 1992.
WORKSHEET:
STAKEHOLDERS AND PURPOSES OF ASSESSMENT
Collect ratings from various constituencies to
assist in determining priorities for
assessment.
Reprinted in "Developing a Report Card: A Summary
Of Research and Practices." KPMG Peat Marwick,
September 1993, p. 4.