|
Section
9.6
Conclusion: The Challenges of Measuring Learning
Outcomes
Developing valid and reliable performance-based measures of learning outcomes is a challenging task, although one with many familiar elements for faculty who assign individual and team projects, cases, preparation and analysis of financial statements, and other multi-dimensional tasks as part of instruction. Refining measures currently in use, and developing shared criteria for judging students' work, are useful ways to establish a system of measurement for program-level assessment.In addition to describing measures for expanded knowledge outcomes and performance measures, this chapter has suggested strategies for measuring "professional orientation," which includes values and attitudes as they relate to professional integrity and the motivation to pursue lifelong learning. Some may argue that attitudes and values can neither be taught nor assessed, and do not fall within the sphere of educational responsibility occupied by the accounting program. Nonetheless, any academic program does communicate values and attitudes, for example, through curricular emphasis, what the faculty evaluates and rewards, how students perceive that faculty allocate their time, and through the quality of day-to-day interactions among students, faculty, staff, employers, and the general public. Explicit attention to these dimensions in the assessment program reinforces their importance in the eyes of the faculty and conveys to students the expectations of both the academic and professional communities.
Consulting with experts on assessment of particular skills (e.g., writing, speaking, ethics and values) can be useful in the initial stages of developing or refining measurements, for example, to help faculty articulate performance criteria and identify cost-effective ways to obtain information of value to them. Equally important is involvement of practicing professionals. Finally, the greatest benefits of measurement are likely to come when faculty examine students' work collectively-not only from the familiar perspective of judging individual performance but also from the perspective of how well students, taken as a whole and in appropriate subgroups, meet faculty expectations and are prepared to meet the needs of a dynamically changing profession.
TABLE 9.1
SAMPLE GOALS, OBJECTIVES, AND POSSIBLE INDICATORS:
AN OVERVIEW
Objectivesb and Performance Criteriac
Sample Indicatorsd
Problem-solving ability
Graduating seniors will be able to identify and solve unstructured real-world problems to the satisfaction of faculty and corporate evaluators.
Students’ performance on written analyses of complex case studies.
Communication skills
Graduating seniors will be able to present, discuss, and defend their views effectively through formal and informal, written and spoken language.
Students’ performance on a series of formal and informal, written and spoken presentations and discussions of their views on topics related to accounting.
Manage challenging pressures
Graduating seniors will be able to select and assign priorities with restricted resources and organize work to meet tight deadlines.
Students’ performance on in-basket exercise.
Faculty rating of students’ organizational ability (e.g., timely completion of class projects and assignments).
a Goals: What general competencies will students acquire?
b Objectives: What action will students be able to perform as a result of instruction?
c Performance Criteria: What characteristics will define acceptable performance?
d Indicators/Evidence: How will an observer know students have achieved the objective? (Adapted from Ewell, 1988).
TABLE 9.2
MEASUREMENT OPTIONS FOR INTELLECTUAL SKILL:
CRITICAL THINKINGObjective: Students will demonstrate the ability to think critically and analytically about controversial subjects in the field of accounting.
Performance Criteria
Indicators/Evidence:
Student formulates the issue clearly and succinctly.
Ratings (using criteria at left) of position papers, memoranda, oral presentations, etc., compiled in a portfolio for review by faculty and/or practicing professionals.
Ratings by faculty of individual student's contributions to class discussions about controversial subjects in accounting.
Student self-assessments of strengths and weaknesses on criteria, obtained through questionnaires.
Employer ratings of graduate's skill in critical and analytical thinking.
Student identifies alternative perspectives on the issue.
Student obtains and evaluates evidence in support of each perspective.
Student clearly articulates a position on the issue (orally or in writing).
Student uses evidence persuasively to support the position chosen.
Student takes into account strengths of opposing positions.
Student uses correct logical reasoning throughout the presentation.
TABLE 9.3
MEASUREMENT PLAN FOR INTELLECTUAL SKILL:
GOAL: PROBLEM-SOLVINGObjective: Students will be able to identify and solve unstructured real-world problems drawn from accounting practice.
Performance Criteria
Indicators/Evidence:
Student identifies central problem(s) in a complex situation drawn from accounting practice.
Cross grading of written case analyses using analytical rating scale based on performance criteria (listed at top).
Ratings of senior projects by faculty and/or practicing professionals.
Ratings of memos, technical reports, or other writing produced in simulated practice situation involving problem analysis and solution.
Employer ratings of intern's or graduate's problem-solving ability.
Observer ratings of student engaged in group problem-solving exercise.
Student recognizes multiple causal factors involved in a problem situation.
Student uses systems thinking to analyze the problem.
Student correctly uses accounting principles to address the problem.
Student generates plausible alternative solutions to address the problem.
Student offers persuasive reasons and evidence in support of solution proposal.
Student adapts solutions to unexpected new information.
TABLE 9.4
MEASUREMENT PLAN FOR ETHICAL REASONINGObjective (AECC): Students will be able to identify ethical issues and apply a value-based reasoning system to ethical questions.
Performance Criteria
Indicators/Evidence:
Student articulates ethical dilemmas of practice in an ill-defined accounting situation.
Faculty ratings (using criteria at top) of written or oral responses to videotaped or written case presentations.
Ratings of student's ethical awareness, analysis, and decision-making in live or videotaped simulations.
Global ratings of student's responses in class discussions of ethical issues.
Employer ratings of graduate's skill in addressing ethical issues.
Student identifies key stakeholders in the situation and their responsibility to the public.
Student applies more value-based frameworks to analyze the situation.
Student offers a fair, equitable and responsible solution proposal.
Student effectively supports the proposed decision or course of action using ethical grounds.
TABLE 9.5
MEASUREMENT PLAN FOR INTERPERSONAL SKILLSObjective: Students will be able to work effectively with others in group situations.
Performance Criteria
Indicators/Evidence:
Student organizes and delegates or accepts tasks for timely completion.
Percentage of assigned tasks completed on time.
Student negotiates effectively in difficult situations.
Faculty, peer ratings of negotiation attempts (with narrative examples).
Group members' ratings of satisfaction with individual's contributions to negotiations.
Student influences and motivates others.
Voluntary involvement of others in task completion.
Student interacts effectively with culturally and intellectually diverse people.
Observer ratings of student's efforts to ensure equal inclusion of all participants.
Sociogram to summarize participant interaction patterns.
Participant satisfaction with interaction is equally high for majority and minority members of the group.
Inclusiveness in analysis of situations involving cultural and/or intellectual diversity.
TABLE 9.6
MEASUREMENT PLAN FOR INTERPERSONAL SKILL
GOAL: CONFLICT RESOLUTION*Objective: Students will be able to analyze professional situations involving conflict and offer appropriate solutions.
Performance Criteria
Indicators/Evidence:
Student clearly identifies issues and interests involved in the conflict.
Analysis: Faculty ratings of student's written analysis of group situations involving interpersonal conflict.
Performance: Peer and/or faculty ratings on checklist based on performance criteria (with narrative examples to support ratings).
Student offers practical solutions.
Student shows clearly how proposed solution will positively affect work in the company.
Student's suggestions facilitate reduction rather than escalation of conflict.
*Adapted from BYU Core, Vol. II, p.48.
TABLE 9.7
MEASUREMENT PLAN FOR GOAL:
AN ATTITUDE OF CONTINUAL INQUIRY AND LIFE-LONG LEARNINGObjective: Students will appreciate the importance of professional involvement.
Performance Criteria
Indicators/Evidence:
Student voluntarily engages in one or more of the following activities (choosing freely):
Joins student chapters of professional organizations
Attends professional meetings
Participates in faculty research
Completes internships in professional accounting firms Engages in professionally-related community service
Observed or self-reported student participation rates under conditions of voluntary involvement (i.e., no program requirement specifies involvement in any of the activities listed).
Student voluntarily reports plans to (affirming publicly):
Pursue further education after graduation
Join professional organizations after graduation
Engage in professionally-related community service after graduation
TABLE 9.8
ASSESSING LEARNING-TO-LEARN SKILLS
OBJECTIVE
PERFORMANCE CRITERIA
MEASUREMENT INDICATORS
MEASUREMENT STRATEGIES
Questioning: Students actively and effectively use questions to advance their understanding of a subject.
Students' questions require analysis, synthesis, application, integration, or evaluation of knowledge.
Cognitive complexity of students' questions based on levels of Bloom's taxonomy or "levels-of-processing" theory.
Rate students' questions in class discussion.
Rate questions submitted by students in preparation for a major project.
Organizing: Students effectively organize information for storage (retention) and subsequent retrieval.
Students' organizing strategies accurately represent relationships among concepts. Students use a variety of organizing strategies for different purposes.
Appropriateness and variety in students' use of organizing strategies (outlines, matrices, flow charts, diagrams, charts, graphs, etc.).
Rate organizational strategies in students' oral and written presentations.
Connecting: Students actively link new concepts and principles to prior learning and experience.
Students identify linkages that accurately reflect concepts and advance understanding if accounting situations.
Ratings of quality, fluency and appropriateness of linkages between concepts and prior learning or experiences.
Rate key-word lists, concept maps responses on paired concept tests (quality of relationships identified for a given pair of terms or phrases).
Reflecting: Students reflect on what they have learned and on their own learning processes.
Students demonstrate ability to extract lessons from experiences and to describe their own learning processes.
Ratings of quality and appropriateness of reflective observations.
Rate debriefing summaries from case discussions and simulations; rate comments in learning journals or self-assessments of strengths and weaknesses in performance on major projects.
Adapting: Students use what they have learned to create new solutions to unstructured problems.
Knowledge base is accurate and appropriate; solution proposals are plausible and inventive.
Ratings of accuracy and appropriateness of knowledge application; ratings of solution effectiveness and inventiveness.
Rate solutions to unstructured case studies, responses in simulations, project proposals, etc.