Assessment for the New Curriculum: A Guide for Professional Accounting Programs PDF Version (for printing) Resources on Change in Accounting Education

Section 9.3 Measuring Skill Outcomes   Measuring complex skills such as problem solving, critical thinking, ethical reasoning, communication, and teamwork involves observing and making judgments about students' performance or the products of their efforts. Performance measures are procedures and instruments that facilitate judgments made by qualified observers. These measures may be familiar to faculty who assign integrative projects such as written case studies, team projects, and oral presentations. For program-level assessment, performance measures can vary from impressionistic ratings of everyday behavior to formal assessments by trained raters reviewing students' written work or observing their performance in complex situations, such as a simulated client presentation or audit team planning meeting. Judgments can be made by faculty, graduate assistants, practicing professionals, and in some cases by students' peers. Just as knowledge outcomes can be assessed using course-embedded measures so too can data on students' intellectual, interpersonal, and communication skills be obtained by selectively compiling and anonymously cross-grading samples of work completed (and separately graded) as part of the normal instructional routine. Current practice favors course-embedded measures over logistically challenging alternatives external to normal coursework. Course-embedded measures are part of a trend toward integrating assessment with instruction to enhance its value to students and the ecological validity of measures used (Ewell, 1991b). This section describes methods for obtaining performance measures and for increasing the validity and reliability of judgments of skill. Examples illustrate assessment of intellectual, interpersonal, and communication skills, and skills of learning to learn. 9.3.1 Procedures for Developing Performance Measures The procedure for obtaining measures of performance is a slight variation of the procedure for measurement of knowledge outcomes outlined in Section 9.l.2: Review objectives and performance criteria and refine if necessary Develop scoring guide or rating scales for judging the product or performance, based on the performance criteria Define measurement situation for the target objective (presentation, simulation, project report, etc.) Observe performance or obtain product from students Rate or describe the performance or product using rating scale or scoring guide Compile results for review Recalibrate rating scale and/or revise measurement situation as needed This section focuses on the use of performance criteria to develop rating scales or guidelines, selecting or defining measurement situations, and developing rater reliability. 9.3.1.1 Establishing Performance Criteria: For each major skill objective, criteria to judge the quality of performance must be identified, as described in Chapter 7. The criteria should be behaviorally explicit. For example, the ability to withstand and resolve conflict is an important objective recommended by the AECC under the category of interpersonal skills; it is also an important element of leadership. Performance criteria based on an established model of negotiation might specify that in a conflict situation the student: Identifies the source of the conflict Identifies principles to guide conflict resolution Maintains a focus on interests, not positions Actively explores the interests of all parties involved Acknowledges but does not react to emotions Actively explores options Seeks criteria of fairness for resolution (Fisher, Ury, and Patton, 1991) Once identified, performance criteria such as these can be translated into guidelines or rating scales for measurement purposes as described below. In addition to facilitating measurement, performance criteria serve a useful instructional purpose. When presented to students, they put forth a recognizable ideal toward which students can strive, and which can be discussed in concrete ways by students as they complete assigned projects and by faculty and students in teaching and advising situations. To reap these advantages, performance criteria should be made known to students early in the instructional process. 9.3.1.2 Defining the Measurement Strategy: The measurement strategy defines how evidence of students' achievement will be obtained. Sources of evidence for measurement of complex skills are of three general types: Judgments of a specific performance or product Judgments of ability Proxy indicators Judgments of performance or product: As noted above, performance measures rely on observation or review of specific performances or products of student efforts such as oral presentations or examinations, written problems or case studies, performance in a team situation, or a portfolio of completed work from one or more courses. These performances must be judged by faculty or other qualified observers. Their judgments are usually expressed as ratings on performance criteria, or described in narrative evaluation. Judgments of ability: Judgments of ability are based on observations of performance over time, for example, in a supervised internship or teamwork situation. Like judgments of specific performance or products, they may be expressed as ratings on performance criteria or as narrative evaluations. Requests for ability judgments can be incorporated into questionnaires administered to students, faculty, alumni, employers, or others who have opportunities to observe students' performance on the targeted objective(s) over time. Proxy indicators provide indirect evidence that the targeted skill has been developed. For example, grades in a communication course could be used as an indicator of general communication skills. As emphasized in Section 9.2, the validity of grades as proxy indicators is greatest when the course is designed to teach specific objectives identified by the accounting faculty. The measurement strategy consists of a prompt to initiate student performance, the performance situation or product to be evaluated and the instrument used to codify reviewers' judgments. Instruments include the scoring guide or "rubric," rating scale, or guidelines for narrative descriptions of the product. To illustrate, the prompt for a performance judgment might be a videotaped factory tour and client interview; the performance situation might be a discussion among students, acting as auditors reviewing the tour and interviewing clients to develop an audit plan (Mohrweis, 1993). Depending on the objective for which the assessment is intended, measurement could involve the use of ratings of individual students' effectiveness in the discussion (interpersonal and communication skills), ratings of the decisions reached by the group (decision-making), and/or scores on subsequent performance when asked to apply audit concepts (higher-level knowledge outcome). Alternatively, the prompt might be a complex accounting problem, the product to be judged being the students' analysis and solution proposals, rated using a scoring guide that emphasizes higher-level knowledge outcomes and problem-solving skills. The concept of "authentic assessment" suggests that insofar as possible, measurement situations should reflect actual conditions of practice. The use of prompts such as complex cases and videotaped prompts depicting real-world situations lend authenticity to the assessment situation. Authenticity is enhanced when a real or hypothetical purpose, audience or client, and context are specified. Examples of objectives, performance criteria, and measurement strategies for each of the major categories of skill categories identified in the AECC Objectives are given in Tables 9.2 through 9.8. 9.3.1.3 Developing Measurement Instruments: Judgments of performances or products can be expressed as numerical ratings or as narrative evaluations. As already noted, the performance criteria provide the foundation for developing an instrument such as a scoring guide or rating scale. A scoring rubric can be designed to facilitate holistic or analytical judgments: Holistic ratings are summary judgments of the product, performance, or ability viewed as a whole. Grades given to student projects exemplify holistic ratings: They summarize in a single letter or number, the degree to which each students' performance responds to the performance criteria. For example, a rating of "6" on a 6-point scale would mean that the students' work exemplified competence on virtually all of the criteria, while a rating of 2 would mean that very few of the criteria were met. Appendix 4 presents a six-point holistic scale to assess critical thinking skills (Facione and others, forthcoming). Holistic ratings provide a general sense of how students are doing, but their diagnostic utility is limited since they do not allow faculty to compile a profile of students' strengths and weaknesses. Analytical ratings are judgments based on each of the performance criteria taken in turn. They reflect specific characteristics of the desired product, performance, or ability. Separate ratings of the accuracy, organization, and supporting evidence in a case analysis illustrate the use of analytical ratings. Analytical ratings are diagnostically useful because they enable faculty to identify patterns in students' performance as a group. For example, faculty review of a sample of case analyses performed by juniors might reveal that in general, students in the junior class receive higher subscores on accuracy of information than on the use of supporting evidence. Such results suggest relatively specific directions for follow-up assessment and curriculum development. The faculty may choose to review writing requirements, syllabi, assignments, and grading criteria to identify changes that could, for example, improve students' ability to use facts as evidence. Occasionally, narrative descriptions of students' performance are preferable, for example, when preparing a dossier on which recommendations to employers or graduate programs will be based, or when the instrument is intended to foster students' self-assessment, as illustrated in Appendix 5. Narrative reports can also supplement numerical ratings. Criteria identified for performance assessment should be made public and used throughout the curriculum to guide and reinforce students' efforts to develop targeted skills. 9.3.1.4 Qualifications of Raters: Whether made holistically or analytically, judgments of products, performances, and ability should be made by raters who are qualified based on their training, expertise, and opportunities for observation. Raters need not be "experts" in the domain of measurement; for example, it is not the case that only English teachers can judge the quality of students' writing. Raters, however expert, should participate in training to improve reliability of their judgments and agreement among judges (Section 9.3.1.5). Although raters need not be "experts," colleagues with appropriate expertise (both academic and professional) are invaluable in developing effective measurement tasks based on performance criteria. Materials developed in consultation with these resource people can then be used by raters trained to apply the criteria, whether faculty, peers, graduate students, or practicing professionals. Training students to rate their own and each other's performance helps them develop self-assessment skills. Students' ratings will usually supplement rather than replace judgments of more experienced observers. 9.3.1.5 Improving Reliability of Qualitative Judgments: Measurement strategies used to assess complex skills generally rely on qualitative judgments of performance translated into numerical ratings or narrative descriptions, rather than quantitative measures such as frequency counts. The subjectivity inherent in qualitative judgments is often seen as a limitation of performance-based measures. However, use of a consensus-based training procedure reduces the subjectivity and increases the reliability of qualitative judgments. In this procedure, referred to here as rater training or "calibration" (of the rating instrument), individuals responsible for making judgments work together to develop consistency in judgment, that is, to improve interrater reliability. Interrater reliability is a measure of the degree to which two independent raters agree on their judgments of performance on a defined task. Interrater reliability is expressed either as a correlation between two ratings or as the percent of cases on which raters' judgments fall within a specified range of agreement (for example, 70%-80% of judgments are within 1 point of each other on a 6-point scale). To develop interrater reliability, two or more raters will: Translate performance criteria into a rating scale, adapting the criteria as necessary to fit the specific performance or product to be rated Independently rate or describe samples of students' work drawn from the same pool of papers or performances that will be used for the final assessment Compare the ratings of all raters; discuss discrepancies until consensus is achieved Score a new sample; continue until the agreement criterion (for example, 75% of judgments by 2 independent raters are within one point of each other) is reached Frequently, discussion of discrepancies between raters leads to clarification of the performance criteria or students' interpretation of the task. In this case it is advisable to revise the rating scale to more closely reflect the task. When the agreement criterion is reached, raters score the remaining students' work independently. Often two raters score each sample; large discrepancies may be resolved by a third rater. Once reliability is firmly established, double ratings are generally not necessary, although periodic checks are advised when large numbers of the student sample must be rated (While, 1985). After scoring is completed, raters should debrief using questions such as the following: What strengths and weaknesses did students exhibit that were not reflected in the scoring rubric? Were there any patterns, for example in students' approach to the task, their understanding of the content, or their value orientations, that might have implications for the curriculum or instruction? Did you notice anything else that the scoring guide did not allow you to record? (adapted from Ewell, 1994, p. 43.) Discussion of raters' observations is an important element in the ongoing review and revision of curriculum and instruction as well as the refinement of assessment materials. "Such discussions may in fact prove more important for encouraging improvement within a faculty team than the scores eventually assigned" (Ewell, 1994, p. 44). Benefits: Rater training can be used to improve consistency of judgments about individual samples or portfolios of students' work, videotaped presentations, or other performance data. The procedure can also be incorporated into instruction to encourage students to develop self-assessment skills and the ability to make sound judgments about the performance of others (see Loacker and others, 1984, for examples). Application to assessment of writing portfolios is described in Belanoff and Elbow (1986). Because cross-grading of students' work is time-consuming, it may be tempting to recruit independent judges, especially when the focus is on written or oral communication skills which faculty may not feel they are trained to judge. This tactic has the advantage of minimizing faculty time on assessment, but correspondingly limits faculty opportunities to develop a good diagnostic understanding of students' strengths and weaknesses as a basis for identifying instructional needs. Faculty involvement in developing and applying criteria also helps to clarify departmental standards, with likely benefits for departmental consistency in evaluating qualitative data such as papers, cases, and essay exams, all of which are likely to be more commonplace in accounting programs in the future. At least in the formative stages of the assessment program, then, key faculty should participate in rating sessions so that the department can benefit from their systematic review of students' work. 9.3.2 Measuring Skills in the Accounting Curriculum The strategies outlined above can be adapted to measure intellectual, communication, interpersonal, and learning-to-learn objectives of the accounting curriculum. Specific examples in each of these categories are provided below. 9.3.2.1 Intellectual Skills: Critical Thinking: Critical thinking may be defined as the open-minded investigation and analysis of a complex issue from a variety of perspectives, resulting in a well-supported position on the issue (Kurfiss, 1988). AECC Objectives related to critical thinking include inductive and deductive reasoning, critical analysis, and the "ability to present, discuss, and defend views effectively" (p.7). Critical thinking is often measured using skills tests that focus on inductive and deductive reasoning, for example, the Watson Glaser Critical Thinking Appraisal, the Cornell Critical Thinking Test (Form Y, for college students), and the California Test of Critical Thinking Skills (CCTST, Facione, 1990a). These tests evaluate students' ability to use correct logical thinking and to avoid fallacies in reasoning about text passages, using a multiple-choice response format. Critical thinking tests tend to be correlated with measures of reading ability and both verbal and quantitative college entrance tests (MacMillan, 1987; MacMillian & McPeck, 1981; Facione, 1990b). Measures that rely on students' responses to open-ended prompts include the Ennis-Weir Critical Thinking Essay Test and subscales of the ACT-COMP instrument. In practice, the skills of logical reasoning are necessary but not sufficient for critical thinking, largely because solving real-world problems depends on discipline-specific and even task-specific knowledge (Glaser, 1984). For example, a student's ability to argue the merits of a particular accounting method depends on knowledge of the options and how each is used (discipline-specific knowledge) and knowledge of the client's situation (task-specific knowledge), as well as the ability to relate general principles to the specific situation (deductive reasoning). Skills of learning to learn (identified by Frances and others for the AECC, forthcoming) enable students to apply their knowledge to critical thinking tasks. Students must question the assumptions of all parties involved and the quality of the evidence for each position; they must organize relevant information to extract essential ideas; they must connect their knowledge to the concerns of various stakeholders in the controversy. They may also have to adapt their knowledge to generate novel alternatives. Lastly, to continue to develop their critical thinking skills, students must reflect on what they have learned from their investigations and from discussion of the issue with others who may not share their views. Assessment of learning-to-learn skills is discussed in section 9.4.2. Assessment of critical thinking, then, should reflect its multifaceted character and dependence on adept use of professional knowledge. Case studies, simulations, and other performance assessments provide opportunities for students to demonstrate critical thinking skills in the accounting context. Table 9.2 and Appendix 4 suggest criteria for evaluating students' performance on critical thinking tasks. 9.3.2.2 Intellectual Skills: Problem-Solving: Problem-solving involves the analysis of a situation in which a discrepancy exists between a current and desired state, development of alternative solutions, and formulation of an appropriate plan of action (Anderson, 1985). The AECC Objectives include the "ability to identify and solve unstructured problems in unfamiliar settings and to apply problem-solving skills in a consultative process" (p.7) among the essential outcomes of accounting education. Unstructured problems are those which require a search for information and generation of alternatives (Bonner and Walker, 1994). Like critical thinking, the solution of unstructured problems requires a blend of general problem-solving ability, knowledge, and learning-to-learn skills. Problem-solving and critical thinking are closely related cognitive processes. Both involve analytical reasoning, induction, deduction, a search for evidence, and synthesis of ideas into a complex whole. Both require professionally-relevant knowledge and application of learning-to-learn skills. They differ primarily in their focus: Critical thinking is use of cognitive processes to develop and support a point of view on an issue Problem-solving is use of cognitive processes to analyze and resolve a troublesome situation (Kurfiss, 1988) For example, critical thinking is involved when students are asked to identify the issues associated with a substantial cost over-run on a contract for new, technologically sophisticated equipment, and to analyze those issues. Complex problem-solving is involved when the students are asked to propose a solution. Critical thinking is involved when students are asked to interpret financial information for an institution that has made some fraudulent transactions and is now on the verge of bankruptcy; complex problem-solving is emphasized when students must develop options and recommend a course of action to restructure the company, including raising new capital. Problem-Solving in Accounting: The AICPA practice analysis described in Chapter 6 delineates tasks and activities related to accounting and auditing and to taxation compliance, consultation, and representation. The AICPA's outline of procedures may be viewed as a guide for approaching the general problem of assessing and completing an engagement. The tasks in accounting and auditing are presented in Figure 6.1 (Chapter 6). Accounting graduates must be prepared to respond effectively to unusual circumstances and problems that may arise in each phase of an engagement. Performance criteria for problem-solving, presented below, suggest a model for the teaching and assessment of problem-solving skills in accounting. Performance Criteria for Problem-Solving Skills: Problem-solving ability may be judged according to performance criteria such as the following: Identifying and diagnosing the problem (clarifying the circumstances that make the situation a problem for the client; identifying constraints; determining client's goals; identifying and ranking possible courses of action; recognizing the limits of what is known and costs of obtaining additional information; and establishing the optimal timetable for resolution of the problem) Generating alternative solutions and strategies (identifying possible solutions to the problem; clarifying possible means to achieve each of the potential solutions; and determining limitations on each potential solution) Developing a plan of action (identifying the full spectrum of possible plans of action; evaluating the relative merits of each possible plan; and tentatively settling on a plan) Implementing the plan (determining who should implement the plan, establishing the optimal timetable for completion, and monitoring and revising the plan) Keeping the planning process open to new information and ideas (quoted with modifications from MacCrate, 1992, pp. 129-135) In addition to the processes indicated above, problem-solving and critical thinking in accounting often involve identification of ethical issues. Although ethical reasoning skills are discussed separately below, they can often be assessed in conjunction with both problem-solving and critical thinking tasks. Sample performance criteria and indicators for measuring problem-solving abilities in the accounting context are provided in Table 9.3. 9.3.2.3 Intellectual Skills: Ethical Reasoning: The AECC Objectives includes among the important intellectual skills the ability to identify ethical issues and make well-founded ethical judgments and decisions in the context of accounting practice. Professional accountants&emdash;both public and corporate&emdash;routinely face ethical dilemmas (Finn and others, 1988). Ethical issues faced by accountants range from fulfilling a broad responsibility to society to the detection of fraud, and from forming an opinion on financial presentations to reporting illegal acts. Incorporating ethical issues into the curriculum, and assessing students' ability to identify ethical dilemmas/questions and to apply a value-based reasoning system, communicates that the faculty places a high priority on professional ethics. Performance Criteria: Performance criteria for ethical reasoning include the ability to analyze the role of the professional accountant in ethically challenging situations, to articulate ethical dilemmas implicit within a case or real-world problem, to identify the stakeholders in the situation, and to apply a specific ethical framework (such as utilitarianism, rights, justice or virtue ethics) to analysis of the problem and development of solution proposals (Velasquez, 1992). Measurement options: Ethical reasoning skills can be assessed using students' written or oral case presentations or observation of simulations that involve ethical issues. Ratings can be made by faculty, peers, and/or professional accountants. Cases can include dilemmas that are relatively straightforward (such as fraud) or more subtle (such as a request by a client to choose accounting methods to minimize taxes or influence stock prices). Another option for measuring ethical reasoning is a research instrument called the Defining Issues Test (DIT). This instrument identifies the considerations people use to make judgments when confronted with moral dilemmas (Rest, 1990). For example, some individuals base their judgments on fear of punishment, others on the desire to conform to social norms, still others on the wish to obey the letter of the law, and others on the basis of "universal" ethical principles such as honesty. According to the underlying theory (Kohlberg, 1976), each of these considerations is associated with a different "stage" of moral reasoning, with universal ethical principles presumed to represent the highest stage. The DIT, or an adaptation of it to an accounting context, could be used to determine the issues students use to reason about ethical dilemmas. Because the DIT asks students to rank-order ethical criteria, it may also be used to measure professional attitudes and values (Section 9.4). The DIT has been used to compare accounting majors to students in other majors (Jeffrey, 1993; St. Pierre, Nelson, and Gabbin, 1990), in experimental studies of ethics interventions in accounting (Ponemon, 1993) and in business programs generally (Conry and Nelson, 1989). A related approach using simplified criteria and focused on the professional accounting environment is presented in Hiltebeitel and Jones (1991). Sample performance criteria, indicators and measurement options for ethical reasoning skills in an accounting context are provided in Table 9.4. 9.3.2.4 Interpersonal Skills: Interpersonal skills enable students and graduates to work effectively with others as clients, co-workers, or government officials; in day-to-day interactions, in teamwork situations, or in negotiations. Interpersonal skills include: Social skills (such as conveying warmth and working compatibly with others) Task-oriented skills (such as helping to achieve consensus) Linguistic skills (such as presenting ideas clearly) Analytical skills (such as the ability to analyze interpersonal dynamics and apply the analysis to facilitate constructive interaction) (adapted from Bayer, 1993, and BYU Core, Vol. II, 1992) Interpersonal vs. Communication Skills: Although interpersonal skills overlap in important ways with communication skills, communication skills are most often assessed in formal situations in which the individual or group presents a product (such as a presentation, written examination or paper). In contrast, interpersonal skills are those called into play while creating the product or working with others to achieve a goal. Value of Teaching and Assessing Interpersonal Skills: Interpersonal skills are frequently cited as important for accounting graduates' chances of professional success, offering a compelling reason to include these skills in the program's goals and the assessment portfolio. However, teaching and assessing interpersonal skills requires a greater departure from customary practice in accounting education than the teaching and assessment of intellectual and communication skills. Faculty in the accounting program may feel that they do not have the expertise to teach such skills, so that assessing them in the accounting context is not appropriate. Nonetheless, incorporating interpersonal skills into the curriculum and its assessment adds an important dimension of quality to the program. Emphasis on this aspect of students' development invites use of active learning and authentic assessment strategies in which students try out professional roles and learn to take the perspectives of future co-workers, managers, clients, IRS agents and the like. Formulating explicit criteria for assessment of interpersonal skills will increase faculty awareness of opportunities to reinforce these skills, while offering benchmarks to students for self-assessment and self-improvement. Performance Criteria for Interpersonal Skills: Performance criteria for assessing teamwork, one form of interpersonal skills, should reflect behaviors that facilitate effective functioning of the group. Examples from the BYU Core (Vol. II, 1992) include: Exhibits a positive disposition to the task at hand Comes to group sessions prepared Completes assigned tasks on a timely basis Additional criteria to consider include: Contributes actively and appropriately Facilitates contributions of others Helps to achieve group goals while maintaining positive relationships Accurately assesses group process and provides behavioral feedback in a form acceptable to the group The functioning of the group as a unit can also be assessed, either by student participants, peer observers, or faculty, for example in a post-task debriefing session or using a brief rating scale. For example, members of an effective group can be expected to work together to: Establish manageable goals Clarify roles and expectation Establish ground rules Develop open communication Resolve conflict Develop consensus (Bayer, 1993) Feedback on group functioning should be discussed by the group to foster both individual and team learning (Johnson, and Smith, 1990). The product of the group's efforts can also be assessed using criteria appropriate to the task. Measurement Strategies for Interpersonal Skills: Interpersonal skill (including teamwork) can be measured by: Instructor observation and rating of interpersonal interactions (live, videotaped) Peer ratings of individual contributions to group process Employer or internship supervisor ratings Instructor ratings of students' analysis of interpersonal and team situations (proxy measure) Since observation and rating of students' behavior provide the most direct basis for judging interpersonal skills, peers are often in the best position to judge interpersonal skills. Peers as raters: Peer ratings are both practical and appropriate in measuring interpersonal and team skills when coursework includes group projects, presentations, or problem-solving activities. Before using a rating scale, the instructor should conduct a practice session in which students use the scale to rate videotaped interactions and discuss their ratings (similar to a reliability training session). Although the discussion is focused on use of the rating scale, it also serves an instructional purpose by directing students' attention to effective and ineffective interpersonal and team behavior. Effective use of peer ratings: Using peer ratings may create anxiety about evaluation. Students are often reluctant to evaluate their peers, so the instructor should clarify the rationale and explain how the information will be used. The instructor can also remind students that professionals are often called upon to evaluate co-workers. Peer ratings should be treated by the instructor as advisory, that is, to supplement the instructor's own judgments and to aid in diagnosing strengths and weaknesses in students' interpersonal abilities. Appendix 6 illustrates a peer rating form that combines ratings of group skills, leadership, and conflict resolution (BYU Core, Vol. 11, 1992). Faculty as raters: Although faculty may be reluctant to judge students' interpersonal skill, use of rating criteria can simplify the task. In large classes, it is difficult for faculty to acquaint themselves with interpersonal skills of individual students. Videotapes of students working in groups can facilitate observation. Faculty can also observe students working in groups during class. However, unless the professor observes groups at work over a period of time, ratings will have little reliability or validity. Analytical measures: Faculty may elect to judge students' ability to analyze actual or hypothetical interpersonal situations as a proxy measure of interpersonal skills. although the ability to analyze interpersonal transactions does not necessarily mean one can handle them in practice, focusing students' attention on interpersonal dynamics provides them with tools they can later use in actual practice situations. Tables 9.5 and 9.6 suggest strategies and criteria for measuring teamwork and conflict resolution skills. Like other performance measures described in this document, the measures suggested here can be incorporated into classroom instruction to facilitate date collection. 9.3.2.5 Communication Skills: Communication skills are those abilities that enable individuals to convey information and ideas to others and to understand the information and ideas others present to them. Professional accountants must be able to discuss technical information with clients and co-workers as well as establish rapport with clients by carrying on informal conversation. Specific skills include the ability to (Greenberg and Smith, pp. 244-245): Ask appropriate questions to determine clients' needs and to obtain necessary information Answer clients' questions Write coherent memos and letters to the IRS, clients and co-workers Oral and written communication skills learned in courses outside the accounting curriculum should be applied and reinforced in accounting courses to ensure transfer of those skills to professional situations. When communication skills are evaluated as an integral part of assignments in accounting, the professional relevance of those skills is conveyed unequivocally to students. The message is amplified when attention given to these skills in individual courses is reinforced by program-level assessments such as reviews of portfolios of students' work. Performance Criteria for Communication Skills: Performance criteria for communication skills depend on the particular aspect of communication to be emphasized. For example, listening skills, identified as important in the AECC statement of objectives, are frequently judged on the basis of the listener's ability to identify the main points in an oral communication and to judge the purpose or intent of the communication. Other criteria for "listening" include formulating questions to obtain additional needed information or the identification of underlying feelings without reacting emotionally oneself. An important communication objective identified in the AECC Objectives is the "ability to present, discuss, and defend views effectively through formal and informal, written and spoken language" (p. 7). In this formulation, the medium of communications is secondary to the ability to present and support ideas (often categorized as the intellectual skill of critical thinking, discussed above). Performance criteria for this aspect of communication skill could require the student to: Clearly state a position on the topic Organize the presentation in terms of a few main points or themes Choose appropriate illustrations, examples, or evidence to support his or her position Use a level of detail appropriate to the audience's need and interests (Loacker and others, 1984; Ewell, 1988) When the focus is on oral communication of ideas, it may be important to emphasize the ability not only to defend ideas but to engage in effective dialogue with others. The following performance criteria have been suggested: Identifies the main points of oral statements by others Identifies points of agreement and disagreement in an oral exchange Modifies his or her own arguments in light of new information (Ewell, 1988) Measurement Strategies for Communication Skills: Opportunities to observe and assess oral communication skills are numerous when the accounting program encourages active student involvement in learning. Formal class presentations, case discussions, role play situations, oral examinations, and conferences with the professor require students to communicate orally in an accounting context. Similarly, written case studies, memoranda, client letters, position papers, research papers, and group and individual projects serve as windows on written communications skills while enabling faculty to assess students' mastery of accounting concepts and their application. Oral presentations can be videotaped for later review and critique by the student and instructor, or for cross-rating by program faculty as part of a program-level assessment. Accounting programs have incorporated assessment of written and oral communication skills in varying degrees. For example, at Brigham Young University, students write numerous papers, give presentations in a variety of settings, and take oral final examinations. A standardized rating sheet is used by faculty to assess both content (adapted to the specific presentation topic) and communication skills in formal written and oral presentation. BYU faculty assess listening skills using exercises that involve following verbal instructions, summarizing a videotaped presentation, and self-diagnosis (BYU, 1992, Vol. 2). Scofied (1994) lists characteristics of oral presentations (Appendix 7). These characteristics can readily be adapted for use as performance criteria and translated into a rating scale. The use of standardized rating scales to assess communication (and other skills) throughout the curriculum serves two important functions. In addition to providing very specific and consistent feedback go students, rating scales allow the faculty periodically to aggregate results for each of the performance criteria to determine whether students show patterns of strengths and weaknesses and whether skills show improvement over time. Care should be taken to ensure that any widely-used rating scales adequately reflect the desired outcomes and performance criteria. Reliability of ratings should be checked periodically using procedures described in Section 9.3. For particular assignments, an instructor can supplement departmental rating scales tailored to the assignment. This method allows the instructor to assess both communication skills and application of knowledge. For example, Scofield and Combes (1993) describe a writing assignment in which students must prepare two alternative balance sheets and write a memorandum to the CEO recommending one of the two formats. Goals of the assignment include: The student will use official pronouncements The student will recognize conflicts within current GAAP The student will place accounting in its business context The student must make an accounting decision (pp. 78-79) Appendix 3, mentioned earlier, presents the assignment, criteria and checklist used to facilitate scoring of this assignment. Previous Continued... Back to Table of Contents