|
Chapter
10
Assessing Contributions of the Educational
Environment
The educational environment includes all the academic influences to which students are exposed. Its assessment, therefore, is essential to understanding observed learning outcomes and planning program improvements.
This section describes assessment of the educational environment from four perspectives:
- Curricular requirements: to document that the means for attainment of curricular goals exist
- Instructional emphasis: to determine whether pedagogical practices support attainment of curricular goals
- Student involvement: to determine the contributions to learning outcomes of students' participation in opportunities afforded by the program
- Stakeholders' perceptions of the program: to determine how students, alumni, faculty, employers, and other stakeholders perceive the program and its graduates, and how they think the program be improved
Figure 10.1 illustrates each of these perspectives.
10.1 Curricular Requirements
When the educational environment and curricular objectives are synchronized, the probability that students will achieve the goals is naturally increased. Conversely, students will have little chance of acquiring knowledge, skills, and professional orientations not emphasized in the curriculum. A review of curricular and instructional requirements enables the faculty to determine whether program goals receive adequate content and methodology attention.
A review of the curriculum can determine whether the means exist within the educational environment for qualified students to achieve the goals of the curriculum. The existence of requirements, however, leaves unanswered the question of whether the goals have in fact been achieved.
A straightforward way to analyze the curriculum is to review curricular requirements set forth in program documents (bulletins, advising materials, etc.). Questions such as the following should guide the review:
- Do the formal requirements of the curriculum reflect a balanced emphasis on program goals and objectives?
- Are learning-to-learn goals formally stated, recognized, and represented as part of the curriculum?
- Do the requirements ensure that students have achieved the desired level of competence (for example, achieving a certain level of performance in a course or on a proficiency examination)?
FIGURE 10.1
ASSESSING CONTRIBUTIONS OF THE EDUCATIONAL ENVIRONMENT
Curricular Requirements:
- Review of curricular requirements: Does the curriculum as described in program documents include requirements related to the targeted outcomes? Examples:
- Completion of a course that emphasizes the target skill
- Inclusion in the curriculum of required experiences such as writing major papers, making oral presentations, or conducting independent research
- Completion of a capstone course or project that involves a comprehensive assessment
Instructional Emphasis:
- Analysis of syllabi and course materials: do course goals, assignments, examinations and projects emphasize the targeted outcomes? Criteria:
- Course activities include instruction, practice, and feedback relevant to program objectives.
- Course grades are based on performance assessments that are consistent with stated program goals.
- Curricular Map: A plan of goal-related activities in each course as taught by each instructor reveals good coverage and balance in use of methods to enhance students' learning-to-learn capabilities.
Student Involvement:
- Review of transcripts: What clusters of coursework, both in the major and the general education curriculum, contribute to student gains on analytical thinking, quantitative reasoning, communication or other valued outcomes? (Ratcliff and Jones, 1993)
- Student self-reports: What educational experiences do students report? How do these relate to a) self-rated progress on program objectives, and b) documented achievement on outcome measures?
Perceptions and Satisfaction:
- Surveys, questionnaires, interviews, focus groups: How is the program perceived by stakeholders? Examples:
- Current Students
- Graduating Seniors
- Alumni
- Employers
- Importance/satisfaction analysis: What priority should be assigned to improvement efforts for various aspects of the program? What strengths should be emphasized in recruitment and employer contacts?
Table 10.1 illustrates partial results of a hypothetical analysis of curricular requirements and objectives related to international accounting.
10.2 Instructional Emphasis
Complementing formal curricular requirements are the goal-related learning experiences required by faculty in their courses. Instructional emphasis can be determined by constructing a curricular map. A matrix with goals as one dimension and courses as the other uses goal-related requirements outlined in course syllabi, assignments, projects, and examinations as indicators of emphasis. Faculty then determine whether important program objectives receive balanced attention within the curriculum. Table 10.2 presents an example of a curricular map.
If the faculty believe graduates should have good writing skills, they should ensure that each semester, students complete writing instruction and assignments relevant to their future roles as professional accountants. And, are these assignments supported by instruction and feedback specific to the type of writing assigned? Similarly, if faculty and future employers value skill in working with clients an co-workers, opportunities for interaction with a wide range of people should be built into the curriculum, followed by relevant feedback.
A number of instructional practices are known to be associated with desirable learning outcomes, such as increased knowledge retention and developed critical thinking skills. One important practice is the use of cooperative learning strategies, in which students work together to solve problems, study new material, or take tests. Students in accounting and other disciplines support the use of cooperative learning (Astin, 1993; Michaelsen & others 1989; King, 1990; Bonsangue and Drew, forthcoming). Many redesigned accounting programs incorporate cooperative learning to foster skills of communication, critical thinking, and teamwork (examples include ASU, USC, and BYU; see Cottell and Mills, 1993, for examples).
Ewell has argued that the use of instructional practices known to be effective is in itself evidence of program quality (1993). However, a more persuasive approach is to include data on students' participation in such practices as part of a multivariate analysis of factors presumed to contribute to observed learning outcomes (as described below).
10.3 Student Involvement
Involvement in learning is an important predictor of student development, with some experiences more likely than others to contribute to each of the program's goals (Astin, 1993). for example, students who take courses that emphasize questioning and the application of knowledge can be expected to show greater progress on learning-to-learn measures than students who avoid such classes. Analysis of relationships between student involvement and learning outcomes, appropriately weighted for entering student characteristics, can help faculty understand patterns of progress and achievement for all students and for particular subgroups such as students of color or transfer students.
Studies that examine the relationship between student characteristics and outcomes, but fail to consider students' educational experiences, may inadvertently reinforce the status quo. For example, the authors of two recent studies using admissions data to predict students' performance in elementary and intermediate accounting find that students with higher admission test scores are more likely to do well in these courses. They conclude that admission test scores should be an important factor in student selection (Booker, 1991; Doran, Boullion, and Smith, 1991). Neither of these studies examined the instructional methods and supports to which students were exposed, precluding the possibility of assessing whether other instructional approaches or the use of support systems might have led to enhanced success for lower-scoring students. This finding is well documented in studies of calculus instruction (e.g., Bonsangue and Drew, forthcoming), and consistent with reports of greater success rates and lower attrition in innovative accounting programs (McKenzie, p. 10; Pincus, Forthcoming).
Standardized test scores may disproportionately handicap students of color, so drawing conclusions about predictors without including measures of the educational environment is risky. Moreover, since the profession seeks a stronger emphasis on producing graduates with conceptual abilities not always addressed in traditionally taught courses, studies that use performance as the dependent variable should include both a description of the instructional methods used and an analysis of the degree to which conceptual thinking is emphasized in evaluating students' performance.
Two primary sources of information on student involvement are self-reports and transcripts.
Self-reports can be obtained using questionnaires on instructional practices and emphasis encountered by students in their courses as well as reports of participation in additional experiences such as internships, work experiences, and research with faculty. Note that self-reports (unlike perception data, described below) should not be obtained anonymously if the unit analysis is the individual student.Review of transcripts: Transcript information can be used to identify clusters of coursework, both in the major and the general education curriculum, that appear to contribute to students' progress toward achievement of curricular objectives (Ratcliff and Jones, 1993). Transcript information can be coupled with data on instructional emphasis in particular for a closer analysis of factors that contribute to observed student learning outcomes.
10.4 Stakeholders' Perceptions of the Program and Program Graduates
Stakeholder feedback is an essential resource for continuous quality improvement. Students (including those who withdraw from the program), alumni, faculty, and employers or recruiters should be periodically surveyed or interviewed to determine:
- Perceptions of the program's strengths and weaknesses
- Satisfaction with various program components and the program as a whole
- Suggestions for how the program can be improved
Students' perceptions can be assessed at strategic points in their experience, for example, when they have just completed an innovative instructional unit, upon completion of a course or coherent segment of the program, upon graduation, and after program completion when they have achieved the status of alumni. Employers and recruiters can also be surveyed at strategic moments, for example, just prior to development of a new curricular emphasis. Periodic assessment of perceptions and satisfaction is particularly important when innovations have been introduced.
Surveying: At many institutions the office of institutional research administers annual surveys of current students, seniors, and alumni. These surveys typically include questions on students' satisfaction with their majors, the advising they have received, and the faculty in their major departments, as well as questions about students' perceived progress toward achievement of curricular goals. Often a department can arrange to include additional questions for its majors and/or graduates as part the survey effort.Importance/Satisfaction analysis: Sifting through copious survey results is daunting without some way to identify priorities of survey respondents. A tool used in strategic marketing simplifies the task by combining data on performance and importance in a visual display of the association between these two dimensions for an array of survey items (Martilla and James, 1977). Students or other survey respondents rate each item on both dimensions, and results are plotted as shown in Figure 10.2. Areas high in importance but low in satisfaction are targets for intensive work; areas high in importance and satisfaction suggest program characteristics to emphasize in recruiting and marketing efforts.
A useful feature of importance/satisfaction analysis is that results can be plotted for subgroups such as transfer students, students of color, employers, and faculty. Displaying the results on transparency film, using different colors for each subgroup, allows easy identification of subgroup differences for many variables simultaneously.
Examples from accounting programs: Several accounting programs have developed surveys and interviews to assess constituents' perceptions and satisfaction. Examples include:
Survey: University of Virginia: Faculty have developed surveys for fourth-year students, alumni, and recruiters (see Appendix 10). The student and alumni surveys assess satisfaction with "personal development" in ten areas based on the AECC objectives: development of skills such as analysis, goal-setting, public speaking, writing, and problem solving and exposure to societal issues including race, gender, ethics, and international business. Surveys also assess perceived effectiveness of various instructional methods used in the McIntire School. Background information is obtained, including students' employment histories, memberships in professional societies and student social groups, GPA, and perceived impact of their interactions with faculty, students, and professionals.
Survey: BYU: At the end of the junior year, students at BYU complete an 80-question survey about all aspects of the program including teaching methods, curricular structure, learning outcomes, evaluation methods, faculty and teaching assistants (BYU Core, Vol. II, 1992, pp. 247&endash;257) Faculty and teaching assistants are also surveyed to determine their degree of satisfaction.
Survey: Arizona State University: Faculty members administer the following open-ended questionnaire during class in the spring term:
- What do you like about the accounting program?
- What specific things do your professors do that are a valuable use of your time and enhance your educational experience?
- What do you not like about the accounting program?
- What specific things do your professors do that are not a valuable use of your time and do not enhance your educational experience?
- Please add here any other comments you wish to make about the ASU accounting program.
Space is provided after each item to encourage written responses. The questionnaire also asks students to check which courses they have completed or are currently taking. This simple feedback mechanism provides considerable information at minimal cost.
Informal meetings: Faculty at BYU also sponsor regular, sack-lunch meetings to discuss the program (BYU Core, Vol. II 1992). Both faculty and students participate.
TABLE 10.1
DOES THE LEARNING ENVIRONMENT SUPPORT PROGRAM GOALS?a
SAMPLE ANALYSIS OF CURRICULAR REQUIREMENTS
LEARNING GOALS: Students will be able to function effectively in an international business environment
LEARNING ENVIRONMENT: The program will improve students' ability to function effectively in an international business environment
PERFORMANCE CRITERIA
EVIDENCE
PROGRAM CRITERIA: PROGRAM INCLUDES:
EVIDENCE
Student is able to:
explain and adapt to differences among major international business contexts
explain and adapt to cultural differences that may affect business transactions
Cross-grading of papers/projects dealing with international business in regular coursework
Ratings of international focus in senior project
Ratings of observed behavior in simulated international business situations
Required courses which contain an identifiable international business component
Courses which use international business examples
Assignments which require familiarity with international business problems and settings
Syllabi and course descriptions
Student self-reports
Course evaluations
Analysis of assignments by faculty and/or practicing professionals
Student is able to:
communicate effectively in a foreign language related to their business interests
Foreign language course grades or test results
Requirements that students must be able to communicate in a major business foreign language
Documentation that foreign language study is required and completed by all students
Graduate are able to perform effectively in international business settings
Placement office inquiries
Self-reports from alumni survey
Employer ratings of performance
Opportunity for students to have access to faculty with international business experience of scholarly interests in international business
Number of faculty with relevant experience and/or research interest
Self-reports of student contact with these faculty
a Adapted from a memorandum to AACSB by Peter Ewell, 1998
TABLE 10.2
A CURRICULUM MAP
OBJECTIVE
Course
101
102
103
104
201
202
203
Information Function of Accounting:
Role of Info Systems
X
Concepts of System Design and Use
X
Methods of Design and Use
X
Application of Methods
X
X
X
Use of Technology:
Info-gathering
X
X
Financial data management
X
X
X
X
Ethical Responsibilities of Accountants:
Ethical principles
X
Professional orientation
X
X
X
Complex Problem Solving:
Problem-solving strategies
Real-world applications
X
X
X
Interpersonal Skills:
Teamwork
X
X
X
X
Negotiation
X
X
X
X
Communication Skills:
Oral
X
X
X
X
X
X
Written
X
X
X
X
X
