AN INVESTIGATION OF NARRATIVES VERSUS QUESTIONNAIRES FOR IDENTIFYING INTERNAL CONTROL WEAKNESSES AND STRENGTHS WITH IMPLICATIONS FOR DECISION AID EFFECTIVENESS

 

 

 

 

James Lloyd Bierstaker

University of Massachusetts-Boston

 

 

Jay C. Thibodeau

Bentley College

 

 

Corresponding Author

James Lloyd Bierstaker

Department of Accounting and Finance

College of Management

University of Massachusetts-Boston

Boston, MA 02125-3393

Phone: (617) 287-7685

Fax: (617) 287-7725

e-mail: james.bierstaker@umb.edu

 

 

 

 

 

 

October 2002

 

Acknowledgements: We thank the auditors who took the time to complete the experimental materials and an anonymous reviewer of the 2002 Auditing Section Mid-Year conference for many helpful comments and suggestions.

AN INVESTIGATION OF NARRATIVES VERSUS QUESTIONNAIRES FOR IDENTIFYING INTERNAL CONTROL WEAKNESSES AND STRENGTHS WITH IMPLICATIONS FOR DECISION AID EFFECTIVENESS

ABSTRACT

Two formats auditors have commonly used in the past to document their understanding of the client’s internal control are the narrative and questionnaire (Cushing and Loebbecke 1986; Bierstaker 1999a). However, recent research suggests that auditors may often rely solely on a narrative to document a client’s internal control and that while the popularity of the narrative has increased since 1995, other formats such as questionnaires are being used less frequently (Bierstaker and Wright 2002). The purpose of this study is to investigate whether the use of a questionnaire or narrative leads to better auditor performance in identifying internal control strengths and weaknesses. The choice of internal control documentation format is important because it guides auditors’ information acquisition, and may affect the way auditors encode (Plumlee 1985), recall (Bierstaker 2002), recognize (Plumlee et al. 1998), and evaluate internal control information (Frederick 1992).

To investigate these issues, 49 auditors identified internal control weaknesses and strengths, and assessed control risk, after preparing either a narrative memorandum or completing an internal control questionnaire (but not both). The results of this study indicate that auditors who completed a questionnaire outperformed those auditors who prepared a narrative memorandum when identifying internal control weaknesses. No significant differences were found between groups for internal control strengths. In addition, auditors who completed a questionnaire assessed control risk higher than auditors who prepared a narrative. The theoretical implications of these results are that auditors who complete an internal control questionnaire appear to access an appropriate problem representation of the auditee’s internal control, allowing them to identify more weaknesses and assess control risk higher than auditors who prepared a narrative. From a practical standpoint, completing a questionnaire may help auditors to identify important control weaknesses, improve error detection, fraud detection, and avoid audit failure. Overall, these results are important because they indicate that the use of a decision aid (the questionnaire) leads to better auditor performance in identifying internal control weaknesses and more conservative performance in evaluating internal control.

Key Words: Internal Control, Questionnaire, Narrative, and Decision Aids

Data Availability: Data used in this study are available from the authors upon request.

AN INVESTIGATION OF NARRATIVES VERSUS QUESTIONNAIRES FOR IDENTIFYING INTERNAL CONTROL WEAKNESSES AND STRENGTHS WITH IMPLICATIONS FOR DECISION AID EFFECTIVENESS

INTRODUCTION

An audit firm uses its existing knowledge to provide services that are demanded by the marketplace. Knowledge is therefore a critical economic resource for an audit firm, and a firm’s profitability will be determined, in large part, by the firm’s capacity to manage its existing firm knowledge. It therefore follows that an audit firm should seek to allocate its existing firm knowledge in a manner that will maximize profitability. One way that an audit firm manages its knowledge is through the development and use of decision aids.

Decision aids allow the knowledge of firm experts to be transferred to its staff auditors. However, the evidence on the effectiveness of decision-aids in auditing is mixed. For example, Holt (1987) found that "auditors actually appeared to integrate base-rates and case-specific evidence less normatively than did the student population subgroups." One possible explanation for this finding is that auditors with a greater amount of relevant task knowledge will choose to rely more on their own knowledge as opposed to base-rate and other case specific knowledge. In addition, checklist decision aids may have harmful effects such as interference (Fischoff et al. 1978; Hoch 1984; Frederick 1991) and information fragmentation (Pincus 1989). In other words, checklists may prevent the auditor from identifying client risks that are not featured on the checklist (Asare and Wright 2001), or from seeing a particular pattern of information. Thus, research investigating the effectiveness of decision aids for experienced auditors is both necessary and important.

One type of decision aid frequently developed by audit firms is the questionnaire. Questionnaires are used for a variety of audit tasks including risk management and fraud detection (Plumlee et al. 1998). Questionnaires are also commonly used to help document an audit firm’s understanding and evaluation of internal control systems. The questionnaire serves as a decision aid because it guides a staff auditor’s search for information about the auditee’s internal control system. The questionnaire is expected to improve a staff auditor’s task performance because it is based on expert knowledge.

Interestingly, recent research suggests that the use of questionnaires is declining relative to the use of the narratives in documenting and evaluating internal control (Bierstaker and Wright 2002). Although it is possible that the choice of documentation format is related to the size and complexity of the system being evaluated, Bierstaker and Wright (2002) suggest that audit firms may be attempting to maximize audit efficiency by choosing to rely on a single documentation format. Professional standards allow the use of either the narrative or the questionnaire to document an auditor’s understanding and evaluation of an auditee's internal control. However, in using the narrative approach, a staff auditor would no longer benefit from the expert knowledge that is embedded in a questionnaire. Further, it is possible that even experienced auditors may overlook important internal control features if they are not highlighted by the documentation format (Purvis 1989). Thus, the recent shift toward the narrative format as the primary means to document internal control may result in decreased audit effectiveness.

The purpose of this study is to investigate whether the use of a questionnaire or narrative leads to better auditor performance in identifying internal control strengths and weaknesses. The choice of documentation format is important because only one of the formats provides an auditor with a decision aid (the questionnaire). For some time, decision aids has been seen as an effective manner for an audit firm to transfer relevant task knowledge to novice auditors (Graham 1993). Thus, it is important to test whether the use of the decision aid (the questionnaire) improves audit task performance.

To investigate these issues, 49 auditors first prepared either a narrative memorandum or completed an internal control questionnaire. Next each auditor identified internal control weaknesses, strengths and then assessed control risk. The results of this study indicate that auditors who completed a questionnaire identified more internal control weaknesses than auditors who prepared a narrative memorandum. No significant differences were found between the groups in identifying internal control strengths. Finally, auditors who completed a questionnaire assessed control risk higher than auditors who prepared a narrative.

The results of this study have both theoretical and practical implications. The theoretical implications are that auditors who complete an internal control questionnaire appear to formulate a more complete problem representation of the auditee’s internal control, which allows them to identify more weaknesses and assess control risk higher than auditors preparing a narrative. From a practical standpoint, the results of this study imply that completing a questionnaire may help auditors to identify important control weaknesses, improve error detection, fraud detection, and avoid audit failure.

DEVELOPMENT OF HYPOTHESES

SAS No. 78 – Consideration of an Auditee’s Internal Control Structure

SAS No. 78 requires that on each audit engagement auditors obtain a "sufficient understanding of internal control" to plan the audit and determine the nature, timing and extent of substantive audit testing (AICPA 1995). Auditors use this understanding to evaluate the capability of an auditee’s internal control system to detect or prevent the occurrence of errors and/or misstatements in an auditee’s financial statements. In recent years, the evaluation of an auditee’s internal control system has received increased attention from practitioners (Arthur Anderson et al. 1992), the business press (Berton 1993) and academics (Hirst and Koonce 1996). This attention has primarily surfaced because practicing auditors have increasingly sought to rely on an auditee’s internal control system as a manner to improve an audit’s efficiency. Indeed, many of the current audit approaches are dependent upon a detailed understanding and evaluation of the internal control systems that exist within an auditee’s business processes (Bell et al., 1997; Eilifsen et al. 2001). Stated simply, as auditors acquire a detailed understanding of an auditee’s business processes, risks, and strategies, it is likely that auditors will be able to use this understanding to take better advantage of an effective internal control system in planning their audit tests (Winograd et al. 2000).

SAS No. 78 also requires that auditors document their understanding and evaluation of an auditee’s internal control system. There are a number of common formats that an auditor can use to document their understanding and evaluation of internal controls, including questionnaires and narratives (Purvis 1989). However, no explicit guidance is provided in SAS No. 78 regarding a specific documentation format that must be used. This lack of professional guidance is surprising, especially when considering that the audit judgment literature consistently shows that the preparation and review of audit documentation can influence an auditor’s evaluation of an auditee’s internal control system. Specifically, auditing research suggests that the format used to document internal control may structure the information processing activities of auditors (Mock and Turner 1981), including information acquisition behavior. In addition, the documentation format may affect the way auditors encode (Plumlee 1985), recall (Bierstaker 2002), recognize (Plumlee et al. 1998), and evaluate information (Frederick 1992). It may be that documentation format impacts the nature of the problem representation created by an auditor, which in turn influences the auditor’s task performance. Indeed, Vera-Munoz et al. (2001) provide evidence that the task presentation format affects an accountants’ problem representation for the task.

A problem representation for a task can be defined as a template in which people organize their informational cues in their working memory that are being processed when making a decision within a task context (Newell and Simon 1972). The problem representation influences the retrieval of knowledge, how current information is evaluated, and ultimately guides the judgments involving the task variables (Newell and Simon 1972). Essentially, a problem representation can be viewed as a cognitive structure that develops during the decision making process. This developmental process will differ, depending on the knowledge retrieved from memory, the informational cues that are acquired, and the subsequent judgments that are made during the decision making process.

Christ (1993) investigated the nature of problem representations created by auditors in a planning task. She noted that a problem representation first develops when an auditor maps the information about a current task into an existing knowledge structure that has been retrieved from memory. At this point, the developmental process is heavily influenced by any additional information that is either provided to (e.g., through a decision aid) or requested by (e.g., from the client) an auditor during the decision making process. Moreover, depending on how the structure develops, there are likely to be differences in subsequent information acquisition behavior and judgments made by an auditor within the task context. Perhaps not surprisingly, the nature and the quality of an auditor’s problem representation for a task have been linked to audit judgment (Bierstaker et al. 1999) and recall performance (Christ 1993).

The relationship between the nature and quality of an auditor’s problem representation and auditor judgments has been addressed within an internal control context. Specifically, Bonner and Pennington (1991) contend that the quality of the representation created by an auditor for the client's internal control system, including the strengths and weaknesses of the system, may impact the auditor’s evaluation of that system. There is also evidence that the choice of internal control documentation format impacts information acquisition behavior. That is, Purvis (1989) suggests that the quantity and type (e.g., procedural, organizational) of internal control information auditors acquire may differ depending on the documentation format. Consequently, the format used to document internal control information can impact auditors’ decision outcomes (Boritz 1985).

Two of the formats that auditors have commonly used in the past to document their understanding and evaluation of an auditee’s internal control system are the narrative and the questionnaire (Cushing and Loebbecke 1986). Interestingly, recent research suggests that while the use of narratives is on the rise, the use of questionnaires is declining. Bierstaker and Wright (2002) report that 88 percent of the auditors surveyed in 1995 reported using narratives compared to 98 percent surveyed in 2000. In contrast, 60 percent of the auditors surveyed in 1995 reported using questionnaires compared to 17 percent surveyed in 2000. In addition, 61 percent of auditors in 2000 reported using a narrative only for internal control documentation compared to 7.7 percent in 1995. The evidence suggests a significant decline in the use of questionnaires, relative to the use of narratives.

The choice of documentation format in an internal control context is important because questionnaires and narratives differ in the nature of knowledge provided to an auditor during the decision making process. Narratives are a very flexible approach towards documentation, as much of the documentation is left to the auditor’s discretion. The advantages of narratives include their specificity to an auditee and their exacting and detailed nature. Disadvantages of narratives include their lack of standardization (Purvis 1989) and their lack of expert guidance. Questionnaires are developed by public accounting firms to explicitly focus auditors’ attention on the critical strengths and weaknesses of the auditee’s internal control system. They are intended to serve as a decision aid developed by firm experts to provide a comprehensive approach to internal control evaluation and documentation. Questionnaires are highly structured and are generally easy to prepare. The potential downside of questionnaires is that they are very general and often lack flexibility.

Questionnaires could also potentially interfere with the auditor’s evidence collecting process (Asare and Wright 2001). Research on comprehensive checklists known as "fault trees," for example, suggests subjects become to reliant on the diagnostic aid and fail to recognize potential problems not listed in the checklist (Fischhoff et al. 1978). Therefore, auditors may attend to aspects of the internal control system addressed by the questionnaire, but ignore other internal control features that are not addressed in the questionnaire.

However, unlike narratives, audit firms standardize questionnaires to ensure quality control. Purvis (1989) found that auditors collected more internal control data when using a questionnaire as opposed to a narrative or flowchart. It is likely that the development of auditors’ problem representations were influenced differently, depending on the documentation format employed. Cognitive theory would suggest that the information and knowledge supplied by a questionnaire (a decision aid) influences subsequent information acquisition behavior and judgments made by auditors. Specifically, it is likely that an internal control questionnaire will serve as a template that contains relevant task-specific knowledge and guides auditors through a maze of internal control information, much the same way that a problem representation would. If this is the case, the use of a questionnaire may lead auditors to a more thorough and focused evaluation of an auditee’s internal control system. Therefore, auditors who complete a questionnaire should identify more internal control weaknesses and strengths than auditors who prepare a narrative memorandum. The first two hypotheses follow:

H1: Auditors who complete an internal control questionnaire will correctly identify more internal control weaknesses than auditors who prepare a narrative.

H2: Auditors who complete an internal control questionnaire will correctly identify more internal control strengths than auditors who prepare a narrative.

Since auditors completing an internal control questionnaire are likely to develop a more thorough understanding of the client’s internal control system, as perhaps illustrated by the identification of more control weaknesses and strengths, than auditors who prepare a narrative memorandum, it is likely that internal control documentation format will influence auditors’ internal control evaluation as well. In addition, there are a number of reasons why an auditor’s internal control evaluation is likely to be impacted more significantly by internal control weaknesses, as compared to internal control strengths. For example, the identification of internal control weaknesses is critically important for error detection, fraud detection, and avoidance of audit failure. Controls identified as weak will not be tested, rather, substantive testing of the related balances will be preformed. Furthermore, SAS No. 60 requires auditors to communicate to management any significant deficiencies in the design or operation of internal control (AICPA 1988). Also, auditors may provide information about potential improvements to internal control in management letter comments.

As predicted in H1 and H2, if auditors’ internal control evaluations are influenced by the questionnaire documentation format and a greater number of strengths and weaknesses are indeed identified, it is likely that this will result in more conservative internal control evaluations, due to the greater focus on weaknesses related to the asymmetrical loss function (Bierstaker 2002). It therefore follows that auditors who complete an internal control questionnaire will have higher overall internal control risk assessments than auditors who prepare narratives. The third hypothesis follows:

H3: Auditors who complete an internal control questionnaire will have higher control risk assessments than auditors who prepare a narrative.

Task-Specific Experience

Previous auditing research suggests that relevant task experience has an important influence on information acquisition and task performance. For example, Mock and Turner (1981) and Biggs et al. (1988) found a positive relationship between experience and the quantity of data collected. Biggs and Mock (1983) and Biggs et al. (1987) linked experience with typical accounting and control systems to the completeness of information search. In other words, more experienced auditors acquired more relevant information than less experienced auditors. In addition, these studies suggest that task-specific experience can influence the quality of auditor decision processes. Similarly, Davis (1996) found that task-specific experience improves auditors’ acquisition of internal control information. These findings suggest that the information acquisition strategy of more experienced auditors is influenced by their knowledge of the important features of a well-designed internal control system (Bonner and Pennington 1991). Moreover, Bonner and Lewis (1990) found that audit experience had a significant effect on auditors’ performance of their internal control task, which asked participants to list two errors that could occur and go undetected for a given internal control weakness, as well as two substantive procedures that would be useful for detecting those errors. Furthermore, Purvis (1989) found that seniors collected more internal control information than less experienced auditors. Finally, Bierstaker and Wright (2001) found that more experienced auditors identified more internal control weaknesses than less experienced auditors. Based on this research, a measure of internal control evaluation experience is included as a covariate in the hypothesis testing models.

METHODS

Participants

An experiment was administered to 49 auditors from two Big Five CPA firms via the internet. Both firms commonly used both narratives and questionnaires to document and evaluate internal control. Auditors had a mean of 55.3 months of audit experience (standard deviation 65.6) and a mean of 34.6 months of experience (standard deviation 53.8 months) evaluating internal control. Although experience varied considerably among the participants, it is controlled for in the hypothesis testing models using ANCOVA (see below).

Experimental Design and Case Materials

Participants received the experimental instrument via the internet. The experimental instrument contains the following: an introduction, general instructions, case materials, the tasks to be completed, answer forms and a questionnaire. The introduction states that the purpose of the study is to examine auditors’ decisions concerning internal control evaluation. Participants were told that all responses would be kept confidential.

After reading the introductory instructions, auditors read the case materials. The case materials include background information that describes a hypothetical client with a good reputation for honesty and integrity and a description of the client’s accounting procedures and controls for the Sales and Collections cycle. The case materials were based on the internal control evaluation task used in Bierstaker and Wright (2001). In addition, a questionnaire was created based on generic internal control questionnaires for the Sales and Collections cycle found in auditing textbooks and audit firm training manuals. The questionnaire was modified based on feedback from auditors at the participating firm to be consistent with the questionnaire used by the firm. The questionnaire was not modified based on the internal control strengths and weaknesses included in the case.

Auditors randomly received one of two documentation formats: 1) auditors in the narrative condition (NAR) were asked to prepare an internal control narrative, and 2) auditors in the questionnaire condition (QUES) were asked to complete an internal control questionnaire. Experimental materials were reviewed to verify that all auditors completed either a questionnaire or narrative. Next, auditors listed internal control strengths and weaknesses on answer forms similar to those used by Plumlee (1985). The answer forms define a "strength" as a control which provides a high likelihood of detecting or preventing certain irregularities or errors, and a "weakness" as the absence of a necessary control or a control which is likely to fail to detect or prevent an irregularity or error. Then auditors assessed control risk from 0 (low) to 100 (high) percent. Finally, auditors completed a questionnaire on their audit experience. The mean amount of time it took auditors to complete the experiment was 69.6 minutes (standard deviation 27.5). The mean amount of time taken by the auditors in the questionnaire condition (73.7 minutes) and the narrative condition (64.7 minutes) was not significantly different (t = 1.11; p = 0.27).

Dependent Variable

There are three dependent variables. The first, WEAK, is based on the number of control weaknesses a participant correctly identifies. The second, STRONG, is based on the number of control strengths a participant correctly identifies. A Delphi panel of three audit managers was used to identify the internal control weaknesses and strengths contained in the case (cf. Leape et al. 1992; Wright 1988). To measure WEAK and STRONG, participants’ lists of control weaknesses and strengths were compared to the list developed by the Delphi panel (see Appendix). The third depended variable, internal control risk, was assessed by auditors on a scale from 0 (low risk) to 100 (maximum risk) percent.

Independent Variables

A dummy variable, FORMAT, was assigned a value of zero for NAR auditors and one for QUES auditors. The experience variable, EVALEXP, is the number of months of experience with internal control evaluation.

RESULTS

Internal Control Weaknesses

The first hypothesis predicted that auditors who complete an internal control questionnaire would correctly identify more internal control weaknesses than auditors who prepare a narrative. As shown on Panel A of Table 1, QUES auditors identified a mean of 4.55 weaknesses (standard deviation 3.8) as compared to NAR auditors who identified a mean of 2.13 weaknesses (standard deviation 1.4). As shown on Panel B of Table 1, the results of ANCOVA reveal that this difference is statistically significant (p < 0.05). Therefore, the first hypothesis is supported. In addition, consistent with previous research, more experienced auditors identified a great number of internal control weaknesses.

____________________

Insert Table 1 about here

____________________

Internal Control Strengths

The second hypothesis predicted that auditors who complete an internal control questionnaire would correctly identify more internal control strengths than auditors who prepare a narrative. Contrary to expectations, there was no difference across the experimental groups. As shown on Panel A of Table 2, QUES auditors identified a mean of 3.75 strengths (standard deviation 2.8) as compared to NAR auditors who identified a mean of 4.05 strengths (standard deviation 2.4). As shown on Panel B of Table 2, the results of ANCOVA reveal that this difference is not statistically significant. One explanation for these results is that due to the asymmetrical loss function, auditors may be more focused on weaknesses than strengths (Bierstaker 2002). If auditors devote most of their attention to identifying internal control weaknesses, it may explain why there were no significant differences for internal control strengths across the two research conditions.

____________________

Insert Table 2 about here

____________________

 

Ratio of Weaknesses to Weaknesses Plus Strengths Identified

In addition to looking at the absolute number of strengths and weaknesses identified, a ratio of the weaknesses identified by each auditor divided by the number of weaknesses plus strengths identified (WEAKRATIO) was examined for each auditor. This was done for two reasons. First, there were more potential strengths (16) in the audit case than weaknesses (12). Second, auditors in the questionnaire condition appeared to identify more weaknesses than strengths, whereas auditors in the narrative condition did not. As shown on Panel A of Table 3, the mean WEAKRATIO is 0.54 (standard deviation 0.14) in the questionnaire condition and 0.35 (standard deviation 0.24) in the narrative condition, indicating that auditors in the questionnaire condition had a more balanced evaluation of internal control than auditors in the narrative condition. As shown on Panel B of Table 3, the results of multiple regression indicate this difference is significant (p = 0.0025). In addition, the main effect of internal control evaluation experience (p = 0.001) and the interaction between format and experience are significant (p = 0.04). Further analysis of the interaction effect indicates that the relationship between experience and WEAKRATIO is strongest in the questionnaire condition. Specifically, the R2 = 0.48 for experience and WEAKRATIO in the questionnaire condition as compared to 0.28 in the narrative condition.

These results suggest that both the questionnaire format and internal control evaluation experience appear to heighten auditors’ awareness of internal control weaknesses. Moreover, the relationship between experience and the WEAKRATIO appears to be at it strongest in the questionnaire condition. One explanation for these findings is that the questionnaire format highlights internal control weaknesses to a greater extent than the narrative format.

____________________

Insert Table 3 about here

____________________

Internal Control Risk

The third hypothesis predicted that auditors who complete an internal control questionnaire would have higher control risk assessments than auditors who prepare a narrative. As shown on Panel A of Table 4, QUES auditors mean control risk assessment was 67.5 percent (standard deviation 17.5) as compared to NAR auditors who had a mean control risk assessment of 43.2 percent (standard deviation 23.1). As shown on Panel B of Table 4, the results of ANCOVA reveal that this difference is statistically significant (p < 0.02). Therefore, the third hypothesis is supported. In addition, the number of weaknesses identified (p = 0.05) and the number of strengths identified (p < 0.005) were significant covariates.

Overall, these results suggest that auditors who completed a questionnaire assessed control risk higher than auditors who prepared a narrative, even when controlling for the number of weaknesses and strengths identified. One explanation for this finding is that the auditors who completed a questionnaire (a decision aid) were able to develop a more thorough understanding of the client’s internal control deficiencies which led them to assess control risk as relatively high, whereas auditors who prepared a narrative may have had a more superficial understanding of internal control such that their assessments of control risk were relatively lower. In fact, while auditors who used a questionnaire identified a similar number of weaknesses and strengths on average (4.55 and 3.75), auditors who used a narrative tended to identify about half as many weaknesses as strengths (2.14 and 4.05). Therefore, it appears that auditors who completed a questionnaire had a more balanced, and less positive, impression of the client’s internal control than auditors who prepared a narrative.

____________________

Insert Table 4 about here

____________________

DISCUSSION

This study has three main findings. First, consistent with expectations, auditors who completed an internal control questionnaire outperformed auditors who prepared a narrative when identifying internal control weaknesses. In addition, auditors in the questionnaire condition had a larger, and more balanced, ratio of weaknesses to weaknesses plus strengths than auditors in the narrative condition. Second, contrary to expectations, there was no difference in the performance of auditors who completed an internal control questionnaire as compared to auditors who prepared a narrative for the evaluation of internal control strengths. Third, auditors who completed a questionnaire assessed control risk higher (exhibiting more conservative behavior) than auditors who prepared a narrative, consistent with the third hypothesis.

Overall, these findings suggest that the use of a decision aid (the questionnaire) to document and evaluate an auditee’s internal control will improve an auditor’s performance in identifying internal control weaknesses, a critical step in evaluating internal control. For some time, audit firms have devoted considerable resources to develop decision aids that are capable of improving the effectiveness of task performance. Our results imply that firms should continue to devote resources towards this end and perhaps even require that their auditors use a questionnaire when documenting and evaluating and auditee’s internal control. It appears that the questionnaire supplies important knowledge for a task that an auditor can then integrate into his/her problem representation for that task. Both experienced auditors, and auditors who used a questionnaire, identified more weaknesses and had a higher ratio of weaknesses to weaknesses plus strengths than less experienced auditors and auditors who used a narrative. These results provide compelling evidence in support of the notion that decision aids can be an effective mechanism to transfer knowledge from a firm’s expert to a novice within an audit firm (Graham 1993). Future research could investigate if the use of a decision aid (questionnaire) improves performance in other audit tasks, such as fraud risk assessment (Plumlee et al. 1998).

The findings of this study are particularly important given recent evidence that audit firms are increasing the use of narratives, relative to questionnaires, in order to document and evaluate an auditee’s internal control system (Bierstaker and Wright 2002). Bierstaker and Wright also provide evidence to suggest that the narrative has emerged as the preferred format because it is an efficient way to document internal control. Our results imply that this trend could reduce audit effectiveness in evaluating internal control, since the questionnaire appears to highlight internal control weaknesses, even for experienced auditors. Although it is possible that the use of a questionnaire could result in over-auditing, auditors who used a questionnaire in this study identified a great number of weaknesses that had been previously identified and rated as important by an expert panel of experienced auditors than auditors who used a narrative. Failure to properly evaluate internal control could lead to increased audit costs in a variety of ways. If weaknesses in the system are not identified, errors or irregularities may be left undetected by the audit. Audit failure, with the ensuing risk of litigation damages and reputation costs may occur. In addition, auditors are required to report significant internal control deficiencies to the audit committee and the board of directors.

Unfortunately, our results do not allow for conclusions about how the expert firm knowledge, supplied by a decision aid, actually improves task performance. It may be that the decision aid acts as a surrogate problem representation, guiding the auditor through the information acquisition process. Another possibility is that the knowledge supplied by a decision aid allows an auditor to access and transfer other relevant task knowledge previously stored in an auditor’s memory (Thibodeau 2002). This additional knowledge may then be integrated into an auditor’s problem representation, which in turn helps to improve auditor performance. Alternatively, the decision aid may act as a type of "hint" that helps an auditor shift his/her problem representation (Bierstaker et al. 1999). Future process-tracing research is needed to investigate the cognitive reasons that would help to explain why a decision aid helps auditor performance. Moreover, verbal protocol analysis would provide more direct evidence regarding auditors’ problem representations.

Finally, since auditors who completed a questionnaire assessed control risk significantly higher than auditors who prepared a narrative, the results of this study imply that auditors who use a questionnaire for internal control evaluation will rely less on internal control, dedicate less audit effort to tests of controls, and place more audit effort on substantive tests than auditors who prepare a narrative. However, future research could explore the effect of the use of a questionnaire versus a narrative for internal control evaluation on audit program planning. In addition, future research could examine whether or not the results of this study generalize to other Big Five firms and non-Big Five accounting firms, and if the findings of this study could be replicated using other audit cases and tasks. For example, it may be that questionnaires are more effective in an internal control evaluation context (Purvis 1989) than in a fraud context (Asare and Wright 2001; Pincus 1989).

 

REFERENCES

Abdolmohammadi, M. J. A Comprehensive Taxonomy of Task Complexity and Knowledge Base Demands in Auditing. Unpublished working paper, Bentley College, 1993.

Alba, J., and L. Hasher. Is Memory Schematic? Psychological Bulletin (1983): 203-31.

American Institute of Certified Public Accountants. 1988. Statement on Auditing Standards No. 60: Communication of Internal Control Structure Related Matters Noted in an Audit. New York: AICPA.

_____. 1995. Consideration of the Internal Control Structure in a Financial Statement Audit: An Amendment to SAS No. 55. New York: AICPA.

Arthur Anderson & Co., Coopers & Lybrand, Deloitte & Touche, Ernst & Young, KPMG Peat Marwick, and Price Waterhouse. 1992. A Liability Crisis in the United States: Impact on the Accounting Profession. Journal of Accountancy 174(5): 19-23.

Asare, S. K. and A. Wright. 2001. The Impact of Risk Checklists and a Standard Audit Program on the Planning of Fraud Detection Procedures. University of Florida Working Paper.

Berton, L. 1993. Big accounting firms weed out risky clients. The Wall Street Journal June 26: B1, B3.

Bell, T., Marrs, F., Solomon, I., and Thomas, H. (1997), Auditing Organizations Through a Strategic-Systems Lens, KPMG Peat Marwick LLP, New York.

Bierstaker, J. L., J. C. Bedard, S. F. Biggs. 1999. The Role of Problem Representation Shifts on Audtior Decision Processes in Analytical Procedures. Auditing: A Journal of Practice & Theory (Spring): 18-36.

Bierstaker, J. L. 1999a. Internal Control Documentation: Which Format is Preferred? The Auditor’s Report (Winter): 12-13.

Bierstaker, J. L. 1999b. A Test of the Split Attention Effect in a Professional Context, Journal of Business and Behavioral Sciences, (Fall): 177-89.

Bierstaker, J. L. and A. Wright. 2002. An Investigation of Internal Control Documentation and Testing: Have Auditors’ Changed Their Approach? University of Massachusetts-Boston working paper.

Bierstaker, J. L. 2002. The Effects of Interference and Knowledge on Auditors’ Recall of Internal Control Strengths and Weaknesses. Managerial Auditing Journal (forthcoming).

Bierstaker, J. L. and S. Wright. 2001. A Research Note Concerning Practical Problem-Solving Ability as a Predictor of Performance in Auditing Tasks. Behavioral Research in Accounting 13: 49-62.

Biggs, S. F. and T. J. Mock. 1983. An investigation of auditor decision processes in the evaluation of internal controls and audit scope decisions. Journal of Accounting Research: 234-55.

Biggs, S. F., W. F. Messier, Jr., and J. V. Hansen. 1987. A descriptive analysis of computer audit specialists' decision-making behavior in advanced computer environments. Auditing: A Journal of Practice and Theory: 1-21.

Biggs, S. F., T. J. Mock and P. R. Watkins. 1988. Auditors’ Use of Analytical Review in Audit Program Design. Accounting Review (January): 148-61.

Bonner, S. E. 1990. Experience effects in auditing: The role of task-specific knowledge. The Accounting Review: 72-92.

Bonner, S. E. and B. L. Lewis. 1990. Determinants of auditor expertise. Journal of Accounting Research 28 (Supplement): 1-20.

Bonner, S. E. and N. Pennington. 1991. Cognitive Processes and Knowledge as Determinants of Auditor Expertise. Journal of Accounting Literature 10: 1-50.

Boritz, E. F. 1985. The effect of information presentation structures on audit planning and review judgments. Contemporary Accounting Research: 193-218.

Chi, M., P. Feltovich, and R. Glaser. 1981. Categorization and representation of physics problems by experts and novices. Cognitive Science 5 (2): 121-152.

Christ, M. Y. and I. Vessey. 1996. The Effect of Presentation Format and Experience on Analytical Review Judgments in a Going-Concern Context. Working paper.

Committee of Sponsoring Organizations of the Treadway Commission. 1992. Internal Control-Integrated Framework. Committee of Sponsoring Organizations of the Treadway Commission.

Cushing, B. E. and J. K. Loebbecke. 1986. Comparison of Audit Methodologies of Large Accounting Firms. Sarasota, FL: American Accounting Association.

Davis, J. T. 1996. Experience and Auditors’ Selection of Relevant Information for Preliminary Control Risk Assessments. Auditing: A Journal of Practice and Theory (Spring): 16-37.

Eilifsen, A., W. R. Knechel, and P. Wallage. 2001. Application of the Business Risk Audit Model: A Field Study. Accounting Horizons (September): 193-207.

Fischoff, B., P. Slovic, and S. Lichtenstein. 1978. Fault trees: Sensitivity of estimated failure probabilities to problem representation. Journal of Experimental Psychology: Human Perception and Performance: 330-344.

Frederick, D. M. 1991. Auditors' representation and retrieval of internal control knowledge. The Accounting Review 66: 240-58.

Frederick, D. M. 1992. The Influence of Experience and Knowledge Retrieval on Audit Judgments. University of Colorado Working Paper.

Graham, L. 1993. Discussion of Expertise in Auditing. Auditing: A Journal Of Practice And Theory 12 (Supplement): 46-50.

Hirst, D.E, and L. Koonce. 1996. Audit analytical procedures: A field investigation. Contemporary Accounting Research 13: 457-86.

Hoch, S. 1984. Availability and interference in predictive judgment. Journal of Experimental Psychology: Learning, Memory and Cognition (October): 649-62.

Holt, D. 1987. Auditors and Base Rates Revisited. Accounting, Organizations and Society 12: 571-578.

Leape, L., M. Freshour, D. Yntema, and W. Hsiao. 1992. Small-Group Judgment Methods for Determining Resource-Based Relative Values. Medical Care:11 NS28-NS39.

Martin, J. W. 1981. Documenting Key Audit Decisions. National Public Accountant (March): 18-22.

Messier, W. F., S. J. Kachelmeier, and K. Jensen. 1997. An Experimental Assessment of Recent Professional Developments in Nonstatistical Audit Sampling Guidance. University of Florida Working Paper.

Mock, T. and J. L. Turner. 1981. Internal Accounting Control Evaluation and Auditor Judgment. Audit Research Monograph no. 3. New York: AICPA.

 

Morris, T., and L. Empson. 1998. Organization and Expertise: An Exploration of Knowledge Bases and the Management of Accounting and Consulting Firms. Accounting, Organizations, and Society 23 (October): 609-624.

Newell, A., and H. Simon. 1972. Human Problem Solving. Englewood Cliffs, N.J.: Prentice-Hall.

Pincus, K. V. 1989. The Efficacy of a Red Flags Questionnaire for Assessing the Possibility of Fraud. Accounting, Organizations and Society 14: 153-163.

Plumlee, R. D. 1985. The Standard of Objectivity for Internal Auditors: Memory and Bias Effects. Journal of Accounting Research: 683-699.

Plumlee, R. D., B. Tuttle, C. L. Moeckel. 1998. Auditors’ Evidence Documentation and Judgment: The Mediating Role of Memory. University of Utah working paper.

Purvis, S. E. C. 1989. The Effect of Audit Documentation Format on Data Collection. Accounting, Organizations and Society : 551-63.

 

Thibodeau, J. 2002. The Development and Transferability of Task Knowledge. Unpublished Manuscript. Bentley College.

Vera-Munoz, S. C., W. R. Kinney, and S. E. Bonner. 2001. The Effects of Domain Experience and Task Presentation Format on Accountants’ Information Relevance Assurance. The Accounting Review (July): 405-430.

Waller, W. S. and W. L. Felix. 1984. Cognition and the Auditor's Opinion Formulation Process: A Schematic Model of Interactions Between Memory and Current Audit Evidence. In Decision Making and Accounting: Current Research, edited by S. Moriarity and E. Joyce, 27-48. Norman, OK: University of Oklahoma Press.

Winograd, B. N., J. S. Gerson, and B. L. Berlin. 2000. Audit Practices of Pricewaterhouse Coopers. Auditing: A Journal of Practice and Theory (Fall): 175-82.

Wright, A., 1988. The Impact of Prior Working Papers on Auditor Evidential Planning Judgments, Accounting, Organizations & Society : 595-606.

 

 

Table 1

Internal Control Weaknesses

Panel A: Means

Format Mean Stan. Dev. T-test P-value

Questionnaire 4.55 3.80 2.78 0.004

Narrative 2.14 1.39

Panel B: ANCOVA

Variable Sum of Squares DF Mean Square F-Value P-Value

Covariate

EVALEXP 168.76 1 168.76 45.24 0.0001

Main Effect

FORMAT 16.66 1 16.66 4.47 0.04

MODEL 219.9 2 110.0 29.47 0.0001

Key:

EVALEXP = participants’ months of internal control evaluation experience

FORMAT = internal control documentation format (questionnaire = 1; narrative = 0)

Table 2

Internal Control Strengths

Panel A: Means

Format Mean Stan. Dev. T-test P-value

Questionnaire 3.75 2.83 0.36 0.72

Narrative 4.05 2.42

Panel B: ANCOVA

Variable Sum of Squares DF Mean Square F-Value P-Value

Covariate

EVALEXP 5.807 1 5.807 0.85 0.363

Main Effect

FORMAT 0.45 1 0.45 0.066 0.799

MODEL 7.83 2 3.92 0.57 0.57

Key:

EVALEXP = participants’ months of internal control evaluation experience

FORMAT = internal control documentation format (questionnaire = 1; narrative = 0)

Table 3

Ratio of Weaknesses to Weaknesses Plus Strengths

Panel A: Means

Format Mean Stan. Dev. T-test P-value

Questionnaire 0.54 0.14 3.09 0.02

Narrative 0.35 0.24

Panel B: Regression

Adjusted R-Square: 0.41 F-Test: 10.2 P-Value: 0.0001

Variable Beta T-Value P-Value

FORMAT 0.45 3.00 0.0025

EVALEXP 1.12 3.41 0.001

FORMAT*EVALEXP 0.76 2.12 0.04

Key:

EVALEXP = participants’ months of internal control evaluation experience

FORMAT = internal control documentation format (questionnaire = 1; narrative = 0)

FORMAT*EVALEXP = the interaction between format and experience

Table 4

Internal Control Risk

Panel A: Means

Format Mean Stan. Dev. T-test P-value

Questionnaire 67.5 17.5 3.86 0.0001

Narrative 43.2 23.1

Panel B: ANCOVA

Variable Sum of Squares DF Mean Square F-Value P-Value

Covariates

EVALEXP 81.94 1 81.94 0.27 0.61

WEAK 881.04 1 881.04 2.86 0.05

STRONG 2487.62 1 2487.62 8.07 0.004

Main Effect

FORMAT 1486.97 1 1486.97 4.82 0.018

MODEL 7498.80 4 1874.70 6.08 0.0005

Key:

EVALEXP = participants’ months of internal control evaluation experience

WEAK = the number of internal control weaknesses identified

STRONG = the number of internal control strengths identified

FORMAT = internal control documentation format (questionnaire = 1; narrative = 0)

Appendix

Delphi Panel Results: Strengths

  1. Credit sales are approved by the cashier from an approved credit list.
  2. Monthly bank statements are reconciled promptly by the accounting department and filed by date.
  3. Separation of duties between mail clerk and A/R clerk.
  4. Separation of duties between A/R clerk and credit manager.
  5. Separation of duties between person making bank deposits (A/R clerk) and person reconciling cash (Cashier).
  6. Relief of inventory based on Sales Invoice.
  7. Remittance advice completed if not available.
  8. Mail clerk is supervised.
  9. Customer receives a validated copy of the sales invoice.
  10. Review function is performed by the Sales/Accounting department supervisor.
  11. A/R clerk reconciles cash received with cash sales invoices.
  12. Separation between credit authorization (Cashier) and person making the sale (Sales Clerk)
  13. Credit manager (Accounting department supervisor) reviews all checks for payments of past due accounts.
  14. Daily deposit of cash.
  15. Separation between inventory clerk and Accounts Receivable
  16. Cashier recaps Sales.

Appendix (continued)

Delphi Panel Results: Weaknesses

1. Checks and remittance advice are not separated/remittance advice created by the mail clerk.

2. The credit function should not be in the accounting department/Accounting department manager is also the credit manager.

3. Accounts receivable clerk is overworked and has incompatible duties.

4. No review of bank reconciliation.

5. No indication of aging A/R.

6. Batch totals for daily sales summary should be compared to bank deposit and ledger totals.

7. Checks are not stamped by the mail clerk when the mail is opened.

8. No pre-listing of checks is prepared.

9. No approval process for credit sales over a certain $ amount.

10. No master price list.

11. A shipping report should be generated and compared to the invoice.

12. No procedures to ensure that all invoices have been accounted for.

 

 

 

ENDNOTES