"Thinking Fast and Slow" about Auditing

My writings in this blog have focused on the absence of linkages between financial reporting and the economic concepts like wealth and value creation/destruction. But, accounting is more than applied economics. Individuals acquire accounting information before making decisions; and heuristics and biases affect the quality of those decisions. I spent much of my academic career thinking about these concepts, and Daniel Kahneman's new book, Thinking, Fast and Slow, brings me back to those halcyon days.

In that vein, I would like share with you this autobiographical vignette from Chapter 7, entitled "A Machine for Jumping to Conclusions":

Early in my career as a professor, I graded students' essay exams in the conventional way. I would pick up one test booklet at it time and read all the students' essays in immediate succession, grading them as I went. I would then compute the total and go on to the next student. I eventually noticed that my evaluations of the essays in each booklet were strikingly homogeneous. I began to suspect that my grading exhibited a halo effect [defined as the tendency to like (or dislike) everything about a person – including things you have not observed], and that the first question I scored had a disproportionate effect on the overall grade. The mechanism was simple: if I had given a high score to the first essay, I gave the student the benefit of the doubt whenever I encountered a vague or ambiguous statement later on. … I told the students that the two essays had equal weight, but that was not true: the first one had a much greater impact on the final grade than the second. This was unacceptable.

I adopted a new procedure. Instead of reading the booklets in sequence, I read and scored all the students' answers to the first question, then went on to the next one. I made sure to write all the scores on the inside back page of the booklet so that I would not be biased (even unconsciously) when I read the second essay. Soon after switching to the new method, I made a disconcerting observation: my confidence in my grading was now much lower than it had been. The reason was that I frequently experienced a discomfort that was new to me. When I was disappointed with a student's second essay and went into the back page of the booklet to enter a poor grade, I occasionally discovered that I had given a top grade to the same student's first essay. I also noted that I was tempted to reduce the discrepancy by changing the grade that I had not yet written down, and found it hard to follow the simple rule of never yielding to that temptation. …

… The consistency I had enjoyed earlier was spurious; it produced a feeling of cognitive ease and … [I] … was happy to lazily accept the final grade. By allowing myself to be strongly influenced by the first question in evaluating subsequent ones, I spared myself decisions of finding the same student doing very well on some questions and badly on others. The uncomfortable inconsistency that was revealed when I switched to the new procedure was real: it reflected both the inadequacy of any single question as a measure of what the student is and the unreliability of my own grading.

The procedure I adopted to tame the halo effect conforms to a general principle: decorrelate error! [pp. 83-84; italics supplied by me]

The lessons that any student or professor can take from the story are obvious and compelling; and grading happens in auditing, too – a lot.

Auditors Have Feelings, Too

The standards that form the foundation for the practice of auditing were laid down long before the study of heuristics and biases in decision making gained traction among psychologists, and eventually economists. But, while a revolution in psychology and behavioral economics has been taking place, the key driver of audit reliability shifted from objective verification to subjective "grading" of judgments made by management when preparing financial statements.

For example, auditor term limits can be regarded by behaviorists as an effort to mitigate serial errors as the auditor accumulates too many positive vibes from the client. The resistance to any any form of auditor firm term limits by auditing firms is effectively either: a denial that a confirmatory bias – as if auditors were inoculated from this human fallability; or perhaps like Kahneman's grading story, a statement that auditors have identified the problem and figured out a solution – much as Kahneman dealt with his exam grading issues.

But, neither argument seems valid. Delving a little deeper into auditing procedures, auditors 'grade' numerous judgments made by management. Yet, there are no explicit policies that I am aware of that have been instituted for decorrelating those grades. According to the confirmatory bias, order matters a great deal. Maybe, auditors should be making and documenting their judgments and estimates independently of management (and even without looking at prior years' work papers), before deciding whether or not a judgment by management "appears reasonable."

Or, perhaps, auditors will maintain that supervisory procedures within the firm ensure that judgments are unbiased and independent. But, that doesn't appear to be the case. Audit managers 'grade' numerous tasks performed by audit staff members, and it seems to me that little or nothing is much different about the way it is done than the way Kahneman performed his grading – until he realized something was amiss.

In the current environment, can it be said that standards governing "independence," "technical competence" and "sufficient evidence" are enough to produce reliable audits? The weight of evidence brought to bear by psychologists suggest that auditing judgments may not be reliable without instituting explicit procedures for decorrelating errors in auditors' judgments.

Kahneman's story is memorable for his self insight in recognizing a halo effect was influencing his own judgments, the creativity to devise a successful strategy for mitigating bias, and the discipline to implement a costly strategy. If only the accounting profession would recognize its duty to the public to strive to act in a similar fashion, we would all be wealthier and wiser.

Comments

[Scott Green]

Another interesting post, Tom. This is a topic that can fuel a life-time of research and study, but I leave a couple remarks.

Besides through auditor rotation, auditor objectivity is supported by clear auditing standards and accounting standards and checklists and other documentation that ensure consistent treatment of similar situations.

Even referring to prior year workpapers helps an auditor maintain consistency, although prior year workpapers are also well known as a trap for the unwary and inexperienced.

[James Ulvog]

I'm a sole-practitioner CPA who provides audits and reviews. I don't think that mandatory auditor rotation is a good idea. However, the author's experience in grading papers is a very unsettling illustration of the halo effect.

Thanks for the post. I will have to ponder this some more.