Low Quality Stats from Center for Audit Quality

Much ado has been made lately of the insubstantial contribution business school professors have made to practice through research.  Accounting research of the last forty years, being no exception, was blasted in a commentary in the Chronicle for Higher Education for not addressing questions relevant to accounting professionals.  The authors, academic accountants themselves, blame universities for not providing more robust incentive systems to encourage a broader range of topics and methodologies. 

I want to add another reason:  practitioner organizations barely give lip service to academic research, because their members actually don't want to know what academics have to say.  That's because the truth can be so darn inconvenient (apologies to A. G.).  Just recently, the Financial Accounting Foundation says it doesn't believe it is necessary anymore to have academic representation on the FASB (see earlier post); and now a new report by the Center for Audit Quality (CAQ) presents the results of their "commissioned" survey of audit committee members.  "Research," by any standard academic, it ain't.

I have two problems with the survey.  First, the survey methods are so crude, that if an academic had tried to present this work to peers as serious research, either his sanity or intelligence would be called into question.  Second, the presentation of the "key findings," such as they are, have been skewed to fit the public relations agenda of the accounting profession:  post-Enron rehabilitation of auditors' reputations, and entrenchment of the revenue-producing aspects of SOX.  Other than these two criticisms, the report is just fine.

Let's start with the methodology:

Non-random sample -- CAQ states, without explanation, that it was "not feasible from a cost or time standpoint" to take a random sample.  How ironic is it that an organization sponsored by auditors would forgo random sampling, the indispensable basis of so many auditing standards, procedures and concepts?  CAQ blithely asks us to assume this is a minor detail:

"With a pure probability sample of this size, one could say with ninety-five percent confidence that the overall results would have a sampling error of +/- 6.2 percentage points.  As this survey is not a pure probability sample, theoretically no sampling error can be calculated."

Setting aside that calculation of a sampling error is not "theoretically" possible (read simply as "not possible in any way, shape or form"), +/- 6.2 percentage points is actually a pretty large number that can affect one's reading of the results.  Maybe that's why it is not mentioned again in the report.

Sample size -- Let's agree that the population of interest, audit committee members of public companies, number very approximately 10,000.  Only 253 of them volunteered in response to invitations to participate.  Given the low participation level, it is highly likely for non-respondents to have significantly different opinions than the self-selected volunteers.  If anything was done to test that proposition (yes, academics routinely apply techniques for doing so), CAQ did not report it.

Sample demographics -- CAQ gathered very little information about the respondents.  We know virtually nothing about their financial or accounting expertise.  For example, how many are "designated financial experts"?  How many are former auditors? 

We do know, however, that 44% took their first position on an audit committee post SOX.  As some of the key survey questions ask their opinions about the pre- and post- SOX environment, their opinions may be different, or even less reliable, than the respondents with longer tenure on audit committees.

Wording of the questions -- Let's say you joined your first audit committee in 2007.  How would you respond to this key question:

"Based on your experience as an audit committee member [italics supplied], how would you rate the overall quality of audits of publicly traded companies being conducted today?

First, I don't know if a respondent could reliably separate their audit committee member experience from their prior experience.  Second, given the variability in length of tenure of audit committee members -- some pre-SOX, and other not -- it appears as if two populations are being combined into one. 

The next survey question is:

"And over the past several years, would you say that the overall quality of audits of publicly traded companies has...?" 

Assuming that the "based on your experience" language carries forward, how is the newbie audit committee member supposed to answer that?  Dunno.  Remember, these folks are 44% of the respondents!

Integrity and Objectivity Red Flag -- Neither the organization that purportedly conducted the survey was disclosed in the survey report, nor the scope of their responsibilities.  Imagine an audit report not identifying the auditor; this is not much different.  Was the contractor a well-respected company that didn't want to be associated with the end product?  Was it an organization with other ties to CAQ?  Just asking.    

Now, on to the reported "key findings"

Current state of audit quality -- The audit committee now appoints the auditor and reviews their work much more closely than in the pre-SOX era.  What board member would be willing to admit, even anonymously, that their choice of auditor was a mistake?  So, it's not surprising that CAQ finds that 95% of respondents rate their auditors' work as "good" or better.  As a "key finding", it's a big fizzle. 

The other side of the coin is more provocative: a client paying top dollar for audit services should not be satisfied with work that was only "good" or "fair." That's how 22% of respondents saw it.  Moreover, 17% said that the situation had not changed or worsened since the passage of SOX. 

Risk of fraudulent financial statements -- CAQ blithely reports that 87% of respondents believe the risk of releasing financial statements that are materially inaccurate due to fraud is not very high.  That's a good thing?  I don't think so, especially when you consider that these are audited financial statements!  Has SOX affected the potential for fraudulent misstatements?  The CAQ would like you to think so, because auditors sure have made a lot of money out of ICFR audits.  Yet, unstated is the fact that 40% of respondents do not believe that the risk of fraudulent misstatement has decreased with SOX. 

Complexity of financial statements -- CAQ would like you to believe that audit committee members think that financial statements are too complicated.  But, the questions don't even begin to address the nature of the complexity, or the sophistication of the respondents as regards financial statements.  And if you still doubt that a self-serving agenda is driving the report, here's the smoking cannon: buried in the back pages (question 17C) is the undiscussed finding that respondents clearly do not feel that GAAP is too complex.  Yet financial statements are too complicated.  Figure that one out!

If you are one of those who believe that the CAQ was created by the powers that be to function as shill for audit firms, then this survey report could be your Exhibit A.  It is also Exhibit A for my point that practitioners with self-serving agendas have good reason to be threatened by the contribution of academics.  But, if truth is the goal, then the participation of academics, trained and paid to apply discipline and rigor to a question, is an asset. 

Comments

[Independent Accountant]

The CAQ is, as I understand it, a successor to the old SECPS [SEC Practice Section] of the AICPA. The AICPA had SECPS employees with nothing to do once the PCAOB was formed. Voila, more makework, create the CAQ. I don't think it was created to shill for the Big Four, just to give the SECPS's "former" employees a place to hang each's hat. I expect no more than we got while they were SECPS employees. Look at the names. Are any of them familar to you?

[Edith Orenstein, FEI]

Tom, you're right, the CAQ report does not identify who conducted the survey, I see in their press release http://www.thecaq.com/newsroom/release_03182008.htm they say it was done by The Glover Park Group, which - according to Glover Park's website http://www.gloverparkgroup.com also does PR work for CAQ.