Summary: The overall CPHQ exam pass rate among US candidates in 2012 was published in the NAHQ 2012 Annual Report, which again suffered from the DRIP (Data Rich, Information Poor) syndrome.
As a consultant, I deal with reports all the time. If there’s one thing I’ve learned, it’s that the devil is in what they don’t tell you. (Much like dealing with salesmen.)
The latest National Association for Healthcare Quality (NAHQ) 2012 Annual Report [PDF, 2 MB], which I recently found time to read, raised alarm bells in my head… again.
A couple of years ago, I had a similar experience with the NAHQ 2010 Annual Report when looking at its graph that presented the CPHQ exam pass rates in 2010.
You might have thought that a quality organization like the NAHQ would pull up their socks when presenting data. You’d be disappointed if you read the 2012 Annual Report.
First things first. Let’s recap why people would want to spend their precious time reading a report in the first place.
- In the case of the NAHQ annual report, I imagine most people would be seeking information that they had not known before they opened the document, i.e. fresh information.
- Further, a chart, graph, or table should tell the reader the whole story without him (or her) having to refer to accompanying text.
- The information provided in a report, especially an annual report, should address all the essential questions. For example, if a company reports a profit of $1 million, it still has to show its total revenue, sources of the revenue and their amounts, total expenditure, and a breakdown of how the money was expensed (among other things). If there were any one-off items that affected the bottomline, these should be clearly stated (not hidden in marketese or charts). In this example, the company could have made a $1 million profit on a $10 million revenue or a $150 million revenue—these two scenarios have vastly different implications.
- There should also be minimal blah-blah text (that few people read anyway).
The image below is a screenshot of Pages 20 and 21 of the NAHQ 2012 Annual Report—click on the image to enlarge it.
There is minimal text (good) which provides very little content (bad).
2012 CPHQ Exam Pass Rate
The three graphs fill one and a half pages—that’s plenty of real estate in a 35-page report, especially when the rest of the report is deficient of real content! Check out the entire report for yourself here [PDF, 2 MB].
Exhibit 1: Total CPHQs (Page 20)
The number of active CPHQs has hardly grown between 2011 and 2012. To be precise, there was an increase of only 75 CPHQs from 2011 to 2012. This is on the back of 984(!) new CPHQs in 2012. In other words, 909(!) CPHQs did not renew their certification in 2012. Why? Furthermore, the numbers of CPHQs reported each year on this graph include both US and international CPHQs—more on this issue below.
There was a small jump in the number of CPHQs between 2010 and 2011 (265 CPHQs, or 3.7%), which remains poorly explained. I have my own hypothesis on the latter issue but I’d prefer to test it thoroughly before I tell.
Note that the scale on the vertical axis starts from 6900, and not zero (at it should). In my commentary of the NAHQ 2011 Annual Report, I provided links to two references that explain why this is not acceptable when presenting data:
- Ezra Klein’s “Lies, damn lies, and the ‘Y’ axis”; and
- Page 2 of this document entitled Misleading Graphs and Statistics.
In case you missed it, Exhibit 1 (the graph on Total CPHQs on Page 20 of the 2012 Annual Report) also solves the problem I had with the NAHQ 2011 Annual Report. Last year, I highlighted the discrepancies in the numbers being reported in the following graph (Page 17 of the NAHQ 2011 Annual Report):
Last year, I wrote:
Bottom line: the numbers do not add up.
Your next question is probably, “Where is the error?”
The simple answer is “I don’t know.” But here are some possibilities:
- The total number of CPHQs in 2011 is not 7810 as stated in the chart. This seems the most likely since the numbers have been flat for several years. By this, I do not mean to imply there was no growth in CPHQs. In fact, based on the demand for my CPHQ exam preparation services, as well as the strong growth in sales of the Teh & Associates online CPHQ exam preparation products, I would be extremely surprised if there was no increase in CPHQ numbers in 2011. I also note from comparing the 2010 and 2011 Annual Reports, “expenditure” on certification activities were more than $110,000 in 2010 compared to 2011. Increase in marketing in 2010 may be part of the reason for an increase in CPHQs in 2011.
- The explanation for the 10% growth in total CPHQs between 2010 and 2011 is incomplete.
- The explanation for the 10% growth in total CPHQs between 2010 and 2011 is inaccurate.
- All of the above.
As it turns out, this year’s report vindicates my suspicions about something being amiss.
The number of CPHQs in 2011 (reported in the 2011 Annual Report) was grossly exaggerated! The NAHQ reported 7810 CPHQs in 2011 but corrected this figure to a more realistic 7364 in their 2012 Annual Report. (Like how the NAHQ quietly made that correction? Especially after boasting about the “10% growth rate”?)
Exhibit 2: CPHQ Candidates (Page 21)
There was a large increase in the number of CPHQ candidates from 2011 to 2012. This could, at least partially, be explained by the NAHQ/HQCC informing (scaring?) everyone with the new CPHQ exam content outline that came into effect in the second quarter of 2013, and people reacting by rushing to take the exam before the transition date.
Again, the numbers on this graph include both US and international candidates—read below to see why this is a problem.
Questions you might ask are:
“What was the total number of US candidates in 2012?”
“What was the total number of international candidates in 2012?”
The graph doesn’t tell us this information, though it should. Why?
Using simple math, I worked out that:
- There were 1074 US CPHQ candidates in 2012, and
- There were 504 international CPHQ candidates in 2012.
I could be out by up to 3 either way, but certainly not by much.
My calculations were based on the following data:
- There was a total of 1578 CPHQ candidates in 2012 (Exhibit 2);
- There were 809 US candidates and 175 candidates who passed the CPHQ exam in 2012 (based on the NAHQ’s list of all 2012 new CPHQ certificants); and
- The overall pass rate among US candidates in 2012 was 75.3% (Exhibit 3).
(I welcome you to do the math yourself.)
Now, do you see why the NAHQ chose not to show the breakdown of US and international candidates in this graph? (By the way, in the reports of recent years, a breakdown of US and international candidates was provided in some form or other.)
In the year before the NAHQ/HQCC amended the CPHQ exam content outline to include US-specific competencies (i.e. 2012), approximately one-third of candidates (504/1578) were based outside the US. CPHQ exam fees (and other related products) represent a large proportion of revenue received by the NAHQ. By amending the CPHQ exam content outline to make it US-specific this year (or claiming to do so… more on that in a separate article), they risk alienating about a third of CPHQ candidates (by the 2012 numbers). How smart was that decision? It’s downright embarrassing!
Exhibit 3: CPHQ Exam Pass Rate (U.S.)
Pass Rate Among US Candidates
Why report only the CPHQ exam pass rate for US candidates? Especially when one-third of your candidate pool is international?
Well, one possibility is that the NAHQ chose to put forward the best pass rate that it could put down on paper (75% does look pretty good). But if that were the case, it’s deceptive.
Why include international CPHQs and international CPHQ candidates in the first two graphs but not the third? Maybe including the international guys in the first two graphs makes the numbers look better but including the international candidates in the third graph gives an overall pass rate of only 62.4%.
Let’s be clear that the quoted 75.3% pass rate applies to all 1074 US candidates in 2012. We know that not all persons who take the exam for the first time will pass. Some candidates who don’t make it on their first attempt, will resit later, maybe three or four months later. (You’re only allowed one attempt within a 90-day period.) Therefore, there would have been a proportion of persons who sat the exam twice or even three times in 2012. The figure of 75.3% includes all candidates, irrespective of their number of attempts. The pass rate among first-timers is presumably lower than that among persons who attempt the exam more than once.
We estimate that the actual pass rate among US candidates sitting the exam for the first time is, say, 65–70%, and perhaps 80–85% among persons who take the exam two or more times in the same year.
Pass Rate Among International Candidates
The overall pass rate among international candidates in 2012 was a dismal 34.7%, i.e. 175/504. This figure, again, includes those who sat the exam multiple times. However, international candidates do not resit the exam as frequently as their US counterparts, hence the elevating effect of candidates attempting the exam more than once is less, compared with US candidates.
A rough estimate of the pass rate among international candidates taking the exam for the first time is about 30%, and that among those who take the exam more than once is about 45%.
CPHQ Exam Pass Rate Among ALL Candidates in 2012
As stated above, the overall pass rate—both US and international candidates included—is 62.4%.
Pages 20 and 21 of the NAHQ 2012 Annual Report leave much to be desired. However, there is enough data in there to make some inferences, when used with other information published by the NAHQ, e.g. the list of all new CPHQ certificants in 2012. But we really shouldn’t have to do all this legwork when the information that people want to know—and expect in the annual report —was readily available to the folks who prepared the report. Looking past the pretty images and average marketing copy, the graphs offered a juicy opportunity for the NAHQ to communicate data/information in a succinct and interesting way. Alas, the NAHQ failed on this count.