Jama article- QUALITY OF CARE Continues to dominate the health policy agenda. Originally engendered by the now multiple reports of the Institute of Medicine (IOM) on quality of care,1 in particular on patient safety,2 and given new impetus by ongoing reports concerning the variable effectiveness of care provided by hospitals and physicians, 3,4 the quality movement has expanding momentum. Perhaps most important, high-quality medical care has become a significant objective for US business, as motivated employers make the point that value purchasing should be as much a rule for medical care as it is for other areas of industry.

1. SPECIAL COMMUNICATION
The Role of Physician Specialty Board
Certification Status in the
Quality Movement
Troyen A. Brennan, MD, JD, MPH
The Institute of Medicine’s reports and discussions on quality of medical
Ralph I. Horwitz, MD
care have focused on a systems-based approach to quality improvement. Our
F. Daniel Duffy, MD
objective is to summarize evidence and theory about the role of a physi-
Christine K. Cassel, MD cian’s current board certification status in quality improvement. The first body
Leslie D. Goode, MHS of evidence includes the validity of board certification demonstrated by the
Rebecca S. Lipner, PhD testing process, the relationship of examination scores with other measures
of physician competence, and the relationship between certification status
Q
UALITY OF CARE CONTIN-
and clinical outcomes. The second body of evidence involves the adapta-
ues to dominate the health
policy agenda. Originally tion of error prevention theory to medical care. Patient safety is enhanced
engendered by the now when problem-solving uses readily accessed habits of behavior, the same
multiple reports of the Institute of Medi- behavior necessary to achieve board certification. The third body of evi-
cine (IOM) on quality of care,1 in par- dence, obtained through a Gallup poll, demonstrates that certification and
ticular on patient safety,2 and given new maintenance of certification are highly valued by the public. The majority of
impetus by ongoing reports concern- respondents thought it important for physicians to be reevaluated on their
ing the variable effectiveness of care
qualifications every few years and that physicians should do more to dem-
provided by hospitals and physi-
cians,3,4 the quality movement has ex- onstrate ongoing competence than is currently required by the profession.
panding momentum. Perhaps most im- We conclude that a physician’s current certification status should be among
portant, high-quality medical care has the evidence-based measures used in the quality movement.
become a significant objective for US JAMA. 2004;292:1038-1043 www.jama.com
business, as motivated employers make
the point that value purchasing should tion of effective and safe health care Remarkably quiet in this quality
be as much a rule for medical care as it and insisting that regulated entities movement is the physician. Indeed,
is for other areas of industry.5 use data about outcomes to improve many architects of the new initiatives
In the wake of the IOM’s advocacy, the care provided. The Leapfrog consider physicians to be impedi-
traditional regulators of quality have Group, an influential collaborative of ments to systematic efforts to improve
renewed their efforts, and they have large employers who have prepared quality. The IOM reports were in-
been joined by a series of new initia- specific criteria to ensure better qual- tended to go directly to the public, for
tives that are intended to hold hospi- ity of the care they purchase, and the fear that an appeal to professionals
tals publicly accountable for quality. National Quality Forum, a private/
For example, the Joint Commission public coalition that aims to sanction Author Affiliations: American Board of Internal Medi-
cine, Philadelphia, Pa (Drs Brennan, Horwitz, Duffy,
on Accreditation of Healthcare Orga- certain measures of quality, are both Cassel, Lipner, and Ms Goode); Brigham and Wom-
nizations and the Centers for Medi- examples of quality promotion that en’s Physicians Organization, Brigham & Women’s
Hospital, Boston, Mass (Dr Brennan); and School of
care & Medicaid Services quality did not exist 5 years ago.6 All regulat- Medicine, Case Western Reserve University, Cleve-
improvement organizations have ing entities are insisting on improved land, Ohio (Dr Horwitz).
Corresponding Author: Troyen A. Brennan, MD, JD,
retooled over the last 5 years, now measurement and implementation of MPH, Brigham & Women’s Hospital, 75 Francis St, Bos-
more explicitly expecting demonstra- mechanisms to improve quality. ton, MA 02115 (tabrennan@partners.org).
1038 JAMA, September 1, 2004—Vol 292, No. 9 (Reprinted) ©2004 American Medical Association. All rights reserved.

2. SPECIALTY BOARD CERTIFICATION AND THE QUALITY MOVEMENT
would fail to overcome physician in- tial IOM report, To Err Is Human.2 The intended result evolved—the nearly
ertia on the question of quality im- subject of this report was in part the epi- solitary focus on systems overshad-
provement.7 demic of medical errors and the inju- owed the important and complemen-
Perhaps more to the point, the tra- ries such errors cause. A notable and tary role of individual physician ac-
ditional physician approach to qual- frequently repeated headline from To countability.
ity, eg, certification, has received mini- Err Is Human reported that 44 000 to The second reason for the minimal
mal notice within the new quality 98000 persons die each year in US hos- inclusion of physician competence in
movement. While physician certifica- pitals as a result of preventable iatro- the quality movement is the percep-
tion is reported by many health plans genic injury. The press reaction was in- tion that limited reliable approaches ex-
and is a component of the National tense and created momentum that still ist to support measuring individual phy-
Committee for Quality Assurance sustains the movement. sician quality. For years, the great hope
Health Plan Employer Data and Infor- The follow-up IOM reports deep- for evidence-based quality measures, es-
mation Set (HEDIS) formula, mainte- ened discussions and understanding pecially related to effectiveness, was that
nance of certification is not routinely about strategies for enhancing the clinical outcome measures could be
considered or reported. Current certi- quality of patient care. A major contri- used to judge the quality of, and per-
fication status appears to be over- bution was the classification of 6 cen- haps to rank, individual physicians. Un-
looked or assumed in the catalogs of tral components of quality: patient- der this approach, the quality of care
measures compiled by the Leapfrog centeredness, safety, effectiveness, provided by physicians would be judged
Group, the National Quality Forum, the efficiency, timeliness, and equity. Most by how effectively their patients’ dis-
Joint Commission on Accreditation of evidence-based measures of quality eases were managed (eg, the rate of gly-
Healthcare Organizations, or the relate to the categories of patient- cosylated hemoglobin levels at goal for
Agency for Healthcare Research and centeredness and effectiveness,6 but diabetes or rate of cholesterol levels at
Quality initiative on evidence-based safety enhancements remained the goal for coronary disease).
quality measures. critical motivation of much of the re- Methodologists have long had con-
A possible part of the reason that cer- newed interest in quality. For ex- cerns about technical obstacles (eg,
tification status has been overlooked or ample, the number of lives that could sample size limitations) that exist when
assumed is based on the accurate per- be saved by reducing errors was the ini- evaluating the performance of indi-
ception that the majority of physi- tial justification for 3 early Leapfrog vidual physicians in their clinical prac-
cians in the United States are certified. Group measures: computerized pro- tices.11,12 There is general agreement
In 2002, more than 85% of licensed vider order entry, full-time intensiv- that, although a worthwhile goal, reli-
physicians held a valid certificate.8 ists in intensive care units, and con- able and valid clinical performance as-
However, this does not address the lack centration of procedures in high- sessment of individual physicians will
of attention to renewing or maintain- volume centers.5 require considerable research and de-
ing certification on the part of regula- To Err Is Human not only focused on velopment.
tors, health plans, and others. safety, but also called for continuous In the interim, to overcome the tech-
We discuss the role of the indi- quality improvement through change nical problems associated with small
vidual physician in the overall quality in systems of medical care. Modern in- numbers, the quality regulators adopted
framework and argue that the mini- dustrial quality improvement prin- approaches that aggregate physicians or
mal attention to the role of the indi- ciples eschew assigning individual providers at the group, health plan, or
vidual physician is a missed opportu- blame as a method for improving qual- hospital levels. In addition, improve-
nity and review data that suggest ity.9 This principle gained greater sig- ment experts have focused on evalua-
patients agree with us. We also out- nificance in light of the IOM’s strate- tions of structural elements in systems
line the prominent role that current and gic recognition that the key regulatory that are related to improved outcomes,
evolving versions of physician certifi- approach to medical injury has tradi- such as report cards indicating whether
cation and maintenance of certifica- tionally been malpractice litigation. a hospital has a computerized order entry
tion can play in advancing quality and Malpractice is founded on individual system and processes of care. These ini-
accountability. blame and is routinely criticized as a tiatives are welcome and hold promise
method of improving care or prevent- for improved care, but the unfortunate
WHERE ARE THE PHYSICIANS? ing injuries by physicians.10 To avoid corollary is that the traditional measure
There are 2 reasons that physicians, and the conundrum of malpractice and of individual physician quality, certifi-
the quality of individual physician care, blame and because significant data from cation status, has been taken for granted
have played a secondary role in the outside the medical profession sup- in the quality movement. This is espe-
quality movement. The first reason port the efficacy of a systems-based ap- cially unfortunate given new policies pro-
arises from the original impetus for the proach to quality improvement, the mulgated by the American Board of
current quality movement, eg, the ini- IOM report focused on systems. An un- Medical Specialties (ABMS) and indi-
©2004 American Medical Association. All rights reserved. (Reprinted) JAMA, September 1, 2004—Vol 292, No. 9 1039

3. SPECIALTY BOARD CERTIFICATION AND THE QUALITY MOVEMENT
vidual certifying boards to expand the tion of certificates.15,16 Realizing that sat- primarily, until recently, on initial cer-
requirements for maintaining certifica- isfactory performance on a single tification. The published evidence on
tion and put a time limit on certificates. examination does not guarantee that the value of certification and mainte-
physicians remain competent through nance of certification takes 3 forms: the
EVOLUTION OF CERTIFICATION their careers, the ABMS has taken on internal validity of the testing process
AND MAINTENANCE the challenge to insist that all member itself, the correlation of examination
OF CERTIFICATION boards’ maintenance of certification scores with other measures of physi-
The evolving requirements for certifi- programs include the 6 certification cian quality, and the correlation of cer-
cation and maintenance of certifica- competencies, organized into a 4-part tification status with practice out-
tion are spurred by many leaders in the framework, now referred to as “main- comes. We review each of these and
profession agreeing that physicians tenance of certification.”8 The ABMS suggest how the evidence of the value
must do more to demonstrate to the maintenance of certification initiative of certification is complemented by
public that they are skilled and knowl- calls for evidence of the following: (1) theories of error prevention.
edgeable. This momentum predates the professional standing, (2) lifelong learn- The first body of evidence concerns
IOM quality reports but is now given ing and periodic self-assessment, (3) the validity of the testing process. Typi-
further impetus by the general activ- cognitive expertise as demonstrated by cally, cognitive examinations are com-
ism surrounding quality. a secure examination, and (4) perfor- posed of questions developed by ex-
Historically, board certification has mance in practice. Each ABMS mem- perts in the discipline and selected to
depended on performance on a proc- ber board has agreed to design meth- fulfill a blueprint for the overall exami-
tored examination of knowledge. Grow- ods to meet these requirements by nation based on importance and fre-
ing from a perceived need to demon- instituting maintenance of certifica- quency with which problems are faced
strate quality and differentiate among tion programs that will be continuous in practice. Most examinations use pre-
specialties, the first specialty board, in nature and include periodic cogni- testing to assure their accuracy and, in
ophthalmology, was founded in 1917. tive examinations, as well as compo- some instances, certified practitioners
Other specialties followed, and in 1933 nents focused on clinical practice as- who are not associated with the board
they organized as a federation called the sessment and quality improvement. rate the relevance of each question to
Advisory Board of Medical Specialists Although each board can design its own clinical practice.18 All ABMS boards set
(renamed the American Board of Medi- methods for compliance with this man- standards for passing the secure exami-
cal Specialties [or ABMS] in 1970).13 date, an ABMS Oversight and Monitor- nations using widely accepted, cred-
Today the ABMS consists of 24 boards. ing Committee has been established to ible standard-setting methods.19,20 Con-
To achieve initial certification, each ensure adherence to the principles.17 tinuous monitoring of the standards set
board requires between 3 and 6 years Most boards believe that there is by the expert question-developers show
of training in an accredited training pro- more to be done before the ambitious them to be credible, valid, and repro-
gram and a passing score on a rigor- agenda set forth by the ABMS has been ducible over time, and different sets
ous cognitive examination. In addi- met. Nonetheless, all 24 boards have ac- of experts arrive at comparable judg-
tion, to assess clinical competence, cepted the challenge, indicating the ments.21-23
some boards require satisfactory pro- medical profession’s commitment to the The second body of evidence for the
gram director evaluations on 6 compe- highest quality care, and specifically to effectiveness of physician certification as
tencies (patient care, medical knowl- the principle that the certified physi- a measure of quality concerns the rela-
edge, practice-based learning and cian is continuously striving to better tionship of examination scores with
improvement, interpersonal and com- serve patients. Given this expanding other measures of physician compe-
munications skills, professionalism, and commitment, it is ironic that the no- tence. A valid measure must be able to
systems-based practice), while others tion of individual physician quality has demonstrate relationships with other cri-
require oral examinations, audits of been overlooked. Review of the evi- terion measures to be believable; groups
medical records, review of case logs, or dence and theory surrounding creden- that should do well on the examina-
observed performance on real or stan- tialing and quality suggests that the am- tion in fact do so. Certification exami-
dardized patients.14 bitious agenda of the ABMS should be nation results are correlated with the
The changing scope of medical in- embraced by the quality movement. type of medical school training (as a
formation, the increased concern of the group, US medical school graduates per-
public for the need to recredential phy- BOARD CERTIFICATION AS A form better than foreign medical school
sicians, and some evidence that knowl- MEASURE OF INDIVIDUAL graduates)24; the amount of formal train-
edge and skills of practicing physi- PHYSICIAN QUALITY ing (those with more training perform
cians decay over time motivated Over the last 30 years, the ABMS boards better on subspecialty examinations than
specialty boards to develop recertifica- and other colleagues have evaluated the those with less training)25; and super-
tion programs and to limit the dura- effectiveness of certification focusing visor assessment of clinical skills (phy-
1040 JAMA, September 1, 2004—Vol 292, No. 9 (Reprinted) ©2004 American Medical Association. All rights reserved.

4. SPECIALTY BOARD CERTIFICATION AND THE QUALITY MOVEMENT
sicians rated independently by their colleagues,32 which found provision of CERTIFICATION AND
training program directors as excellent preventive care services and a few out- GREATER PATIENT SAFETY?
trainees perform better on the certifica- comes (eg, lower mean glycosylated he- The theory of error prevention sug-
tion examinations than those less highly moglobin levels for diabetic patients) gests that certification may be more im-
rated).26,27 favoring board-certified physicians. In portant for the safety domain of qual-
Physicians specializing in an area (eg, addition, board-certified surgeons had ity than the currently available empirical
geriatrics or critical care medicine) per- lower peptic ulcer surgical mortality evidence suggests. The quality move-
form better on those portions of a re- rates, but rates did not differ from non- ment, especially the part focused on pa-
certifying examination compared with certified surgeons for stomach cancer tient safety, has relied as much on cog-
those who do not have such inter- surgery or abdominal aneurysm.33 In a nitive psychology concepts, guided as
ests.16 Also, a positive relationship ex- study of physicians disciplined by the much by theory and common sense, as
ists between recertification examina- state of California, Morrison and Wick- by evidence of outcomes.42
tion performance and patient volume ersham34 found that disciplinary ac- The safety domain of the quality
as well as complexity of patient prob- tion was negatively associated with spe- movement owes a great deal to the im-
lems reportedly seen in practice. 28 cialty board certification. portation by Leape et al43 of basic er-
Performance on an open-book, take- Literature published after 1999 also ror prevention theory into medical care
home self-assessment examination used shows mixed findings. In a series of stud- proposed by Reason. Reason’s most
in the American Board of Internal Medi- ies in Pennsylvania, certified cardiolo- accessible work differentiates rule-
cine (ABIM) maintenance of certifica- gists were shown to have lower in- based behavior (prone to lapses and
tion program shows that the scores are hospital mortality rates independent of slips) from knowledge-based behav-
as reproducible as a 60-item licensing volume of patients.35-37 A retrospective ior (prone to mistakes).44 These in-
or certifying examination and having study of patients in northern Illinois re- sights are built on years of cognitive
small but significant positive correla- vealed that board certification in sur- psychological research, which empha-
tions with length of training, initial cer- gery was associated with reduced size similar dichotomies, including the
tification examination scores, and the mortality for colon resection, but sub- skill-, rule-, and knowledge-based lev-
composition of the clinical practice.29 specialty certification in colorectal sur- els of cognition of Rasmussen and
Likewise, the patient and peer self- gery was not related to outcomes.38 Sil- Jensen45; the symptomatic and topo-
assessment measure is as reproduc- ber39 studied patients who underwent graphic rules of Rouse46; and Reason’s
ible as other survey measures of its kind surgical procedures in Pennsylvania and own sophisticated differentiation be-
and has small but significant positive found that the lack of board certifica- tween attentional and schematic modes
correlations with the internal medi- tion was related to higher mortality rates; of processing decisions.44 While not all
cine program director ratings of over- however, type of hospital was not con- of the cognitive psychology literature
all clinical performance and commu- trolled in the study. A study of family supports this paradigm, the interpre-
nication skills rendered nearly 10 years physicians in Quebec showed a posi- tation of Reason’s theory by Leape et al
previously.30 tive relationship, sustained over 4 to 7 has proven to be very intuitive to phy-
The third body of evidence regard- years out in practice, between certifica- sicians and policy makers.
ing certification as a measure of phy- tion examination scores and mammog- In each of these areas of psychologi-
sician quality attests to the relation- raphy screening, consultation rate, but cal investigation, theorists recognize a
ship between certification status and not continuity of care.40 A recent study complex interaction between problem-
various clinical outcomes; conclu- of physicians disciplined by the Medi- solving that relies on readily accessed
sions in this area are mixed. In a sys- cal Board of California showed that lack habits of behavior and problem-
tematic review of the literature on stud- of board certification was related to a solving that involves slower interroga-
ies published between 1966 and 1999,31 greater risk of disciplinary action (prac- tion and processing of a knowledge
only 5% of the studies used research tice suspension, public reprimand, pro- base. Error prevention depends on rec-
methods that were appropriate for as- bation, and license revocation).41 ognizing that different behaviors are
sessing the research question and, Although the evidence on clinical necessary to prevent mistakes or over-
among these, more than half support outcomes is mixed, it is nonetheless sights arising from these respective
a positive relationship between board promising that better outcomes are as- types of problem-solving.
certification status and clinical out- sociated with physician certification and Certification and maintenance of cer-
comes.31 Of the studies that did not maintenance of certification in many tification evaluate a physician’s evi-
demonstrate a positive association, the studies. Obviously, more research is dence of possessing the requisite hab-
majority showed no association be- needed to focus on the maintenance of its of practice (practice performance
tween certification and clinical out- certification process and to assess its assessment) and robust knowledge base
come measures. Examples from this re- value to the public and the profession (cognitive examination) needed to pre-
view include the work by Ramsey and as a measure in the quality movement. vent both types of errors. A physician
©2004 American Medical Association. All rights reserved. (Reprinted) JAMA, September 1, 2004—Vol 292, No. 9 1041

5. SPECIALTY BOARD CERTIFICATION AND THE QUALITY MOVEMENT
who performs well on a certification ex- in the quality movement, often be- ment and recent changes in require-
amination and who maintains certifica- cause of what and how information is ments by specialty boards, support cer-
tion by routine review of the medical lit- presented.47 The key question that pa- tification as a measure of quality.
erature presumably has demonstrated tients ask with regard to quality is, “How
ability to access a base of clinical knowl- do I find a good physician?”48 The an- CONCLUSION
edge and uses this same skill and knowl- swer often involves certification status. The ABMS continues to work on be-
edge when faced with a patient prob- To test our hypothesis that mea- half of its ambitious agenda to im-
lem. Common sense suggests that the sures of physician quality used in cer- prove physician quality through its
physician with a broad and readily ma- tification and maintenance of certifica- maintenance of certification program.
nipulated knowledge base will be more tion matter to patients, the ABIM Reasonable empirical evidence sug-
likely to arrive at the correct answer to commissioned the Gallup organization gests that certification and mainte-
a clinical question, although no empiri- to poll the general public about their nance of certification programs will im-
cal studies are available on this point. views on physician certification and prove quality, and more research is
The ABMS member boards’ mea- maintenance of certification. Among the under way. That evidence is sup-
sures of performance in practice (part major findings, the survey revealed that ported by the theory of error preven-
4 of the ABMS maintenance of certifi- certification and maintenance of certi- tion and even by common sense as-
cation framework) are intended to dem- fication are highly valued by the pub- sumptions about medical practice. Our
onstrate and improve the extent to lic, patients expect and would prefer that polling data suggest the public is con-
which a physician practices within es- physicians demonstrate skills that are vinced that there is a connection, no
tablished national guidelines. For ex- just beginning to be addressed by the doubt swayed by common sense.
ample, a person’s habits of behavior can ABMS requirements in their mainte- Maintenance of certification is essen-
be judged by overall compliance with nance of certification programs, and that tially self-regulation by the profession.
widely accepted guidelines: failure to physicians should be evaluated more fre- It is not intended to replace or sup-
prescribe ␤-blockers or aspirin after a quently than is currently required by any plant those efforts to improve quality
patient suffers myocardial infarction board (all require certificate renewal be- that are generated outside the profes-
may reflect poor habits of care, not a tween 6 and 10 years). Perhaps most sig- sion. There is every reason to believe that
knowledge deficit. These deficits in ex- nificant, respondents indicated that they regulation by the profession and other
ecuting known guidelines for care can would be likely to change their own be- organizations can be synergistic.
be ameliorated by incorporating re- havior to ensure that they are treated by Therefore, the answer to the ques-
minders in medical records. The ABIM, a certified physician. Most claimed they tion, “where are the physicians?” should
for example, provides practice improve- would change physicians if their cur- be that they are engaged in efforts to en-
ment modules for use in the mainte- rent physician or specialist failed to sure professional quality using meth-
nance of certification program that maintain certification, and when given ods that comport with much of the rest
stimulate awareness of intended prac- the choice between a board-certified of the quality movement and in con-
tice and provides suggestions for im- physician or a noncertified physician rec- junction with other organizations that
provement in office settings. In addi- ommended by a trusted friend or fam- are actively pursuing quality improve-
tion, peers and patients will likely have ily member, the majority reported that ment. Indeed, our professional com-
well-grounded observations about a they would choose the former (unpub- mitment to patients and each other de-
physician’s habits in practice. Physi- lished data, July 2003). mands nothing less.
cians report that feedback received from Based on evidence that consumers
peer and patient assessments is help- make limited use of quality mea- REFERENCES
ful.30 Thus, the criteria on which cer- sures,48 it was not surprising to find that
1. Committee on Quality of Health Care in America.
tification or maintenance of certifica- only a minority of respondents ever di- Crossing the Quality Chasm: A New Health System
tion are based will, at least in time, rectly researched or inquired about a for the 21st Century. Washington, DC: Institute of
Medicine; 2001.
increase the likelihood that certified physician’s credentials. Nonetheless, 2. Kohn KT, Corrigan JM, Donaldson, MS, eds. To Err
physicians provide recommended care, they intuitively and highly favor what Is Human: Building a Safer Health System. Washing-
ton, DC: Committee on Quality of Health Care in
leading to improved quality. the credential of certification repre- America, Institute of Medicine; 1999.
sents and have strong and consistent 3. McGlynn EA, Asch SM, Adams J, et al. Quality of
CERTIFICATION IN views about the extent to which phy- health care delivered to adults in the United States.
N Engl J Med. 2003;348:2635-2645.
THE PUBLIC’S EYE sicians should demonstrate ongoing 4. Fisher ES, Wennberg DE, Stukel TA, Gottlieb DJ,
Patients generally agree with these theo- competence. Is the public’s confi- Lucas FL, Pinder EL. The implication of regional varia-
tion in medicare spending, part 1: the content, qual-
retical and common sense insights into dence in certification misplaced? We ity, and accessibility of care. Ann Intern Med. 2003;
certification. Research suggests that pa- believe it is not, and that several de- 138:273-286.
5. Leapfrog Group for Patient Safety Web site. Avail-
tients pay very little attention to the scor- cades of empirical evidence, as well as able at: http://www.leapfroggroup.org/purchase
ecards and measures that predominate modern theories of safety improve- .htm. Accessed March 7, 2004.
1042 JAMA, September 1, 2004—Vol 292, No. 9 (Reprinted) ©2004 American Medical Association. All rights reserved.

6. SPECIALTY BOARD CERTIFICATION AND THE QUALITY MOVEMENT
6. Mello MM, Brennan TA. Regulation of quality of fication: what should the standards be? N Engl J Med. plined by a state medical board. JAMA. 1998;279:
care: the critical assessment. J Health Polit Policy Law. 1997;337:43-44. 1889-1893.
In press. 21. Shea JA, Reshetar RA, Dawson BD, Norcini JJ. Sen- 35. Norcini JJ, Kimball HR, Lipner RS. Certification and
7. Lamb RM, Studdert DM, Bohmer RMJ, Berwick DM, sitivity of the modified Angoff standard setting method specialization: do they matter in the outcome of acute
Brennan TA. Hospital disclosure practices: results of a to variations in item content. Int J Teach Learn Med. myocardial infarction? Acad Med. 2000;75:1193-
national survey. Health Aff (Millwood). 2003;22:73- 1994;6:288-292. 1198.
83. 22. Norcini JJ, Shea JA. The reproducibility of stan- 36. Norcini JJ, Lipner RS, Kimball HR. The certifica-
8. Horowitz SD, Miller SH, Miles PV. Board certifica- dards over groups and occasions. Appl Meas Educ. tion status of generalist physicians and the mortality
tion and physician quality. Med Educ. 2004;38:10- 1992;5:63-72. of their patients after acute myocardial infarction. Acad
11. 23. Norcini JJ, Lipner RS, Langdon LO, Strecker CA. Med. 2001;76:S21-S23.
9. Brennan TA, Berwick DM. New Rules: Regula- A comparison of three variations on a standard- 37. Norcini JJ, Lipner RS, Kimball HR. Certifying ex-
tion, Markets and the Quality of American Health setting method. J Educ Meas. 1987;24:56-64. amination performance and patient outcomes follow-
Care. San Francisco, Calif; Jossey-Bass: 1996. 24. Norcini JJ, Shea JA, Benson JA. Changes in medi- ing acute myocardial infarction. Med Educ. 2002;36:
10. Studdert DM, Mello MM, Brennan, TA. Medical cal knowledge of candidates for certification in inter- 853-859.
malpractice. N Engl J Med. 2004;350:283-292. nal medicine. Ann Intern Med. 1991;114:33-35. 38. Prystowsky JB. Patient outcomes for segmental
11. Hofer TP, Hayward RA, Greenfield S, Wagner EH, 25. Norcini JJ, Shea JA, Langdon LO, Hudson LD. First colon resection according to surgeon’s training, cer-
Kaplan SH, Manning WG. The unreliability of indi- American Board of Medicine critical care examina- tification, and experience. Surgery. 2002;132:663-
vidual physician “report cards” for assessing the costs tion: process and results. Crit Care Med. 1989;17: 670.
of quality of care of a chronic disease. JAMA. 1999; 695-698. 39. Silber JH. Anesthesiologist board certification and
281:2098-2105. 26. Norcini JJ, Grosso LJ, Shea JA, Webster GD. The patient outcomes. Anesthesiology. 2002;96:1044-
12. Landon BE, Normand ST, Blumenthal D, Daley J. relationship between features for residency training 1052.
Physician clinical perfomance assessment: prospects and ABIM certifying examination performance. J Gen 40. Tamblyn R, Abrahamowicz M, Dauphinee WD,
and barriers. JAMA. 2003;290:1183-1189. Intern Med. 1987;2:330-336. et al. Association between licensure examination scores
13. American Board of Medical Specialties. 2003 27. Norcini JJ, Maihoff NA, Day SC, Benson JA. Trends and practice in primary care. JAMA. 2002;288:3019-
Annual Report and Reference Handbook. Evanston, in medical knowledge is assessed by the certifying ex- 3026.
Ill: American Board of Medical Specialties; March amination of internal medicine. JAMA. 1989;262: 41. Kohatsu ND, Gould D, Ross LK, Fox PJ. Charac-
2003. 2402-2404. teristics associated with physician discipline. Arch In-
14. Lynch DC, Swing SR, Horowitz SD, Holt K, Messer 28. Norcini JJ, Lipner RS. The relationship between tern Med. 2004;164:653-658.
JV. Assessing practice-based learning and improve- the nature of practice and performance on a cogni- 42. Leape LL, Berwick DF, Bates DW. What practices
ment. Teach Learn Med. 2004;16:85-92. tive examination. Acad Med. 2000;75:S68-S70. will most improve safety? evidence-based medicine
15. Glassock, RJ, Benson JA, Copeland RB, et al. Time- 29. Norcini JJ, Lipner R, Downing SM. How mean- meets patient safety. JAMA. 2002;288:501-507.
limited certification and recertification: the program ingful are scores on a take-home recertification ex- 43. Leape LL, Lawthers AG, Brennan TA, Johnson WG.
of the American Board of Internal Medicine. Ann In- amination? Acad Med. 1996;71(10 suppl):S71-S73. Preventing medical injury. QRB Qual Rev Bull. 1993;
tern Med. 1991;114:59-62. 30. Lipner RS, Linda LB, Leas BF, Fortna GS. The value 19:144-149.
16. Norcini JJ, Lipner RS, Benson Jr JA, Webster GD. of patient and peer ratings in recertification. Acad Med. 44. Reason J. Human Error. Cambridge, England: Cam-
An analysis of the knowledge base of practicing in- 2002;77:S64-S66. bridge University Press; 1990.
ternists as measured by the 1980 recertification ex- 31. Sharp LK, Bashook PG, Lipsky MS, Horowitz SD, 45. Rasmussen J, Jensen A. Mental procedures in real-
amination. Ann Intern Med. 1985;102:385-389. Miller SH. Specialty board certification and clinical out- life tasks: a case study of electronic troubleshooting.
17. American Board of Medical Specialties. Annual Re- comes: the missing link. Acad Med. 2002;77:534- Ergonomics. 1974;17:293-307.
port and Reference Handbook. Evanston, Ill: Ameri- 542. 46. Rouse WB. Models of human problem solving:
can Board of Medical Specialties; 2002. 32. Ramsey PG, Carline JD, Inui TS, Larson EB, Lo- detection, diagnosis, and compensation for system fail-
18. Norcini JJ, Day SC, Popp RL, et al. The relevance gerfo JP, Weinrich MD. Predictive validity of certifi- ure. Automatica. 1983;19:413-425.
to clinical practice of the certifying examination in in- cation by the American Board of Internal Medicine. 47. Epstein AM. Rolling down the runway: the chal-
ternal medicine. J Gen Intern Med. 1993;8:82-85. Ann Intern Med. 1989;110:719-726. lenges ahead for quality report cards. JAMA. 1998;
19. Cizek GJ. Setting Performance Standards: Con- 33. Kelly JV, Hellinger FJ. Physician and hospital fac- 279:1691-1696.
cepts, Methods, and Perspectives. Newark, NJ: tors associated with mortality of surgical patients. Med 48. Hibbard JH, Peters E, Slovic P, Finucane ML, Tu-
Lawrence Erlbaum Associates; 2001. Care. 1986;24:785-800. sler M. Making healthcare easier to use. Jt Comm J
20. Kassirer JP. The new surrogates for board certi- 34. Morrison J, Wickersham MS. Physicians disci- Qual Improv. 2001;27:591-604.
©2004 American Medical Association. All rights reserved. (Reprinted) JAMA, September 1, 2004—Vol 292, No. 9 1043