Pro Publica and Rating Surgeons: Garbage IN equals Garbage OUT.

Recently, the independent investigative journalism research group known as Pro Publica has received a great deal of attention for a newly published project for rating surgeons. This particular project has been met with a great deal of scrutiny by the medical community and has stirred significant debate since it was released in early July 2015. While providing patients with an objective metric by which they can evaluate physicians before deciding where and when to receive care is a lofty goal, Pro Publica’s attempt has fallen significantly short—and if anything—has created more confusion and frustration among both doctors and patients.  Several of my colleagues have written blogs and commented publicly about this rating system–Here is my take.

Helping the public make informed decisions about healthcare is an essential part of the process of improving the quality of care throughout the US—However, any information provided to the public must be easily digestible, RELIABLE and ACCURATE and supported by solid, well-conducted research. The methodology by which any data is assembled and any particular meaningful conclusions are reached is critical in determining whether or not the conclusions are of ANY use to the intended audience.

What Exactly is Pro Publica?

According to their website, Pro Publica is an “an independent, non-profit newsroom that produces investigative journalism in the public interest”. It is labeled as a non-profit organization and is funded by philanthropic donations as well as advertising dollars. The organization is staffed by nearly 45 investigative journalists and is located in Manhattan. Their mission statement is published as follows:

“To expose abuses of power and betrayals of the public trust by government, business, and other institutions, using the moral force of investigative journalism to spur reform through the sustained spotlighting of wrongdoing.”

What exactly was the Surgeon Rating project?

In the surgeon rating project, Pro Publica sought to produce a mechanism by which healthcare consumers could examine outcomes data prior to choosing a medical professional. In the project, journalists collected data from Medicare billing databases—these data were limited to a few selected low risk procedures and only included patients admitted to hospitals for these procedures. The analysis did not consider outpatient surgical center locations, nor did it evaluate data from patients admitted from the Emergency Department.   Doctors in particular specialties such as orthopedics, neurosurgery, general surgery and urology from all over the US were rated based on two outcomes—death and readmission—and all data was mined from a Medicare billing database. The journalists clearly state their methodology on their website. They examined records from in patient hospital stays from 2009-13 and looked at complications from what they defined as low risk surgeries. While Pro Publica does insist that they consulted with “experts” in each area evaluated, they do not mention if they had any guidance from statisticians, and other experts in the design of medical research investigations.

In their project, Pro Publica journalists collected information from nearly 3600 hospitals involving almost 17,000 surgeons. The outcomes of almost 64,000 Medicare patients were evaluated. A searchable “surgeon scorecard” was created online and has been available to the public since early July.

What DOES make a good surgeon?

The most important quality of a successful surgeon is judgement. Ironically, many outcomes are determined by the surgeon before operating—choosing an appropriate surgical candidate is often the most difficult decision a surgeon will make. This database does not take any of the pre surgery decision making into consideration. A good surgeon certainly has inherent physical talent and dexterity—most great surgeons have exceptional skill with a scalpel. In addition, patient volume is a huge determinant in surgical outcome. A surgeon with a large volume practice is much more adept at performing a routine procedure and is also much more skilled at handling intra operative surprises. Moreover, surgeons who have been well trained and well educated at academic institutions during their residencies and fellowships are more likely to practice evidence-based medicine and adhere to “best practices” and clinical guidelines.

Why is the Pro Publica project fundamentally flawed?

The project, while well intended, is of no practical use. The methodology is flawed at the beginning. Mining Medicare databases assumes that the data entered is in fact accurate. There are coding errors and data entry errors that occur every single day in hospitals and at CMS. The project does not take into consideration surgeons who accept patients that others would turn away—due to high risk, etc. In addition, the data analysis only takes into consideration inpatient procedures. Many of these low risk procedures are also performed in outpatient surgical centers—including these numbers would certainly bolster a surgeon’s “scorecard”. The research presented by Pro Publica was NOT peer reviewed. Any reputable medical journal requires that all investigations are reviewed by a minimum of two or three independent and anonymous EXPERTS in the discipline of interest. These experts are charged with evaluating not only the quality of the research and its findings but also the methods that were utilized in the study. Other than mentioning a group of physician experts that were consulted to identify complications, there is no mention of a peer review process and no mention of a methodology review.

Epilogue: Pending Impact of the Pro Publica Surgeon rating project?

 Ultimately time will tell. This database unfairly evaluates surgeons. While not all physicians are created equal, the methodology involved in arriving at the Pro Publica ratings has produced unreliable and inaccurate data. I fear that rating projects of this sort—that are not based on good, sound science—will result in many physicians refusing to treat sicker, higher risk patients. Ironically, it is the BEST surgeons that must take on the toughest cases. I, along with many of my colleagues, applaud any effort that can potentially empower patients to make informed decisions about their healthcare. However, I cannot endorse faulty science. If you put garbage data into a statistical program, you simply get well analyzed garbage out. As British economist Ronald Coase once said….”If you torture the data long enough, it will confess to anything…”

Unknown

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s