Tag Archives: journalism

Pro Publica and Rating Surgeons: Garbage IN equals Garbage OUT.

Recently, the independent investigative journalism research group known as Pro Publica has received a great deal of attention for a newly published project for rating surgeons. This particular project has been met with a great deal of scrutiny by the medical community and has stirred significant debate since it was released in early July 2015. While providing patients with an objective metric by which they can evaluate physicians before deciding where and when to receive care is a lofty goal, Pro Publica’s attempt has fallen significantly short—and if anything—has created more confusion and frustration among both doctors and patients.  Several of my colleagues have written blogs and commented publicly about this rating system–Here is my take.

Helping the public make informed decisions about healthcare is an essential part of the process of improving the quality of care throughout the US—However, any information provided to the public must be easily digestible, RELIABLE and ACCURATE and supported by solid, well-conducted research. The methodology by which any data is assembled and any particular meaningful conclusions are reached is critical in determining whether or not the conclusions are of ANY use to the intended audience.

What Exactly is Pro Publica?

According to their website, Pro Publica is an “an independent, non-profit newsroom that produces investigative journalism in the public interest”. It is labeled as a non-profit organization and is funded by philanthropic donations as well as advertising dollars. The organization is staffed by nearly 45 investigative journalists and is located in Manhattan. Their mission statement is published as follows:

“To expose abuses of power and betrayals of the public trust by government, business, and other institutions, using the moral force of investigative journalism to spur reform through the sustained spotlighting of wrongdoing.”

What exactly was the Surgeon Rating project?

In the surgeon rating project, Pro Publica sought to produce a mechanism by which healthcare consumers could examine outcomes data prior to choosing a medical professional. In the project, journalists collected data from Medicare billing databases—these data were limited to a few selected low risk procedures and only included patients admitted to hospitals for these procedures. The analysis did not consider outpatient surgical center locations, nor did it evaluate data from patients admitted from the Emergency Department.   Doctors in particular specialties such as orthopedics, neurosurgery, general surgery and urology from all over the US were rated based on two outcomes—death and readmission—and all data was mined from a Medicare billing database. The journalists clearly state their methodology on their website. They examined records from in patient hospital stays from 2009-13 and looked at complications from what they defined as low risk surgeries. While Pro Publica does insist that they consulted with “experts” in each area evaluated, they do not mention if they had any guidance from statisticians, and other experts in the design of medical research investigations.

In their project, Pro Publica journalists collected information from nearly 3600 hospitals involving almost 17,000 surgeons. The outcomes of almost 64,000 Medicare patients were evaluated. A searchable “surgeon scorecard” was created online and has been available to the public since early July.

What DOES make a good surgeon?

The most important quality of a successful surgeon is judgement. Ironically, many outcomes are determined by the surgeon before operating—choosing an appropriate surgical candidate is often the most difficult decision a surgeon will make. This database does not take any of the pre surgery decision making into consideration. A good surgeon certainly has inherent physical talent and dexterity—most great surgeons have exceptional skill with a scalpel. In addition, patient volume is a huge determinant in surgical outcome. A surgeon with a large volume practice is much more adept at performing a routine procedure and is also much more skilled at handling intra operative surprises. Moreover, surgeons who have been well trained and well educated at academic institutions during their residencies and fellowships are more likely to practice evidence-based medicine and adhere to “best practices” and clinical guidelines.

Why is the Pro Publica project fundamentally flawed?

The project, while well intended, is of no practical use. The methodology is flawed at the beginning. Mining Medicare databases assumes that the data entered is in fact accurate. There are coding errors and data entry errors that occur every single day in hospitals and at CMS. The project does not take into consideration surgeons who accept patients that others would turn away—due to high risk, etc. In addition, the data analysis only takes into consideration inpatient procedures. Many of these low risk procedures are also performed in outpatient surgical centers—including these numbers would certainly bolster a surgeon’s “scorecard”. The research presented by Pro Publica was NOT peer reviewed. Any reputable medical journal requires that all investigations are reviewed by a minimum of two or three independent and anonymous EXPERTS in the discipline of interest. These experts are charged with evaluating not only the quality of the research and its findings but also the methods that were utilized in the study. Other than mentioning a group of physician experts that were consulted to identify complications, there is no mention of a peer review process and no mention of a methodology review.

Epilogue: Pending Impact of the Pro Publica Surgeon rating project?

 Ultimately time will tell. This database unfairly evaluates surgeons. While not all physicians are created equal, the methodology involved in arriving at the Pro Publica ratings has produced unreliable and inaccurate data. I fear that rating projects of this sort—that are not based on good, sound science—will result in many physicians refusing to treat sicker, higher risk patients. Ironically, it is the BEST surgeons that must take on the toughest cases. I, along with many of my colleagues, applaud any effort that can potentially empower patients to make informed decisions about their healthcare. However, I cannot endorse faulty science. If you put garbage data into a statistical program, you simply get well analyzed garbage out. As British economist Ronald Coase once said….”If you torture the data long enough, it will confess to anything…”

Unknown

Physicians and Journalism: Responsibly Meeting the Challenge

As a physician journalist I find myself in a very fortunate and quite unique position—I am able to reach vast numbers of Americans on a daily basis and provide them with credible (and hopefully impactful) news on health and wellness. Medical journalism is similar to the practice of medicine in that we must put the PATIENT first. Just as physicians provide patients with information they need to better understand their disease state and treatment options in a clinical interaction, physician journalists must carefully choose their words when on camera or quoted in print. In a clinical situation, there is time for questions and two-way interaction between doctor and patient. In contrast, medical reporting in broadcast media is a very different situation–there is no opportunity for patient interaction and what is said MUST be something that will stimulate further conversation between viewers and their OWN private physicians. Statements must be clear, evidence based and stories must be reported without bias.

I entered the world of medical journalism nearly five years ago. It is my job to carefully dissect and interpret new studies and provide candid and accurate commentary. It is essential that as a physician, I am able to communicate new research findings on new treatments or new health risks in a way that is non-biased and free from any external influence. Moreover, it is vital that I am able to report stories in a way that does not sensationalize or overstate the effectiveness of any particular therapy. In the last several years, we have seen numerous examples in the media in which medical journalists have behaved in ways that have not met these lofty expectations.   From Dr Mehmet Oz and his overstated claims on herbal remedies to Dr Sanjay Gupta and his heroic involvement in surgical cases while covering stories in Nepal and in Haiti, there are numerous examples from which we can all learn. Dr Oz ultimately testified before Congress concerning his choice of words when discussing non-proven therapies for weight loss and other common maladies. Dr Gupta, a well-respected neurosurgeon and medical reporter, admits that when he is covering a story in a disaster area, he always is a “doctor first” and will respond to an emergency while reporting—even though ethics dictate that journalists should never be “part of the story”. For medical journalists, it can be difficult to decide exactly where the boundaries exist between the responsibilities of being a doctor and serving as a reporter.

The Society of Professional Journalists lists four major tenets in their Code of Ethics that I think MUST be upheld by any medical journalist in order to ensure that patients are protected from mis-information and sensationalism on television as well as in the print media. I believe that any physician who is contemplating entering the world of the media must be aware of these guidelines and think about how each can specifically apply to medical journalism. Below, I have listed each of these principles (as they are listed by the Society) and shared my thoughts on how they may apply to each of us when serving as medical reporters.

1. Seek the Truth and Report

As physicians it is our duty to carefully examine new findings and analyze studies in order to determine their scientific merit. It is important to understand exactly how researchers conducted their studies and arrived at conclusions prior to reporting on any new medical “breakthroughs.” While it may be a great headline to report on a new “revolutionary” treatment, it is far better to temper excitement with the facts—while a new finding may be promising, it takes time to determine whether or not it will truly be a groundbreaking new therapy. It is important that medical journalists describe the basics of any study to the audience—sample size, randomization, and design methods—in order to help viewers understand exactly what conclusions can be drawn for a particular bit of research. Once the data is reported, it is essential that the physician journalist place the findings in context—how can the study be applied to patients and how might it impact lives.

2. Minimize Harm (Primum non nocere)

Certainly, all physicians take an oath to first do no harm when caring for patients. This principle should also apply to physicians who are reporting the news. It is essential to remember that physicians, by their very title are given a certain level of elevated credibility. Physicians who are featured on television are provided an even higher level of credibility and believability. When a physician with well respected credentials speaks to a national television or radio audience, most viewers believe what is said and do not question the source—this requires a physician journalist to carefully choose the words that they use to communicate complex ideas in order to leave no room for ambiguous interpretation. Sensationalization can produce confusion and may result in patients running for treatments that are not proven to be safe and effective in randomized controlled clinical trials. In addition, if a physician journalist is involved in debating policy or healthcare politics, he or she must remain respectful to the opposition and remember that, even though we may not agree with others, all involved are human beings.

3. Act Independently

Conflicts of interest can destroy credibility and can also lead to perceived professional misconduct. It is essential that the physician journalist is careful to avoid any outside influence when reporting on a new device or treatment. Pharmaceutical and medical device companies can significantly influence the way in which data or breaking news stories may be reported. In order to remain and perceived as unbiased reporters, physician journalists must carefully disclose ANY relationships with industry and ideally avoid accepting ANY payments or gifts from industry partners. Avoid any form of “advertising” when reporting and always use trade names rather than brand names when appropriate. Always mention alternatives and competitive drugs or treatments when discussing a particular branded device or drug in order to provide the viewer with a more complete view of the story.

4. Be Accountable

A credible and successful physician journalist must accept responsibility for your words when reporting. We all must be willing to respond to challenges and criticism in a respectful, professional way. Not all viewers will agree with your assessment of a particular story—and most certainly will not always agree with your position in a healthcare policy debate. Be ready to defend your position with vigor but also be willing to admit if you have made a mistake or error in your reporting or in any conclusion that you may have drawn. Clarify your position when required and be very transparent with your sources of information when appropriate.  Carefully determine the impact of your words–as a physician on television, you are given an elevated level of credibility.  Avoid the Dr Oz example of sensationalization and over-blowing stories.  If medical journalists are conscientious and honest, they will not likely be required to testify before Congress as in the case of Dr Oz.

What is the Bottom Line??

The practice of medicine is an honor and a privilege—every person with the degree of Medical Doctor is very fortunate to be able to utilize a particular set of gifts and skills to help others. Providing care to patients and offering treatment and even cures for chronic disease is incredibly rewarding. For me, as a physician journalist, it is equally as important to educate the public and improve awareness of diseases and their treatments. Television, radio and print media provide the opportunity for physicians to serve the pubic in an entirely different way. By discussing medical advances and drawing attention to common symptoms and medical problems, physician journalists have the chance to make a real impact on overall public health. Just as the physician has a responsibility to provide their very best to the patient when involved in patient care, the physician journalist also has an enormous responsibility to provide credible, non biased and accurate information to the public when reporting.

Screen Shot 2015-07-22 at 7.11.26 AM IMG_8643 Screen Shot 2015-03-25 at 8.22.17 AM