The ProofPilot Blog - Design, Launch & Participant in Research Studies

How to Avoid “Evidence-Based Cynicism” in Health Innovation

Late Friday, December 15th, the Washington Post reported a story detailing seven terms the CDC (and likely other health agency) should avoid in budget documents. Shortly after the news broke, Dr. Brenda Fitzgerald, the new CDC Director reached out to concerned staff via an all agency e-mail. The e-mail explained the terms were in fact, not banned, but suggestions to improve the likelihood of budget passage in Congress. These terms are: “vulnerable,” “entitlement,” “diversity,” “transgender,” “fetus,” “evidence-based” and “science-based.”

The CDC (and other health agency) budget processes are highly political. It’s not uncommon to make language adjustments to improve congressional support. In our current political environment, while reprehensible, it’s not surprising to see terms like “transgender” and “fetus” in such a list.

But why are government budget analysts suggesting health agencies avoid the term “evidence-based?” How did we get here? The situation is a warning call for any innovators in healthcare. Learning from CDC’s mistakes could mitigate already problematic trends in the digital and innovation health ecosystem.

Ideally, our entire healthcare, public health, and social welfare system should gravitate towards programs, treatments, and campaigns that are evidence-based. Insurers and other payers prefer treatments that have evidence. That evidence strongly suggests the investments in those initiatives will have the desired effect. In fact, over the past ten years, 42 states have passed more than 100 laws prioritizing funding to those programs, treatments, and interventions that have evidence.

Now, the Trump administration has seemingly thrown that all away. Here at ProofPilot, an online platform to create this evidence, we’ve seen the writing on the wall for a long time. It would be a mistake to classify these requested terminology changes as only political ideology. In fact, some of the underlying problems with “evidence-based” inspired what ProofPilot is today.

Everyone agrees that we should prioritize our resources towards things that work. Over the past two decades, the CDC (and other health agencies) have gone all in on “evidence-based.” But they’ve done so without developing a sufficient ecosystem to produce that evidence. Identifying what works and what doesn’t is highly problematic.

An “evidence-base” is created using techniques like the randomized controlled trial, outcome studies, and programmatic evaluation. These methods are incredibly expensive and time-consuming. Very few of them get done. Those that do are conducted by a small set of very well funded institutions on a limited set of situations and populations.

Unique cultures, behaviors, environments, available resources, and expertise have a significant impact on health outcomes. But, given the focus on paying only for “what works,” professionals without the resources to run their own trials, are forced to implement a limited set of “evidence-based practices,” defined in entirely different circumstances. In some cases, those professionals are also barred from implementing new, locally relevant innovations that aren’t part of the approved “library”.

This situation has lead to widespread cynicism. The “evidence-base” is so limited and prescriptive, it’s lead to a backlash. That backlash is evident in the CDC’s budget presentation guidance.

At ProofPilot, we hear potential customers saying something to the effect of, “Just because an initiative worked there, at that point, doesn’t mean it’ll work here now. Our situation is entirely different.” They continue with, “I have ideas that might work better. No one will give me money to evaluate it for efficacy, so I’ve got to do it myself.” That’s why they come to ProofPilot.

What we’re hearing, and our experience suggests the government funding model of developing and pushing a limited set of evidence-based practices didn’t work. The recent requests from the Trump administration suggest it may have backfired.

Here’s what health innovators can learn from these mistakes to avoid the same controversy.

Decisions based on evidence aren’t going to go away. The digital health community has come under fire for being “digital snake oil.” Regardless of current political whims, health and social welfare decision makers are risks averse. They want to see evidence before risking their limited resources. Be prepared.

Evidence isn’t evidenced until it’s local, specific and timely. Many health disruptors are located in big cities and at major academic medical centers. As innovations move to very different environments, don’t be naive and assume outcomes will be the same. Culture changes rapidly, it differs organization to organization.

Repeat and repeat again. Just because you have one study result that provides verifiable evidence doesn’t mean you’re good forever. Reproducing study results is a key part of the scientific process. Be open about your process. Let others try and replicate your study in and see if they get the same result.

Offer pilots with an outcome evaluation component to all your customers/users. Many digital health pilots today focus on limited items like operations and acceptability. They rarely focus on outcomes. But outcomes are ultimately what people want. These efforts do not need to be fancy third-party studies. Tracking patients over time within an app or simple solutions will do the trick. Provide these outcomes to users, customers and participants.

Micro-Target your innovation. Protect your innovation as it moves into different situations. Be open and honest that it hasn’t been tested in unique circumstances. Partner in new environments to run trials and repeat.

ProofPilot believes in fostering health innovation and giving everyone the tools to evaluate new ideas and perspectives for efficacy (not just a select well funded few). We believe in democratizing the tools previously only possible in well-funded pharma and academia.

Matthew Amsden is CEO of ProofPilot, a venture-backed online platform to design, launch and participant in studies that determine what works to improve lives.

You'll Also Be Interested In

Other Stories in the ProofPilot Blog