ProofPilot How To

Creating a Study When You Don’t Know What to Study

Here’s an 8 step by step guide to get past the ProofPilot equivalent of writer’s block? Take a study that’s already been done and replicate it for your own purposes. Here’s how.

Like staring at any blank page, an empty ProofPilot study flow page can be overwhelming, even for those of us who use ProofPilot every day. There are so many possibilities. Where to start?

Replicating a study is not plagiarism. In fact, it’s an important part of the scientific process. Reproducing a study helps ensure the first set of results are correct. It’s also the basis for generating new knowledge by piggybacking on previous findings and study designs.

Here’s a step by step on how you can use this process with ProofPilot

  • to replicate an existing study
  • take advantage of technology to make the study more efficient and participant-friendly
  • adapt the design for your own purposes

Step 1: Find a Topic of Interest

A lot of us have a very specific topic of interest. You might even have something specific you want to do a study on. If so, you can skip to Step 2.

But, let’s not make assumptions. Maybe you’re interested in designing a study, but don’t have any idea the topic or structure.

Study ideas are all around us. We might see an advertisement, hear a claim from a politician, or experience something in our lives. At ProofPilot, a lot of times we’ll see an article in the newspaper or magazine. It might be something we’d never thought about before …

.. like the effect cashews have on cholesterol. Who knew while cashews are the third most consumed nut in the US, they are disqualified from the FDAs approved health claim that nuts reduce cholesterol only because they have a fat amount above the FDAs somewhat arbitrary cut off?

So here’s a study. Do cashews lower cholesterol? I don’t know anything about cholesterol, and I eat cashews occasionally … so where do I start?

Step 2: Find an Existing Study Design

You need to boil down your topic of interest, whatever it is, into a couple of keywords. For our example, we’ll boil down our interests to two keywords “cashew and cholesterol.”

Researchers at academic institutions measure their success, in large part, by publishing. A steady stream of results published in the top academic journals, and cited by peers, makes a career. Therefore, there are millions of articles across tens of thousands of journals.

Luckily there are several resources where you can search across all these research articles:

  • PubMed: The US National Library of Medicine collects almost 30 million research article citations and makes them searchable in one spot.
  • Google Scholar: Doing a plain Google search on a topic includes everything. What we want are scholarly articles. Luckily, Google makes it a little easier by providing a separate academic search functionality.
  • ClinicalTrials.gov: The clearinghouse for completed and in process clinical trials funded and/or conducted in the United States. The World Health Organization has a similar database.

In your search results narrow in on articles that include the following terms in their title:

  • Trial or clinical trial
  • Experiment
  • Evaluation of
  • Outcomes
  • Study or clinical study

Spend less time reviewing (maybe even skip) articles that have the following terms in their title. They likely won’t contain study protocol designs:

  • Meta-Analysis
  • Review of the literature
  • Case reports
  • Opinion, Perspectives or Statements
  • Any article that appears to cover bench science (research studies that don’t include actual people).

For our cashew example, we found: Cashew consumption reduces total and LDL cholesterol: a randomized, crossover, controlled-feeding trial. The formatting makes it really hard to read. So we copied the title into Google and got the actual journal website version with better formatting.

In many cases (our example included), you only get an abstract of the article for free. That’s all you need. At ProofPilot, we’ve never paid to access a full journal article. Everything you need is typically included in the abstract. If it’s not, look for another.

Step 3: Find a Couple of Key Sections in the Journal Article

These academic journal articles don’t just focus on the results. A research journal article also must discuss the methods to get those results. While the results may be really cool and interesting, our interest is replicating those methods. It’s those process components of the article we’re most interested in.

Here are the sections to look for:

Methods or Design:

  • What was the overall design of the study? Was it a randomized controlled trial? Was it a randomized cross-over trial?
  • How often did the participant take the intervention or treatments (aka dosing)? What was the periodicity between measurements?
  • How long did the study last?

Outcome Measures:

  • What are the primary and secondary measurements of the study?
  • Does the abstract list specific measurements (an assessment like “The SF-36”), or specific lab tests?

Inclusion/Exclusion Criteria:

  • What situations make a participant not a good fit for the study?
  • What unique participant characteristics are important characteristics for the study?
 

The cashew and cholesterol example study is fairly simple. Most studies will have many more complex interventions and a broader selection of outcome measures.

Step 4: Breaking the Study Design (or Methods) into Study Tasks

The basic element of a ProofPilot study is a study task. Study tasks are tied together with rules and organized into arms to create a full study. In doing so, not only do you get a nice organized visual representation of a study, once launched, the study can run in a relatively automated fashion.

Once you’ve found the appropriate sections that describe your study you can start breaking it down into the individual pieces. In the case of the cashew study, those basic components are:

  • Baseline (measurement at the start) Cholesterol
  • Potato Chips for the control
  • Cashews for the experiment
  • Final measurement of Cholesterol

Step 5: Organize those Tasks into Arms, and Tie Them Together with Rules — Start Designing Your Study on ProofPilot

Choose a study template and then go in and enter your first task. If you want to have all your arms laid out you can do that. But if not, don’t worry, you can add arms at any point in your study.

In the cashew study, I take a stab (ignoring the crossover design and washout period for a moment) to produce this in a couple of minutes:

 

Step 6: Adapting to a More Customer Service Orientation, Technology and Resource Constraints

Every study, large or small has resource constraints. Many studies are difficult and burdensome for participants to engage in. Technology generally (and ProofPilot specifically) adds opportunities and limitations. You need to consider these opportunities and limitations as you design your study.

Ask yourself the following questions:

  • Most studies are conducted in an in-person setting. Is there any reason the task, as it is defined in the original design, needs to be done in person with a study professional? Can a participant complete it themselves?
  • As great as ProofPilot is, in some cases it can’t eliminate a cost or high touch in person activity. Is there an alternative technique that will accomplish the same goals?
  • IF you don’t have the money to replicate the study as it exists are there alternative techniques to give participants the opportunity to comply and pull the data from existing sources?

The cashew study, if replicated in exactly the same way as described in the journal article, would be expensive. For 100 participants it would cost almost $20,000. You need to pay staff to see patients and conduct the study. You need to budget for at least two cholesterol tests (at baseline and at 30 days) per participant, plus a supply of cashews, and a supply of potato chips.

What are the alternatives?

When you create a new task in ProofPilot, the first step is to select a task template from our library. There are hundreds of templates that provide unique measurement and intervention functionality that may spark ideas.

  • Can participants purchase their own cashews and potato chips with a discount via a local or national retailer (like ProofPilot partner Thrive Market?)This would also make it possible to confirm a purchase was made)
  • Is there available electronic medical record data that can sub in for tests and diagnosis that would otherwise require additional staff time? If so, then go ahead and add the ProofPilot EMR Task as a measurement.
  • Are there valid at-home and/or self service versions of things we typically consider require an office visit. For example, the FDA approved at-home cholesterol tests in 1993. Some brands cost as little as $12. Would participants pay for them out of their own pocket?

If you made every change described here, the entire study would cost under $5,000 for 100 participants, and it’s possible to defer all or a majority of the costs to participants themselves for around $50 each.

 

Step 8: Filling in the Blanks, Making Assumptions, and Improving the Study Design

Many research study journal articles don’t go into sufficient detail to replicate the study exactly. And sometimes, limited resources mean some important elements don’t get included in that academic study. And, you may have slightly different goals than the original study author.

This is where you start thinking critically and logically.

  • What measurements are missing that take into account participant behaviors and environment? Add a new measurement task. Browse around. What’s missing? In the cashew study, it seems obvious you’d want to know what a participant’s general dietary habits are, so I add the NHAINES dietary screener, which asks a participant to recollect their diet over a specified period of time.
  • Did participants really adhere to the intervention or treatment? There are many ways to measure adherence. If your intervention is a digital health app, you may be able to see usage data in the app itself. You might use ProofPilot’s SMS Group Based Confirmation Attendance task to easily measure attendance at an event or class. There are high tech ways to measure medication adherence, but like general behaviors, it might just be easier to ask on a periodic basis: Did you do the intervention?
  • What additional interventions or treatments are taking place? Can they be combined to create a more realistic study? There’s no one silver bullet. The classic question here is does medication PLUS lifestyle change have a better effect than lifestyle change alone? Think about isolating certain combinations of treatments across arms. In the cashew study, I’ve added various arms that look at cashews plus traditional treatments.
  • Most academic studies are exacting. Is it necessary or even realistic? With drug or supplement trials, obviously, the dosing needs to be 100% spot on. However, most of life isn’t exact. You’ll notice in the example cashew and cholesterol study methods section the term “isocaloric.” It means the study staff took great pains to control the calories exactly in each serving of cashews. This likely wouldn’t happen in real life. Therefore, just saying a small handful might be good enough for our purposes.
  • What did participants think about the treatment or intervention? The placebo effect in humans is amazingly strong. The entire process of a double blind study is to eliminate our minds ability to change our body. Still at ProofPilot, we almost always ask at least at the end of the study how participants views the intervention, and whether they thought it made any difference. While the results might not be scientifically valid, it they might provide important marketing and acceptance information.

Step 7: Adding Imagery, Color, Rewards and Participant Friendly Language

At ProofPilot we’re working to turn the research study into a new form of online media. That means studies are more than tasks, instructions, and data entry elements. They include language that instructs and engages along with images and colors that add a visual style.

The Final Cashews and Cholesterol Design

Over the course of four days, in between e-mails, conference calls and everything else that makes up my day as a startup founder, I probably spent a total of four hours on replicating and adapting this study. I ended up making some significant changes from the original design, but I think it creates a more interesting study:

  • I added two additional arms that include both the cashews and the standard of care (likely a popular cholesterol-lowering drug). This will allow us to look at whether cashews in addition to standard medical treatments improve outcomes.
  • I added back the crossover design. Notice there is an immediate arm and a delayed arm. At the crossover point, the 30 days of cashews should be all done for the immediate arm, and just beginning for the delayed. This let’s us get a treatment vs no treatment effect at the early portion of the study without a blinded controlled trial (which would be tough in this case).
  • I extended the study from 28 days to 60 days. The Thrive Market cashews have 15 servings each, so 30 days of “treatment” is a nice neat number.
  • I included a number of assessments for diet, and physical activity to control lifestyle factors.
  • I removed the potato chip alternative to cashews finding it somewhat superfluous with the added diet questions.
  • I also added 15 day follow-ups during the “cashew treatment phase.” At the point of these follow-ups given one bag of cashews should be empty, and the participant should be moving on to the next. This creates a nice neat marker to assess adherence.
  • I included 2 at home cholesterol test kits, plus an option for participants to share their electronic medical records (which likely has various cholesterol readings in it).
  • Finally, I added a participant perception at the end of the study. While we will get purchase data from Thrive Market, participants can buy cashews anywhere. We want to know whether the participant is keeping this up.
 
The participant experience still needs work. But even the addition of a simple image of cashews and a subtitle it’s looking like something people might want to participate in.
 

Final Thoughts

It’s likely that a research purist would argue with some of the decisions made in replicating this study. The reality is no research design is ever perfect. It’s about working within your resources and constraints to control for as many variables as possible.

Designing a study shouldn’t take years. Nor should it be confined to the few well funded researchers at academic institutions. You don’t need a big IT budget. You don’t even need to be an expert in a particular field.

Simply by using existing studies in the academic research literature, you’ll be able to overcome the, “I don’t know how to design my study paralysis” every time.

(This is an adaption of a slightly modified version of an article originally posted on the ProofPilot Blog)