Is this product evidence-based?

Welcome to #BeyondTheMean! Check out this post to see what this blog is all about.

Our schools are filled with products that help us run our institutions, streamline instructional processes, and accelerate student learning. As class sizes grow and student needs increase, educational products are becoming more necessary to help teachers fulfill their duties. But – as any teacher can tell you – not all products are the same. When selecting products to release in the classroom, we must take steps to ensure that they will meet our needs and support our students. One of the most important things to consider is whether a product is evidence-based.

What makes a product evidence-based? Good question! In brief, a product is evidence-based if it has been tested on a group of students similar to yours and was shown (usually through statistical analysis) that it achieved its stated objective. The Every Student Succeeds Act of 2015, and its subsequent regulations, give us a more detailed definition of evidence. You can read more about how the law defines evidence on the US Department of Education’s website.

One key challenge to adopting new educational products and programs is that everything seems to be evidence-based. Every marketing brochure, cold call conversation, and conference vendor area is splashed with the phrase – usually in bold red letters designed to capture your attention. So how is an educator supposed to know whether a project really is evidence-based? Here are five steps to take.

1. Start with evidence before choosing a product.

In a perfect world, we would all have the time to start our search for new products or programs in the literature. By identifying a problem of practice and turning to the literature to find answers, we can make more intentional decisions. We may find that programs you have never heard of have demonstrated success with your populations, or conversely, you may find that products everyone around you is using has been shown not to work. By starting with the evidence, we can make smarter decisions.

For some, research literature can be intimidating. Many of us lack access to research databases and the skills necessary to apply the findings of research studies. If that’s you, never fear. I have a variety of resources in The Repository that are designed to help you fill this skill gap. One resource you might find particularly helpful is my video series “Finding Evidence-based Practices in Education”.

2. Talk with your salesperson.

Modern thinking about evidence-based practices in education became solidified in 2015 with the passage of the Every Student Succeeds Act. By now, any salesperson worth their salt should be able to eloquently explain the research that supports the product they are there to sell. Take what they tell you in their marketing materials and sales emails and craft some probing questions. If the salesperson cannot, or will not, explain the research to you, it is probably a good sign that its time to move on. Here are some questions that you might consider when you prepare for your next pitch meeting:

  • What research supports the theoretical structures of this product?

  • Has the effectiveness of this product been independently evaluated?

  • How many studies have been conducted using this product?

  • Has this product been proven to impact achievement on students like mine?

  • Have any studies demonstrated that this product doesn’t work?

3. Seek out published studies about the product.

A lot of us have been conditioned to focus our evidence-consumption on syntheses or summaries provided to us by vendors or professional associations. This is helpful to us as we seek to embed research use into our busy schedules. However, when we rely on syntheses created by others, we exchange our own professional authority and autonomy for convenience. Depending on where you source your research summaries, you may not be getting a full and detailed picture of the product before you.

Take time to find the original studies that support the product you are considering and verify the findings with your own eyes. Vendors pushing evidence-based products should be able to provide you with those documents. If you find studies you cannot access on your own, consider reaching out to the researcher directly and asking for a copy.