Welcome to #BeyondTheMean! Check out this post to see what this blog is all about.
Our schools are filled with products that help us run our institutions, streamline instructional processes, and accelerate student learning. As class sizes grow and student needs increase, educational products are becoming more necessary to help teachers fulfill their duties. But – as any teacher can tell you – not all products are the same. When selecting products to release in the classroom, we must take steps to ensure that they will meet our needs and support our students. One of the most important things to consider is whether a product is evidence-based.
What makes a product evidence-based? Good question! In brief, a product is evidence-based if it has been tested on a group of students similar to yours and was shown (usually through statistical analysis) that it achieved its stated objective. The Every Student Succeeds Act of 2015, and its subsequent regulations, give us a more detailed definition of evidence. You can read more about how the law defines evidence on the US Department of Education’s website.
One key challenge to adopting new educational products and programs is that everything seems to be evidence-based. Every marketing brochure, cold call conversation, and conference vendor area is splashed with the phrase – usually in bold red letters designed to capture your attention. So how is an educator supposed to know whether a project really is evidence-based? Here are five steps to take.
1. Start with evidence before choosing a product.
In a perfect world, we would all have the time to start our search for new products or programs in the literature. By identifying a problem of practice and turning to the literature to find answers, we can make more intentional decisions. We may find that programs you have never heard of have demonstrated success with your populations, or conversely, you may find that products everyone around you is using has been shown not to work. By starting with the evidence, we can make smarter decisions.
For some, research literature can be intimidating. Many of us lack access to research databases and the skills necessary to apply the findings of research studies. If that’s you, never fear. I have a variety of resources in The Repository that are designed to help you fill this skill gap. One resource you might find particularly helpful is my video series “Finding Evidence-based Practices in Education”.
2. Talk with your salesperson.
Modern thinking about evidence-based practices in education became solidified in 2015 with the passage of the Every Student Succeeds Act. By now, any salesperson worth their salt should be able to eloquently explain the research that supports the product they are there to sell. Take what they tell you in their marketing materials and sales emails and craft some probing questions. If the salesperson cannot, or will not, explain the research to you, it is probably a good sign that its time to move on. Here are some questions that you might consider when you prepare for your next pitch meeting:
What research supports the theoretical structures of this product?
Has the effectiveness of this product been independently evaluated?
How many studies have been conducted using this product?
Has this product been proven to impact achievement on students like mine?
Have any studies demonstrated that this product doesn’t work?
3. Seek out published studies about the product.
A lot of us have been conditioned to focus our evidence-consumption on syntheses or summaries provided to us by vendors or professional associations. This is helpful to us as we seek to embed research use into our busy schedules. However, when we rely on syntheses created by others, we exchange our own professional authority and autonomy for convenience. Depending on where you source your research summaries, you may not be getting a full and detailed picture of the product before you.
Take time to find the original studies that support the product you are considering and verify the findings with your own eyes. Vendors pushing evidence-based products should be able to provide you with those documents. If you find studies you cannot access on your own, consider reaching out to the researcher directly and asking for a copy.
4. Consider its alignment with your institution and its goals.
It is not enough to simply check that a product is proven to work, we need to be sure to check the studies alignment with your institution. If you’re working in a rural elementary school, research conducted in an urban high school is not going to give you a reliable understanding of how the product might impact your students. Consider the full demographic profile of your school and make sure that you have research that aligns closely to your population.
You should also consider how the research aligns to the goals of your school. It is easy for a vendor to say that a product is evidence-based, but the real question is – evidence-based to do what? Let’s say you are looking for a program to increase student attendance at your school. You want to make sure that the research that underpins the product shows improvement in attendance. Evidence can be manipulated by savvy marketers and you might end up with an evidence-based program that has been shown to improve social-emotional outcomes instead of attendance outcomes. To make a wise evidence-based decision, you must dig deep into the studies and check for alignment with your goals.
5. Check out the What Works Clearinghouse.
If you are really strapped for time, check out the What Works Clearinghouse (WWC). The WWC is kind of like a Consumer Reports page for education products. The team at the WWC evaluates studies aligned to a variety of educational products and transparently and independently summarizes the research so that you can make a better decision. On their page, you will find that many of the questions I’ve prompted you to ask have been clearly answered. They also provide links to the direct studies so that you can dig deeper and reach your own conclusion.
The issue with the WWC is that it is not complete. The team can only review a limited number of new programs each year – so you should still work to develop the skills necessary to evaluate research and draw your own conclusions. If you rely solely on the WWC to solve your problems, you might not be considering all of the possible options.
I hope this short post has given you somewhere to start the next time you are considering adopting a new educational product in your school. Through careful vetting and deep consideration, we can make sure that our educators are equipped with the best tools to do the most important job in the country.
Good luck on your journey friends!