Welcome to #BeyondTheMean! Check out this post to see what this blog is all about.
When working with research and data to drive continuous improvement in a school, education decision makers must be aware of the various types of bias that may be introduced into research. I’m not talking here about bias that may impact instruction or access, such as implicit bias, racial bias, or gender bias. I am talking about research biases that may impact the way a study was performed, the group of participants included in the study, or the analysis procedures applied to a body of data. While some degree of bias is unavoidable – no researcher or study is ever perfect – education decision makers must be on the lookout for cases of extreme bias and be prepared to consider the impact of that bias on their decision making processes. Let’s examine three categories of research bias.
Sampling and Selection Bias
Let’s begin our discussion by looking at sampling and selection bias. These two biases are related to how we select the data or groups of people that we include in our study. Sampling bias occurs when a researcher selects participants for a study and does not apply appropriate randomized controls. It also occurs when a study sample includes only a specific type of participant (inclusion bias) or doesn’t include an impacted population (omission bias); when it only includes participants who have been identified as needing a specific intervention (channeling bias); or when it includes a group of individuals already assembled and available to the researcher (convenience bias).
The term selection bias is often applied to research studies that utilize archival data rather than data collected as a direct result of an experiment. All the aforementioned concerns apply to selection bias. Any time a researcher selects certain archival data points to consider while intentionally or unintentionally ignoring other data points, they are opening their study up to selection bias.
Let’s look at an example. If you are examining the impact of a new curriculum on your student body and you have randomly assigned your fifth graders into intervention and control groups, you have taken appropriate steps to limit sampling bias. If, however, you think that Sarah would benefit from experiencing the new curriculum and place her in the intervention group as a result of your opinion, you have engaged in channeling bias that is inappropriate and will skew the results of your study.
Sampling and selection bias aren’t inherently bad. Sometimes they are necessary. If you are studying the impact of a behavior intervention on one single student using a single-case study design, you are engaging in channeling bias. You are performing a study on a student who was selected due to their previously established need to undergo an intervention. There is no way around that, and the single-case study design is appropriate in this circumstance.
When using research to inform a continuous improvement decision, your job is to carefully examine how the samples were collected, to look for bias in those samples, and consider how that bias impacts the results of the study.
Response Bias
Response bias represents another category of bias that is common in educational research – especially research designs that rely on interviews, observations, or survey instruments. Response bias is introduced when the respondent changes their answers or behaviors due to action taken by the researcher. Two common forms of response bias are acquiescence bias and social desirability bias. Acquiescence bias is when a respondent gives the answer, they think the researcher is looking for. If it is very clear that the researcher is hoping for one outcome over another, respondents may unintentionally change their answer to a question in an unconscious act of empathy – hoping to help the researcher achieve their desired goals.
Social desirability bias is similar in that a respondent may provide an answer that they think is preferable based on societal norms, even if that answer doesn’t reflect their true feelings on a situation. Social desirability bias is common in settings where groups of participants are interviewed together. A respondent with a contrary point of view may be less willing to voice that opinion due to the unconscious social pressures created by the group. Similarly, activities such as student or teacher observation are prone to social desirability bias. If a teacher knows that the researcher is counting the number of higher order questions asked in a specific thirty minute period, they are likely to unconsciously boost the number of higher order questions during their observation period in an effort to respond to the social pressures.
Interviewer or observer bias is similar in that respondents may change their answers in an attempt to please the interviewer or observer. The difference here is that the change in response comes as a reaction to a behavior performed by the researcher. Simple things, like vocal inflection or facial expressions, can influence the behaviors of the study participants. Researchers must take care to maintain professional decorum to ensure that their behaviors do not influence the outcome of a study.
It is more challenging to identify these types of biases when reading a published study. A well-constructed research study should explain the steps that an individual researcher took to overcome these response biases. They may not always use the word bias in their description, but their discussions on interview, survey, or observation protocols should be thorough enough to give the reader a good understanding of what they did. You can then apply your own professional judgement to consider how the results of the study may impact your continuous improvement decision.
Result Bias
The final category of research bias I will discuss in this article is results bias. Researchers and the publication industry may introduce bias when they communicate the results of studies. The first type of bias introduced at this stage is called confirmation bias. Confirmation bias is when the researcher spins the results of the study to tell the narrative that they want to tell. It can also be introduced in other points along the way – such as in the selection or research design process. Confirmation bias can be difficult to spot, but education decision makers should take time to check for it when reading the results and subsequent discussion in a published study.
Another common form of bias in research is publication bias. Publication bias occurs when a researcher or academic journal chooses not to publish negative results from a study. This is sometimes called “file folder bias” in the trade because the researcher just leaves their results in a file folder never to be heard from again. It is an extremely problematic issue in our field because negative results are just as important, if not more important, than the positive ones. Education decision makers need access to study results that tell the full story of an intervention. If an intervention doesn’t work with a certain population or under certain conditions, the educators in the field need to know that.
Education decision makers can overcome the challenges posed by results biases by ensuring that they have performed a deep and comprehensive scan of the literature on a given topic. If you have only read three studies on a desired intervention, and all three of those studies showed promising results, you have probably not seen the full scope of research available. The challenge of an education decision maker is to weigh the good with the bad and put it all together to create an action plan that addresses your current problem of practice. That is what continuous improvement is all about! It is about making well informed decisions, then, implementing and monitoring those decisions along the way.
When reviewing research for decision making, be sure to scan for some of these common biases. Ask yourself how those biases may have impacted the results of the study. Then, read another study! By synthesizing as much information as possible about a topic, you can feel confident that you have filtered out the noise and made the best decision for your kiddos.
When you’re ready to take your research journey to the next step, stop by The Repository and check out some of the resources I have made available there. You will find videos, how-to guides, and auto-analysis tools that will help you build and maintain systemic processes. I want to make it easier for you to use research to guide continuous improvement decision making in your school, and I hope the resources in The Repository will do just that. Good luck on your journey friends, and let me know how I can help.