Research Bias and Continuous Improvement

Updated: Jul 9

Welcome to #BeyondTheMean! Check out this post to see what this blog is all about.

When working with research and data to drive continuous improvement in a school, education decision makers must be aware of the various types of bias that may be introduced into research. I’m not talking here about bias that may impact instruction or access, such as implicit bias, racial bias, or gender bias. I am talking about research biases that may impact the way a study was performed, the group of participants included in the study, or the analysis procedures applied to a body of data. While some degree of bias is unavoidable – no researcher or study is ever perfect – education decision makers must be on the lookout for cases of extreme bias and be prepared to consider the impact of that bias on their decision making processes. Let’s examine three categories of research bias.

Sampling and Selection Bias

Let’s begin our discussion by looking at sampling and selection bias. These two biases are related to how we select the data or groups of people that we include in our study. Sampling bias occurs when a researcher selects participants for a study and does not apply appropriate randomized controls. It also occurs when a study sample includes only a specific type of participant (inclusion bias) or doesn’t include an impacted population (omission bias); when it only includes participants who have been identified as needing a specific intervention (channeling bias); or when it includes a group of individuals already assembled and available to the researcher (convenience bias).

The term selection bias is often applied to research studies that utilize archival data rather than data collected as a direct result of an experiment. All the aforementioned concerns apply to selection bias. Any time a researcher selects certain archival data points to consider while intentionally or unintentionally ignoring other data points, they are opening their study up to selection bias.

Let’s look at an example. If you are examining the impact of a new curriculum on your student body and you have randomly assigned your fifth graders into intervention and control groups, you have taken appropriate steps to limit sampling bias. If, however, you think that Sarah would benefit from experiencing the new curriculum and place her in the intervention group as a result of your opinion, you have engaged in channeling bias that is inappropriate and will skew the results of your study.

Sampling and selection bias aren’t inherently bad. Sometimes they are necessary. If you are studying the impact of a behavior intervention on one single student using a single-case study design, you are engaging in channeling bias. You are performing a study on a student who was selected due to their previously established need to undergo an intervention. There is no way around that, and the single-case study design is appropriate in this circumstance.

When using research to inform a continuous improvement decision, your job is to carefully examine how the samples were collected, to look for bias in those samples, and consider how that bias impacts the results of the study.

Response Bias

Response bias represents another category of bias that is common in educational research – especially research designs that rely on interviews, observations, or survey instruments. Response bias is introduced when the respondent changes their answers or behaviors due to action taken by the researcher. Two common forms of response bias are acquiescence bias and social desirability bias. Acquiescence bias is when a respondent gives the answer, they think the researcher is looking for. If it is very clear that the researcher is hoping for one outcome over another, respondents may unintentionally change their answer to a question in an unconscious act of empathy – hoping to help the researcher achieve their desired goals.

Social desirability bias is similar in that a respondent may provide an answer that they think is preferable based on societal norms, even if that answer doesn’t reflect their true feelings on a situation. Social desirability bias is common in settings where groups of participants are interviewed together. A respondent with a contrary point of view may be less willing to voice that opinion due to the unconscious social pressures created by the group. Similarly, activities such as student or teacher observation are prone to social desirability bias. If a teacher knows that the researcher is counting the number of higher order questions asked in a specific thirty minute period, they are likely to unconsciously boost the number of higher order questions during their observation period in an effort to respond to the social pressures.

Interviewer or observer bias is similar in that respondents may change their answers in an attempt to please the interviewer or observer. The difference here is that the change in response comes as a reaction to a behavior performed by the researcher. Simple things, like vocal inflection or facial expressions, can influence the behaviors of the study participants. Researchers must take care to maintain professional decorum to ensure that their behaviors do not influence the outcome of a study.

It is more challenging to identify these types of biases when reading a published study. A well-constructed research study should explain the steps that an individual researcher took to overcome these response biases. They may not always use the word bias in their description, but their discussions on interview, survey, or observation protocols should be thorough enough to give the reader a good understanding of what they did. You can then apply your own professional judgement to consider how the results of the study may impact your continuous improvement decision.