Welcome to #BeyondTheMean! Check out this post to see what this blog is all about.
Data is information that we turn into knowledge and insight. Through the careful collection, curation, and analysis of data, education decision makers can develop a new understanding of current problems of practice and inform continuous improvement decisions. In this post, I want to examine the different types of data in an effort to broaden your perception of the various types of data available to you as you pursue continuous improvement activities.
Quantitative and Qualitative Data
Let’s start broad by discussing the two major categories of data – quantitative and qualitative. Quantitative data are data that are used to uncover facts about a given phenomenon. They are data collected through rigorous, sometimes experimental means, and are analyzed using statistical inferences. Usually, quantitative data are numbers, but can also include fixed items, like survey responses that range from “Least Satisfied” to “Most Satisfied”.
Quantitative data can be further divided into variables that are continuous or discrete. A continuous variable is a variable that exists on a spectrum, such as student test scores. A discrete variable is a variable that can be categorized or counted, like the number of male or female identified students in a class.
Qualitative data, on the other hand, are data that are concerned with explaining and understanding a phenomenon. Qualitative data are collected by the researcher using less definitive methods, such as interviews or observation, and they rely on the researcher to interpret and give context to the data. Case studies or focus groups are common qualitative study designs deployed in education. They are reports designed to tell you about something that happened and explain some of the nuances of the phenomenon through the eyes of the participants.
When considering data analysis tasks for continuous improvement efforts, education decision makers should strive to examine a wide variety of data from both the qualitative and quantitative categories. Remember that your goal is to fully understand the breadth and scope of a problem of practice. You cannot fully understand your problem without examining it from all angles. Ensuring that you have a variety of data from both categories helps you do that.
Administrative Data
Administrative data are the data points that we collect every day as a part of administering a system, school, or classroom. Education produces an insane amount of administrative data. Think for a moment about the volume of data we collect on a student every day. We record whether the student is on time for school, absent, or tardy. We record whether they buy breakfast or lunch in the cafeteria. We record their behavior infractions, including the time, location, duration, impact, and consequence of the infraction. We record when they visit the nurse. We record the score they earned on their homework assignment. We record their summative test scores. We record their benchmark test scores. We record their address, phone number, guardian names, email addresses, the past schools they attended, their gender, their race/ethnicity, their age, their birthdate, their zip code. I can go on – but that won’t make for a very interesting post.
All of this administrative data is stored and archived forever in our student information systems. Some of it is also reported and stored by the US Department of Education and the Institute for Educational Sciences. USED and IES also have a variety of surveys that are designed to collect even more administrative data.
Administrative data is a continuous improvement specialists’ best friend. It can help us spot trends and examine the health of a school or system. It also makes it easy for us to build action research projects because we already have the information we need to establish a baseline – all you need to do is add in some post-intervention results and POW you have actional information. Educational researchers rely heavily on administrative data. When combined with administrative data from other fields, such as unemployment data or US Census data, researchers can uncover valuable information about our schools.
Assessment Data
Assessment data is often lumped in with administrative data, but I think it warrants its own discussion. Assessment data tells us about the performance of our students. When combined with other forms of administrative data, assessment data can help us spot inequities in our systems and monitor the impact of our continuous improvement decisions.
Usually when I use the phrase assessment data, educators immediately think of the end of the year federally required assessments – sometimes referred to as high-stakes standardized assessment. In my opinion, these assessments have an unearned bad reputation. Standardized assessments of the past were often filled with bias and were dripping with white privilege. They measured social status more than academic achievement. They also frequently came with high stakes that meant bad things for teachers and principals. I wont even touch the criticisms of the testing industry which is often viewed as predatory and profit crazy (that’s a topic that deserves its own post).
But standardized assessments have come a long way. While they still have bias and occasionally ask problematic questions, they are much more heavily vetted than in the past and provide real actionable results for use at the system level. One common criticism of standardized assessment data is that they don’t provide meaningful information for the teachers working with individual kids. I agree – that really isn’t their purpose. When used properly, these end-of-the-year assessments help decision makers understand what is going on in a school or system over time.
These assessments aren’t the only assessment data that we have though! Many schools give regular benchmark tests that provide actionable, student level data to educators on a quarterly basis. These tests are designed to diagnose gaps in student learning and help teachers understand where the holes in instruction may be. They are incredibly valuable for grouping students, planning future instruction, and informing multi-tiered systems of support (MTSS).
We also have teacher created formative and summative assessments. These are the assessments that classroom teachers use to measure individual progress and growth on the standards being taught as they are being taught. These teacher created assessments allow individual educators to adjust their instruction to meet the needs of all students in their care.
When using assessment data to drive continuous improvement efforts, it is important that leaders consider the role of the assessment and what information the data was designed to convey. Educators who try to use standardized end-of-the-year assessment data to inform the teaching of their volcano unit will be understandably frustrated; teacher created formative assessments are more appropriate in this instance. Remember, more data isn’t always better. We want the right data to solve the current problem at hand.
Stakeholder Engagement Data
Schools that are engaged in effective continuous improvement efforts are also rich with stakeholder engagement data. This is the data that you collect when you survey parents, hold a community forum, or meet with small groups of students to discuss a new policy in your school. Schools are the center of the community. In many communities, the school is the largest employer, the first point for pediatric medical care, the primary provider of youth mental health services, the most frequented restaurant in town, and the most reliable clothing bank.
Schools should regularly, consistently, and systematically seek community stakeholder feedback throughout the year. But what do you do with all that data? Stakeholder engagement data is often qualitative data, which means it requires the researcher (or in this case, the educational decision maker) to interpret the data, provide context, and derive meaning. This is done through a process called coding. Coding is when the researcher groups comments from stakeholder groups into themes that are then used to inform decisions.
Stakeholder engagement data is valuable, but it can also be heavily biased. Each stakeholder brings their own agenda to the party and its important that the decision maker weighs the feedback evenly. It is also important to review stakeholder feedback without emotion. Often, we defer to the stakeholder group that is loudest or angriest, even if the summation of the data points don’t support their opinion. Apply the same rigorous analysis to stakeholder data that you apply to administrative or assessment data and you will make better decisions.
Community Data
Finally, continuous improvement leaders have access to a wide variety of community data points. Just like the education system, the other civic systems in a community collect data too! Think about how you might access and use data from the workforce development agency, the justice system, or the child welfare office.
Let’s say that you want to add a new set of courses designed to prepare graduates for immediate entry into the workforce. Your first stop should be to the workforce development agency. They will have data that will help you understand the future outlook for various professions in your community. It won’t benefit your students to create a certified nursing assistant program if the local hospital is already overstaffed. Using the rich data collected by other agencies within your community can help you make these kinds of decisions.
As I mentioned before, accessing administrative data from other federal agencies can also be of benefit. Consider the data collected by the US Census Bureau. This data may help you to see a pattern of changing demographics in your community. Those changing demographics may inform your future recruitment and hiring efforts, the targeted programs taught in your school, or even the makeup of your school calendar.
I hope this post has given you some food for thought and helped you think of new data sources that you have available to you. Remember that your continuous improvement data is more than the 20 or 30 variables you have on your spreadsheet or dashboard. By weaving together administrative data, assessment data, stakeholder data, and community data you can get a deeper picture of the needs in your school and the possible solutions you could implement.
If you need help analyzing data in a meaningful way, check out the resources I have pulled together in The Repository. I have included a variety of tools, videos, and eBooks to help you jump-start your data analysis processes. Good luck on your journey friends, and let me know how I can help.