Welcome to #BeyondTheMean! Check out this post to see what this blog is all about.
Regular assessment is the key to effective instructional planning – but only if you take the time to look at the data and properly adjust your instruction. For many of us, this means simply comparing a pre-test and post-test score for each student to see if, and how much, they grew throughout the course of a unit. While this may work for a homeroom teacher with only 25-30 students, if you are analyzing summative assessment data for a case load of 125 students spread across five preps, or 500 students spread across an entire grade level team, this granular process is simply not going to cut it. In this post, I will walk you through the steps to carefully examine pre-test/post-test data to inform instructional plans for a large group of students.
Step One: Gather the Data
Before you can analyze the data, you have to gather your data and tidy it up. If you use a student information system (SIS) to store student grades, this may be as easy as exporting a spreadsheet from your system. If you keep your data manually, you may need to copy and paste some columns into a fresh spreadsheet for analysis. If you still keep a paper gradebook – call me and let me help you digitize!
When you prepare your spreadsheet for analysis, you want to have a separate column for each piece of information. I suggest you have a column for the student name, student ID number, student gender, student race/ethnicity, the section the student is enrolled in, their pre-test, and their post-test scores. At a minimum, you need to have student names and test scores – so three columns.
Make sure your data is orderly. That means that each column only contains one piece of information and each student only has one row. Also, check for repeats – we don’t want to analyze a student multiple times.
Step Two: Find the Averages
Once you have your data neat and tidy, the next step is to find the average pre-test and post-test scores. This is easily done by using the AVERAGE formula in most spreadsheet software. Simply type “=AVERAGE” into an empty cell, then highlight the column with your pre-test data. Do the same for your post-test data. Then, see how far apart they are. Ideally, the average score on your post-test should be higher than that of your pre-test.
Next, you should disaggregate your data and examine the averages between different groups. I recommend always disaggregating by gender, race/ethnicity, and course section. This will help you see discrepancies and inequities in your data. There are several ways to do this. In spreadsheets, I like to use the AVERGEIF formula. This allows you to quickly average a subset of data and works in most spreadsheet software. A breakdown of this formula can be found in my eBook entitled “Essential Spreadsheet Formulas”.
Step Three: Dig Deeper
Once you have examined the averages between groups, you need to dig a little deeper to really understand what is going on. This opinion shouldn’t surprise you; this blog is called #BeyondTheMean after all.
The next step in this process is to calculate some other simple descriptive statistics. These include the median, mode, standard deviation, and range. Again, these can all be calculated easily in spreadsheets with a few basic formulas, a breakdown of which is available in the aforementioned eBook.
As you examine the differences among these simple statistics, consider how they vary between groups. Look for changes in the standard deviation – a statistical measure that tells you how spread out your scores are. Ideally, you would like to see the same or a lower standard deviation on the post-test than you saw on the pre-test. This would indicate that your students are moving as a group. If your post-test standard deviation is higher than your pre-test, that is an indication that some kids may have been left behind.
The same is true for the range. When calculating the range, you must first find the maximum score and the minimum score. Take a look to see how these changed over time. This is where you will find your kiddos who are ready for enrichment and those who need additional support.
Step Four: Make Some Graphs
Data visualization is an important step in the analysis process. It helps you see your data more clearly and communicate about changes to stakeholders. I recommend making a histogram and a boxplot of your post-test scores. A histogram is a type of graph that shows how many students scored within a given range. It uses vertical bars to show the count of the students in each “bin” or category. Boxplots also help summarize your data by showing you the average score, the interquartile range, the full range, and any outliers. It can be especially useful to create a boxplot of the pre-test and post-test scores on a single axis. This will make it very clear what kind of changes your students experienced.
Step Five: Determine the Significance
When you’re ready to get even deeper, run a couple of tests for statistical and practical significance. This will set your data analysis apart and show stakeholders that you really know what you’re doing. For statistical significance, I recommend using a paired-sample t-test. The t-test compares the means of two distributions of scores. The paired-sample version of this test is best when you have two scores for each student. While the t-stat may yield useful information for the most nerdy of us, what you really are looking for here is the p-value. It is generally accepted that the results of a t-test are statistically significant if the p-value is lower than 0.05. (Side note: there is a lot to unpack in the literature related to p-value, something I will examine in a future post.)
Another useful test for significance is effect size. Effect size is a test of practical significance. If the p-value tells you whether or not the difference between two means is significance, the effect size tells you how significant. The most common formula for effect size is called Cohen’s d. To calculate the effect size, you simply subtract the mean of your post-test from the mean of your pre-test and then divide that by the pooled standard deviation. It is generally accepted that the results of Cohen’s d test are small if they are around d=0.02, medium if they are around d=0.05, and large if they are around d=0.08. (Another side note: there are tons of people who will disagree with me on this step, more on that in a future post too.)
Most spreadsheet packages include a statistical add on that will allow you to perform these tests with a few clicks, but for a shorter learning curve, I recommend uploading your data into my Pre-test, Post-test Analysis Tool. I have designed this tool to run a complete pre-test/post-test analysis on whatever data you upload. To use the tool, you simply upload your spreadsheet, then select your pre-test column and post-test column from the drop down menus. Remember to remove any personally identifiable information (PII), such as student names, before uploading. While your data isn’t stored in the tool, you can never be too careful. Once you upload your data, you will see a full analysis on the screen, this can be easily copied and pasted into a report as you complete the final step below.
Step Six: Share the Results
After spending time on your analysis, it is time to share your results with your peers and other important stakeholders. With your team, carefully look at the data you have collected. Examine the differences between groups. Consider how statistically or practically significant the difference between your pre-test and post-test are. Is there a sub-population of students who did better or worse than their peers? Is that sub-population your third period class, your male students, or your English learners? How will you adjust instruction to meet the unique needs of your students?
Engaging in a rich data conversation is the key to truly improving instruction. By taking the time to carefully disaggregate your pre-test/post-test data and preparing a handful of statistical outputs, you can jumpstart your data conversations and get straight to the point – helping your students.
As you prepare for future data conversations, please check out the free tools and resources available to you in The Repository. I regularly update and upload new tools, eBooks, video tutorials, and engagement opportunities to help the #BeyondTheMean community grow in their continuous improvement skills.
Good luck on your journey friends and let me know how I can help.