This post and accompanying video is part three in a series of four by Aaron Dewald, Associate Director, Center for Innovation in Legal Education, University of Utah College of Law. Watch his second video, and then read his article below regarding how to calculate the index of discrimination.
The index of discrimination goes a step beyond item difficulty. Simply put, the index of discrimination determines how well the question can tell the difference between high and low performers. The index of discrimination will really come in handy when we need to decide whether or not our question is good at telling who is understanding the topic, and who is not – thus discrimination.
The index of discrimination begins by calculating the total quiz score for each student and sorting them from high to low.
Next, based on the final test rankings, we group the top 27% and call them high performers and the bottom 27% and call them low performers.
Then, for each question, we calculate the item difficulty for each group. The number in each group who answered it correctly over the total number in the group.
Finally, you subtract the calculated difficulty of the bottom group from the top group. Voila! You have your index of discrimination for the item.
So, what’s a good index of discrimination? As we talked about for item difficulty, you’ll never pin anyone down into telling you there are definite cutoffs, however, we can start off with some general rules of thumb. These rules are based on Crocker and Algina (2008).
Anything lower than 0 should be rewritten or discarded. An item that is is said to be negatively discriminating.
An index of discrimination that hovers around .2 is generally a bottom threshold. Anything between 0 and .20 -ish isn’t really doing a very good job at telling us the difference between high and low performers. But, this doesn’t mean an question that has a .07 discrimination index is bad.
As we get up to .4, .5, or .6 we’re in great discrimination territory. These questions do a great job of differentiating high and low performers. Realistically, you might see a lot of your best questions somewhere in this range.
An index of discrimination that equals 1 is said to be perfectly discriminating. This is our golden unicorn because, let’s be realistic, you’re rarely going to have questions that are perfectly discriminating.
Examples using the Index of Discrimination
Let’s assume for columns two and three that we’ve already taken the top and bottom 27% based on their total quiz score and calculated the difficulty level for each group. The fourth column, the index of discrimination, is found by subtracting the bottom difficulty from the top difficulty.
Question 1 is negatively discriminating. A greater proportion of the bottom 27% got it right when compared to the top. This question likely needs to be rewritten or thrown out.
Question 2 isn’t bad. It’s right against that bottom threshold we’ve set, so we might want to take a quick look at it to see if there are any improvements we can make.
Question 3 is a great question as it is realistically able to tell top performers from bottom performers. I’m going to look at the topic of this question, how I wrote the question, and use it to understand what my good students know that my lower ones don’t, and also try to emulate the style of the question on others to improve all of my items.
Question 4 could be a pretty lousy question as it’s not telling us much between top and bottom – but! this is ok if it’s a mastery level question, but otherwise we might want to look at it to see
Question 5 is our golden unicorn… nearly perfectly discriminating.
Stay tuned for the last video/blog post. Subscribe above to have everything delivered straight to your inbox, or, register here for an upcoming Twitter Chat hosted by Aaron himself!
About the Author
Aaron is currently a Ph.D. candidate in learning science, which gives him a unique perspective on technology use in pedagogical situations. Aaron received his B.S. in Information Systems from North Dakota State University in 2001, and his M.Ed. in Instructional Design and Educational Technology from the University of Utah in 2010.Follow on Twitter More Content by Aaron Dewald